Orion: a lab project laying the groundwork for Meta's future smart glasses (#16 of 2024)

This fortnight I am bringing another single-topic article. This time it is about Meta's Orion project, augmented reality glasses that project holograms onto the real world, that cost more than $10,000, that have captivated part of the tech press, and that we will probably never see on the market.

Thank you for reading me!

Mark Zuckerberg in a demo of the Orion glasses.

The Orion project

Last September 25, Meta presented at its annual Meta Connect 2024 conference a device that has sparked curiosity and positive comments among some specialized media: the Orion augmented reality glasses. A pair of augmented reality glasses that, unlike Apple's headset, look like normal glasses and could in principle be worn by anyone on the street.

But let us not fool ourselves. Although Meta titles its announcement “Orion: The real AR glasses are here,” the text itself makes clear that no, they are not going to end up in consumers' hands. At least not as they are shown today. What this is really about is testing technology that will be integrated into future products.

While Orion won’t make it into consumers’ hands, make no mistake: This is not a research prototype. It’s the most polished product prototype we’ve ever developed, and it’s really representative of something that could ship to consumers. Instead of rushing it onto shelves, we’ve decided to focus on internal development first, which means we can keep building quickly and keep pushing the boundaries of the technology.

And that means we’ll get to an even better consumer product faster.

In fact, the prototype still has many shortcomings and, if it were a real product, it would cost around $10,000. There are many things left to refine, and Zuckerberg himself says in his interview with Alex Heath of The Verge that:

We still want it to be a little bit smaller, a little bit brighter, a little bit higher resolution, and much more affordable before we launch it as a product.

Some outlets are convinced

Even though the glasses are not going to go on sale any time soon, Zuck wants to show them and wants some journalists to try them. He has learned. He has realized that it is necessary to go beyond videos and the toy avatars, in Antonio Ortiz's phrase, from barely three years ago, and that he needs to open the doors of his labs to show something that can be touched and experienced.

It is curious how much Zuckerberg's image has changed in just a couple of years, and how this new version of him seems to be succeeding: a down-to-earth, self-confident young TikToker concerned with real experiences, making everyone forget the failed Metaverse stage. He himself presented the entire Meta Connect keynote, wearing a black T-shirt with the Latin phrase “aut Zuck aut nihil.” “Either Zuck or nothing,” a phrase derived from the Roman-era “aut Caesar aut nihil,” which gives him a tone that is somewhere between funny and irreverent.

Back to Orion: Zuck prepared a couple of demos that have completely won over some media outlets and analysts. People deeply immersed in the Apple world, such as Ben Thompson or Marco Arment, who say this is the real path Apple should have taken, instead of what it did with Vision Pro. It feels like we may be witnessing a new version of Jobs's famous “reality distortion field.”

For example, Ben Thompson says in his article things as emphatic as:

Orion makes every other virtual or augmented reality device I’ve tried feel like a mistake, including the Apple Vision Pro.

As for technical aspects, even though the image quality is not as good as on Vision Pro, for some reason, one he does not explain in much detail, Meta's proposal seems to have impressed him much more:

The obvious limitations, particularly the low resolution, feel irrelevant. The difference versus the Quest or Vision Pro is that actually seeing reality is so dramatically different from even the best passthrough capabilities of the Vision Pro that the quality of the holographic video doesn’t matter nearly as much.

He can even imagine replacing the iPhone with Orion glasses:

Orion’s image quality is good enough. In fact, it’s impressive. In fact — and I don’t say this lightly — it’s so good that, for the first time, it made me imagine a world where I’m not carrying a smartphone.

In short, these glasses are what Apple should be trying to build:

With all of that in mind, the big question in the next few years is the race between Apple to build something this good, and Meta to figure out how to build something that is already excellent at scale and at an affordable price.

Arment also likes Meta's glasses and, in the latest episode of ATP, uses the occasion to criticize Apple once again for its headset, for its lack of support for developers, and for its lack of interest in the product, which he openly considers a failure.

Others, not so much

Other media emphasize the limitations of the demos more, or are more convinced that Apple's approach is the correct one.

For example, Adam Savage mentions in his YouTube review several negative points:

• To get the eye tracking to work, the glasses had to be fitted precisely to the face, and they needed to stay fixed on the ears and nose. If they moved slightly, that calibration was lost and hologram tracking stopped working.

• The room where the demo takes place is a room without bright lights, with lots of angles that help SLAM and world mapping.

• The projected images have some latency, and persistence and anchoring of virtual objects still do not work fully. Windows and messages do not stay entirely fixed in the real world, and they disappear when we look away and then look back in the original direction.

In the following images we can see the room where the demo was done and an example of virtual labels positioned over real objects.

More criticism of the displays:

It has a density of 13 pixels per degree. You can read some text, look at a web page, but it isn’t sharp. There are other versions with twice the resolution, but the images aren’t as opaque as the ones in this resolution. They’re looking for a balance. They know they need to get to 30 pixels per degree before they can ship this. Objects look fuzzy and you are not going to confuse a hologram with a real object.

For example, one of the demos is an eighties-style video game.

They do the demo with the user looking at a gray wall. Honestly, I doubt that a demo like this would work in a normally lit room, with light-colored walls and objects.

And one last criticism targets the idea of reducing the weight of the glasses by putting all the processing in an external unit, a kind of small puck, that wirelessly sends the images to the glasses:

The approach of putting the processor outside the glasses and sending the image over a wireless connection is also debatable. What frame rate can you get with this? What is the maximum resolution you can get? I have my doubts that this can scale to things like watching a movie or having a crisp video call.

Analysts like Jason Snell, in his article at Six Colors, or John Siracusa, in the same ATP episode mentioned above, argue that Apple's approach is the right one.

I agree with them. Apple keeps secret what it is developing until the product is truly finished. This approach avoids creating expectations that cannot be met and protects its reputation from possible disappointment. Although this strategy sometimes gives the impression that Apple is “behind,” it may in fact be working on advanced technologies behind the scenes.

Unlike Meta with Orion, Apple already has a finished product on the market and is building a software platform and an ecosystem of applications and experiences. Apple knew how to recognize when the technology was not ready for certain innovations. Tim Cook's initial goal was always glasses like Orion, but the moment they realized that those were not feasible, they shifted their focus toward Vision Pro. In fact, we could consider Vision Pro, with elements such as the external eye display, as a “simulation” of future augmented-reality glasses. But it is a “real” simulation, usable and capable of providing real experiences rather than toy ones.

Meta's real strategy

If we agree that Orion is not going to be a real product, at least not in the next 10 years, why present it now? What is Meta's real strategy? To answer those questions, we have to study the real products Meta currently has on sale.

A headset 10 times cheaper than Apple's

The only real product Meta presented at Connect was its Quest 3S headset, an improved version of the Quest 2, with passthrough similar to that of the Quest 3. In fact, they are practically the same model, with slightly lower screen resolution, 773 PPI on the Quest 3S compared with 1,218 PPI on the Quest 3.

The Quest 3S promotional video clearly shows applications that are a clear copy of Vision Pro apps: immersive environments, a huge movie screen, or floating displays we can interact with.

However, the Quest 3S specifications are vastly worse than those of Vision Pro.

For example, Quest does not have eye tracking, its screen resolution is dramatically lower, and I very much doubt it has SLAM and object anchoring good enough to let you stand up, move around, and look at objects and windows from different perspectives. We will see what the reviews say when it officially goes on sale. It looks like a good entry-level device for games and virtual reality experiences, but my impression is that it will fall far short for work or for immersive experiences. Still, it costs ten times less than Vision Pro.

Meta is going to try to use this headset to reach a large number of people and build a user base that Apple is not managing to build. That will be good for Apple, because it will force it to move, to launch a more affordable headset, and to produce more experiences. It is very good that there is another strong competitor playing in the same space. Perhaps another duopoly is being formed, similar to iOS vs Android, but this time in virtual or extended reality: Vision OS vs Horizon OS.

Glasses as an intelligent assistant

Meta's other major bet right now is AI, with all the open-source Llama models that I have mentioned many times in this newsletter. For now that bet is taking shape in the integration of AI into applications like WhatsApp and Instagram, though we still do not have it available in the EU.

Meta wants to go further and play an important role in the new trend everyone is now chasing: devices with intelligent assistants. For now it has developed Ray-Ban glasses with a camera, microphone, and audio, connected to the network and allowing us to take photos and talk to a remote intelligent assistant.

Antonio Ortiz has tried them, was delighted with them, and wrote an excellent analysis on error500. Antonio comments that the field is already mature enough to become a consumer technology:

The possibility that an AI could assist you with what you are seeing and hearing, thanks to the new multimodal models and using natural language, is something that is within reach in the coming months.

But Meta runs into the problem of privacy and social acceptance:

How much we will accept having a camera pointed at us all the time, even if it warns us with a light when it is recording; how closely we will watch whether the person we are talking to is no longer paying attention because they have some other content overlaid in their glasses that interests them more; how alienated we will feel, unable to escape the hacking of our dopamine cycle that current content platforms have achieved.

Google Glass fell off the board more than 10 years ago, leaving a scar on this emerging technology. The Orion project leans on the promise of the Metaverse and on the reality of Vision Pro to try to erase that mark. As Antonio says, it is trying to ensure that the adoption of glasses is not forever linked to Scoble in the shower.

Orion presents us with a socially acceptable technological utopia, one where virtual interactions augment reality and allow us to stay more connected with other people. The aim is that, once the narrative has changed, some of its simpler elements, such as floating panels or video calls, can be introduced into new versions of smart glasses without being perceived as invasive features that threaten our privacy.

I think Meta has a very difficult path ahead. News such as the recent story about students who, using only Meta's glasses, extracted lots of information from strangers they passed on the street are not going to help at all.

In addition, Apple is not going to stand still and has already announced similar features on the iPhone with its Apple Intelligence. For now it is only getting started, but it will surely push very hard and try to make the iPhone the smart device that helps us, and the device we use to capture the environment when we want to ask something, as shown in this Apple video promoting a feature called Visual Intelligence, not yet available:

Will it be more socially acceptable to wear glasses than to take out your phone and ask it? Smartwatches have accustomed us to wearing something that lets us avoid pulling out the phone. They make us feel better because they let us disconnect from the phone. Will the same happen with glasses? I do not know. I think not. I think it is going to be very difficult to convince people to spend more than €1,000 on glasses that will do things very similar to what we can already do with a phone and a pair of AirPods.

In 2030

Other times I have ended articles with questions about the future. I like to think of this newsletter as a place I will keep revisiting in a few years, rereading it, reviewing the questions, and checking what has been answered and what has changed.

This time I am going to commit and make 5 predictions for early 2030:

1️⃣ The Orion project, as it is presented now, will still be in the lab. Meta will not manage to build glasses that do augmented reality in a way similar to Vision Pro. For example, it will not be possible to have virtual elements anchored in the real world, virtual screens able to compete with a monitor, or virtual games in which you interact with remote people while sharing fixed and persistent virtual elements as you move around.

2️⃣ Some elements of the Orion project, such as floating notification panels or signs, will have been incorporated into smart glasses, an advanced version of today's Meta Ray-Bans. They will cost somewhat more than €1,000 and some of the influencers of the moment will wear them, but they will not become especially popular. Their use will remain quite limited because of privacy concerns, and many people will prefer to use the phone to point and query. Or perhaps some new device that OpenAI releases as a result of its collaboration with Jony Ive.

3️⃣ Apple will have consolidated its Vision Pro platform. Two models will be sold, a cheaper one, around €1,500, and a more expensive one, around €2,300. Meta will have raised the price of its virtual reality headsets and will sell them at €600 and €1,000. Meta will have many more users than Apple, but both platforms will be in continuous growth and their future will not be questioned.

4️⃣ Cameras for filming immersive experiences will be common in the industry, and there will be production companies specialized in filming this kind of experience, with films, plays, and so on. They will be published on both Apple's and Meta's platforms.

5️⃣ Trials will begin with the first sporting events and shows broadcast live in immersive format. YouTube and Netflix will end up betting on these new formats and will have native applications on both Apple's and Meta's platforms.

In 5 years we will review the predictions 😜

Until the next fortnight, see you then! 👋👋