The rumour that Apple is developing AR glasses isn’t new; claims that the company had its sights set on such a device have been circulating for a few years now. But things got interesting in 2017 when the announcement of ARKit at WWDC made it clear that Apple is deeply interested in this area. And no wonder: with Google and Microsoft and others actively developing AR devices, it seems like a party Apple can’t afford to miss.
It also hasn’t escaped our notice that for a number of years now Apple’s smartphones have been well-equipped for AR, thanks to bionic chipsets, which enable AR processing.
Apple certainly looks like it is gearing up to be a big player in the AR field. But what exactly does Apple have in its sights and when are we going to see the new glasses?
A couple of patents granted in January 2020 give some insight into Apple’s plans for a headset. Read our section on design below for more information. In fact, there may be two headsets in the pipleline: a AR/VR headset as well as sunglasses-like AR specs – more on that below.
In this feature, we look at all the latest Apple AR glasses rumours, patents and possible release date details, including iOS 13 support for AR glasses.
What is augmented reality?
First, a quick refresher on terms, as many get AR and VR confused. VR headsets are mounted on the head in a similar way to Ski goggles, and completely block your view of the outside world. The VR headset tracks your head movement, and the 3D image displayed inside the headset moves accordingly. This makes it appear as if you’re wholly inside a 3D ‘virtual’ world.
Augmented reality, on the other hand, is where the glasses are see-through and you can still see the world around you, but an image is displayed in front of your eyes.
AR hit the headlines in recent years, first thanks to Google Glass (which displays 2D images) and more recently with a headset developed by Microsoft called HoloLens that embeds 3D images in the world around you.
Microsoft’s second-gen HoloLens glasses are already available for developers – although anyone can buy one. However, it costs the astronomical price of $3,500 and is focused on the enterprise market rather than consumer.
As for Google Glass, development of the consumer-focused product was halted back in 2015, but Google revealed a new-and-improved Google Glass Enterprise Edition 2 in 2019.
If Google and Microsoft are all working on AR products, you can bet your bottom dollar that Apple also has an AR headset prototype in its labs. Apple CEO Tim Cook has gone on record discussing the big possibilities he sees for AR, as we discuss below.
However, it’s an industry that is in its infancy. With that in mind, when can we expect to see Apple’s AR headset or glasses?
Apple AR glasses: Release date
Assuming that Apple is working on an AR headset or AR glasses – there is plenty of evidence (gathered below) that Apple is working on an AR headset – when is such a device likely to launch?
A 2020 release date was looking likely but it now appears that Apple has pushed the date back to 2022.
The Information is reporting that at an internal Apple meeting Apple executives have shared a roadmap which includes plans for two augmented reality devices. An augmented-reality headset set to appear in 2022 and AR glasses coming a year later in 2023.
We may get official confirmation of Apple’s plans in this field before that date, however. According to the report, Apple intends to reach out to developers in 2021 with the hope that they will design software for the headset.
At the same time as the Information report appeared, Bloomberg also published a report stating that sources had confirmed that Apple would be delaying the release of its headset from 2020.
The first report of a 2020 release date for the glasses came way back in November 2017 via Bloomberg, and was backed up by reputable sources in the months and years since.
An April 2018 Cnet report suggested that 2020 would be the launch date, according to ‘sources’.
Then, in March 2019, analyst Ming-Chi Kuo reported that Apple’s first AR product is set to go into mass production as soon as Q4 2019, with a deadline of Q2 2020. He also detailed features of the headset, but we’ll go into that in more detail below.
At the beginning of September 2019, the rumours started up again after code was spotted in the iOS 13 beta. MacRumours reported that documentation in an internal iOS 13 build indicated that AR headset support was being built into iOS 13, there was even a reference for a device codenamed ‘Garta’. A clear indication that something is in the works at Apple and that it may soon launch.
It’s not only iOS that’s revealing clues to the presence of an AR device. In September 2019 9to5Mac’s Guilherme Rambo tweeted that code in Xcode 11 included a framework for an ARDisplayDevice with files for AR headsets codenamed ‘Franc’ and ‘Luck’ and HoloKit codenamed ‘Garta’.
So the Xcode 11 GM released today includes the ARDisplayDevice framework, including data files for Apple’s AR headsets (Franc, Luck) and HoloKit (Garta). WHAT? https://t.co/TxaX0un1dk
— Guilherme Rambo (@_inside)
September 10, 2019
Then, towards the end of October 2019 Bloomberg published a report indicating that Apple had a date in 2020 in mind for the release of an AR headset.
However long we have to wait, Loup Ventures analyst Gene Munster believes that augmented reality will be the next big thing for Apple. Munster is the analyst who was once convinced that Apple would launch a television set.
Munster expects that Apple will release Apple Glasses in late 2021, a year later than Kuo’s predicted release date but closer to what the sources at Apple are now indicating. It’s something Munster says we aren’t yet ready for, “but eventually we will be”.
Price
We expect that when it launches Apple’s AR headset will have a high price, as is Apple’s custom. But we hope that the price will be a little more down-to-earth than that offered by the competition currently.
Microsoft’s HoloLens 2 glasses cost $3,500 (approx £2,661), while the Google Glass Enterprise Edition costs $1,828 (approx £1,336).
We estimate a starting price of £999/$999 for a consumer-focused product, but with no existing products to compare to, it’s a seller’s market.
Design
It appears that there are actually two different AR products in the pipeline. A headset (codenamed N301) which will combine AR and VR features. This is said to resemble the Oculus Quest (which launched in May) and feature a 3D sensor system that builds on the technology of the Face ID camera set up to create a map of a user’s surroundings. The headset will be used for “gaming, watching video and virtual meetings,” according to a Bloomberg report in November 2019.
In addition, there will be smaller AR glasses more suited to being worn for longer periods of time. According to The Information’s source (also in November 2019) the current prototypes “look like high-priced sunglasses with thick frames that house the battery and chips.”
Back in June 2017, an iPhone Hacks report claimed that Apple’s smart glasses would have a unique design, according to their source.
Their insider provided details on the design – featuring a bone conduction earpiece, a microphone, an accelerometer and a capacitive frame for navigation and interaction, the glasses sound impressive.
Apple has been awarded patents that give some insight as to how the new glasses could look. One patent describes a way to conceal optical components behind an electrically-controlled layer so that they are hiden from view when not required. Another patent indicates that earphones could be removed from a head-mounted display and used as speakers (via this AppleInsider report.)
Some creative people have already come up with potential designs for Apple’s glasses. For example, this YouTube video (which enthusiastically claimed that the specs would launch back in 2018) has some neat designs.
Partners
In October 2019 analyst Ming-Chi Ku claimed that Apple will be collaborating with ‘third-party brands’ to design the headset, as per an October 2019 report (in Chinese).
A month later, in November 2019, a DigiTimes report suggested that Apple is partnering with Valve to make the AR glasses. DigiTimes reported back in July 2019 that Apple was ceasing development of the glasses, but it seems that the change was a transition to the partnership with Valve.
Another potential partner is Imagination Technologies. Apple and Imagination Technologies had a bit of a falling out a few years ago when Apple ended its licensing contract with the British firm. But now it appears that Apple has struck a deal with Imagination Technologies due to their expertise in ray tracing, which is a rechnology that renders graphics with realistic lighting, reflections, and shadowing, so that they look real to the human eye. This kind of technology will be invaluable for AR applications. (Via Venture Beat)
Specs
We don’t know much about the technologies being used in Apple’s AR specs (get it) but there are a few rumours circulating that could give an indication about what to expect.
Display
Apple has a number of patents related to AR glasses, leaving little doubt that the company is interested in the technology. One patent published by the US Patent and Trademark Office in October 2019 refers to Photochromic Lenses, which are the kind of glasses lenses that get darker when in bright sunlight.
In the patent Apple notes that the “Head-mounted device such as a pair of augmented reality glasses” may have an adjustable opacity system, such as a photochromic layer. We assume that this technology would make it easier to read the screen in bright sunlight.
According to a cnet report in April 2018, Apple’s AR headset will offer an 8K display for each eye.
The 8K displays will make the VR and AR images look more lifelike and will help avoid the nauseous feeling that can be the result of using AR/VR headsets.
An older report from June 2017 suggested that the smart glass being used in Apple’s AR headset was being manufactured using a Kopin NED Acetate frame.
There were also claims that Apple is looking to sell its AR glasses with prescription lenses or even polarised lenses in partnership with Zeiss.
Processor
A cnet report in April 2018 suggested that the AR glasses won’t require a computer or smartphone to run.
Cnet’s source claimed that the headset will require a dedicated box powered by a custom Apple processor to which it will connect via a “high-speed, short-range wireless technology”. The box will use the wireless technology 60GHz WiGig, according to cnet’s source. The 60GHz WiGig signal cannot typically penetrate walls but it can propagate off reflections from walls, ceilings, floors and objects using beam-forming. This may not matter if the headset is used in just one location.
This does mean that users won’t be required to install various base stations around a room to enable to headset to determine their location (as is the case with HTC Vive). Everything will be included in the one box and headset.
According to cnet, the box will feature a 5-nanometer processor. Apple is designing the chip in-house and the chip will be similar to the chips Apple is said to be developing for its Macs, with a switch from Intel planned for 2020.
However, that conflicts with the more recent claims of Ming-Chi Kuo, a man with a proven record of accurate predictions. Kuo states that the first-gen AR glasses will be heavily dependent on the iPhone, with the AR glasses essentially only providing the display. That leaves the iPhone to handle the computing, rendering, internet connectivity and location services from the users pocket – hinting that, like with the Apple Watch, Android users won’t be able to get in on Apple’s AR fun.
While we assume that all this computing will be done wirelessly, it’s not explicitly mentioned in Kuo’s report.
If the iPhone handles all the heavy lifting, Apple can create a headset that is light and stylish, but there are also concerns about the impact to the battery life of your iPhone if it has to power the AR experience.
Features
So far the evidence seems to point to the AR glasses being dependent on the iPhone, as the Apple Watch was when that launched. They will work as a display to deliver maps, messages and games directly to your field of vision.
Bloomberg’s sources claim that the glasses will wirelessly connect to an iPhone and will display not only information but images and other data directly in front of the wearer’s field of view.
KGI analyst Ming-Chi Kuo believes that Apple could integrate AR to redefine key product lines. For example, an AR-powered interface could change the way that users interact with small-screen devices like the Apple Watch, eliminating issues with clunky controls.
Software
A new operating system, dubbed rOS (r for reality), is being built for the AR headset. It is based on iOS, according to Bloomberg.
rOS will power the AR headset. Apparently, the ‘rOS’ software group is headed by Geoff Stahl, formerly a software manager for games and graphics at Apple.
As for how you will get apps, Apple has discussed pairing the headset with its own version of the App Store, where users would be able to download content, just as they do with the iPhone, Watch, Apple TV and Mac, according to Bloomberg.
Apple’s AR device is here now!
We don’t actually have to wait for Apple to launch an AR headset. The company already sells an AR device – the iPhone.
For most people, their first taste of AR came when they started playing the popular Pokemon Go app on their iPhone in the summer of 2016.
Now, thanks to the TrueDepth cameras in the iPhones X, XS, XR, and 11-series, the bionic chipsets offered by the A12 processor and beyond, iOS 12 – which was described as “the world’s largest augmented reality platform,” and now iOS 13, which arrived with ARKit 3. The iPhone is taking augmented reality to the next level.
When it arrived in 2017, the iPhone X’s AR credentials weren’t a surprise, a Bloomberg report back in March 2017 claimed that AR would be introduced on the iPhone first, with glasses coming later down the line.
That report claimed that there are “hundreds of engineers devoted to the cause” including a number of people from Apple’s iPhone camera team working on AR-related features for the iPhone.
The Bloomberg report was supported by a claim from a Business Insider source (in November 2016) that Apple has teams from several acquired startups working on iPhone AR, and this reflects Apple’s short-term desire to put AR into the hands of consumers.
By adding AR capabilities to the iPhone Apple allows consumers to point the phone at a real-world object, like a statue or a car, and be offered contextual information overlaid onto the real world.
Integrating AR into a smartphone that millions of people already have introduces the concept of AR, and may make them more likely to buy Apple-made glasses once they go on sale.
Speaking of which…
Evidence that Apple’s making AR glasses
Claims that an Apple AR headset is in the works at Apple stretch back some years and a number of reports have appeared presenting evidence that the project is well underway at Apple. In fact, the project even has a code-name – T288.
In-house team
There have been rumours that Apple has been making AR/VR related hires some time. Back in November 2016 a Business Insider source claimed that Apple has hired an expert in head-mounted displays (HMDs).
Also in November 2016, a Bloomberg report claimed Apple had ordered a small number of “near-eye displays” for internal testing displays.
Citing people with knowledge of the project, in a March 2017 report, the FT claimed that Apple had started to assemble a team of augmented reality experts a year previously. Apparently, more resources have been devoted to the AR project than Apple’s rumoured driverless car.
Then in April 2017, more evidence of some sort of AR project came from leaked Apple employee injury reports obtained by Gizmodo. While they’re mostly about mundane workplace injuries like cafeteria burns, there were two that stand out:
One involved someone testing an unknown prototype device who “experienced discomfort in her eye and said she was able to see the laser flash at several points during the study.” She was later referred to an optometrist.
Another involved an employee who “reported eye pain after working with a new prototype, though it may be associated with use”.
While these could reflect either an AR or VR device, an internal source told Gizmodo that the injuries were likely linked to an AR prototype Apple is working on, speculating that it could be “something like glasses with an overhead display”.
A Bloomberg report, from November 2017, claimed there is a team of several hundred engineers working on AR-related projects, according to its sources.
The team, led by Mike Rockwell, who previously ran engineering at Dolby Lab, is working on several hardware and software projects under the umbrella code name of ‘T288’, according to Bloomberg.
The team also worked on ARKit. The tool that lets software developers build AR apps for iPhone and iPad. We discuss ARKit in more detail below.
More recently, as of July 2019, Kim Vorrath was said to have joined Apple’s AT team. This news came via a report in The Information that claimed Apple software executive Vorrath had been moved within Apple to the AR team. She’d been tasked with “bringing some order” to the team of about a dozen.
Earlier in July 2019 it had been rumoured (via DigiTimes) that Apple was shutting down the project, but Vorrath’s move seems to suggest otherwise.
Then in September 2019 it emerged that Apple had hired Arthur van Hoff founder of VR content startup Jaunt. Hoff was hired as a Senior Architect, suggesting he will be placed in the VR/AR team.
Not everyone Apple has hired into AR/VR related roles has remained at the company. HoloLens co-creator Avi Bar-Zeev had been working at Apple for three years but had left as of around July 2019.
Acquisitions
Apple’s also been acquiring companies in the AR space over the past few years.
Back in May 2015, Apple acquired a German company called Metaio. It was a relatively well-known company with regards to augmented reality, powering many of the popular AR applications being used today – including Ikea’s virtual catalogue, and Ferarri’s AR showroom app.
Since being bought out by Apple, Metaio has gone quiet – heading to its website will offer you a brief overview of the company, a contact email address and not much else. Mysterious.
Apple has confirmed the acquisition of Flyby Media, a company well-known for developing technology that allows smartphones to ‘see’ the world around them. Notably, the company has worked with Google in the past to develop the image-recognition abilities found in Google’s Project Tango.
Though the website is now down (typical of an Apple acquisition), Flyby Media described itself as “dedicated to building new technology that can elevate, rather than replace, our real-world experiences.” and boasted a team with knowledge in the areas of large-scale SLAM, indoor navigation, sensor fusion, image recognition as well as 3D tracking.
Apple at CES
Another Bloomberg report, back in January 2018, presented more evidence that Apple has plans in the AR space. The publication claims that, despite not exhibiting at the show, Apple was present at CES 2018, with execs meeting suppliers to discuss AR glasses, according to people familiar with the meetings.
Apple’s Tim Cook on AR
There is yet more evidence that Apple is working on something in the AR space – this time coming from Apple itself.
Apple CEO Tim Cook said in an August 2016 interview with The Washington Post that Apple is “doing a lot of things” in the augmented reality space. He didn’t stop there though, as the Apple boss went on to claim that he thinks of it as a “core technology” for the company.
“I think AR is extremely interesting and sort of a core technology,” Cook stated during the interview. “So yes, it’s something we’re doing a lot of things on behind that curtain we talked about.”
Following on from these comments, Cook took part in a rare interview session at Sen. Orrin Hatch’s Utah Tech Tour in Salt Lake City in October 2016 where he again discussed augmented reality, although this time he gave us a better idea on what Apple may be working on.
“AR I think is going to become really big,” said Cook. “VR, I think, is not gonna be that big, compared to AR … How long will it take? AR gonna take a little while, because there’s some really hard technology challenges there. But it will happen. It will happen in a big way. And we will wonder, when it does [happen], how we lived without it. Kind of how we wonder how we lived without our [smartphones] today.”
It’s interesting that Apple is focusing more on augmented reality than virtual reality, although it does make sense – especially with the booming popularity of apps that feature the technology like Pokemon Go, and the introduction of ARKit which makes it much easier for developers to offer AR capabilities in their apps.
Cook sees AR as becoming something that a huge part of the population will use on a daily basis, going on to state that he thinks “that a significant part of the population, of developed companies, and eventually all countries, will have AR experiences every day”.
Speaking to The Independent during a trip to the UK in February 2017, Cook said that he prefers AR over VR because it “allows individuals to be present in the world but hopefully allows an improvement on what’s happening presently.”
Considering that VR seems to provide the more immersive and enjoyable experience of the two platforms at the moment, why does Cook believe that AR is the future? People don’t want to be cut off from the world, apparently.
“Most people don’t want to lock themselves out from the world for a long period of time and today you can’t do that because you get sick from it,” he said, discussing virtual reality. “With AR you can, not be engrossed in something, but have it be a part of your world, of your conversation. That has resonance.”
Cook even went as far as to liken it to the smartphone, and how that has changed the world. “I regard it as a big idea like the smartphone” he claimed. “The smartphone is for everyone, we don’t have to think the iPhone is about a certain demographic, or country or vertical market: it’s for everyone. I think AR is that big, it’s huge.”
Tim Cook also addressed AR and ARKit during the Q3 2017 earnings call. When asked a question about how people will be using ARKit, Cook replied “I could not be more excited about AR and what we’re seeing with ARKit in the early going.” He continued to explain the variety of ways he has already seen AR used, and that he thinks “AR is big and profound and this is one of those huge things that we’ll look back at and marvel on the start of it.”
iOS & AR
Yet more evidence that Apple is interested in AR: Apple added ARKit as part of iOS 11 and has since updated ARKit in iOS 12 and again as part of iOS 13.
ARKit allows developers to offer high-end augmented reality capabilities in their apps without having to develop the AR system, and improves existing AR solutions too – for example, Pokemon GO running ARKit looked much smoother than it did previously.
Per Apple’s ARKit page on its website, it claims that ARKit is a “cutting-edge platform for developing augmented reality apps for iPhone and iPad” and it offers “powerful capabilities for positional tracking and scene understanding”.
It makes sense for Apple to offer AR as part of the iPhone and iPad experience, as it’s a device already in the hands of millions of consumers. It also encourages developers to create AR apps that could in future be ported to an AR headset, if that’s the route that Apple decides to go down.
Plus, it makes iOS the biggest AR platform in the world. Not too bad for emerging technology!
Interestingly, iOS 13 actually introduces support for stereo AR glasses. The MacRumours team inspected the most recent iOS 13 beta and spotted an app that suggests Apple’s AR glasses are actively in development and could appear sooner rather than later.
First up, the iOS 13 beta includes the STARTester app which MacRumours claims can switch in and out of a head-mounted mode “presumably to replicate the functionality of an augmented reality headset on an iPhone for testing purposes”. Per the report, the app features two modes – worn and held – which further supports the idea that Apple is working on a pair of AR glasses.
There’s also an internal Readme file hidden within the iOS 13 code that describes a Starboard system shell for use with the vast collection of existing AR-powered iOS apps, and also indicates a possible internal prototype codename for the AR glasses – Garta. The Starboard mode is referenced in several places in the iOS 13 code, with “views” and “scenes” and, somewhat more blatantly, “ARStarBoardViewController” and “ARStarBoardSceneManager”.
There was no mention of Apple’s AR glasses during the company’s September 2019 event, but it looks like Apple is preparing for something big…
MacOS and AR
There is also evidence in Xcode that an AR device is in the pipeline. Code in Xcode on the Mac has confirmed Apple’s work on some kind of AR headset.
In September 2019, developer Steve Troughton-Smith tweeted that Xcode 11 referred to codenamed test devices as well as frameworks and a system shell related to these devices.
There was also a ReadMe file that referenced an Apple “HME” headset.
Apple AR Patents
In January 2017, Apple was granted a pair of patents that could give us a glimpse at the company’s AR aspirations. Patent no. 9,560,273 details the hardware framework of an AR device that can understand its environment thanks to enhanced computer vision capabilities, while patent no. 9,558,581 details the method of overlaying virtual information on a physical environment.
Both patents were filed back in 2015 by German AR specialist Metaio shortly before Apple acquired the firm, and transferred to Apple in November 2016.
Patent 273 describes a “wearable information system having at least one camera” but goes further and discusses the possibilities of having more cameras, a screen, a UI and even internal components dedication to computer vision. The patent details a head-mounted display as the primary platform for AR, but it also suggests that a smartphone (aka the iPhone) could serve as a decent stand-in.
However, on the whole, the patents deal more with object recognition than it does the visualisation of the AR data. The issue of object recognition is a barrier in the AR industry on the whole, as existing solutions require a large amount of power and thus, aren’t ideal for real-world use.
In Apple’s case, the ‘invention’ maintains a low-power scanning mode for the majority of its operation, with high-power modes triggered in relatively short bursts.
Then, in May 2017, a patent surfaced depicting the head-mounted display of Apple’s upcoming augmented reality headset. The patent details a device and techniques for projecting a source image in a HMD comprised of left and right displays, suggesting that they may look more like standard glasses than Google’s failed Glass.
In the filing, Apple notes that some HMDs can be used to view a transparent image imposed upon a real-world view. Sound familiar to augmented reality? We think so too.
Another Apple patent filing, discovered in July 2017, describes how a pair of AR-enabled glasses could interact with a smartphone. The user is said to be able to interact with the objects that appear on the glasses using their smartphone. Of course, the phone would need to be equipped with the necessary hardware and software to be compatible with the glasses, and we’re unsure if the tech is already featured on the iPhone 7 or not.
Another concept in the patent describes how the user can interact with objects that appear on the AR glasses by pointing at them – a similar mechanic is used by Microsoft in its HoloLens glasses, so it’s not a far-fetched idea. The patent claims that it’ll be possible via an array of cameras attached to the glasses, as the cameras will map and track the user’s finger as it moves towards the on-screen object.
However, a patent published in Q3 2018 suggests that the AR headset could be controlled via a touch-enabled device like the iPhone. As well as depicting a person wearing a headset and interacting with a touchscreen device, the patent specifically details how “the user interacts with the GUI in part through stylus and/or finger contacts and gestures on the touch-sensitive surface.”
With more advanced patents appearing, it suggests that Apple is taking augmented reality very seriously, and we might be seeing Apple’s AR-enabled device on the market sooner than originally thought.
KGI analyst Ming-Chi Kuo told investors that he believes that Apple’s track record of delivering innovative and high-quality user experiences will help the company move into – and succeed – in the AR market.
Via MACWORLD