Apple CEO Tim Cook believes augmented reality's rise will be as "dramatic" as that of the App Store, but he doesn't believe AR glasses or similar wearables are ready for the market yet, according to a sit-down interview with The Independent. Much of Cook's interview focused on the prospects of augmented reality and Apple's justification for making it a focus in both iOS and the iPhone 8.

He said this to The Independent:

Think back to 2008, when the App Store went live. There was the initial round of apps, and people looked at them and said, "this is not anything, mobile apps are not going to take off." And then, step by step, things start to move. And it is sort of a curve, it was just exponential–and now you couldn't imagine your life without apps. Your health is on one app, your financials, your shopping, your news, your entertainment–it's everything. AR is like that. It will be that dramatic.

iOS 11, the latest software release for iPhones and iPads, included ARKit, a framework for developing augmented reality applications around the iPhone's robust suite of sensors and cameras. It doesn't enable anything that has never been done before in AR, but it is intended to greatly increase ease of development of AR applications for one of the most robust software markets in the world—the iOS App Store.

Because handsets as far back as the iPhone 6S are supported, smartly designed AR apps based on ARKit will find a large and mature market of consumers. Cook believes his company is providing a sort of garden for growing great AR ideas. He's quoted saying: "The way that you get lots of great ideas is for us to do the heavy lifting of the complexity of locational things and software, and put those in the operating system... And then you have all the developers that are able to put their energy into their passion." Cook also said that he believes day-to-day experiences like shopping will be "entirely" changed by AR.

Ars' iPhone 8 review called it "the best mass-market AR platform we've yet seen." But the best mass-market platform is not necessarily the most advanced. Even as many developers are creating AR apps for ARKit on iPhones or for Google's ARCore platform, others are focused on platforms like Microsoft's HoloLens and Windows Mixed Reality platform, which have some capabilities the iPhones don't. Some of these competing concepts are worn like goggles over your eyes.

Cook didn't address all of these technical differences in the interview, but he did talk about the prospects for AR goggles or glasses. He believes the tech isn't ready:

The technology itself doesn't exist to do that in a quality way. The display technology required, as well as putting enough stuff around your face–there [are] huge challenges with that. The field of view, the quality of the display itself—it’s not there yet.

Using HoloLens shows the potential of AR glasses, both in the sense of them being wearable and in the sense that HoloLens offers some technologies that aren't yet on mass-market consumer phones—like better mapping of 3D assets over varied terrain rather than just flat surfaces. But at the same time, there are terrible limitations. HoloLens is bulky and unattractive. The field of view is pitifully small, breaking all potential immersion in what's being displayed. Granted, things have moved forward a bit since HoloLens was introduced, but they haven't moved that far.

But even with those limitations, it's easy to conclude while trying HoloLens that wearables are where AR ultimately needs to go in the long run to achieve its full potential, despite the poor reception of the now ancient-seeming Google Glass concept. Holding up a phone just isn't the same. But judging from Cook's comments, Apple nevertheless believes the phone AR experience is enough for a revolution in its App Store. As attractive as the longterm vision of wearable AR might be to many, he's probably not wrong.

Samuel Axon
Based in Los Angeles, Samuel is the Senior Reviews Editor at Ars Technica, where he covers Apple products, display technology, internal PC hardware, and more. He is a reformed media executive who has been writing about technology for 10 years at Ars Technica, Engadget, Mashable, PC World, and many others. Emailsamuel.axon@arstechnica.com//Twitter@SamuelAxon

137 Reader Comments

I can see a potential path forward where people start to use AR devices on smartphones and a general population feeling will organically build of people thinking "wouldn't it be great if we could use this app without having to point our phone at it" and then suddenly people who were skeptical about Google Glass will give a warm welcome to its descendants with nary an objection.

Cook is basically stating the obvious, minus the self-congratulation on the app store.

Pokemon Go is arguably the most widely used AR mobile app right now, and it shows that battery on phones is remains a problem for AR applications. I've never used it, but any of my friends who do are always either low on battery or looking for a plug/their battery pack to maintain.

As for AR goggles, it seems like that'll be the future eventually, but we are still pretty far away from sci-fi movies where people have what looks like traditional glasses doing it, or even more far-fetched (for now), contact lenses doing it.

Cook is basically stating the obvious, minus the self-congratulation on the app store.

Pokemon Go is arguably the most widely used AR mobile app right now, and it shows that battery on phones is remains a problem for AR applications. I've never used it, but any of my friends who do are always either low on battery or looking for a plug/their battery pack to maintain.

As for AR goggles, it seems like that'll be the future eventually, but we are still pretty far away from sci-fi movies where people have what looks like traditional glasses doing it, or even more far-fetched (for now), contact lenses doing it.

That might actually not be a problem today. Pokemon Go is hard on phone battery mainly because the developers had to cobble together their own AR implementation using the existing phone API. Which probably isn't very battery optimized.

Having ARKit (and ARCore) implemented in the OS (and common AR functions optimized) should alleviate that greatly.

I'd be curious to see if the developers of Pokemon Go bother to swap their code-base to ARKit/ARCore.

That might actually not be a problem today. Pokemon Go is hard on phone battery mainly because the developers had to cobble together their own AR implementation using the existing phone API. Which probably isn't very battery optimized.

Having ARKit (and ARCore) implemented in the OS (and common AR functions optimized) should alleviate that greatly.

I really don't see it. GPS location tracking plus an always-on screen (likely at a high brightness level to compete with sunlight) will drain the battery regardless of processing overhead.

That might actually not be a problem today. Pokemon Go is hard on phone battery mainly because the developers had to cobble together their own AR implementation using the existing phone API. Which probably isn't very battery optimized.

Having ARKit (and ARCore) implemented in the OS (and common AR functions optimized) should alleviate that greatly.

I really don't see it. GPS plus always-on screen (likely at a high brightness level to compete with sunlight) will drain the battery regardless of processing overhead.

I don't know, I played Pokemon Go casually for a few months (maybe 2 hours a day total, pulling it up whenever I went somewhere) and never really even noticed being short on battery. Sure, if you're more hardcore and are going to go out for long sessions I'm sure it would be different, but many AR use cases are not multi-hour events. Some examples I've seen are to point your phone at a food court and have menus overlaid onto each restaurant.

I've been enthusiastic about a lot of mobile technology, but I can't say I'm thrilled by AR. It seems the only people that are singing its praises are those involved in developing and producing it, while the average consumer probably still isn't eager to live in a world where a camera is pointed at them during every human interaction. We've seen it with Google Glass, and it isn't a problem that is easily solved (or if it's one that even should be solved). I think the tech world really should try to sober up a bit from the high they got from smart phones taking off the way they did in the past ten years or so.

I'm clearly not part of his target demographic, because all I could think of when I read this was "No."

Indeed. If anything, the web will win (as usual) over the App Store. Apple has been hesitant to implement the technologies that make progressive web apps possible, precisely because it will cannibalise native apps, but no matter how much they drag their heels it will happen eventually.

I've been enthusiastic about a lot of mobile technology, but I can't say I'm thrilled by AR. It seems the only people that are singing its praises are those involved in developing and producing it, while the average consumer probably still isn't eager to live in a world where a camera is pointed at them during every human interaction. We've seen it with Google Glass, and it isn't a problem that is easily solved (or if it's one that even should be solved). I think the tech world really should try to sober up a bit from the high they got from smart phones taking off the way they did in the past ten years or so.

I've seen some really cool uses for AR that don't have much to do with interacting directly with other people. Some of them are head-mounted, but the point is the same that they're AR enabled. Things like overlaying instructions/part labels when working on a car, pointing your phone at a cubicle area and getting labels for who sits where, the thing I mentioned earlier about pointing a phone at a food court and having menus or restaurant wait times overlaid. Being able to see through walls or objects. All information that can be gotten elsewhere but overlaying information with the world can be really useful and present interesting new workflows.

For the privacy issue, you can make use of AR without necessarily equipping people with "google glass" style always-on HUD. Maybe a doctor or mechanic wears it while they are working.

I've been enthusiastic about a lot of mobile technology, but I can't say I'm thrilled by AR. It seems the only people that are singing its praises are those involved in developing and producing it, while the average consumer probably still isn't eager to live in a world where a camera is pointed at them during every human interaction. We've seen it with Google Glass, and it isn't a problem that is easily solved (or if it's one that even should be solved). I think the tech world really should try to sober up a bit from the high they got from smart phones taking off the way they did in the past ten years or so.

I've seen some really cool uses for AR that don't have much to do with interacting directly with other people. Some of them are head-mounted, but the point is the same that they're AR enabled. Things like overlaying instructions/part labels when working on a car, pointing your phone at a cubicle area and getting labels for who sits where, the thing I mentioned earlier about pointing a phone at a food court and having menus or restaurant wait times overlaid. Being able to see through walls or objects. All information that can be gotten elsewhere but overlaying information with the world can be really useful and present interesting new workflows.

For the privacy issue, you can make use of AR without necessarily equipping people with "google glass" style always-on HUD. Maybe a doctor or mechanic wears it while they are working.

I totally agree and actually work in a field where this will be revolutionary. I'm referring to everyday consumer use though, which is where the tech world seems to want it to go.

I'm clearly not part of his target demographic, because all I could think of when I read this was "No."

Indeed. If anything, the web will win (as usual) over the App Store. Apple has been hesitant to implement the technologies that make progressive web apps possible, precisely because it will cannibalise native apps, but no matter how much they drag their heels it will happen eventually.

Apple initially bet entirely on the web for the iPhone. Apps were added because they made a lot of sense, but their initial direction may end up proving the correct one. We're a long way from there yet.

You seem to be conveying that if everyone were like Apple there would be no pushing the envelope or progress. I think it's folly to assume they aren't playing with these things and testing the waters just like other companies are. They just aren't open about it or letting the public guinea pig the ideas. When Tim says the tech isn't there yet, I'm assuming that means they've actually built some google-glass or hololens like prototypes and weren't happy with them, perhaps constantly rebuilding and reevaluating, especially with how bullish they seem to be on AR.

I've been enthusiastic about a lot of mobile technology, but I can't say I'm thrilled by AR. It seems the only people that are singing its praises are those involved in developing and producing it, while the average consumer probably still isn't eager to live in a world where a camera is pointed at them during every human interaction. We've seen it with Google Glass, and it isn't a problem that is easily solved (or if it's one that even should be solved). I think the tech world really should try to sober up a bit from the high they got from smart phones taking off the way they did in the past ten years or so.

AR has the potential for corporate/government malice in the form of DRM and/or soft censorship.

The former in the form of interactive content blocking ie. You go at a large Comic Con event and every time you take pictures/videos of posters, toys on display, movie trailers, you get nothing but black splotches and a text line telling you certain images/sounds are blocked on copyright grounds.

And the latter as a way to keep users from recording police/military/government/corporate abuse without resorting to something as harsh as a kill switch for your phone.

Apple has always followed the "Strong Opinions, Lightly Held" approach to product development.

Having a strong opinion means you invest heavily in what you think is right, and avoid wasting resources on everything else.

But you must hold onto those opinions "lightly", because everyone is fallible. No matter how strong your opinion is, you have to easily recognise when you're wrong and switch focus abruptly when you change your mind.

An example of that is finger print sensors. Earlier this year Apple was expanding fingerprint sensors to new product lines (such as the MacBook Pro), but now they've dropped it from their flagship product and in interviews they've basically said fingerprint sensors are bullshit, face recognition is where it's at, and they want to remove all their finger print sensors ASAP.

Exactly. He'll say the tech isn't there up until the exact moment his company's product is ready to hit market, and then the story will change to "you won't believe what we've done here."

He's speaking as a salesman and an advocate for his particular company, not as a neutral and authoritative source on what technology is capable of in 2017. That's okay, that's his job, but everything he says should be seen with that context in mind.

I can imagine one of the first apps developed for AR glasses will be ‘x-ray vision’ that superimposes a prediction of everyones naked body over the top of their clothing. As a result anyone that wears the glasses in public will be automatically labelled as a creep.

I can imagine one of the first apps developed for AR glasses will be ‘x-ray vision’ that superimposes a prediction of everyones naked body over the top of their clothing. As a result anyone that wears the glasses in public will be automatically labelled as a creep.

Leakers inside Foxconn recently discussed prototype AR Glasses that Apple had them building and mentioned that the compromises needed to miniaturize down to something that fit in normal glasses frames were pretty big.

It's hardly the first time Apple took a shot at building something and decided the tech wasn't far enough along, YET.

Movement would play some role in the function of the glasses: Foxconninsider noted that for an app like Tinder, a user could shake their head for no, or nod for yes. A “small capacitive strip on” the arm would allow users to accept calls and control the volume, while Siri could also be used to make calls. The device would use a prism that “conveys NED display image to lens,” and would have 428 x 240 resolution.

The insiders noted that they only saw a “transparent reader lens but I understand they can procure polarised or prescription lenses,” that the frames were “Cellulose acetate Injection mold frames into an aluminum mold,” that they came in two sizes (men’s and women’s), and that they had a P3 frame design. There was also the possibility that there would be seasonal design updates. The materials cost for the devices would be around $150, which Foxconninsider noted would likely place the retail price in the $600.

It’s not clear if Mirrorshades will go forward, however. Foxconninsider explained that while they were a “unique design,” there was a 65 percent chance that the entire project would be canceled because the batteries are difficult to produce.

That might actually not be a problem today. Pokemon Go is hard on phone battery mainly because the developers had to cobble together their own AR implementation using the existing phone API. Which probably isn't very battery optimized.

Having ARKit (and ARCore) implemented in the OS (and common AR functions optimized) should alleviate that greatly.

I really don't see it. GPS location tracking plus an always-on screen (likely at a high brightness level to compete with sunlight) will drain the battery regardless of processing overhead.

My experience with gaming is that the game CPU/GPU usage can be a huge battery drain, but that the screen being on is certainly not a trivial factor.

Two examples:

1) NYTimes crossword puzzle app. It's not doing much of anything on a moment-to-moment basis. No 3D rendering. Minimal animation. Only occasionally hitting the network to sync progress. Still causes noticeable battery drain over time. I haven't paid attention to exactly how much drain, but I've noticed "wow, I didn't expect it to use batteries anything like that". Unless it's really badly programmed, it's battery usage must mostly be the screen with "some" amount due to not being able to sleep the CPU as often/long.

2) Simcity BuildIt. Can't imagine that the game logic is using a ton of CPU power, but the 3D rendering seems to be using all kinds of lighting/luminance effects, and often it's hitting the network to update the player-to-player store. This thing eats batteries. Like can run down half the battery on my 6S in a matter of 30-45 minutes.

My point being that I agree one shouldn't discount the battery usage from simply having the screen on and the phone active. But that computation specific to the game can still push power usage way, way beyond what the screen uses.

Actually, the trend is that Apple will take a shot at building something and then decide it's just not good enough to release.

In that way, they are sort of the Anti-Samsung, because Samsung is willing to shit the bed endlessly and throw it up against the wall to see if anybody will buy it anyway.

Hell, look at what the NY Times review had to say about the Galaxy 8's biometrics.

Quote:

Some of the biometrics, including the ability to unlock your phone by scanning your face or irises, are so poorly executed that they feel like marketing gimmicks as opposed to actual security features.

Exactly. He'll say the tech isn't there up until the exact moment his company's product is ready to hit market, and then the story will change to "you won't believe what we've done here."

He's speaking as a salesman and an advocate for his particular company, not as a neutral and authoritative source on what technology is capable of in 2017. That's okay, that's his job, but everything he says should be seen with that context in mind.

I appreciate the way you put this into context without making it a company-specific attack, as we often see against Apple (or against Microsoft, or against Google, or etc). Thanks for a smart, non-adversarial analysis.

The moment AR becomes mainstream is when VR and AR merge in a single small package with good battery life and looks. Microsoft is ahead in that regard, since Alex Kipman confirmed in the MR event that we will see such a device in the ‘near future’, likely next year.

But in terms of getting both AR and VR mainstream, there needs to be a standard that is cross form factor.

The moment AR becomes mainstream is when VR and AR merge in a single small package with good battery life and looks. Microsoft is ahead in that regard, since Alex Kipman confirmed in the MR event that we will see such a device in the ‘near future’, likely next year.

But in terms of getting both AR and VR mainstream, there needs to be a standard that is cross form factor.

Don't forget about magic leap. As dishonest as some claim they've been, they too have a working prototype, even if it's also currently tethered. And there's some other company whose name I forget who focuses on B2B AR who is supposed to be very advanced as well.

I'm clearly not part of his target demographic, because all I could think of when I read this was "No."

It explains why they're so bullish on not giving up info to authorities though. If you assume all your users have their financials, health data, shopping lists and more tied to the phone that's an insane amount of data.

I've been enthusiastic about a lot of mobile technology, but I can't say I'm thrilled by AR. It seems the only people that are singing its praises are those involved in developing and producing it, while the average consumer probably still isn't eager to live in a world where a camera is pointed at them during every human interaction. We've seen it with Google Glass, and it isn't a problem that is easily solved (or if it's one that even should be solved). I think the tech world really should try to sober up a bit from the high they got from smart phones taking off the way they did in the past ten years or so.

AR has the potential for corporate/government malice in the form of DRM and/or soft censorship.

The former in the form of interactive content blocking ie. You go at a large Comic Con event and every time you take pictures/videos of posters, toys on display, movie trailers, you get nothing but black splotches and a text line telling you certain images/sounds are blocked on copyright grounds.

And the latter as a way to keep users from recording police/military/government/corporate abuse without resorting to something as harsh as a kill switch for your phone.

That's all very unlikely to happen, both users and corps would push back hard against that kind of abuse.

A more likely scenario of concern is bubbling yourself. People are already tying themselves to ecosystems for disparate types of information, and then they go further and personalise those information feeds for themselves specifically. If this takes off people run the risk of literally living in their own world.

Seriously? I think the kit looks nice, and while I would love to have something as lightweight as my current pair of glasses, Hololens is arguably non-bulky considering what it can do.

You can cover what the man said without going the extra mile of trying to back his claims up with odd statements about tech from competing companies. BTW AR/VR done the traditional Apple way is a non starter. You need to be open, early, and with many devs and users involved from the get go (ya know, like what ever who isn't Apple is doing with AR and VR).

A thing you can do is take two pictures and put them on the end of a stereopticon. The person with that thing on their face either gets eyestrain or sees a 3D pic.The tech has been around for a while, and is better this century.

I kinda agree, he doesn't have good tech yet so it just !isn't! in his world but his ain't the world.

In other words: Apple is lagging behind the competition in this area, too, so they need a couple more years to catch up to at least today's level. Until they do, they will dismiss this technology area as unimportant, not something they can't even enter yet.