The Vogue Augmented Reality Experience: How We Did It

Augmented Reality, or AR, refers to technology that overlays the user’s view of the world with digitally rendered visual elements. Think Pokemon Go, where the little creatures appear, to the user, as part of the surrounding environment. AR has become an increasingly popular way to create unique experiences in the digital realm. For our brands, AR presents a great opportunity to engage with readers in new ways. For example, a New Yorker article might be enhanced with a scale model of the article's topic that you can place in your living room, or an Epicurious recipe might use AR to guide people through cooking techniques on their kitchen counters.

With the release of iOS 11 last September, Apple shipped ARKit, a native AR platform on iOS, making it easier than ever to create immersive 3-D AR experiences. Since we have many popular native iOS apps already in the App Store, the barrier to delivering high quality AR experiences for our brands was significantly lowered: it became a matter of shipping updates to these apps with ARKit content included.

In October of last year, Vogue collaborated with Apple’s developer relations team to come up with an idea for an AR experience (to be integrated into Vogue’s existing iOS app) that would leverage the new face-tracking capabilities of the iPhone X. The goal was to have it ready to go by November, to coincide with the launch of the new phone. A small internal team, comprised of an AR specialist and two iOS developers (myself included), had a month for ideation, design, and development.

The Disco Effect

First, the idea. We knew we wanted to create a glamorous and immersive experience for Vogue; the requirement that we use the new iPhone X face-tracking technology aligned well with this goal. We set out to create a vibrant, colorful, and fun experience, like the user is inside of a disco club. We also wanted it to be fun and shareable, so there had to be some way to record what it looks like to the user.

We settled on an immersive experience that would use series of strobe lights that look to be projected directly on the user’s face, as if they are moving through beams projected from the position of the phone. When the experience launches, the user is launched into this "club" world, with lights appearing over their face. When they hit "record", a soundtrack starts, and a pulsating light appears (that is positioned according to the orientation of the phone) and moves to the beat of the included audio track.

Here is what the final experience looks like:

Integration into the Vogue app

Working closely with our AR specialist, who had no prior experience with iOS, we built a separate, standalone app in Swift, to experiment with what was possible in ARKit. Once we felt comfortable with the frameworks and code involved and had the effect working well, we were ready to integrate the experience into the Vogue app.

Then, we hit a roadblock.

Unlike many of our native apps at Condé Nast, the Vogue app is written in React Native, which leverages Javascript for building cross-platform mobile apps. Most native APIs are bridged to Javascript, which makes React Native a powerful solution for building native applications. One of the significant drawbacks of building apps this way is that the ability to use new native technologies lags behind their release, as every new technology needs to be adapted and wrapped for use within the React Native Javascript environment. Though some solutions for wrapping ARKit APIs have already been developed, such as react-native-arkit , they were not sufficient for our purposes. For one, none of them yet support the face-tracking capabilities of the iPhone X. Moreover, we needed to have total flexibility to experiment with the APIs.

Luckily, we had already built a custom iMessage app extension for the Vogue app, written in Swift. If ARKit would work in an iMessage app, we could embed the experience in the existing app, as an extra option. After some experimentation, we found that we could get ARKit to function as expected in iMessage. We were in business.

While embedding the experience within iMessage presented some limitations, and was certainly not ideal from a UX perspective, it simplified the requirements for what we wanted to accomplish: the user could record a video inside the iMessage app and be able to immediately share it, in an iMessage.

The UI we settled on was a toggle at the top of the iMessage app UI, which would only be visible to a user with an iPhone X (Apple provides an API to check if face tracking is possible, which we can use to ensure the user is running the right hardware). If the user opens the app and they don’t have an iPhone X, they are presented with our existing iMessage interface, which shows a selection of stickers:

However, if they have an iPhone X they will have an extra option at the top of the interface:

Tapping "Effect" unveils a new screen, prepping the user for the experience:

From there, tapping "Start" or expanding the iMessage interface will launch the experience.

Implementing AR functionality with ARKit

Now to the fun part: how did we actually use ARKit to bring this experience to life?

The good news for us is that ARKit is a very well-designed set of APIs that are exceedingly simple to get up and running. The first step in creating the experience is using SceneKit, Apple’s set of tools for rendering 3-D graphics and effects. In iOS 11 a new class was added to SceneKit, ARSCNView, a subclass of the existing SCNView, which facilitates working with ARKit with 3-D scenes.

In our iMessage app, when the user taps the “Experience” tab within the iMessage app, we pull up a new view, which is an instance of ARSCNView. Once our view is active, we configure the sceneView like so:

This code is all that is needed for us to get AR started and running! The sceneView has a reference to a session, which is an instance of ARSession, the class that drives and handles all AR events. We tell the session to run the ARFaceTrackingConfiguration (only available on iPhone X), which recognizes participants' faces and generates a 3D mesh that matches its contour.

The class handling all of this is set as the “delegate” of the sceneView and the ARSession, allowing our code to fully synchronize the AR meta-data and the information about the 3-D space of the SceneView. As the delegate, our class will receive relevant events which contain information about the Three-dimensional space, the position of the face -- everything we need to precisely calibrate and style the lights and create a sense of 3-D space that our users will occupy.

And with all of that in place, we are in business! All in all, the ARKit code is only about 500 lines in total, most of which is setting up the lights and tweaking the effects.

Conclusion

In delivering our first ARKit experience, we worked under a tight deadline, with a small team, and were able to go from ideation to execution to delivery of our app update to the App Store in about two months. It wasn’t easy or always pretty, but we were able to make it happen, for a few reasons:

First, the scope of the project was manageable and focused: we wanted to use the technology in an interesting way and create a great experience for our users, but we weren’t fundamentally reconstructing our app. We were given enough leeway by the brand to create an engaging experience that made sense within the framework of augmented reality.

Second, our existing expertise in iOS development allowed us to get up and ready for the experience quickly, including on-boarding our AR specialist who was not familiar with iOS development.

Third, ARKit was designed and implemented to make it easy to get started and iterate quickly. It was as easy as implementing a few methods and tweaking our parameters to get the effect looking and feeling as we envisioned.

We’re proud of what we were able to accomplish, and we’re excited to work on further AR projects in the future.

Use of this site constitutes acceptance of our User Agreement (effective 1/2/2014) and Privacy Policy (effective 1/2/2014). Your California Privacy Rights. The material on this site may not be reproduced, distributed, transmitted, cached or otherwise used, except with the prior written permission of Condé Nast.