The Ultimate Augmented Reality Test: The Shake

Early 2017, we went to visit our customer Schneider Electric in Grenoble, France, the site of their major R&D division. To say the least, the purpose was to present augmented reality for field worker applications. Shapetrace had prepared for this demo, and Sonal and I were excited to be there. We would spend an entire day doing presentations to a rotating group of people from each research division. We were gonna rock them out of their shoes and show them the future... or so we thought.

The day before, naturally we debriefed with the team who invited us. We showed off our prototype with pride, talking about the technical innovations we executed, and how we could further work together. Romain Gassion said there one more thing he needed to do.

Our perfectly aligned 3D BIM model drifted into oblivion, and that's when we understood the ruggedness that AR re-localization needed to be. A field worker (or even a consumer) won't have any patience to play around with a delicate device. At the end of the day, if it causes frustration, then it's not worth anything to them.

When you shake the device violently, you are doing two things. First, you are causing extreme movements of the inertial sensors, which the device assumes you're suddenly travelling on the back of a bumblebee. As a results, your device (camera pose) thinks you're somewhere else, and your 3D asset appears somewhere.

Secondly, you're stress testing the vision algorithms to ensure there are sufficient image-based anchors to re-localize when your camera re-stabilizes.

From this day forward, we've used the AR Shake test on every piece of AR software we create, every new device we see, and every new SLAM engine that people present us with. We've taken joy in watching other people's jaw drop in horror and happy to pass on this stress test to them.