Fresh From TED: A Mind-Blowing App That Could Remake Mobile Retail

Jinha Lee, an MIT Media Lab alum, gives Co.Design a first look at an invention that allows you to try on clothes virtually.

To be perfectly honest, Jinha Lee—the MIT Media Lab alumnus and TED Fellow who you may know best for inventing interactive floating pixels—didn’t even want to share this clip. He thought it might be too brief. But when we heard him describe the idea, we insisted. And we’re really glad we did.

What You Click Is What You Wear, which Lee developed alongside Daewung Kim, is a means to try on clothing virtually. It’s not intended to be a shopping platform, but rather a working prototype of what’s possible "when information lives in our space."

What you’re actually looking at is an arm behind a transparent display, but the system could just as easily work as a HUD in something like Google Glass. The real-time simulation looks so accurate because the system tracks motion in two ways. A depth camera, like a Kinect, follows the general position of your body in space. Meanwhile, a phone, which the user holds, passes along gyroscope readings (the rotational position of your appendage).

"With just a camera, you can’t rotate your arm. The watch will never follow you," Lee explains. "But if you’re holding a mobile phone, you can get the orientation."

Bigger picture, the project is grounded in progressive philosophy. Of course we’ve seen some decent augmented-reality apps that use taped-on QR code markers. But when you consider the practical future of wearables, in which several discrete sensors will already be tracking various parts of your body at all times, it’s a natural evolution that some umbrella systems will be able to combine many of these measurements to produce simulations of incredible fidelity. (Lee points out that this idea of combining phone sensors with camera sensors seems to have been created by Microsoft Research’s Andy Wilson.)

But I think what makes this young project so fun is its dead-simple interaction model. Click a single button on your phone and, presto, there’s a watch on your arm. It’s like magic.

"The inspiration behind these projects started when I was young. Growing up, my mother was a paper artist doing origami," he recounts. "So I witnessed this beautiful connection of information display and how physical information is made into a shape."