Mobile

Mobile Phone, Meet TV

By Larry Hardesty, April 09, 2010

'Surround Vision' is to what you see as 'Surround Sound' is to what you hear

Augmented reality is an emerging discipline that uses handheld devices to superimpose digital data on the real world: If, say, you're in Paris and point your phone at the Eiffel Tower, the tower's image would pop up on-screen, along with, perhaps, information about its history or the hours that it's open. With a TV enhancement called "Surround Vision", however, researchers at MIT's Media Lab are bringing the same technology into the living room.

Surround Vision is intended to work with standard, Internet-connected handheld devices. If viewers want to see what is happening off the left edge of the television screen, they could simply point their cell phone in that direction, and an image would pop up on its screen. The technology could also allow a guest at a Super Bowl party, for instance, to consult several different camera angles on a particular play, without affecting what the other guests see on the TV screen.

To demonstrate the idea, Santiago Alfaro, a graduate student in the lab of Media Lab research scientist Michael Bove, shot video footage of the street in front of the Media Lab from three angles simultaneously. A television set replays the footage from the center camera. If a viewer points a motion-sensitive handheld device directly at the TV, the same footage appears on the device's screen. But if the viewer swings the device either right or left, it switches to one of the other perspectives. The viewer can, for instance, watch a bus approach on the small screen before it appears on the large screen.

Alfaro and Bove envision that, if the system were commercialized, the video playing on the handheld device would stream over the Internet: TV service providers wouldn't have to modify their broadcasts or their set-top boxes. The handheld would simply use its built-in camera to determine its orientation toward the TV and the selected channel; for both purposes, it might cue off of the insignias that most stations display in the lower right corner of the TV screen. Viewers who opted not to use the enhanced programming need never know they were missing anything. Those who did use it wouldn't need anything other than a smartphone and a standard Wi-Fi connection.

Many existing handheld devices have built-in motion detectors called "accelerometers". But switching between viewing angles with the Surround Vision system seems to involve motion too subtle for accelerometers to register. To get his prototype up and running, Alfaro had to attach a magnetometer -- a compass -- to an existing handheld device and to write software that incorporated its data with that from the device's other sensors. But Alfaro and Bove say that devices now on the market -- including the most recent version of the iPhone -- have magnetometers built in. The researchers have answered most of the technical questions about the system; now their chief concern is determining how best to use it.

To that end, they plan a series of user studies in the spring and summer, which will employ content developed in conjunction with a number of partners. Bove won't name names, but, he says, "We're looking at sports; we're looking at children's programming, both live action and cartoons; we're looking at, let's say, ordinary entertainment programs, as well as programs shot in a studio like talk shows. And we hope to have examples of several of these fairly soon. There are also one or two other things that defy categorization right now, that you sort of have to see in order to understand what they are."

Since sports broadcasts and other live television shows already feature footage taken from multiple camera angles, they're a natural fit for the system. Many children's shows already encourage a kind of audience participation that could be enhanced, Bove says, and viewers of the type of criminal-forensics shows now popular could use Surround Vision to, say, see what the show's protagonists are looking at through the microscope lens.

Of course, today's TV shows frequently feature short scenes with quick cuts between different camera angles, which would allow little time for exploration of the virtual environment. But "the system as it stands should not be a constant feature," Alfaro says. "We can't ask a user to constantly hold a device up at arm's length: They will quickly tire and not use it at all." Alfaro suggests that as TV directors grew to appreciate the system's strengths, they would likely modify their styles to take advantage of them.

If the user studies conducted by Bove's group converge on some applications that resonate with viewers, however, Surround Vision could prove to be something more than a spur to the imagination. "In the Media Lab, and even my group, there's a combination of far-off-in-the-future stuff and very, very near-term stuff, and this is an example of the latter," Bove says. "This could be in your home next year if a network decided to do it."

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task.
However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Video

This month's Dr. Dobb's Journal

This month,
Dr. Dobb's Journal is devoted to mobile programming. We introduce you to Apple's new Swift programming language, discuss the perils of being the third-most-popular mobile platform, revisit SQLite on Android
, and much more!