MIT creates video you can touch

The Massachusetts Institute of Technology (MIT) recently announced the development of Interactive Dynamic Video (IDV). This breakthrough, created by researchers in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), allows you to reach in and touch objects in videos.

The best part? These are videos that can be shot using ordinary video cameras – even the one in your smartphone.

However, the key lies in the algorithms.

How does MIT’s IDV work?

“This technique lets us capture the physical behavior of objects, which gives us a way to play with them in virtual space,” Abe Davis, a CSAIL PhD student who will be publishing the work this month for his final dissertation, told the MIT website. “By making videos interactive, we can predict how objects will respond to unknown forces and explore new ways to engage with videos.”

To simulate the objects, the researchers analysed video clips to find “vibration modes”. At different frequencies, each vibration mode represents the distinct ways that an object can move. By identifying these modes’ shapes, the team can predict how these objects will move in new situations.

“Davis and his colleagues have provided a simple and clever way to extract a useful dynamics model from very tiny vibrations in video, and shown how to use it to animate an image,” said Doug James, a professor of computer science at Stanford University.

New era of CGI

One of the applications for IDV Davis cites is film-making. When attempting to make CGI characters interact with real-world objects (in, say, Alvin and the Chipmunks) green screens need to be used and detailed models of virtual objects need to be constructed. This is both time-consuming and expensive.

Using IDV, a videographer can shoot an existing, real-world environment and apply minor edits such as masking, matting and shading to achieve a similar effect in a fraction of the time and cost.

More advanced AR?

With AR apps such as Pokemon GO, virtual object (e.g. a Ratata) does not interact with the real objects around it (a plant pot behind the Ratata).

If IDV was applied to Pokemon Go, you could see the leaves of the plant on which the Ratata is standing rustle and move as it dodges your fifth Pokeball. Davis explains this in a little more detail in this video:

IDV engineering

Another potentially more useful application of IDV could be in engineering. When the next Burj Khalifa is built here, architects and structural engineers could test the building’s resistance to wind and earthquakes.

“The ability to put real-world objects into virtual models is valuable for not just the obvious entertainment applications, but also for being able to test the stress in a safe virtual environment, in a way that doesn’t harm the real-world counterpart,” said Davis.

New forms of VR

Davis wants to other applications emerge, from studying sports film to creating new forms of virtual reality.

“When you look at VR companies like Oculus, they are often simulating virtual objects in real spaces,” he said. “This sort of work turns that on its head, allowing us to see how far we can go in terms of capturing and manipulating real objects in virtual space.”