Kinect Hack Controls the Web With Hand Gestures

This site may earn affiliate commissions from the links on this page. Terms of use.

Kinect hacks are pouring in left and right, but this one really grabbed our attention. A group of MIT engineers have hacked Microsoft’s Kinect for Xbox 360 to control a Windows 7 PC with open-air hand gestures—just like that thing in that old space movie we like.

The open-source system, called DepthJS, was developed by four MIT engineers in MIT Media Labs’ Fluid Interfaces Group: Aaron Zinman, Doug Fritz, Roy Shilkrot, and Greg Elliot. At its core, DepthJS is a browser extension for Google Chrome. It lets JavaScript talk to the Microsoft Kinect in such a way that you can scroll Web pages, cycle through open browser tabs, select links, and more, all without touching a mouse or keyboard. It also works with panning and zooming on Web pages.

To make the setup work, the engineers placed the Kinect for Xbox 360 on top of a box on a table, aimed at a position in front of two LCD monitors in an extended desktop configuration. They wrote the Kinect driver and vision software on top of Open Frameworks and OpenCV in C++, according to their project page.

Here’s a one-minute, 30-second video from MIT’s Media Lab showing it in action with a minimum of talking and a maximum of awesomeness.

“Navigating the web is only one application of the framework we built—that is, we envision all sorts of applications that run in the browser, from games to specific utilities for specific sites,” the group wrote on their Vimeo page. “The great part is that now web developers who specialize in JavaScript can work with the Kinect without having to learn any special languages or code.”

Next up for the project: adding recognition for two hands, for pushing and pulling, for individual fingers (these aren’t the droids… never mind), and for finger-based hand-gestures like a peace sign. Which we’d use right before we fragged someone online, because we’re slick like that.