Kinect to VR

What do you get when bring together the cutting-edge virtual reality (VR) of the HTC Vive system and the Kinect for Xbox One’s unparalleled ability to track multiple people in three-dimensional space? You get an experience that allows you to see not only what another player is doing in the VR space, but also to see a virtual representation of yourself. This is new and exciting, as current HTC Vive multiplayer games only show you representations of the other players, not yourself.

The work of Dutch developers Jasper Brekelmans (a Microsoft Most Valued Professional) and Jeroen de Mooij, the experimental process uses the Kinect sensor to track the bodies of two users in 3D space. By taking the Kinect data and aligning it with the Vive’s positional tracking, the developers have created a unique experience that lets users see a video representation of themselves and other players in a VR environment.

The setup that Brekelmans and de Mooij used in the video shown above consisted of two HTC Vive headsets and controllers, two workstation PCs, a Kinect for Xbox One sensor, and a laptop PC, all running on a local area network. But the developers point out that the same process could be used via the Internet, provided that both users use the exact same setup of HTC Vive components and Kinect sensors. And, as Brekelmans happily notes, “we managed to keep the data bandwidth to about 4 to 5 megabits per second, which took quite a bit of sweat.” …

Recently we had the opportunity to play with two Valve/HTC Vive headsets and controllers so we opted to experiment a bit for a few days.

Since I strongly believe the Virtual Reality revolution needs a social element in order to really take of we implemented the following things to play with:– multi user collaborative experience– interaction with virtual objects using Vive’s hand controllers– implemented using two (networked) machines running Unity– a live pointcloud of the users streamed in over the network

Note that in this example both users are in the same physical space, but we’ve designed all components so everything can be done over an internet connection (about 5 Mbps needed).