Virtual Self Makes Full Embodiment In VR A Reality

When game developer CloudGate Studio got its hands on HTC Vive Tracker peripherals earlier this year, it led to players kicking dinosaurs in virtual reality. It was a fun use case, but the company has much bigger plans for this tech. Its Virtual Self plugin enables developers to use one to three Trackers to provide full-body motion-capture in VR. It will be available to demo in CloudGate’s dinosaur survival game Island 359 on December 1, and at a later date, the plugin will launch for Unreal Engine 4, with a Unity version plugin to come.

Virtual Self coincides with HTC’s announcement that players can purchase Trackers for $100 each starting today. The Tracker can turn anything into a VR peripheral. Attaching one to a tennis racquet or baseball bat will turn that object into a controller for a round of immersive sports indoors. Starting in March, developers could buy Trackers to begin experimenting with them.

But even before CloudGate got its dev kit, it was already working on a full-body solution.

Cofounder and president Steve Bowler says that they’ve been floating around the idea of figuring out full embodiment in VR since November 2015. However, CloudGate only seriously pursued it once HTC announced the Trackers at the International Consumer Electronics Show (CES) in January.

Bowler began working on a prototype immediately, creating makeshift Trackers using the controllers that come with the Vive headset. He and cofounder Jeremy Chapman both have decades of experience with motion capture, and they figured that they only needed to track six points on a player’s body to create a realistic representation of it in VR: the hands, head, two feet, and hips.

Supplementing tracking with animation

Optimally, Virtual Self works with three Trackers — that’s six points when combined with the three existing in the Vive’s head-mounted display (HMD) and two controllers — but CloudGate has it working with two and even just one of these devices. Bowler says that while VR arcades might have three Trackers, it’s doubtful that most general consumers will have so many at home.

It’s developed different solutions depending on how many Trackers someone is using: A player with one can place it on their hips; two Trackers go on the feet; and three go on the hips and feet. It works because Virtual Self can figure out where all the other body parts are in relation to the hips or feet, especially since it knows where your head and hands are because of the HMD and controllers.

But capturing player movements is only one part of the solution.

“Even the two-Tracker solve, where I just had them on my feet, that worked well,” said Bowler in a phone call with GamesBeat. “So we realized, what if we could make this scalable? What if we could start finding ways to make it so everybody could enjoy it? It’s a full body solution, so let’s use our animation experience to do it with no trackers. Can we show you your legs animating as if you really had human legs under you? Your feet aren’t being tracked, but we can show you a body in VR that works.”

CloudGate began working on a template that would automatically scale with the user based on factors like height and where known points are. This was handy for dealing with occlusion, which is when a camera loses sight of what’s going on. If a Tracker is on your foot, but you move it so that the camera can’t see it, CloudGate replaces real-time tracking with realistic animation to approximate your movements.

“We know what your body’s doing based on where your HMD and your controllers are,” said Bowler. “That means — are you crouched? Your center of balance changes when you’re crouched and we have to procedurally reposition your body in VR if we’re not tracking it. We know, well, there’s only one thing you could be doing if your height just went to half. We know you’re crouching. So we blend into the crouch procedural solve from there.”

CloudGate tested Virtual Self’s template on people of various heights, ages, and genders to make sure that it could scale. The template configures where everything is in relation to each other, which affects tracking as well as animation. Bowler says that the focus testing was primarily to make sure that the calibration could fit as closely as possible to whomever is playing.

“We have this weird math thing running where we’re trying to guess how much of this is your head and how much of this is the rest of your body that we’re scaling,” said Bowler. “Our scalers adjust accordingly to try to catch it like—kids, there’s a certain ratio, and then as you blend from kid to adult that ratio changes. As you get closer to 15 and 16, you’re really a lot closer to your adult head size.”

Why is full embodiment important?

CloudGate already implemented Virtual Self in Island 359, and it’s demoed some possible VR esports use cases such as a spectator mode. But Bowler says that full embodiment in VR has many more applications that they’re excited to explore.

“We were contacted about phantom limb research and we’d love to actually work on that, for phantom pain, for people who’ve lost a limb and experienced phantom pain,” said Bowler. “How can embodiment, seeing their arm in VR, help them deal with that pain?”

Bowler is excited about helping researchers gather data for bionic limbs, using VR and simulated physics to quickly iterate before creating prosthetics. He lists other possible use cases, like tracking gait in therapy and helping patients with memory issues, which can be helped with a sense of presence and accurate tracking in VR.

Social VR could also benefit from incorporating more realistic gestures and body movements. Bowler says that Rec Room is a strong example of what social VR is already capable of. It feels like you’re there with your friends, even though they’re not rendered well.

CloudGate isn’t the only company that believes in the magic of full-body tracking. Motion capture studio Ikinema has also been working on a six-point solution, and as more people get a hold of Trackers, there will likely be more innovation in the space.

“We think that’s the future of VR,” said Bowler. “We think that’s how people are going to watch people playing in VR. Once you experience seeing your body in VR, you just have to have it every time. You miss it when it’s not there, once you’ve experienced it. We’re excited to get it out there.”

With technology like the Vive Trackers and the Virtual Self plugin, virtual reality might start looking more like actual reality very soon.