The Cutting Edge Of Motion Capture: David Cage On Kara and the Technology Behind Emotional Virtual Performances

You might remember back in 2006 when David Cage and his team at Quantic Dream released a short video called “The Casting.” It was a brief yet powerful scene designed to show off the developer’s latest experiments with performance capture. It had nothing to do with the game they were working on at the time—which ended up being Heavy Rain—but rather it was a way to demonstrate their vision for what new technology could allow them to deliver in terms of storytelling and performance; for a refresher, you can watch it here.

At GDC 2012, Cage took to the stage to give the world its first look at Quantic Dream’s newest test piece entitled “Kara.” Similar to “The Casting,” the video is a standalone story that doesn’t (but would be so awesome if it did) represent what the team is working on next. It’s just something designed to showcase what Quantic Dream has been up to since it finished development on Heavy Rain back in 2010. And if you’ve seen the piece, then you know it packs a mighty impressive punch.

Speaking of “Kara,” here are a few choice details Cage discussed during his panel:

The video is a year old, and runs entirely on PS3 hardware. It was built using the first version of Quantic Dream’s new engine that they build from scratch after Heavy Rain; they are currently on version three. Cage described the demo as “crap” compared to what the engine is capable of now. In fact, “Kara” only contains around 50 percent of the engine’s current features.

They used two different actresses to record the French, German, and Japanese portions, and then just used those facial maps with the lead actresses model. Cage included these other languages to test the engine’s retargeting capability.

The robotic arms were simulated by two other actors pulling and grabbing the actress during the scene. Cage explained that this level of interaction generates nuances in muscle tension and weight that you wouldn’t otherwise get in traditional animation.

The actress playing Kara is Valorie Curry; she beat out 70 other actress for the role.

During his panel, Cage discussed how Quantic Dream has evolved over the years in terms of its approach to bringing virtual actors to life. When the team developed its first game, Omikron: The Nomad Soul back in 1999, it used keyframe animations for character motions and a rather crude but effective method for lip syncing: a midi keyboard mapped with “phonemes” that represented various mouth shapes. They would then have a “pianist” essentially use the keyboard to do the lip syncing.

For QD’s next game, Indigo Prophecy, the team used a glove-based system where the phonemes were mapped to the fingers of the left hand and facial expressions were mapped to the right. By combining them, the animation “puppeteer” was able to bring the characters to life more believably (even it sounds like a rather peculiar solution).

Today, the team is fully invested in full performance capture. Unlike the common technique of recording an actor’s face and voice separate from their body and then meshing the two later—like QD did in Heavy Rain—the new method, which has become more and more popular in games and movies over the past few years, captures an actor’s entire performance in one take. If you’ve ever seen behind-the-scenes videos of people in mo-cap suits, covered in tiny dots and surrounded by scores of infrared cameras then you’ve seen this system in action.

Quantic Dream used a setup of 28 cameras for Heavy Rain; their new studio has 65 and the actors wear 90 markers on their bodies and faces. Unlike full motion scan technology (like that used in L.A. Noire), this method doesn’t capture a video of the performance, but rather creates a highly detailed map of the face and body. The team uses a body scanning technology to create identical digital “clones” of the actors that, when combined with the mesh created from the performance capture, creates a perfect virtual replica of the actor.

Cage explained that this new method allows for a much truer performance and adds a level of complexity and fidelity that is almost entirely lost when face and body are captured separately. See above video for why this is true. “Kara” was born out of a desire to understand the potential of this new technology and what it could allow Quantic Dream to deliver with its next project, not just in terms of the tech involved, but in terms of how the team approached everything from casting, to actor prep, to direction.

Interestingly enough, Cage ended his panel discussing his vision for the future of games and the technology that drives them, suggesting that soon, at least it’s his hope, that developers will spend less money on tech and will share from the same tool pool and invest more on creating meaningful content; gaming’s version of “content is king” sort of idea. If you look at the ubiquity of something like Epic Games’ Unreal engine and the wide variety of experiences and styles of games that have been built with, it’s not such an unreasonable prediction.

Cage also sees there being a real market for “adult” games—games that provide meaningful and emotionally charged experiences—in the mainstream. Cage believes these types of games could compete with casual, family, and hardcore games, three ways games can become mainstream, according to Cage.

Judging from the level of intensity and quality present in “Kara,” it’s clear that Cage and Quantic Dream are certainly on track to push the boundaries of the medium even further than they did with Heavy Rain. And if their next game doesn’t let you play as a female android who gains consciousness, breaks out of the android factory, and goes on the run, then I’ll be about as devastated as Kara was as she was when she was being disassembled.