The Special Effects of Dawn of the Planet of the Apes

"We actually call it performance capture." That's how Matt Reeves, director of Dawn of the Planet of the Apes, quickly corrected me when I asked him a question about the state of motion capture technology. This was two weeks ago, at the press junket for the new film. Reeves, along with the film's actors and visual effects supervisors, fielded questions for an hour from a packed hotel suite of entertainment reporters. It was the first junket for a big hollywood film that we've been invited to, and the experience was a little surreal. It felt a lot of like a Comic-Con panel, but for just 30 people instead of 3,000. And we had all seen the movie at an early screening the night before (it's really good). And with that opportunity to speak directly to the filmmakers, I wanted to learn about the process of filming a movie on location with the latest in motion performance capture technology. Unlike some films that use performance capture with primarily green screened sets (think Avatar), Reeves chose to build out many of the film's locations as actual physical sets, like the massive ape village, for the actors--both for human and ape characters--to interact in. And the computer generated characters were maybe the best I've ever seen in a live action movie. At this junket, I was able to ask a few of these technical questions to Reeves, actor Andy Serkis, and the film's Visual Effects Supervisors Joe Letteri and Dan Lemmon. Here's what they had to say.

Credit: 20th Century Fox

Tested: Matt, you've talked about how directing this movie differs from directing one without performance capture, without a lot of CG, and how that required you to shoot scenes many times over. Can you go over that process?

Director Matt Reeves: My biggest fear, having never done this [kind of movie] before, and being such an admirer of Andy’s--specifically being affected so deeply by his performance in Rise of the Planet of the Apes--was “How was that done?” I didn’t really understand [the process]. As much as I understood the technical side from the outside, I had this fear that somehow the technology would get in the way with my interaction with Andy. Because there would be this technology between us.

So then, I looked at all the footage, and what I saw was Andy, working with the other actors, and he’s amazing. The trick to what gives Caesar such soul is that Andy has soul. So that part of it was immediately demystified. I was very happy to see that. The hard part comes after that [inital filming].

So [first], I’m working with Andy, we’re talking through a scene, and then he does this beautiful scene with the actors. Then we’ve got that shot in reference. And then we’ve got to shoot that shot again. Sometimes, when there aren’t humans in [the scene], we have to shoot it with no one in the shot. The camera operator has to try to reproduce what he did when he was trying to follow Andy, including the sometimes surprising moves that he would make. And other times, I would have to get the humans, who had just played a very beautiful scene with Andy, to play the scene by themselves. Because those shots, were then used to put Andy, [rendered] as Caesar, into the scenes.

So the shots where the actors have had that beautiful connection with Andy, often were not the shots that were going to be used [in the final film], so I had to let them know “Your performance is still not in the movie yet. We have to get it right now--you have to remember what Andy did. And Andy, then, would get on a microphone, and try to talk the other actors through, the beats they’d just experienced together. It was a very unusual process.

After you get those shots, and animators then go into tweak that performance, Andy, how do you reconcile what you do on set--for example your facial movements--with what gets tweaked in post-production? What is it of your performance you want to get through?

Andy Serkis (Caesar): Well, they’re not so much tweaks, as such. What Matt wanted to create on set, was the emotional truth of every single beat, and find the drama of the scene. And so what would happen then, after Matt had found the moment that he wanted--a series of choices--is that he would move on and then put the shots together in the cut. That would organize the pacing, the storytelling, the drama, and the emotional intensity of the scene, using that raw footage of all of us actors together.

What happens then in the post-production process--the brilliance of Weta and the technology the software that’ve evolved--is the artistry of the animators, who honor exactly what those performances are in the edit.

What is one thing in that original performance that you really want to make it through that entire process, from first shot to post?

"The challenge for Weta becomes how to take Andy's emotional search and translate it to an anatomy that is not the same as Andy’s."

Matt Reeves: What you want to get through is what Andy’s done. We go through an emotional search. He does something--discovers something--in the moment and it feels right. We put that in the editing. The challenge for Weta becomes how to take that and translate it to an anatomy that is not the same as Andy’s. So that’s the genius of what they do. They find a way to take the shapes--which are all purely Andy’s--and put them into the face of an Ape who doesn’t share the same anatomy.

What’s so weird about it is now, I can’t see Caesar without seeing Andy, because [the characters] take on certain details [from the actors]. I can’t see the apes without seeing the actors, and that’s because Weta is so brilliant at figuring out how to take certain details of his face and expression and translate them. For example, certainly Caesar’s mouth is very different from Andy’s, but they’ve found a way to take those shapes that come from Andy’s emotional expressions and translate them to Caesar’s face. It’s not a button, it’s a translation. Sometimes I would say something like “Andy’s eyes look sadder, guys, the brow’s not the same shape,” and Weta would go back to figure that out. The whole pursuit in post becomes trying to find a way to go after every ounce of emotion that Andy has given, and that’s a crazy process.

I wanted to ask you about the character rendering, because it looks so much better in this movie than in the first. In terms of creating a realistic CG character or creature, what are the new rendering technologies allow you to do that?

Visual Effects Supervisor Dan Lemmon: Like the first time, all of our characters are sort of built on our performance capture methodologies. And one of the big things we did for this film was take that performance capture toolset and moved it out into the live action production environment. So, in this story, a big part is set in the woods and the rain and mud. We needed to make sure our tools would be able to get all the detail from the actors’ performance--capture that data and bring it back home. We kind of revamped our whole toolset so that it could hold up to the rain and the mud.

Are there certain scenes or environments where you can make an ape look more realistic? Where rendering fur and skin looks more life-like?

Director of Weta, Joe Letteri: There’s nothing inherently easier about rendering for one environment versus another. We’re used to seeing realism in so many different kinds of lighting--broad daylight or an overcast day or deep inside a forest. You have to get it right no matter what the circumstances are, so all of our tools are built with a physical basis to them. So when you do it, you are doing things like light transport correctly for sub-surface scattering and for light.

We actually wrote new algorithms for doing light transport through hair, and we wrote new algorithms for doing the dynamics of hair. So the physics are also very important to get right, underneath everything we do. So that’s the level of realism for which we apply the final rendering.

Dan Lemmon: The great thing about keeping everything realistic is that we can take our cues from the real actors in the shot. The same things that the cinematographer looks at to make that shots look good and to make the actors look good--we can match that.

Weta has done CG characters that range from humans to fantastical creatures like dragons. Is there something about animating and rendering an ape--something in our minds and perception--that makes it easier or tougher to do than a human?

Joe Letteri: It’s tougher to do in a lot of ways because everyone has seen chimps. We’ve seen them in real life, in places like zoos. Plus they are so close to being human, that you would [as an animator] almost think that you should treat them like a human and that would make them look good. But in fact it doesn’t. You have to walk this fine line of how do you make it look exactly like a real chimp but how do you interpret the performance so it doesn’t look like a man in a suit.

Dan Lemmon: Because the story we’re telling in this movie is very much a human story, a story that humans can relate to. And these characters that these chimps represent have emotions that are very human-like. And so translating those emotions played by human actors over on to the chimp bodies is part of the whole challenge.

Dawn of the Planet of the Apes is out in theaters now, and we'll be doing a spoilercast about the movie on next week's Still Untitled! We'll also be putting up videos from my junket interviews on the site for Tested members.