The first film in the trilogy, Rise of the Planet of the Apes, wowed audiences with incredible lifelike characters, made in part possible by a breakthrough in motion capture technology. Andy Serkis was able to wear that now-famous mo-cap suit, all black and covered in what seems to be ping pong balls, and act out the movements of Caesar, the first modified smart ape. It was rightly hailed as a huge innovation, but WETA, the cutting-edge, New Zealand-based VFX studio that worked on the series, did not stop there.

Motion capture, after all, was just the start of the ape-making process. "The part that you see on screen creating the character comes after [motion capture acting]," Joe Letteri, the series' Oscar-winning visual effects supervisor and director of WETA, told SYFY WIRE. "It's what we do to take all that information and interpret it and translate it to the character, both for the body and face."

Over the course of the franchise's six year run, WETA continued to innovate, making giant leaps in the technology involved, as well as the artistry required to make the apes look as realistic as possible while making changes necessary to tell the story director Matt Reeves set out to tell in the two sequels, Dawn of the Planet of the Apes and War for the Planet of the Apes.

Letteri laid out how the technology developed over the half-decade, as well as the changes WETA made to the apes along the way.

When you started out, was the goal to create completely realistic apes, or was there some wiggle room there?

When we were doing the first film, we had a lot of stuff to figure out. The apes have to start off looking completely realistic because that's the origin story, they’re current day apes. But we knew that Caesar was going to be very expressive and that he was going to say something in the film. You think of apes being very much like humans, but the way their muscles are constructed they can't really talk. Even Caesar, we actually made his muscle a little smaller and a little more human like so he could do the dialogue, and his eyes we made a little more like Andy's eyes so you can understand the expressiveness.

So you start making these little choices in the design that work their way through into animating the character. On the first film we spent a lot of time trying to understand that translation. How do you turn Karin Konoval into Maurice, an orangutan?

Video of War for the Planet of the Apes | Face Of Caesar | 20th Century FOX

Did the apes, over time, change visually as they got smarter and became more active characters?

We started with all the apes except for Caesar being really true to realistic apes that we could find and photograph and reference. But as we got into the sequels, where we knew that the apes were going to have to speak, we did make some adjustments to them. We were able to hide that in the fact that they were aging throughout the story so we made slight adjustments, mostly to their muscles but also to their eyes a little bit. So you know it's the same character, but in the same way that a human ages over the course of 15 or 20 years. We kind of buried it in there so that when did they have to speak they were better equipped for it.

As you developed new ape characters, how did that impact their design?

Bad Ape is a good example. Matt had this photograph of a chimp that he just absolutely loved the expression of, and he wanted us to recreate that. So you're working from one photograph from the front that gives you this kind of unique expression. We started to build it, but then we found other photographs of the same chimp and when you start to reconstruct him in a natural way you realize whatever was unique about that one expression, from every other angle he looks more just like a normal chimp.

That expression made it look like he didn't really have a wide muscle, that it was narrower and centered and elongated, so we actually built that ape that way so that from every angle he had that personality and the character from that one image that Matt loves.

You look at him and you think he's a chimp, but if you were to see him side by side next to a real chimp, you’d see we took a lot of liberties there to give him that expressiveness. And obviously he had a lot of dialogue, so we had to take that into account.

Video of War for the Planet of the Apes | &quot;Bad Ape and Maurice&quot; Clip | 20th Century FOX

So which elements of the technology changed?

The performance capture itself. We were able to go from doing it live on set, which was a breakthrough for us, to actually moving it outdoor scenes. On Rise we shot on set and we did a little bit of exterior work, but not too much, and in pretty controlled conditions. On Dawn, Matt wanted to take it out into the deep forest where the apes had set up their community and had been living for 15 years, so it was very primitive and very remote.

This is pretty sensitive equipment; you're tracking micro-movements on these characters and you've got to set up a lot of cameras to do that, make sure they're all calibrated, and we're hauling them out into this wet woods every day to do this. And we got it to work. We had to figure out how to make them wireless and waterproof and do all these technical things to handle that.

By the third film Matt really wanted to push it. They were really going out on this journey and we were way up in the mountains and it was rainy and cold and it was snowing. And if there wasn't enough rain they would bring out rain bars and add more, it was pretty grueling conditions. We just kept pushing the technology to make sure we could get to all work.

On the side of creating the characters we build everything, so that there is a physical basis to them. All the chimps have a skeleton, they have a muscle system, they have fat, they have a skin layer, they have fur, and the fur is all dynamically driven so we simulate what happens when they move, when they interact with each other, when they're rolling around in the snow. They have to accumulate rain and snow into the fur. We have to recreate all the cinematography, we have to understand what all the lights are doing in the real world, how the camera behaves and we wrote a whole new piece of software that we call Manuka, which is a renderer that actually makes the images by calculating all the physical light transport.

How does Manuka do that?

So we build in the computer an analog of everything that would happen in the real world — and not just what you see, but everything that causes what you see to happen. All the physics and all the dynamics all have to get calculated on top of the artist input that we put into it to create the performances and the emotional aspect that we're looking for. Both in animation and lighting and cinematography because they go together like in a real film. You have to stage everything to get the effect that you want.

By the time we got to War, this was an ape-centric story, it was entirely from an ape’s point of view. Apes were like in 90% of the shots and not just Caesar and his immediate band. A lot of shots had hundred of apes in them. As the technology evolves and improves what we get asked to do also becomes bigger and bigger in scope. There's always sort of a balance there.

Back on Rise we were just trying to figure out is this the right thing. If Karin raises her eyebrow by three millimeters does Maurice raise his by three or does it go up by four? What's the translation to an ape? Does it change the expression because all these micro-movements do actually change how you perceive the performance. Having gotten that under our belt we were much more able to concentrate on just the details of the performances.