Now, we catch up with Joe Letteri, Senior Visual Effects Supervisor on The Hobbit movies, who geeks out with GeekDad about motion capture, 3-D, motion blur, the 48 frames-per-second debate, and how his latest innovations affect the always astonishing field of digital effects.

Letteri is also the director of Weta Digital, in Wellington, New Zealand, where his leadership has helped make the studio one of world’s best visual effects houses. His work has earned him four Academy Awards for Best Visual Effects — for the films Avatar, The Lord of the Rings: The Two Towers, The Lord of the Rings: The Return of the King and King Kong. He has also received the Academy’s Technical Achievement Award for co-developing the subsurface scattering technique which helps create realistic rendering of translucent materials, like skin, that brought Gollum to life.

Letteri got his start as an animator on such films as Star Wars (the 1997 special edition), Jurassic Park, Star Trek VI: The Undiscovered Country and The Abyss. In addition to the Rings movies, he also worked on I, Robot, The Adventures of Tintin and Rise of the Planet of the Apes. You might say he specializes in creating highly realistic creatures, from dinosaurs to Gollum, the Na’vi to apes. As senior visual effects supervisor on Avatar, Letteri helped launch a new approach to filmmaking called virtual production.

I had first spoken to Letteri some years ago, in 2003, when The Lord of the Rings: The Return of the King had just come out. This time around, I wanted to ask him about advances in digital effects since then, the evolution of performance capture, and of course his take on this whole 48 frames-per-second issue.

Gilsdorf: We spoke a long time ago, probably ten years ago, when Lord of the Rings came out. You probably don’t remember me.

Letteri: We’ll pick up where we left off.

Gilsdorf: Ha! Yes. Can you clarify your title? You are senior visual effects supervisor on The Hobbit, but also running Weta Digital?

Letteri: I’m one of the partners with Peter [Jackson]. I’m basically responsible for running the company, but my day job is doing effects for film.

Gilsdorf: You’re American. Are you full time in New Zealand or do you go back and forth?

Letteri: I went down 11 years ago on a two year contract to work on the Lord of the Rings and I’m still there.

Gilsdorf: It sounds like [Conceptual Designers] Alan Lee and John Howe. They went down to New Zealand and never left.

Letteri: Well, they left for a little while and of course they came back for The Hobbit.

Gilsdorf: That’s quite a commitment on your part, to go down and stay all this time.

Letteri: It’s been great. Everything we’ve been doing at Weta and getting it built up, the kind of shows we’ve been able to do, and working with Peter, it’s all been a progression for us.

“Hopefully we’re keeping a little bit ahead.”

Gilsdorf: Peter, you, and Richard [Taylor, head of Weta Workshop] have put New Zealand on the map as having the preeminent effects houses in the world. Has it been hard staying on top? Keeping up with the Joneses, whoever the Joneses are?

Letteri: Hopefully we’re keeping a little bit ahead. It’s a unique environment. We’ve just been fortunate with Lord of the Rings starting everything off, to have this whole landscape of characters and creatures that we had to create, that led just directly into King Kong, where we had a similar mandate, and of course with Avatar, that was entirely the mandate. To create this whole world, to create all these characters. We had to do the same thing with Tintin to where it was just, “OK, there’s no live action, now let’s just do the whole movie this way.” It’s been a kind a good progression where we’ve learned a lot along the way. So when we got to do The Hobbit, it’s kind of this homecoming. Well, here’s kind of where it all started. Now let’s take what we’ve learned and do it again.

Building the Barad-dûr ”bigature” for Lord of the Rings (Image: Weta Workshop)

Gilsdorf: I was just speaking with Richard Taylor, who told me that with shooting in 3-D, and the 48 frames-per-second format, there’s no need for Weta Workshop to make miniature models anymore. Or their famous “bigatures” (oversized miniature models). Now they are modeled entirely in computer graphics.

Letteri: It’s easier these days to create everything entirely digitally than it is to build miniatures and to try to figure out how to shoot miniatures. Just the logistics of shooting in stereo, getting certain camera moves on the stage, and dealing with the lighting, there’s just a lot to think about. Whereas in a digital world, it all becomes kinda the same. You have seen our scene inside the goblin caverns [in The Hobbit], where you’re flying the camera around in this three dimensional space. And you don’t have to worry about building pieces of the set that will fly in or out, or that you’re going to have to match up the various pieces that are going to have to be shot at different scales. This thing’s far away and it needs a big camera move, so you’re going to shoot it at 1/24th scale. [This things is] closer so you need a 1/8th scale. All that sort of thing.

Gilsdorf: The goblin town scene is a good example of the long shot you’re talking about. When the actors (playing dwarves and Gandalf) are running across the bridges and gangplanks, are the actors completely modeled in CG at that point? Are they motion captured and imported into a digital model?

Letteri: Yeah, the actors are all digital. But we do use a lot of motion capture to create the performances. Motion capture, if you’re doing a human kind of character, you know dwarf or goblin… that’s still kind of our animation tool of choice. It’s still the best animation technique for doing believable character [motion].

Gollum (Image: Warner Bros. Pictures)

Gilsdorf: Regarding Andy Serkis and his performance as Gollum: In what ways has which motion capture has gotten better? Is it actually tracking more parts of his face and body than before, than in Lord of the Rings?

Letteri: It’s more that we can do it more accurately and with lighter-weight gear. The markers that he has to wear are a lot smaller because the camera’s more accurate. That has certain advantages. It’s not the weight of the markers that’s such a big deal but the smaller they are, the less likely that they get knocked around. The less likely they impair the performance. With things like that, we’re trying to make the technology as invisible as we can.

“[Motion capture is] still the best animation technique for doing believable character [motion].”

Gilsdorf: I remember there was an internal dispute during Lord of the Rings between those who wanted to animate Gollum using motion capture versus the key-frame animators. At this point, is it almost entirely done as motion capture?

Letteri: It’s still a blend. But the motion capture tools have gotten sophisticated enough, not just on the capture side, which is really just part of the equation, but on the animation side. How you actually create the character once you have that captured data is still important. So for example, one of the things we can do much more easily now is we can take motion capture and we can edit it much more easily. That happens all the time.

Even though you are looking at a performance of Gollum, there’s still a lot of detail that has to be done by hand. The hand, the fingers, the foot, the way the toes bend, all those contact points you can’t capture accurately. Those all have to still be animated. In the past it was really hard to do that onto motion capture data, because it was so dense. Now, we have tools that allow you to do that.

If you’re doing an impossible jump over a rock, or crawling down the side of a wall, there are things where, even Andy as physical as he is, still can’t perform it. We go and change the animation and touch it up. In a character like Gollum, because he basically is a one-to-one human performance, that’s much less of an issue. Most of it is at this point cleanup work. Whereas ten years ago, there was actually a lot more animation you were seeing, even though we were trying to copy Andy’s performance. A lot of it was done by hand. This time, we’re recording a lot of what Andy’s doing directly and that’s what you’re seeing on screen.

The Goblin King (Image: Warner Bros. Pictures)

But then when you see Goblin King where there’s such a big difference in proportion between the actor’s body, Barry Humphries in this case, and the character, then you still have to go in and edit it, right? If you had a normal person performing that, just the fact that he’s moving his arm around, his elbow is going to be inside the Goblin King’s body the whole time. No part of his elbow will go out to clear the body. Well, an actor can’t act that way. So we want them to just act naturally so we understand the gesture of performance, and we go back and an animate on top of that to make it fit the body, to fit the character. So we sort of operate on a spectrum, almost, from 100 percent motion capture [to] it could be full key-framed animated. Hopefully you don’t know the difference. The animators need to be as fully aware of the characters as the actors are so that they can take over and blend all these techniques.

Gilsdorf: What’s the future? Will motion capture get so sophisticated, especially for special effects-heavy genre movies, sci-fi, fantasy, that increasingly these movies are going to be MoCap? Will MoCap eventually be able to capture individual toe joints?

Letteri: That is certainly is possible sometime in the future. We haven’t put so much effort into that because it’s sort of diminishing returns. The resolution we need to get that, we can capture enough data that we can just go in add that more easily than taking the effort to capture it. And we know that what we added in is correct because we have enough of the surrounding information. At some point, if you’ve got 100 megapixel cameras the size of an iPhone in five years, who knows, you might get that kind of information, who knows.

“Even though you are looking at a performance of Gollum, there’s still a lot of detail that has to be done by hand. The hand, the fingers, the foot, the way the toes bend.”

Gilsdorf: In terms of special digital effects shots, does the first Hobbit film compare to the last of the Lord of the Rings films, in terms of shots with digital elements in them? Does The Hobbit now hold the new record? I had thought at one point Return of the King had held the record.

Letteri: I forget the actual number [for Return of the King]. I think around 1,900 something. This one was 2,176 shots. I think a little bit more than The Return of the King.

Gilsdorf: Wow. And that’s just for the first movie?

Letteri: That’s just for the first movie. We’ve got two more to do.

Gilsdorf: And does creating digital effects for 48 frames-per-second, does that create headaches for you guys?

Letteri: 48 frames-per-second creates more work, but it’s spread throughout the effects. For example, you’ve got to render everything twice, there are twice as many frames. Four times if you consider the stereo. There are twice as many frames to render, so from a technical point of view, there’s that you have to deal with. Some things aren’t affected by it so much. Lighting doesn’t really matter what the frame rate is; animation, a lot of body movements are not that sharp that it really matters. However things like hand contacts, eye blinks, it’s actually nice to have the extra frames because you can really refine the points at which all those things really happen. Things that are very hand intensive … there’s twice as much work to do there. There’s no way around that. But it doesn’t add up to twice as much work but in the end, you have twice as many frames you have to create.

Gilsdorf: Are you a fan of the 48 frames, as a medium, or does it not do much for you?

Letteri: I like it both ways, because it what it does is remove the motion blur artifact from stereo. It allows your eye to resolve things spatially that you couldn’t see before if you were just shooting at normal 24 frames. I like that aspect of it. Having grown up with film at 24 frames, I’m still completely comfortable with seeing things on film. With me, I get to see it all four ways, every time. I’m sort of spoiled.

“People who are skeptical of 48 felt the loss when they went back to 24.”

I would suggest, if people are on the fence about it, to really just look at that approach. If you’re curious and you think you really want to see what this would really look like, like in a hyper-real sense, go for the 48. It might take you a few minutes to get used to, and when I say get used to, it’s just because it’s a familiarity. We all go in having seen film with an expectation of what a movie should look like. This will look a little bit different. But again, that’s the effect that we are going for, so you’ll get to see that. And you can always then go back to look at the 24 to experience the difference.

People that have done both have told me that the 24 has actually been a little bit of a letdown, even though they did have that initial adjustment period that they weren’t planning for, because you see everything so much more vividly in 48. I would say, if you’re skeptical of the whole thing, and you’re happy with the way film looks in 24, go see it that way. If you like the film and want to see it again, which we hope you do, then maybe it’s worth having a look at 48, because it will build up from there.

Gilsdorf: Whereas the other way, 48 to 24, it will feel like you are losing something.

Letteri: Right… People who are skeptical of 48 felt the loss when they went back to 24.

“It’s not 3-D, it’s two fixed images, but your eye is fooled into thinking it’s 3-D.”

Gilsdorf: The other issue being 3-D. You can also see this movie in 3-D, or not in 3-D. And when you combine 3-D with 48 frames per second, does that add another layer of issues?

Letteri: The formats you can see it in are: 3-D 48 frames, 3-D 24 frames, 2-D 24 frames, and 2-D film. The 48 frames, even though you can certainly do it and you get some advantages if you weren’t shooting stereo, the real reason to do it [in 3-D] is because it solves this motion blur problem with stereo. It’s one of the things that tends to give people a headache, if you’ve got stereo with a lot of fast action, and your eye can’t follow it because you can’t focus on it, this solves that problem.

Gilsdorf: So the motion blur is a 3-D problem, and the 48 can solve it.

Letteri: Motion blur is not a problem in 2-D, because you’re looking at a single image on a screen. Your eye is used to seeing motion blur in a photograph or in a film. But in 3-D your eye is fooled into thinking — see, it’s actually stereo, it’s not 3-D, it’s two fixed images, but your eye is fooled into thinking it’s 3-D, and of course it’s marketed as 3-D. But it’s not really 3-D. You can’t look around and focus on anything you want to. The photography’s already [set]. So if you try to focus on something that’s fast-moving, it’s just going to be blurry, and that’ll give you a headache. It’s like trying to focus on something from a fast moving car, when you’re not the driver, and you don’t know where you supposed to be looking, and everything just blurs, and your eye is trying to find something to lock onto? That’s the problem. This eliminates that problem.

Gilsdorf: So if you see The Hobbit in 24 frames per second, but in 3-D, that problem’s going to be there.

Letteri: Exactly. The motion will have a little more blur. So the fast action scenes will be a little bit harder to focus on than they will be at 48. They still play fine. We were careful to make sure we still delivered a good 24 frame movie. 48 just really takes it one step further.

Gilsdorf: That just sounds like, from a technical point of view, to make sure that all the formats this film will be released in will be a satisfactory viewing experience. A more than satisfactory viewing experience.

Dwarven-style 3-D glasses to watch The Hobbit. Yes these are for sale.

Letteri: We were careful to make sure that everything had its own unique finish. They’re all made from the master, but every one is finished differently so it’s optimized for its own format.

Gilsdorf: What was your biggest surprise in making The Hobbit, what was your main takeaway from the experience for you?

Letteri: I think the biggest surprise for me was actually Gollum. Because, after having done him 10 years ago, there’s always the sense that you’re going to come back to something that you’re familiar with and it’s not going to have the same impact. He’s still my favorite character and my favorite scene to watch. It just goes to show the power of not only performance — Andy brings this great performance. But when you have all aspects of the character working together, you’ve got realism on top of performance. You’ve actually got a digital actor there. You’re not just creating a creature like, I love the eagles, they’re big, they’re majestic, they feel like this amazing creature that you see in Middle-earth. With Gollum, you really understand his whole journey and his whole story, just in that scene with him.

“[Gollum is] still my favorite character and my favorite scene to watch.”

Gilsdorf: Sequentially, you first learn about Gollum in The Hobbit. But many poeple who see The Hobbit will have already seen Lord of the Rings. So this will be a flashback to a previous time in Gollum’s life. I had read somewhere that the younger Gollum, 60 years ago, has more teeth.

Letteri: We didn’t do that, though. We decided, we tried it, and just thought it made him look too different. We tried giving him more hair, too. We thought he looked too different, we thought he looked like a different character, so we put him back to the way he was. He’s completely rebuilt, the amount of detail, the physiology, everything is taking advantage of everything we know how to do now with characters ten years on. But the core is the same. You can immediately recognize him as the same as when you last saw him.

Gilsdorf: This is great. I really appreciate this. Best of luck and congratulations, and thanks again for taking some time to speak to me.

Letteri: Thanks. Talk to you next time.

Writer, journalist, poet and geek Ethan Gilsdorf is the author of Fantasy Freaks and Gaming Geeks: An Epic Quest for Reality Among Role Players, Online Gamers, and Other Dwellers of Imaginary Realms. More info fantasyfreaksbook.com or follow on Facebook fantasyfreaksbook .