Q&A: How Digital Effects Gave 'The Muppets' New Freedom

Credit: Walt Disney Studios Motion PicturesRest assured, Muppet purists: You won't see any computer-generated Kermits, Gonzos, or Fozzies in The Muppets, which opened in theaters last week. Every time the Muppets appear on screen in the movie, they're the real deal: fuzz, felt, and fur creatures given their voices, movements, and expressions by human puppeteers.

But you also won't see any visual evidence of those puppeteers in the film, despite the fact that some scenes required several puppeteers to be in full view of the camera. This is where the film's extensive visual effects make their mark.

For the most part, the Muppets appear as autonomous, stand-alone beings in the film, free of their strings, rods, wires, and puppeteer operators. Unlike most effects-heavy movies, The Muppets uses most of its digital trickery to conceal what's actually there instead of adding things to the scene.

To get a behind-the-scenes look at what was involved in giving the Muppets their on-screen autonomy, I spoke with Max Ivins, visual effects supervisor for Look Effects, whose team worked on hundreds of scenes in the film.

PCWorld: When working on the digital effects for The Muppets, did you feel a lot of pressure to make the effects conform to viewers' expectations and to the legacy of the show and the previous films?

Max Ivins: I didn't feel a lot of added pressure. At the beginning of the project, when we were talking about doing the effects for the movie, I was like, "Really? What are we going to do? Put legs on them? Are we doing CG Muppets? I don't get it." But that's not what they wanted to do.

The biggest factor, in terms of what we were used for, was to give the puppeteers more freedom to do the puppeteering. There were a lot of characters shot on a blue stage with blue props, and puppeteers dressed head-to-toe in blue standing behind the puppets. We removed the puppeteers later, so it gave the puppeteers a lot more freedom in that they didn't have to hide from the camera to do everything.

Credit: Walt Disney Studios Motion PicturesAnd the best thing about the movie is that it's about the Muppets--it's not about these spectacular effects. In a way, our job was to make it seem as if we were never present. There was sort of a conscious effort to remove the digital look from things. All of it needed to feel tangible, even if it was obviously not "real." It's a puppet. But they didn't want it to seem as if there was an extraordinary leap in technology, even though we definitely used that.

PCW: Did the digital-effects team have to follow any sort of rules as to what you could and could not do to enhance some of the things the Muppets were doing? For example, adding digital effects to their eye movements or expressions or anything like that?

Credit: Walt Disney Studios Motion PicturesIvins: We didn't do any expressions or change any of the puppeteering. There was one shot involving a reflection in a mirror, and they shot the action from a camera that was just off to the side of the mirror. In that scene, the mirror wasn't as distorted as they wanted--it was like a fun-house mirror. So we took the reverse angle and put [the character] into that and ended up changing the eye line to make it work as a reflection. That was the sole thing we ever did to any Muppet's face.

Otherwise, it was pretty much just rod removal--the only times we ever retouched any of the Muppets was to take out a rod that was in front of them. We didn't add any limbs or arms. If we did use something to repair where a rod had been or anything, we just took the actual photography of that limb or whatever, and cloned it over into the right place.

We didn't really go back and reference any of the movies--we referenced more the television shows, following what the director gave us as a reference. That was mostly about reproducing the "arches" shot, which was the opening for the TV show.

There's a scene with a wall about five arches tall, with a Muppet in every arch--about 46 different Muppets in it, all on blue screen. It was one of those shots where it's like, "Oh my gosh, it's endless."

PCW: So you had to digitally remove 40-something puppeteers from that shot for the movie?

Ivins: Oh, no--that was shot all on blue screen and composited into the final shot. For that scene, the puppeteers did it traditionally. Most of the Muppets are waist-down in the original, so it was more of a "traditional puppeteering" thing where the puppeteers are all below the frame we were going to show. So the only thing we had to do for that was rod removal on almost every single one of them [laughs].

That's one thing we did for the whole movie: Remove all the rods for the hands. We didn't leave any rods in on purpose--I'm sure there are rods that are visible, but you probably can't tell if they're rods or not.

Credit: Walt Disney Studios Motion PicturesI think two weeks before we were done, we were still finding puppeteers' heads in shots. "Wait a minute, what's that thing over here? That's somebody's head!" [laughs] "Oh no, not another one." Some of the shots, you've got like 15, 20 puppeteers crouching down, kneeling, lying on their backs, operating a big crowd of puppets ... this giant crowd under the crowd. A big crowd of Muppets, and underneath that, a big crowd of puppeteers, all looking at a monitor to see how their Muppets are positioned. So they might set up and think, "That's good, no one sees my head," but then another puppet moves during the shot, and boom, there's their head. It was amazing how long into the process we got before we found the last puppeteer head in the shot.

PCW: You mentioned using blue screens. Did you need to use blue screens because Kermit the Frog is about the same color as a green screen?

Credit: Walt Disney Studios Motion PicturesIvins: Basically. Characters with blue on them were less problematic to pull keys for, and there aren't that many principal characters that had any blue on them at all. Gonzo has some blue feathers on his head, which was a nightmare, but other than that, blue isn't in the Muppet palette very much. There are a couple of Muppets that appear in the arches shots that are blue...

PCW: Sam the Eagle!

Ivins: Right--he was shot on blue screen, believe it or not. It really wasn't a big problem. But Kermit on green screen is a disaster!

They shot tests on blue and green, and sent them to us, and we pulled keys on them. They didn't want to carry around a green screen and a blue screen, or make a green stage and a blue stage, so in the end we just decided on blue as the less problematic color.

I didn't realize before the project how many Muppets are hairy and furry. They are furry! And that is not friendly to blue screens. That ended up being the most challenging thing about it--the intricacy of getting fine detail in hair, you have to pull a key that's very specific to it. If the character is lit brighter on one side than on the other side, or if there's shadow down by the legs or whatever, the key has to be different there. So a lot of times, there are multiple keys on each character, basically.

And the puppeteers in blue suits, they're not lit very well if they're behind a character, so you have to do a lot of rotoscoping and luminance keying to fix those things.

PCW: Were the puppeteers always controlling the Muppets from underneath, or did it depend on the specific scene or Muppet?

Ivins: For most of the project they were [underneath], but there were quite a few head-to-toe shots of the Muppets. For those scenes, puppeteers were in the shots, because it sometimes took four puppeteers to make them walk and move their arms and their head. It was quite an undertaking for the puppeteers to stay behind the puppet and get the right form of motion and everything. For those head-to-toe scenes, the [characters] were shot on a blue-screen stage and composited into the shots later.

They ended up building several pretty involved rigs--in one scene in the movie, Beaker gets shrunk down to about 5 inches tall, and we see him from above, running in a circle. To puppeteer that, they basically built a merry-go-round with a platform about 4 or 5 feet off the ground so that they could have puppeteers control him to run around in a circle while somebody spun the whole thing around. They went to great lengths to compose some of these shots.

PCW: Were all the Muppet scenes filmed separately on a blue-screen stage? Or did the project also include scenes in which the Muppets and real-life actors were interacting in front of the camera?

Credit: Walt Disney Studios Motion PicturesIvins: Most of the shots done on the blue-screen stage were solo shots, kind of hero shots, and the puppeteers would react to the plates of the previous footage. There's a scene where the new Muppet ("Walter") was shot in this sequence on a blue-screen stage: He's climbing up on a cabinet, jumping onto a doorknob, swinging into the kitchen, and flying across the room. Jason Segel is in the scene, watching him fly by and talking to him, but Segel's part was all shot beforehand. Then they just puppeteer to [that action] while looking at it. That was sort of the formula: Give the puppeteers what they're going to have to "fit to" and "match to."

But a lot of times it was shot traditionally. The puppeteers are masters at getting really low and using one of those dollies that mechanics use to roll under cars; they would use those and push themselves really low to the ground for the shots where the Muppets would be walking around. A guy would be pushing himself along with his feet on one of these things underneath. They're really good at that [laughs]. I was really impressed. They're great at getting into these odd angles. They had a lot of driving scenes where they rigged a car so that the puppeteers could get inside of it and do the puppeteering--they took out the bottom of the car, almost.

In some scenes the actors and the puppets were together on a blue screen. In one shot a Rolls Royce comes driving out of the ocean onto Cannes beach in France, which they shot at Lake Castaic near us [in Los Angeles]. They put a Rolls Royce in the water and pulled it out with a cable, onto a beach that they hauled sand in to build. They had a bunch of people in thong bikinis and Speedos. It was about 7 o'clock in the morning, so it was about 56 degrees or something. We put a big matte painting of the Cannes beach hotel behind them, and then they shot the actors and the Muppets on a blue-screen stage with a rig that looked like a Fred Flinstone vehicle--just boxes for seats, and basically a dolly to mimic the movement. So we tracked that all inside the car so it looked as if they were sitting in it coming out of the water.

In a lot of ways, that was a traditional effects job. It just so happened that there were puppets in it.