Movie special effects expose engineers to star technologies

For engineers who are also movie buffs, helping make Star Wars,Jurassic Park and Forrest Gump might seem like a dream job, where they could see special effects as they're developed and bump into famous movie stars and directors while they're working. But to many engineers at Industrial Light & Magic, the biggest excitement is getting to work with hot new technologies, often enhancing state-of-the-art products so they meet the needs of filmmaker George Lucas and others who are leading the charge to digital movie-making.

"In a job prior to this, back on the East Coast, one of my interview questions was 'What is your dream job?' My semi-joking answer was sweeping the floor for George Lucas," said Raleigh Mann, manager of network systems at ILM (San Rafael, Calif.). "I moved out here [to California] to get married, and fate kept playing me a wonderful hand. At a job fair, a Darth Vader cutout caught my eye, and I was going to tell them that it was a copyright violation. It turned out [the exhibitor] was ILM, and I spent an hour talking to the human-resources guy."

For chief engineer Gary Meyer, what's been a 10-year stint began as a three-day job. A friend asked if he would help combine a video camera with the film camera so directors could get an immediate look at their shots. Now he's creating systems that let them view production from anywhere in the world.

Marty Brenneis was finishing up some wiring at Francis Ford Coppola's Zoetroppe studios when he got a call from ILM. In the 20 years since Brenneis became the first electronics engineer hired by ILM, the technical staff has grown enormously. Engineers used to work there mainly on a production basis, but once Jurassic Park marked a changeover to digital-effects editing, the permanent engineering staff ballooned.

Today, most of ILM's 1,200 employees are engineers, and there are more than 50 engineers assigned to improving and maintaining the production system that handles every cinematic work that flows through the operation. Just how much data they're processing is evident in the workload they put on the main network that links them together.

"We're moving huge files to multiple areas multiple times," said Mann. "When I started here in June of last year, we averaged 2 terabytes/day on the production side. That peaked at 14 Tbytes/day, which is what we used to see at AOL. The average during the Star Wars [Episode I: The Phantom Menace] work was about 5 Tbytes/day. Now that Star Wars is wrapped up, it's back down to around 2 Tbytes/day."

That pipeline would have to be even larger if peer-to-peer networks didn't carry a lot of the load. The systems attached to those networks are mostly the 600 Silicon Graphics systems housed in a couple of huge computer rooms, replete with raised floors and powerful air conditioning, not to mention several uninterruptible power supplies. As in many parts of ILM, that hardware has some specialized components designed to handle the work load and keep things running smoothly.

"We have an architecture that's built so that when there are hardware faults, a failover kicks in and the animators don't know it happened," said Joe Takai, director of production engineering. "That's a challenge with our system, which has everything from Macs to PCs to the SGI servers to systems that are entirely custom. We're continually doing system-level R&D, and we hold several patents as a company."

Though the special-effects house is pushing technology's boundaries on the engineering side, it isn't stretching the limits of the engineers at the same time. Most say that they work harder than at previous jobs, but the hours aren't as demanding as with other area companies.

"Once in a while we work 12-hour days, but the rest of the time it's 8- to10-hour days," said Jeff King, a senior system administrator who traded ties for shorts and T-shirts when he joined ILM two years ago. "That's a lot different than when I used to work in aerospace and you always had to work long hours. When we're on call during production, we may have to come in and work until 3 a.m. [But] they have figured out how to get a lot of things done without burning people out."

When they're working, the engineers have challenges unlike anywhere else. Movie fans easily get blas once they've seen something, and that pressure ripples down to the system designers.

"We've got a twofold challenge. We're continuously pushing the envelope to create a pipeline big enough to get things done and get a lot of film out at once," said Ken Beyer, manager of Unix server systems. "Once the film is out there, it's 'been there, done that,' and the animators have to create an even better effect.

"During the production, there are definite cutoff times that can't be missed, so our other job is to minimize downtime to nothing. Computer systems will always break, so our job is to bring them back up quickly."

That was a particular challenge with the latest Star Wars film.

"Star Wars was a trump card, where we really found out that things could be broken. Although we had to scurry to fix them, everything went along without a hitch," Mann said. "Star Wars was a huge endeavor and a milestone for us, not just because it is the boss' show but because it is such a big show. We wanted to eliminate any delays, and despite the size of The Phantom Menace, we can still say we haven't missed a deadline for any movie, ever."