Python Success Stories

Background

Industrial Light & Magic (ILM) was started in 1975 by filmmaker George Lucas,
in order to create the special effects for the original Star Wars film. Since
then, ILM has grown into a visual effects powerhouse that has contributed not
just to the entire Star Wars series, but also to films as diverse as Forrest
Gump, Jurassic Park, Who Framed Roger Rabbit, Raiders of the Lost Ark, and
Terminator 2. ILM has won numerous Academy Awards for Best Visual Effects, not
to mention a string of Clio awards for its work on television advertisements.

While much of ILM's early work was done with miniature models and motion
controlled cameras, ILM has long been on the bleeding edge of computer
generated visual effects. Its computer graphics division dates back to 1979,
and its first CG production was the 1982 Genesis sequence from Star Trek II:
The Wrath of Khan.

In the early days, ILM was involved with the creation of custom computer
graphics hardware and software for scanning, modeling, rendering, and
compositing (the process of joining rendered and scanned images together).
Some of these systems made significant advances in areas such as morphing
and simulating muscles and hair.

Naturally, as time went by many of the early innovations at ILM made it into
the commercial realm, but the company's position on the cutting edge of visual
effects technology continues to rely on an ever-changing combination of custom
in-house technologies and commercial products.

Today, ILM runs a batch processing environment capable of modeling, rendering
and compositing tens of thousands of motion picture frames per day. Thousands
of machines running Linux, IRIX, Compaq Tru64, OS X, Solaris, and Windows
join together to provide a production pipeline that is used by approximately
eight hundred users daily, many of whom write or modify code that controls
every step of the production process. In this context, hundreds of commercial
and in-house software components are combined to create and process each frame
of computer-generated or enhanced film.

Making all this work, and keeping it working, requires a certain degree of
technical wizardry, as well as a tool set that is up to the task of integrating
diverse and frequently changing systems.

Enter Python

Back in 1996, in the 101 Dalmation days, ILM was exclusively an SGI IRIX shop,
and the production pipeline was controlled by Unix shell scripting. At that
time, ILM was producing 15-30 shots per show, typically only a small part of
each feature length film to which they were contributing.

Looking ahead towards more CG-intensive films, ILM staff began to search for
ways to control an increasingly complex and compute-intensive production
process.

It was around this time that Python version 1.4 came out, and Python
was coming into its own as a powerful yet simple language that could be
used to replace Unix shell scripting. Python was evaluated, compared to
other technologies available at the time (such as Tcl and Perl), and
chosen as an easier to learn and use language with which to
incrementally replace older scripts.

At ILM, speed of development is key, and Python was a faster way to code (and
re-code) the programs that controlled the production pipeline.

Python Streamlines Production

But Python was not designed just as a replacement for shell scripting and, as
it turns out, Python enabled much more for ILM than just process control.

Unlike Unix shell scripting, Python can be embedded whole as a scripting
language within a larger software system. In this case, Python code can
invoke specific functions of that system, even if those functions are written
in C or C++. And C and C++ code can easily make calls back into Python code
as well.

Using this capability, ILM integrated Python into custom applications
written in C or C++, such as ILM's in-house lighting tool, which is
used to place light sources into a 3D scene and to facilitate the
writing, generation, and previewing of shaders and materials used on CG
elements. It is the lighting tool that is ultimately responsible for
writing the 3D scene out to a format that a renderer can interpret and
render.

At the same time, more and more components, such as those responsible
for ILM's many custom file formats and data structures, were re-wrapped
as Python extension modules

As Python was used more widely, extending and customizing in-house
software became a lot easier. By writing in Python, users could
recombine wrapped software components and extend or enhance standard
CG applications needed for each new image production run. This let
ILM staff to do exactly what a production needed at any given time,
whether that meant allowing for a specific look for an entire show, or
just a single CG character or element.

As it turned out, even some of ILM's non-technical users were able to
learn enough Python to develop simple plug-ins and to create and modify
production control scripts along side the technical users.

Python Unifies the Toolset

Encouraged by its successes in batch process control and in scripting applications
and software components, ILM started to use Python in other applications as well.

Python is now used for tracking and auditing functionality within the
production pipeline, where an Oracle database keeps track of the hundreds of
thousands of images that are created and processed for each film. DCOracle2,
one of the Oracle integration libraries available for Python, has performed
well in this task and is now in use on Linux, IRIX, Tru64, and Solaris.

Python is also used to develop the CG artist's interface to ILM's asset
management system. Designed to be modular, this tool has been enhanced
by a large group of developers and non-developers alike to extend well
beyond its original mandate. The application is now used not only to
manage CG assets and elements, but also in daily shot review, as a
network-based whiteboard, as an instant messenger, and even allows an
occasional game of chess.

As Python was applied in more ways, it slowly crowded out a plethora of
competing technologies for shell scripting and batch control, embedded
scripting, component software development, database application development,
and so forth. Python's versatility ultimately simplified the developers'
toolset and reduced the number of technologies that needed to be deployed to
ILM's thousands of production computers. This new, simpler toolset translated
directly into an easier to manage and more reliable development and production
process.

Hardware Costs Reduced

Although chosen initially for its ease of use and integration capabilities,
Python's portability to many other operating systems eventually became one of
its key strengths.

Once Python was in use, it made the production control system portable.
This gave ILM additional freedom in making hardware technology choices,
including a large-scale introduction of commodity PC hardware and
Linux, a move that has saved the company substantial amounts of money
in recent years.

Source Code Access Important

After having used Python intensively for six years, ILM has yet to run into
significant bugs or portability issues with the language. As a result, ILM has
since Python 1.5 been able to rely on stock distributions in unmodified form.

However, availability of source code for the language acts as an important
insurance policy should problems arise in the future, or should custom
extensions or improvements become necessary. Without this, ILM could never
have bought into Python so heavily for its mission-critical production
process.

One case where access to source has already been beneficial was in ILM's
continued use of Python 1.4, which is generally considered obsolete. Because
the production facility is under continuous use, upgrading systems to new
Python versions would result in significant disruption of the production
process.

Instead, ILM installs new systems with newer versions of Python but
maintains older systems only so they can run the same scripts as the
newer systems. Supporting this mix has relied on access to the Python
sources in order to back-port some changes found in newer Python
versions, and to reimplement portions of newer support libraries under
older versions of Python. ILM is currently running a mix of Python 1.4,
1.5, and 2.1.

Python Tested by Time

The visual effects industry is intensely competitive. To stay on top of the
heap, ILM continuously reviews its production methods and evaluates new
technologies as they become available.

Since its adoption in 1996, the use of Python has also been reviewed numerous
times. Each time, ILM failed to find a more compelling solution. Python's
unique mix of simplicity and power continues to be the best available choice
for controlling ILM's complex and changing computing environment.

About the Author

Tim Fortenberry joined Industrial Light & Magic in 1999 as an intern.
Later that same year he began to work full time in the Resources
department. He worked as a scripts/tools programmer. Shortly after,
Fortenberry joined the Research and Development department. He is one of
the founding members of the Pipeline and TD Tools groups that helped
bridge the gap between artists and technology.

As an engineer, Fortenberry is responsible for developing and maintaining
the myriad of applications used for rendering and pipline control flow of
images at ILM. Prior to joining ILM, Fortenberry worked as a Linux
systems administrator for VA Linux Systems.

Originally from Southern California, Fortenberry received his Bachelor of
Arts degree from the University of California at Berkeley in Anthropology
with an emphasis in Archaeology.