Postmortem: Lucas Arts' Star Wars Starfighter

1. Good
team communication. I've read many Game Developer Postmortems
that blamed failures on a lack of communication, so I'm particularly proud
that we got this one right. From the beginning of the project, Daron worked
hard to impress on the programmers that it was the level designers and
artists that would ultimately secure the success of Europa. As I became
fond of saying, programmers build the picture frame, but it's up to the
rest of the team to provide the most important part, the picture.

To bring
this to fruition, the programming team needed to understand as best we
could the way the rest of the team worked. While Andrew worked with lead
artist Jim Rice and our world-builders to understand their workflows,
Brett Douville, our AI and mission programmer, filled a similar role with
the level designers. Brett scheduled regular "LD Days" with
each individual member of the level design staff. This gave each designer
the opportunity to meet with Brett on a regular basis and show him the
specific challenges and problems that they were tackling in their missions.

This image depicts the design schematics for one of the Lok missions.
Level designers made elaborate plans such as these for every level
that appears in the game.

Europa periodically
had full-blown team meetings where we could get together and kibitz about
the overall state of the project. However, the most valuable meetings
were at the subteam level. Both the programming team and the art teams
would meet weekly to discuss the issues of the day, and each of these
meetings would have an attendee from the other camp -- a role we referred
to as the "exchange student." This meant that if questions came
up in the art meeting, for example, that required answers or input from
a programmer, there would always be someone present that could give an
informed opinion. Likewise, as programmers would discuss issues or new
features in their weekly meeting, the art or level design representative
would be able to disseminate this information among the other team members.

Finally,
we relied on an internally maintained web site as a pivotal communications
tool. We tried to make the site as comprehensive as possible, organizing
areas along the lines of programming, art, level design, project management,
and so on. When artists had questions about how to implement a particular
effect, or a level designer needed a refresher on our class-file script
syntax, there was usually a web page that they could be directed to that
would answer many of their questions.

2. Project
discipline.Star Wars Starfighter was a well-organized project.
In the heat of battle it's all too easy to let requirement lists and schedules
get lost in the shuffle of the moment. We were determined not to let this
happen. As soon as our technology began to take shape we started to follow
an iterative process of milestone planning and execution. These milestones
were typically four to six weeks in duration, with no milestone extending
longer than eight weeks. Milestones were also required to demonstrate
some visual or gameplay aspect of the game. As a consequence, we had very
few milestone tasks that looked like "complete the Foobar class";
instead we would have a milestone task that might read "Explosion
smoke trails," and the assigned programmer would know that completing
the Foobar class was an implied requirement. By keeping our attention
focused on a discrete and relatively small body of work, we were able
to avoid the cumulative errors that invariably creep into longer schedules,
while still allowing for demonstrable progress.

Most of
the milestones were driven by the progress of the technical team. Programmers
were solely responsible for estimating the duration of their tasks. We
would occasionally adjust these estimates outward but would never change
an estimate to be shorter. Tasks were structured so that the shortest
scheduled task was never shorter than a half-day. Even if a programmer
was certain that a task could be completed in less than half a day, experience
clearly showed that the time would be lost elsewhere. Using these simple
rules of thumb, we were consistently able to build schedules that were
fair and accurate. Out of eight scheduled milestones, we never missed
one by more than a handful of days. Best of all, most team members completely
avoided extended periods of crunch time. Like most game teams as they
approach their ship date, everyone was working hard and often into the
evenings; however, this period of time was short, and we never had to
resort to all-nighters.

We also
closely managed the process we used to distribute new binaries to the
team at large. Since most of our development occurred on the PC even after
making the decision to ship on the PS2, it was important that team members
have timely access to stable builds of the game. We accomplished this
through weekly public builds. Once a week we would package and distribute
the current code as a full-blown Install Shield-compiled install. This
provided team members with debug and production versions of the game,
along with level design tool and art exporter updates. Predicting that
public builds would become critically important, we tried to be as ruthless
as possible about maintaining the build schedule. As we got closer to
our ship date, the frequency of these public builds increased until we
were performing new builds as often as three times a week. By this point
we had a full-time staff member dedicated to managing the public build
process and ensuring that the distributed code met quality and functionality
expectations.

3. A
well-executed PC-to-PS2 transition. Making the decision to move the
project to the PS2 could have been a complete disaster. Yet, despite paying
little attention to portability during the earliest stages of the project,
the Europa code base was well positioned to make the jump to the PS2 platform.
With the aid of strong and generally stable development tools provided
by Metrowerks, the core port went off without a hitch. The biggest trick
was on the graphics side, because this was clearly where we were most
vulnerable. None of our programmers had any console experience, and none
of us was up to the task of tackling the PS2's infamous low-level vector
units. Enter LEC gl.

LEC gl was
the brainchild of Eric Johnston and Mark Blattel, two of Lucas Arts' most
senior console programmers. They had recently shipped Star Wars: Episode
I Racer for the N64, and they welcomed the opportunity to tackle a problem
temporarily that was one step removed from the day-to-day pressures of
a project team. Although Europa was the most immediate recipient of their
efforts, Eric and Mark were never officially on the project. Instead they
worked in a support role, providing us with regular LEC gl library drops
and immediate "on-call" PS2 graphics support.

There was
another, more subtle problem that we had to conquer when we made the decision
to adopt the PS2 as our primary platform. Most of the team members were
big PC game players, but very few of us played console games. Intellectually,
we knew that there were huge philosophical differences in game design
between consoles and the PC. Much of our original game design had used
the X-Wing games as a conceptual leaping-off point, wandering into
the arcade action of i only when it suited us. Now that we were
on the PS2 we recognized that our design priorities needed to be completely
flipped. Instead we would use Rogue as our primary point of reference
and work from there, layering on gameplay elements borrowed from X-Wing
as needed. As such, I think the final game demonstrates our successful
indoctrination into the console mindset. We were having so much fun blowing
things up that we had little desire to start adding sim-like features
to the gameplay experience.

4. Macromedia
Flash. As we approached the end of summer in 2000, we realized that
we had a serious problem on our hands. Despite our best efforts, we still
had not addressed the issue of our out-of-game user interface. We had
a 2D virtual-page system that we were using for our HUD (heads-up display)
symbology, and we had always planned to evolve that into something that
could be used for what we called the "administrative interface."
However, in August, with the quality assurance department nipping at our
heels and our ship date looming ominously in the distance, things were
not looking good.

We had heard
that a small San Francisco-based company named Secret Level was adapting
Macromedia's Flash technology for use in PS2 games. After meeting with
company representatives, we were excited by the prospect. The Macromedia
content-authoring tools were far more elaborate than anything we could
come up with in the same time frame. We also suspected that there was
a wealth of Flash authoring expertise available from out-of-house contractors,
which would help us smooth out the work load. Most importantly, we were
very impressed by the intelligence and games savvy of the Secret Level
staff. When we realized that building our user interface in Flash would
significantly ease our localization efforts, we decided to take the plunge.

Early concept art for the mercenary Vana.

Soon afterward,
we hired a design firm named Orange Design to help us implement our administrative
interface in Flash. Orange not only had a ton of experience with Flash,
but they also brought a technical perspective to the table. We knew that
this technical emphasis would be critical for working with our programming
team on integration issues.

Integrating
Flash into the Europa engine was not a completely smooth process, however.
Performance in the first-generation Flash Player was poor (current generations
of the Player are now much faster), and we had to spend a lot of time
integrating the user interface Flash movie with the core game systems.
That said, the five months that a single half-time programmer spent on
this task ended up yielding a user interface that was far beyond what
we would have been able to custom-code in the same period of time.

5. Good
debugging systems. Our programmers built several tools that greatly
helped our pursuit of high-quality code. One of the most instrumental
was a Windows-only library that provided detailed stack-tracing information.
This library was largely based on the code and concepts covered by John
Robbins' "Bugslayer" column published in the Microsoft Systems
Journal.

As is standard
practice on many games, we built a custom memory manager that could detect
when the application was leaking memory. However, unlike most implementations,
when our memory manager detected a leak it could provide a comprehensive
stack trace of arbitrary depth, leading directly to the leaking code statement.
This capability represented a significant advantage over other implementations
that could only provide the immediate location of the allocation request.
If the memory allocation was being made by the Standard Template Library
(STL) or one of our widely used utility classes, it was usually not enough
to know what part of the STL or which one of our utility classes was the
culprit. What we really needed to know was what class called the STL method
that caused the leak. In fact, the leak was usually several steps up the
call chain. Our stack-tracing library made finding these cases almost
trivial.

We also
incorporated stack tracing into our exception- and assert-handling systems.
When the game encountered a hard crash, we trapped the exception and generated
a complete stack trace; a similar process occurred when our code asserted.
This information was initially reported back to the user in dialog form.
However, we also packaged up this same data and had the game send the
programmers an

e-mail detailing
exactly where the problem occurred. This ended up being an invaluable
tool for us. As a matter of practice, the Europa programmers got into
the habit of checking the assert mailbox regularly. In addition to appraising
the current stability of the code, we could also use this data to spot
trends and note when people weren't being diligent about installing new
builds.

In the end,
we had an exceptionally smooth QA process because the bugs we did have
were generally easy to track down and fix. There were no last-minute "heart
attack" bugs that required us to set up camp and track a single problem
for hours or days at a time. This made life easier on the programmers,
but it also made things easier for the testing team and improved morale
across the entire project.