By the Books: Solid Software Engineering for Games

GDC2003 roundtable by Noel Llopis

Thanks again to everybody who attended the roundtable this
year. It was great to see the amount of interest in the topic, as well as all
the people who repeated from last year.

The roundtable was held at three different times, and even
though the general topic was the same, all three sessions narrowed down on
slightly different subjects. Here is a summary of what was discussed.

Attendance

All three sessions ended up being packed, with a total of
around 120 people (50, 35, and 35), which is pretty amazing considering the
last roundtable was going up against Fred Brooks' keynote :-)

Of the people who attended, very few were straight managers
(about 5-6%), but there were a lot of lead engineers (roughly 30%), and the
rest regular programmers/engineers.

The split between consoles and PC development was a rough
50-50 with a tiny fraction of people doing handheld development.

Like last year, C++ was by far the dominant language of
choice, with just a few people using straight C, Java, or even C# in the case
of one person. Nobody this year admitted to using only assembly, so maybe
things are moving in the right direction.

Software engineering

What exactly is software engineering and why do we care as
game programmers? People pointed out that better software engineering doesn't
necessarily make for a better game. As a matter of fact, there are plenty of
examples of terrible development processes that end up creating a good game.
What software engineering does is help make the development of the game faster,
more predictable, and especially less painful for everybody involved.

People pointed out that the larger a team becomes, the more
important software engineering becomes to keep development in track and keep
everybody working together. With teams of 25+ programmers and schedules over 18
months, this is becoming more and more important every day. One person
mentioned that there were 140 programmers in his company, split into 4-5
different projects, and a team that creates a common code base and tools for
those teams.

Methodologies

Some people associated software engineering with highly
structured, pre-defined approaches, with lots of documentation and overhead.
Something to be feared even. Clearly, not all software engineering approaches
are that way, and some try to be quite at the other end of the spectrum (agile
methods).

Mostly everybody described their development process as
being somewhat unstructured and very iterative, although some claimed they were
doing nothing more than plain “hack and slash”. Only a few people used a
traditional waterfall model, and even so, with some iteration to it.

A few people were either using, or had tried in the past,
agile methodologies like extreme programming. Not everybody adopts all the
practices from extreme programming, and some of them always encounter a lot of
resistance (pair programming, or no code ownership). Somebody mentioned they
used to do XP in his company, but they had to stop because all the programmers
hated it and threatened to quit.

Unlike last year, we didn’t get into agile methodologies on
purpose because there was another roundtable dealing entirely with that topic.

Technical design

Mostly everybody did some sort of technical design. Only two
or three people claimed they jumped into coding something without doing any
prior analysis.

There was quite a range of how people did design though. For
some, design meant running some ideas by a co-worker, maybe using a whiteboard
and drawing some UML diagrams. On the other extreme, some people were very
happy with tools like Together
that allow for a UML design to be converted into code and then have the code
changes reflected back in the UML diagram.

Most people fell somewhere in the middle. A lot of people
used UML, but mostly as a communication aid rather than as a starting point for
code. Most people did not keep the initial design documents up to date since
their purpose was for the initial design only.

Sharing information

One of the problems with trying to do any design with a team
of people is the question of how to actually share that information, keep it up
to date, and make sure everybody is aware of it.

Several people brought up code reviews, but the response to
them was very mixed. Only about 15 people were doing some form of code reviews.
Some thought it was a waste of time, but most of the people who were doing them
found them useful. Most of the code reviews were done by a senior engineer on a
one-on-one basis, or even reviewing the code alone and then emailing the
original programmer or talking to him in person. One of the side benefits
mentioned of code reviews is that people know somebody is going to be looking
at their code, so they end up writing better quality code to start with.

Somewhat related to code reviews was the issue of pair
programming. It was brought up that pair programming was also a way to share
and spread knowledge about the design and the code base.

Weekly meetings were also mentioned as a good way to keep
the team in synch, knowing what everybody else is working with, but it's not
enough to explain all the details. One person said everybody in their team did
everything together, in and outside of work, so they were always talking about
work issues and information dissemination was never a problem.

Some specific ways of sharing information that were brought
up were:

Internal
project web pages.

Internal
mailing lists and forums.

Use of
Wiki pages. Several people brought
this up. One of them even had the artists and designers using it. It can
track changes to the web pages so it's useful to keep a history.

Code ownership

This is another hot issue that nobody seems to manage to
agree on. Most people expressed that they were using a system with fairly
strong code ownership. There was one or two people who were clearly responsible
for knowing a particular system, and were the people to go to if anything had
to be changed.

On the other extreme, a few people preferred the
no-ownership approach. Everybody can modify any files, make improvements to
them, fix bugs etc. This has the advantage that knowledge about the code is
spread among the whole team and that issues won't become a problem of egos, but
problems with the code. It might also improve team spirit and communication. It
also makes the approach to development much more flexible since there might be
several people qualified to work in different areas.

On the other hand, strong code ownership might be more
appropriate for areas where specialized knowledge is required (for example,
physics code or pathfinding). Also, with strong code ownership it is possible
for people to identify more with a particular system and do a better job with
it as opposed to letting the broken
windows syndrome settle in.

One team used an approach of no-ownership, but had a person
full-time refactoring the code, fixing problems, and improving the overall
quality.

Coding standards

A fairly large percentage of the people who attended had
coding standards in their company; around 30-40%. However, probably only a bit
over half of them actually followed them.

Like in the case of code reviews, the people who currently
had coding standards found them invaluable, while the rest were mostly
skeptical about their usefulness. Somebody described them painful to implement,
but good in the long run. A coding standard can be particularly beneficial when
bringing in new members.

Some people used macros in their environment to make it
easier to follow a particular convention. For example, macros could create new
skeletons for classes and function definitions. One person mentioned the tool Artistic
Style to automatically format the code in the correct way for the company
standard (although it might cause some problems with source control history).

A hotly debated issue is how much should a coding standard
cover. Everybody agreed that it shouldn't be too nit-picky, and only
concentrate on the things that matter. Unfortunately, for some a consistent
bracing style matter a lot, while others are not willing to give up their own
style. It seems that one of the most beneficial things that could be covered in
a standard is not so much the formatting of the code, but the design principles
behind it, and other high-level issues, but usually none of those things are
part of a standard.

Source control

Like last year, everybody working in a team was using source
control. Only two people who were working by themselves were not using it (but
several other people who were also working by themselves were using source
control regularly).

Interestingly, the large majority of the people only used
source control to keep the history of the files. Only a small number of people
(10%?) were taking full advantage of branching during development (virtually
none of them were SourceSafe users).

Another source control strategy presented was that each programmer
has a private branch. This gives great flexibility, but merging is more
complicated. Some teams had scripts that would email specific people if files
in a particular section of the code changes so they are aware of the changes.

Unit testing

Unit testing
was brought up briefly. Only a handful of people were actually doing any real
unit tests, but they were really pleased with the results.

A common concern with unit tests is how to effectively test many
of the systems involved in game development since they might not be tested
easily in isolation, or they might need full levels to be tested. Also, testing
high-level code is more complicated than low-level math libraries.

Testing of visual aspects is particular difficult with unit
tests, but one team (working on a game engine) reported to use image diffs to
verify that the engine was working correctly.

The beneficial side effect of serving as documentation for
other people was brought up, as well as a brief mention of test-driven
development.

Automated builds

Automated builds were much more popular this year with a
fair amount of people reporting that they had them at their companies (maybe
15%-20%).

The advantages cited for automated builds were:

You
know as soon as something breaks the build and it can be fixed right away.
Particularly important with multi-platform development.

Allows
you to keep a historical record of past builds.

Easier
to hook up automated testing after the build completes.

A common problem was trying to avoid breaking the build.
Some teams used tactics to discourage people from breaking the build such as
having to contribute to a common fund (which would be used for ice cream at the
end of the month), or wearing the “sombrero of shame.” Some teams had more
complex procedures to prevent breaking the build, such as doing full local
builds before committing the changes, or even submitting the changes automatically
to a machine that would do a tentative build and email back the programmer with
the results.

Everybody was using a custom build systems (in Python, Perl,
plain batch files, or even in .NET). One person was using Ant, an open source, Java-based automated
build system

There were different strategies for automated builds. Some
people built once every day, some twice a day (once at night, once at lunch),
some once every hour, and some as soon as one build was finished started
another one.