Using Linux at Left Field Productions

Left Field Productions, Inc. is a game
developer in Westlake Village, California, which concentrates on
the console market, specifically the Nintendo N64 and Gameboy
systems. Their web page is at
http://www.left.com/. I
joined Left Field in September of 1998 to do Gameboy programming.
Since that time, Linux has been used on the company-wide network
offering various services, all running on a lowly Pentium 90 with
16MB of RAM and a single 6.4GB hard drive. The machine is known by
all as “the Linux box”.

The company uses 10BaseT for all networking. The Linux box
also provides the main gateway to the Internet. It is running the
named name server, and the Apache
web server for company internal web pages. The Linux box has dual
mechanisms for connecting to the Internet: one through ISDN, the
other through a conventional 56K modem. ISDN is the preferred
method, but during occasional outages, it is necessary to downgrade
to the modem.

The Linux box provides public Samba services, making storage
area available for file backups and exchange among the artists and
programmers. The fetchpop program
was used to pull mail off our ISP and forward it to individual
accounts on the Linux box. When some problems with fetchpop arose,
mostly due to the lack of a timeout feature during mail retrieval,
I replaced it with fetchmail,
which we are still using. Periodically, employees' preferred
desktop mail client connects to the Linux box to retrieve their
mail and send any outgoing mail off to the Internet. Sendmail also
provides outgoing mail service.

The Linux box has enjoyed up times of more than 60 days. The
few times it has been rebooted were due to planned outages for
power utility service, and upgrades such as adding a new hard
drive. As far as I know, there has never been a system crash. I
think it is sometimes rebooted unnecessarily by some of the other
Linux-literate employees, as it can be more convenient than
specifically restarting a single task after modifying a
configuration file. Employees with lots of DOS/Windows experience
have a kind of “when in doubt, reboot” philosophy.

When I first started at Left Field, we used a free
assembler/linker combination for generating the code for the
Gameboy ROM images, and we used commercial painting and graphic
arts programs for art generation. The tools for converting and
compressing graphics files were all developed in-house. We used
Win32-based machines for everything. On my Pentium II 400 machine,
a complete rebuild, where all source files are assembled and then
linked, would take from 30 seconds to 2 minutes, depending on the
project. We were working on a basketball game called Kobe
Bryant 3 on 3 and a Disney title called Beauty
and the Beast, A Boardgame Adventure.

Once the ROM image was generated, we could download it to the
Nintendo debugger system for testing, or more frequently, we would
run it on a Gameboy emulator. The emulator was very usable, but it
had quite a personality and was temperamental at best. I found if I
exited the emulator and reloaded it a few times, it would
invariably crash the system. Also, it was unusable for testing the
sound aspects of the games—the quality of its sound emulation was
painful to hear. The emulator was actually a DOS program working
with an extender, and knew nothing about the Windows 95 windowing
environment. I think legacy is the most likely cause for its
instability.

After finishing up these first two projects, I found myself
with some free time before the next project began in earnest. I had
long been thinking of writing our own emulator because of my
dissatisfaction with the emulator we were using—the frequent
crashes and inability to get the source in order to repair it
ourselves was quite frustrating. So, within a couple of months of
starting at Left Field, I began some minor work writing an emulator
for the Gameboy CPU. All my programming was done in C. I installed
the public-domain DJGPP compiler and used it under DOS. In fact, I
was able to significantly speed up build times by using the MAKE
utility from DJGPP, replacing the Watcom MAKE we had been
using.

Unfortunately, the demands of the project caused me to stop
work on the emulator before I could even test it. Then, about a
year later, I had some free time again and was able to get back to
the code. This time, I chose to move away from Win32/DOS and switch
to gcc under Linux—a move that
made the programming infinitely more enjoyable. Surprisingly, there
were only a few mistakes in the emulation code, and in a short
time, I had the CPU “working”. It appeared to be behaving
correctly. The next step was to emulate the video hardware of the
Gameboy. For display output, I chose to use the SDL library, which
is a multi-platform gaming library. One of the supported platforms
is Win32, so the benefit there was that any code I wrote could be
used by other employees who were still running Windows. I was using
SDL under an X Window System environment. After some solid work, I
had the video emulation working quite well, and it was a joy to see
ROM images actually work.

Finally, I had to add the sound-emulation code. This proved
to be the easiest task, and after a short time, the emulator was
producing quite accurate and acceptable sound, again using the SDL
library. With a simple recompile using a cross compiler, an .EXE
executable could be built. The emulator worked under Win32 as well.
There were a few quirks related to sound under Windows 95 that had
to be worked out. Windows proved incapable of servicing the audio
interrupts at the 64Hz rate I had been using without problems under
Linux. I had to compromise and lower the rate to 32Hz so Windows
could keep up. I never determined whether the problem was in SDL's
Win32 code or in Windows itself.

The assembler/linker we had been using offered a version for
Linux, but I wasn't happy with it. The Linux source wasn't as up to
date as the DOS version—the two versions were based off different
source trees, and it was clear the DOS version had priority. My
options were to use the older Linux version or port the DOS code to
Linux. I chose instead to abandon the assembler and write my own.
Using core code that originated from my own ACC C-like compiler, I
managed to create an assembler with a syntax similar enough to the
assembler we had been using. I took the opportunity to make changes
in syntax when it was convenient. I knew how the assembler was
going to be used, and some features weren't important, so I never
implemented them.

In the end, my own assembler reported lines-per-minute
assembly rates over 30 times faster than the old assembler. With
the small source files, assembling each was practically
instantaneous. The next part I needed was the linker. Again, I
began with the ACC code and modified it to suit. Linking was also
much faster than before.

Now, to test the assembler/linker combination, I took our
game source trees and made the necessary syntax modifications in
the source files. I used the Beauty and the
Beast code, and spent about four hours going through all
the files to get something without linker errors. Naturally, the
resulting ROM image didn't work, but after a day of hunting for
bugs in all parts of the system, I got a ROM image that actually
came up on the emulator looking like the real game—very
encouraging.

In a project like this, debugging problems can be tricky.
When no single part has really been tested, a bug can be anywhere;
thus, I found myself frequently hunting in the wrong places for
bugs. Sometimes, I was surprised to find the assembler actually did
something right, and the emulator was to blame—and vice
versa.

In testing the ROM images with the emulator, it became
apparent I needed some debugging functions built into the emulator.
Even before beginning work on the assembler, I had added
disassembly capability to the emulator. Doing so had been very
helpful in finding bugs in the emulator. After I had the assembler
working, I added some nice features like symbolic debugging, break
points, expression evaluation, memory viewing and instruction
execution history. For text display and entry, I added a scrollback
buffer, name completion and
tcsh-style line editing.

Bugs became more and more rare, and were easier and easier to
find. It was clear the new system was completely viable for
developing, and an in-house suite of tools offered very strong
advantages that we would never get by using outside software. For
example, any desired feature could be added easily, since the
source was ours. For portability, I had written everything in
standard C. One thing I kept in mind was that at any moment I might
have to retreat from Linux and switch back to DOS, and I wanted the
tools to work there as well. The assembler and linker compiled
perfectly with DJGPP.

I was pleased to note build times were reduced to almost
nothing. A complete rebuild that had taken 30 seconds before took
three seconds now, on the same machine. It must be noted that those
times reflect two different operating systems as well as two
different assembler/linker pairs, so an actual breakdown of how the
speedup was occurring can't be made. I never bothered to do a
detailed analysis; I was happy just to be using the new
system.

On the other hand, my emulator, written in pure C, placed
significantly more demands on the CPU than the DOS emulator we had
been using. The author probably had hand-coded x86 code sprinkled
liberally throughout. I also assumed this was the source of most of
the crashes caused by that emulator.

To complete the Linux Gameboy development environment, I had
to port the various tools used for converting graphics files and
dealing with data files in general. Some I had created myself, and
these ported almost without modification because I had used DJGPP
as the C compiler. The only change required was related to the DOS
custom of having CR/LF (carriage return/line feed) as the end of a
line, rather than the UNIX-style LF only. DJGPP header files define
the flag O_BINARY which must be used when
opening a file, to specify that CR/LF should not be converted.
Under Linux, O_BINARY is not defined, so
compiler errors would result. The solution was to place the three
lines

#ifndef O_BINARY
#define O_BINARY 0
#endif

early in the source files, so the source would work without
change under both DJGPP and gcc/Linux.

The main graphics manipulation tools had been written by
someone else at the company, and I had to make more extensive
changes to get them to compile under gcc/Linux. There was the
O_BINARY problem as before, but in addition,
some of the programs did wild-card expansion in the program itself.
The DOS shell doesn't do wild-card expansion, so DOS programs must
provide that service for themselves. Unfortunately, the functions
for performing this were nonstandard C and so wouldn't carry over.
In the end, I hacked out those sections of the code and relied on
standard UNIX shell wild-card expansion to do the job. Also, the
header file “windows.h” was included frequently, and that
dependence had to be removed as well as the structures and system
calls in the code that required it.

Another problem which came up is the DOS/Win32 standard
practice of file names being case insensitive. Under UNIX, case is
significant, so errors popped up in source files where a file such
as “Elmer.h” was being requested, when the file in the directory
was actually elmer.h (or even worse, ELMER.H). Also throughout the
code, programs would create an output file with an extension and
the extension would be in upper case, so I'd generate files with
names such as “cpaused.CHR”. To me, this was jarring and
unattractive, so I modified all the files to produce lower case
extensions. This required another round of changes when the wrong
file name was being referred to, and error messages were
appearing.

In the end, I managed to mirror on Linux every tool we had
under DOS/Win32. Developing code under Linux was vastly more
enjoyable than under DOS/Win32, mostly because it was faster and
more stable. There is also something very rewarding about using
your own tools in any work. In total, from start to finish, the
time to bring everything up on Linux, as well as writing the new
emulator, took about three to four weeks.

Shortly after the next project began taking form, one of the
artists asked if he could do builds himself, as he wanted to tinker
with animation frames and see how they looked in actual use. Rather
than go to the trouble of getting all the tools set up on his
machine, as a quick solution I copied the source tree to a
directory path that my machine had been making public with Samba.
The directory /d on my machine appeared as //dave/d on the network.
I set up a simple script to check for the existence of the ROM
image file. If the file wasn't there, it would do a rebuild. So, to
get a new ROM image, the artist would copy over the modified data
files, then delete the ROM image and wait a moment for the new
build to appear magically. With Linux's excellent disk caching, I
was usually unaware of when this happened.

One final problem caused me some worry. I was currently the
only person actually using Linux as a development platform. What
would happen if one of the other programmers wanted to contribute
to the project? I didn't want to force my Linux preference onto
anyone else, so I had to be able to deal with this. Already, all
our tools had equivalent Linux/DOS versions, so that wasn't a
problem. The problem would be from multiple programmers merging
code changes to the same source tree. Under Win32, we had been
using Microsoft Visual Source Safe for that. Although I suspected
there must be a Linux client for talking to Source Safe, I never
looked. Instead, I studied how to use
CVS (Concurrent Versions System),
the traditional source-code revision-control system used in a vast
array of open-source projects. After experimenting with CVS and
learning enough about its quirks, I found it performed at least as
well as Source Safe. It conveniently handles the end-of-line
problem by storing the source files in the repository in UNIX
format and adding/removing the CR as needed when dealing with a
Win32/DOS client. I'm using CVS on the command line, and I prefer
that to mouse clicks; CVS seems faster at updating the modified
files.

Even though I'm the only programmer on the current project, I
am checking my code changes into the network CVS server, just to
get into the habit of doing so, as well as to provide an additional
level of backup and modification history. As you might have
guessed, the CVS server is running on the Linux box.

David Ashley
(dash@xdr.com) has been working with
computers and electronics for over 24 years. He has written several
free games for Linux, including Scavenger, XBomber and Sdlshanghai.
He pays his bills by working at Left Field Productions, Inc. of
Westlake Village, California, doing game programming for Nintendo
consoles. Having recently become a father, he is finding himself
with less and less time for his addiction—programming in Linux. He
tells us, “All opinions expressed in this article are my own and
do not represent the opinions or official policy of Left Field
Productions, Inc.”

Trending Topics

Webinar: 8 Signs You’re Beyond Cron

Scheduling Crontabs With an Enterprise Scheduler
11am CDT, April 29th

Join Linux Journal and Pat Cameron, Director of Automation Technology at HelpSystems, as they discuss the eight primary advantages of moving beyond cron job scheduling. In this webinar, you’ll learn about integrating cron with an enterprise scheduler.