Who

Pharaoh Speaks

Continuing our focus on A Tale in the Desert, here is an interview with the
Pharaoh of this particular Egypt, Andrew (“Teppy”) Tepper, the game’s designer
and President of eGenesis, its publisher.

Typically, interviews like this focus on the gameplay design or community
development. I’m less interested in that (in part because I think you can
often explore those aspects by simply playing the game). Instead, my aim for
this interview was to focus specifically on the actual technical development
challenges in creating a complex online game with a small team, something that
few of us have done. If you’re want details about the game itself, you might
want to look at my recent
review. I reached Andrew
at home, just after he had finished up work for the evening. peterb: A
Tale in the Desert has been released on multiple platforms (Windows and
Linux) with more on the way. How early did you decide to go multi-platform?
Did it impact development?

Andrew Tepper: We did it immediately, so right from the start we did things
like separating the rendering code so that in theory there was one file that
needed to be changed to go to a different platform, or to a different graphics
API (openGL to DirectX, for example) and then a single #ifdef to change to a
system that was different byte order. In practice, it worked pretty well.
There were a few bugs, but it’s been pretty easy. For the most part there were
just a half dozen or so bugs where we made assumptions about byte order…the
whole idea was that stuff that hooks in to the operating system is isolated.
It was pretty clean. The Linux port took about 3 weeks. And the Mac port has
been underway for about 5 weeks.

How big is the team, and what development environment are you using?

We have 2 full time coders, and 3 full time graphics artists. We use Visual C
(a slightly older version) on the Windows side and just gcc in Linux and, I
don’t even know what we use in Mac. gcc, I think?

How do you track bugs?

We just started using bugzilla for script code kinda bugs. With two
programmers, tracking bugs isn’t a big deal – if there’s a bug, you just
scribble down a note and then fix the bug.

How about version control?

We use cvs. That’s been helpful. Sometimes, i’ll go off on a project kind of
experimentally, or my partner will, and 20% of the time it won’t work and
we’ll just throw it all away.

What’s the single hardest bug you’ve encountered in developing A Tale in the Desert? How did you fix it?

The hardest bug was probably – well, I have to explain this. Our execution
model is that the client has synchronous and an asynchronous, or we sometimes
say a predictive, model of the game world. The server has a master world
model, distributed over many physical CPUs, and a predictive world model for
each client. The synchronous model is the subset of the master model that a
particular client sees. The invariant is that the server model for a given
client, and the synchronous model on a client, always stay in lockstep. So for
a given time, the synchronous models must match exactly. So we had one or
probably a few bugs where for some reason the synchronous model on the client
got out of sync with the model on the server. So that required changing the
memory manager and rewriting a whole bunch of subsystems in the server so that
we were able to log every byte going in to one of the cpus and every byte
coming out. So we wrote a replay facility which we still use which simulates
the bytes going in to a server. The solution to finding the bug was replaying
on both the client and server kind of at the same time and figuring out where
the two synchronous models diverged. It turned out to be something like
uninitialized memory. It was not a very glorious bug. But because our
execution model is pretty unique, it took some custom tools.

The replay tool is so handy that now when we get a server crash, 90-plus
percent of the time, the solution is you take the replay log, rerun it in a
debugger, the server crashes in the same place, and then you say “aha!” For
something as complicated as a massively parallel system as this – boy, I
can’t imagine a better way to do it. The server itself, by the way, is a
single-threaded application. the server is designed to handle a million
pseudo-threads at once – every building in the game gets its own pseudo-
thread. On the client we use three threads.

A pseudo-thread isn’t actually sharing memory with other threads, so there aren’t any tricky concurrency issues?

Yeah, it’s not an OS-level construct. It’s just an internal thing. That’s just
what we call it.

Is there any feature you implemented that you wish, from a timeline standpoint, that you wouldn’t have implemented? Something that you look back on and say “Wow, that really blew the schedule for not a lot of benefit?”

Josh [Yelon] would have different answers than me. For instance, I bet Josh
would say bump mapping was more effort than it was worth, but I would say just
the opposite. For me, there were a bunch of kind of specialized things we
added, not big architectural things that were a pain in the ass.

Are any of those still in the codebase?

That still remain in the code? No, because usually when something like that
happens we rip it out. For instance we had a conversation system built in, so
if you had NPCs in the game you could talk to them. We didn’t actually start
out to write A Tale in the Desert. We were writing a system for creating
massively multiplayer RPGs, so you had a character in the middle of the
screen, and there are other characters running around, and objects to interact
with. We knew we were a small company, so we spent a long time writing
sophisticated tools to let us quickly develop content, because we knew we’d
never be able to dump millions of dollars into developing the content. We
spent 2 and 1⁄2 years not writing a game, just developing a toolset. If you
ask me what part of the toolset is the biggest thing, it’s being able to
change the code on the fly without kicking people off. That’s as big a change
to developing as going from punch cards to interactive editing. Instead of the
big testing cycle, you can put your code out as soon as its done, and the
really bleeding edge players will run into bugs, and you can fix them in
minutes. It’s a giant leap forward.

A lot of the players in A Tale in the Desert seem fairly sophisticated, and they’re very vocal. So it’s clear that a lot of the content has been developed with these more sophisticated players in mind. Certainly, most of them have played other MMORPGs before. So how do you gauge how the game is received by Joe Average who has never played an online game before? Do you capture those guys, or do they wander away confused?

It’s very hard, because they’re kinda shy. They don’t talk as much as the MMO
veterans. I’ve done things like kinda gone incognito and played the start of
the game, to try to find out. But you don’t know if someone’s dropping out
because their connection went down, or they don’t like it and they’re
embarassed to say it, or they feel stupid. I don’t know. It’s very hard.

You’re extremely accessible to your users; players and GMs have your cell phone number, and often lobby for features vocally (even outside of the in-game ‘election’ protocol). Is there a downside to giving such easy access to your users? Has pressure for a particular feature ever blown the schedule?

In 18 months I’ve only had 10 inappropriate calls. As for the questions on the
schedule, players will always ask for features, but ultimately I’m in control.

I’ve tried to stick to questions about development, but do you have any comments on the game design in terms of the choice to go with economic model vs. monster killing?

I consider this an adventure game. I think Tale is kinda the MMO side of
adventure games, where the puzzles are all social puzzles… Think about the
Test of the Covered Cartouche, for example. [Ed: In this test, participate in
successive rounds of creating a building in secret, all throughout Egypt. At
the end of each round, the player whose building is the smallest is
eliminated. Since creating larger buildings is quite expensive (which can
spoil your chances for future rounds), the challenge is to spend as little as
possible on your building while not spending the least.] The puzzles in A
Tale in the Desert are all like the old-style adventure games, but with the
smartest NPCs you ever came across. If you think about the game, about
unlocking technologies, it’s an adventure game. We add some stats so it’s RPG-
like, but I think Tale is much more an adventure game then everquest.

To me the boring stuff is where there’s just a single plodding path – find a
sheep, grow onions, feed the sheep – but, y’know, the whole game can’t be a
chess match. There has to be some mindless stuff, then some thinking stuff.
But the coolest thing to me is where I make up a challenge where I don’t know
what the best solution is. A few times, I’ve screwed up and the players
haven’t stumbled on a solution. For example, the Test of the Seven Phoenix in
_ Tale 1_ _ [Ed: The Test of the Seven Phoenix involves building expensive
statues of a specific type in named locations within a one hour period.
“Waypoint” travel during the test is forbidden. Which statues need to be built
in which locations is a secret before the test begins.]_ Either I just made it
too hard, or there is no good solution, but it hasn’t been a great success.

If you could work on any game next, and it couldn’t be an MMO, what would it be?

If it couldn’t be an MMO, I like physics games. But what those have in common
with Tale is that they’re both sorts of sandboxes where the designer ahead
of time doesn’t know what the best way to solve it is.