25 November 2007

A Decade of Ultima Online, Part 2

Since I have UO on the brain now, I think I'll list off some of my best memories both as a player and as a developer.

Player

My first character: a smithy/miner. This was back near launch in Oct 1997. Mining was tough because of player killers (PKs) and you always had more ore than you could carry, so you would have to set it on the ground and move it a tile at a time. Pack animals didn't exist yet so your precious ore was always in danger. Players actually paid me to fix their armor.

The first time I (successfully) went from a city to another city. When the game was very new the only map you had was a little mini-map in the game and the cloth map that came with it. Tools like UO Auto-Map didn't exist yet, so getting lost was the nature of exploring.

Dying to forest creatures. Rabbits and deer were merciless to new characters.

Sailing around the world. When I came back after a few years away, the first thing I did was buy a boat and sail all over. It provided hours of enjoyment exploring the world, with nothing to show except to be able to say I had been there.

Joining a guild. While I was wandering in the dangerous non-consentual-PVP world of Felucca, I met a nice guy named Alydar. We chatted for hours while killing creatures and discovered that we were both programmers. I had started writing some tools for UO and he was interested in learning the same things that I was discovering. He invited me to join his guild, the Defenders of Virtue. He became my mentor and later sent me on a quest to discover the Virtues in order to become a Knight within the guild.

Animal Taming. I tamed white wolves for hours to get my Animal Taming skill up, but it was all worth it the first time I tamed a dragon. Stealing the not-so-original name Trogdor, my dragon and I could solo Ancient Wyrms and Balrons.

Age of Shadows. With the help of my pal Alydar, I was able to obtain a house plot inside the brand new city of Luna in the new continent of Malas. I still own that house to this day and use it as a vendor location, allowing other players to set up vendors at my house. This expansion also launched with a critical bandwidth usage problem that eventually got me a job working on UO.

Developer

Fixing the bandwidth problem mentioned above in less than two weeks on the job. It was actually a very simple problem. The Age of Shadows expansion introduced real item tooltips to UO for the first time. Tooltips were always the same for everyone and had a global 'version'. Whenever something would change the tooltip was invalidated and the new version number would be sent out to everyone in range. This would cause all of the clients to request the new version. However, the problem was that tooltips were being invalidated whenever anyone new would come into range of the item. Instead of sending the current version to just the new in-range player, it would spam everyone in range when someone new came close. In busy areas the net traffic got ridiculous.

The Day the Servers (almost) Died. UO's player backup format is very prone to bloating the backup file with empty space. Someone made a change to the way player characters were stored that broke the compression utility that removed said empty space. Right before Christmas. Everyone was gone right before Christmas break started, and player backups on production servers were quickly approaching their 4GB limit, above which Bad ThingsTM happen. I had to make a very timely change to the compression utility and get the fix out to all the servers without dropping them by myself after having been on the job for about a month.

Fixing the 'insurance bug'. Targeting in UO is request/response. The server tells the client that it needs a target. The client then changes the cursor to a targeting cursor and sends a response back to the server when the player picks something. The server then fires a callback on the item that requested the target. The problem was that the item was determined from information in the target response, so the server was stupidly trusting the client in this respect. In most cases, the scripts that controlled targeting had their own "security" but it was up to the developer to script it in every case. In the case of insurance, the 'item' the callback fired on was the player requesting to insure/uninsure something and there was no security. Now the security of all request/response formats is ensured by the server at a global level.

Character Transfer system. I was pulled in to the character transfer team to wrap it up with my good friends Deathwish, AviStetto and MrTact. It was the last major feature that we did before Origin was dismantled and it was a lot of fun.

Moving to Redwood City. Although I miss Origin, moving to CA with EA has been one of the best things to happen to me, even though I really didn't enjoy working at EA nearly as much as Origin.

Fixing the 'chunk egg crash'. Each 8x8 tile section ("chunk") of the UO world has an invisible item called a 'chunk egg'. These are immovable (even by GMs) items that store all of the resources for the chunk (wood, metal, etc) that can be harvested by players. However, periodically these chunk eggs could end up inexplicably inside a moving crate in someone's house. The server would crash when the players would try to re-organize their house. The workaround was for a developer to go and remove everything from the moving crate, lock it down in their house and then delete the moving crate. Oddly and seemingly unrelated, people were also complaining about items disappearing from their houses. It turns out that the two problems were related. Object IDs in UO are 32-bit numbers that can be reused and the reference to the moving crate wasn't cleared when a player cleaned it out. This would leave a 'dangling reference' which was normally invalid. Sometimes the ID referenced would be reused to be a container in someone else's house or a chunk egg. Summoning the moving crate would pull that container from someone else's house or crash the server due to attempting to summon a chunk egg.

Samurai Empire expansion. I did most of the coding for getting in the new wearables. I also did the a lot of the scripting side of the new Bushido and Ninjitsu skills. I did a proof-of-concept for the roof tile system which was later turned over to an intern. I also did a proof-of-concept for a Unreal Tournament-style PvP battlefield system that was rejected (and continues to be a bit of sore spot with the point of view of management).

Unannounced game revamp. This isn't a 'great memory' so I don't know why I'm mentioning it. Yes, we were working on UO2 for the third time. We had a 3D client and were doing awesome things with breaking the skills into specialization and quest-based learning. The server technology was mostly staying the same, but we had finer-granularity movement actually working that allowed you to move within a tile. The scope of the project grew huge and we didn't have time to finish it nor wherewithal to cut part of it. Everything was scrapped and the idea eventually became Kingdom Reborn.

Mondain's Legacy expansion. Mostly support work here. I supported the rest of the engineering team and picked up a lot of the "smaller" tasks like the world collections (museum, zoo, etc) and parrots. I also worked on an asset replacement system that was supposed to help our artists be able to test out new content in the game engine faster. I also helped MrTact and EvilMantis design the guts of the Quest system. I left EA before this expansion went live.

Wombat. For some reason, I had great love for this proprietary scripting language. I took it under my wing and made some major improvements to it:

An integrated debugger that allowed breakpoints, stepping and printing out variables.

The ability to do negative numbers. Yeah. Really.

Floating point support.

Compiled binary support. Greatly reduced server start up times.

A simple optimizer.

Support for array and dictionary container types. Previously the only type was linked-list.

Global variable support (saved on memory usage)

Reference-counted containers. Saved time when large amounts of data were thrown around. Previously everything was copied. A lot.

7 comments:

An interesting trip through UO's history, from someone who was on both sides of the story through most of it.

This is the most we've heard about the cancelled third attempt at UO2, ever.

From what little I'd heard at the time, I thought it was just another attempt at UO3D, using an existing game engine that shall remain nameless. From what you're saying it sure sounds like it was going to be something bigger.

Was its cancellation the catalyst for the Mass UO Team Exodus of 2005? Vex, Oaks, Hanse, Fertbert, you, Leurocian, Toes, then later SunSword, and Binky?

Anyway, thanks for some insights into those years. I'm sorry I didn't get a chance to meet you personally at the 2004 Community Day, even though we were both there.

Holy cow, what memories. I got the chance to meet you several times and we've spoken many, many times through the years. You were always very friendly and informative, and I've been teased mercilessly through the years by my husband that I had some sort of secret crush on you!

The UO2.3 started at Origin and was originally using a custom engine, then we evaluated several different EA engines and eventually went with the Sims 2 Engine (which is actually Maxis' engine, called the Gonzo-Rizzo Framework [yeah, no idea]). It was big and bloated and I hated working on it.

We were toying around with being able to play several of your characters at the same time (on one account) and actually had a concept video made up for it. In the end we were going to call it "Kingdoms" (probably the 'Kingdom' in the Kingdom Reborn name) and it was going to have an RTS element where your house was actually a much larger estate and you had workers, fighters, etc.

In the end, Design went overboard, Production didn't hold the reigns very tightly, we had no budget to redo all of the Art as 3D models and Code had two different methodologies between the Origin folks and the EA folks that joined us. Friction abounded.

The cancellation of it certainly was a factor, but there were many things to consider: The Origin team didn't see eye-to-eye with our then -producer and people's housing subsidies from the TX->CA move started running out. NoCal is expensive :)

Personally, I think EA figured that we'd all leave when our 1- or 2- year contracts ran out. They were pretty much right. In fact, some theories say that they intended to drive us away.

Yeah, UO 2.3 originally started back in Austin, and actually I think the very first ever mention of it was a hint in Introducing STLdude back in July '03. After the move to EA, it started to get much more serious and we picked up team members from the UX:O team that needed stuff to do. The bulk of the team (which included new team members from EA) were working on UO2.3 while the rest of us did UO:SE.

You don't really find any mentions of the project by the dev team on the web, but I know that there were several hints dropped and coy answers given whenever people talked face-to-face.

Fortunately, nothing was ever officially announced. In the game biz, projects like this typically (and often) start up and die without anyone on the outside ever knowing. It's harder to maintain the illusion on a live game especially with a vocal team.

I'm sad that UO2.3 didn't come to fruition, but I'm not convinced that EA could've done it right.