Demigod: Day 1 Status Report

Frogboy

I broke my 56 hour day this morning and took a nap before coming back. One of the nice features of Stardock is that it has developers in the US and in Europe so we can, technically, work on things around the clock if necessary which is something we’re taking advantage right now.

So where are we at?

Today

The system works pretty well if you have a few thousand people online at once. The system works…less well if there are tens of thousands of people online at once. And if there are over 100,000 people, well, you get horrific results such as the game being incredibly unresponsive due to simple web service calls that were considered pretty benign during the beta that suddenly start to bring down firewalls and such due to the sheer massive number of calls that are being made.

Sadly, most of the ~120,000 connections are not customers but via warez. About 18,000 are legitimate. So anyway, we spent a lot of time today trying to isolate out the warez users from the legitimate users (it would require a lot of surgery to actually break them and even if we did, there’d be no friendly “ha ha pirate” message which would result in people just saying the game is buggy). Mind you, the game makes relatively few server calls, it’s just the sheer number of people.

Our stress tests had counted on having maybe 50,000 people playing at once at peak and that wouldn’t be reached for a few weeks by which time we would have slowly seen things becoming problematic. With Sins of a Solar Empire, the game was incredibly successful but its user base grew slowly and steadily over time. Sometimes on a peak time the server would start to get slow and we would adjust so that it would be better next time.

But here, when you’re getting that many connections at once, you’re no longer talking about dealing with the basics like having a good SAN and lots of redundant servers. You instead fall into all kinds of weird secondary issues that start to pop up like yesterday’s case where an older network card couldn’t handle all the packets (not bandwidth limited but # of tiny packets being sent out at once).

So during the day today, people couldn’t even logon and in some cases, the Demigod forums, which use one of the affected databases for some piddly thing were even down. Even getting the game running was a pain today because a simple HTTP call to see what the latest version would get hung leaving people looking at a black screen. Stuff of nightmares.

Before the game shipped, I wrote a scary email to our team saying how disastrous things would be and predicted doom for us and GPG if there were problems with multiplayer. At the time, my worry was about things like disconnects and CVP. It didn’t occur to me that we’d have near MMO user connections to throw in.

Tonight

Last night and all day today we’ve been working on multiple fronts.

First, we’ve been trying to shuttle off the warez users so that they’re not slamming our servers. Well, not slamming the same servers that legitimate customers are using. I’m feeling pretty confident that that is working better because there’s a lot more people in games with lots of people in them.

Second, we’ve developed a new connection system (pictured below). The connection stuff in Demigod is not part of Demigod but is rather part of Impulse Reactor which in turn is talking to the various connection servers we have set up to get player A to connect to player B and so on. It’s not done yet but amongst its new features is the ability for the host to eject players who just can’t connect.

This whole new connection service has been written in the past 48 hours.

Third, we’ve greatly expanded our capacity. I tell you, anyone who wants to know why we delayed Society (MMO RTS we are slowly working on) need look no further than how painful the first 24 hours since the official release of Demigod have been. That was done today and hopefully users have noticed an improvement.

We’re in the process of testing out this new update to put up tonight (less than 24 hours after the street date) that should largely fix this.

But the GameSpot review nailed us because of the connectivity with a 6.5. I’m pretty disappointed about that. [Begin Rant] First, I totally understand that connectivity is central to a game like this. I totally agree. But I think that should be weighted with what the average user who gets Demigod will experience and in reality, as annoying as this issue is, it’s not something that’s going to be an ongoing issue, it’s something that is likely to be taken care of in the next day or two. So this time next week, players will be happily playing but GameSpot’s review will live on. I suppose the solution going forward isn’t to send out review copies until the game is released and we make sure there’s no network issues. Afterall, in a week from now, the guy buying it from retail will have a great experience. [End Rant]

The IGN and 1Up reviews are coming up. So we could be looking at a lot of reviews like that if every reviewer decides to take the same approach Gamespot did which I hope they don’t (luckily for every other on-line centric game that’s had a rough first day or two most reviewers haven’t taken Gamespot’s approach).

Video

We took a video about some of the stuff we’re working on and a little quick tour of Stardock. You can see it here:

I was going to tell you to stop complaining about the gamespot review because you deserved it for having unplayable multiplayer on day 1 but then I changed my mind. I remember that Halo 3 day 1 was unplayable online, the same for Battlefield bad company and I am sure many other games because of the number of connections. Why did these games not face the same wrath in review scores as Demigod on gamespot? Probably because they were played on private servers for the review Really unfair when you think about it....

Exile 3 was a great game. I actually have the source code to it at home (long story). I really got into Blades of Exile, too, but that was about the time I got into programming and also had to concentrate on schoolwork.

I was going to tell you to stop complaining about the gamespot review because you deserved it for having unplayable multiplayer on day 1 but then I changed my mind. I remember that Halo 3 day 1 was unplayable online, the same for Battlefield bad company and I am sure many other games because of the number of connections. Why did these games not face the same wrath in review scores as Demigod on gamespot? Probably because they were played on private servers for the review Really unfair when you think about it....

Hahahahaha I remembered that as well. The Halo 3 servers were overloaded, you had to basically wait 2 hours or so, hopefully to get in the server. My reaction was the same as many are having now that I almost gave up on Halo3 and send it back.

The differense is that Halo is Halo and people will always look past Halo's flaw. Demigod, you can say is for many something new. The gamespot reviewer is not familiar with the style and the game has no tutorial, so he despite the game being very simplistic it will still take a while for him to know what he is actually playing.

Is that windows 7 I see? Does that mean you're accepting crash reports from it? Cause I am switching to vista when I play to avoid some crashes. Obviously not something that would demand your immediate attention, but... thought I might ask.

Also, I would like to salute you because we all know the happy smile of someone who's been awake way more than he ought to be the last few days.

Exile 3 was a great game. I actually have the source code to it at home (long story). I really got into Blades of Exile, too, but that was about the time I got into programming and also had to concentrate on schoolwork.

Blades was OK...I just didn't like the fact that it was only 3 short adventures. Don't get me wrong the three missions we're awesome and really well calibrated but it no where nears equals the EPIC SCALE of Exile 3 (PROMOTION FTW). The fact you can create your own scenarios was fun but Exile 3 was EXCELLENT!

Magic Barrier + Quick Fire Combo FTW

or..my personnal favourite Simulacrum of your favourite enemy +_+.

So if you played Exile 3...then I suspect you also played games like Geneforge, Nethergate...etc?

I realize this game doesn't use DRM, but doesn't it do a CD Key check when you log into online? That seems like a no-brainer when it comes to protection against the sort of problems you're having. Sure, people can make a CD Key generator or something eventually, but I'm sure having a unique CD Key check would at least prevent the 80k plus pirates you have playing the game online days after release.

I think it's a mistake for demigod to connect to anything before the menu screen, even if it was to check for updates. All this should be done after the person clicks on multiplayer. Well it's too late to change now with the warez dvds out there.

So 18,000 legitimate users which means 18,000 first days sales and preorder sales. Not too bad. I wonder how many are retail vs online. I think you'll get #1 NPD pc sales for this week.

Exile 3 was a great game. I actually have the source code to it at home (long story). I really got into Blades of Exile, too, but that was about the time I got into programming and also had to concentrate on schoolwork.

Crikey! I have Exile 3 on my desktop right now! I'm still playing through Exile 1 which I never finished but my favorite memory was using simulacrum to spawm vahnatai lords. Took me over 10 minutes of saving and reloading to get the capture but I eventually did it !

Sadly, most of the ~120,000 connections are not customers but via warez. About 18,000 are legitimate. So anyway, we spent a lot of time today trying to isolate out the warez users from the legitimate users (it would require a lot of surgery to actually break them and even if we did, there’d be no friendly “ha ha pirate” message which would result in people just saying the game is buggy). Mind you, the game makes relatively few server calls, it’s just the sheer number of people. Our stress tests had counted on having maybe 50,000 people playing at once at peak and that wouldn’t be reached for a few weeks by which time we would have slowly seen things becoming problematic. With Sins of a Solar Empire, the game was incredibly successful but its user base grew slowly and steadily over time. Sometimes on a peak time the server would start to get slow and we would adjust so that it would be better next time.

I like how in the end of the video... you disconnect from the game and AIs take over for the other three players.

At least you kept it in the video, and I'm glad you're being so open about the issues. I'm really upset with gamespot, and I hope that enough people ask for the game to be re-reviewed once the new servers are up. Fantasic game, I'll be addicted for ages.

The balance seems a little wonky in some cases, and I'm just talking about single-player. Try playing as the Unclean Beast, with allies Rook and Regulus, versus Sedna, The Oak, and Lord Erebus. With normal AI, you'll still probably lose. It seems like Generals are roughly equivelant to Assassins in one-on-one strength, add Minions and they're drastically stronger.

Am I right in assuming that at some point in development they considered making Generals and Assassins just as strong, but making Generals more about buffs, and let everybody have Minions? Because it kind of seems like that now, except not everybody gets minions.