Author: benadams

Running a massively multiplayer game is difficult. You need good up-time, but also need to continuously update the game and add new content. The server is busy. We’ve had over 900 million page views in our three years since launch and we currently average 800 page views a minute with over 1.2 million a day – and that’s only active pages being viewed; not including game notifications, passive chat, or any ingame popups. We’ve just done our 815th releaseto live; I imagine even our players don’t know we release something on average every 1.4 days. Some are minor bug fixes, others full UI refreshes, of which we have done 3:

We are available on many platforms, with an HTML5 website game, a Chrome Webstore App, Firefox App, a Facebook game, and Windows 8 App; all of which run on the same single sharded world, and all of which need to be kept up to date and in sync. We have to ensure with each change everything keeps working on all the different platforms, different browsers and different devices – Illyriad even works on a Kindle Touch! We run a full Microsoft stack, have changed CDN 4 times, and during these three years we’ve had 1 day of cumulative downtime, mostly for security patches, although our longest period was for completely moving hosting providers to Hivelocity for our dedicated servers and Windows Azure for our cloud based needs and CDN – lots of live data to move. All through this the players keep playing. Blissfully unaware of all the changes going on. Sending their troops on 58 million combat missions, their traders on 41 million missions and building their 33 million buildings, all the while chatting 213 million words together. Each update is transparent and doesn’t interrupt the players’ play time for patching. People live busy lives, and they have chosen to give their precious time to play the game and that must be respected. They don’t want to spend that time waiting for downloads, patching and updates. They just want to play. Maintaining a uninterrupted service, with huge concurrency and transparent updates is hard, but its worth it!

Winter has come. The days have grown short and the nights long.

High above the lands, the atmosphere is almost serene. The wars and daily conflict seem far away; but even here the busy activity and effects of players can be clearly seen!

At night the lights of the settlements and sovereignty glow bright:

Illyriad at Night (click to zoom)

Flying into central Elgea you can see the ring of settlements formed by new players:

Central Elgea - New Player Ring (click to zoom)

Looking to the south-west, the outline of the ocean is visible as well as the smaller settled islands. Also some almost constellation-like joined up lights can be picked out:

South-West Elgea (click to zoom)

Turning up the contrast and zooming in shows some interesting player land claims. All over the map an Alliance is claiming road like structures:

Road building?

While another owns land in a shape that resembles a person with a hat and a cape?

Person with a hat and a cape?

Reviewing the bill of land ownership lodged at King Sigurd’s Council Of Illyria; indeed this land is claimed by the alliances: Fairy Road Authority and Silly Hats respectively. Creative expansion plans!

Academic Aside: An interesting analysis, when just considering player towns (excluding sovereignty), their arrangement resembles a Poisson disk sample. 99% of these towns outside the “new player ring” are consciously placed by thousands of individual players acting independently.

Lilli Thompson from Google asked us how we were doing texture decompression in the pixel shaders and what algorithm we were using. We thought we would share our answer…

Texture compression was a bit of journey – as no one at Illyriad had ever implemented anything in 3d before; to us texture compression was mostly a tick box on a graphics card.

It started when we found out our 90MB of jpegs expanded to 2GB of on-board memory and we were worried we’d made a terrible mistake, as this was certainly beyond the possibilities of low-end hardware! Half of it this was due to Three.js keeping a reference to the image and the image also being copied to the GPU process – so essentially the required texture memory doubled.

Dropping the reference Three.js held after the texture was bound to WebGL resolved this. I’m not sure how this will play out with context lost events – as I assume we will have lost the texture at that point – but local caching in the file system and reloading may help with recreating them at speed.

With 1GB of memory remaining we were faced with three choices – either deciding what were were trying to do wasn’t possible; reducing the texture sizes and losing fidelity or trying to implement texture de-compression in the shader. Naturally we opted for the last.

We were originally planning to use 24bit S3TC/DX1; however this proved overly complex in the time we had available as the pixel shaders have no integer masking or bitshifts and everything needs to be worked in floats. The wonders we could unleash with binary operators and type casting (not conversion) – but I digress…

In the end we compromised on 256 colour pallettized textures (using AMD’s The Compressonator to generate P8 .DDS textures). This reduced the texture to one byte per pixel – not as small or high colour as DX1 – but already 4 times smaller than our original uncompressed RGBA textures.

It took a while to divine the file format; which we load via XMLHttpRequest into an arraybuffer. The files have 128 bytes of header which we ignore, followed by the 256×4 byte palette which we load into a lookup table texture RGBA. The rest we load into a Luminance texture. Both textures need to use NearestFilter sampling and not use mipmapping to be interpreted sensibly.

We have created our own compressed texture loaders – the colour texture loader looks a little like this:

When we first did the decompression in the pixel shader, it was very blocky as we had turned off filtering to read the correct values from the texture. To get around this we had to add our own bilinearSample function to do the blending for us. In this function it uses the diffuse texture with the colour look up table and using the texture size and texture pixel interval samples the surrounding pixels. The other gotcha is that the lookup texture is in BGRA format so the colours need to be swizzeled. This makes that portion of the shader look like this:

This performs fairly well; and certainly better than when your computer feels some virtual memory is required because you are using too much! However, I’m sure on-board graphics card decompression should be swifter and hopefully open up the more complex S3TC/DX1-5 compression formats.

There is a major downside however with decompressing this way in the pixel shader. You have to turn off mipmapping! Not only does turning off mipmapping cause a performance hit as you always have to read the full-size textures – but more importantly it doesn’t look good. In fact in the demo – we had to use full-size textures for the grass so we could apply mipmapping as otherwise in the distance it was a wall of static!

Unfortunately, as far as I’m aware, WebGL while you can create mipmaps with generateMipmap – you can’t supply your own. Again, real compressed textures should help here.

EDIT: Benoit Jacob has pointed out this is possible by passing a non-zero ‘level’ parameter to texImage2D – one to look into.

What can WebGL do? Can it do what we want? We were wondering.. and so decided to put it to the test…

To test the upper bounds of WebGL we put together a rough and ready demo [caution – it’s bandwidth hungry]. It’s very rough, not optimized and currently only runs on Chrome [working in all browsers that support WebGL is our priority]; but that’s kind of the point – its a technology tester to ensure we weren’t making a mistake.

The results speak of themselves – it definitely proves itself! Sure it needs a bit more polish, but we are now confident that the actual in-game libraries we are building have a lot of head room to use. Below are a couple screenshots of the town during the day:

And another at night:

Of course there were many trials along the way and things that didn’t quite work as we’d planned as can seen below:

We learnt the importance of GPU compressed textures and had to write a pixel shader decompressor of our own, as WebGL doesn’t currently support them natively – but with a cost. The loss of mip-maping this causes it can clearly be seen; and we will have to work around this if they are not supported soon.

Overall we are very pleased with the result, which you can check out here. Remember to press space to unlock your mouse to look around – if you aren’t fond of reading on-screen instructions 😉

Naturally this is just a taster of what we have waiting in the wings. We’ll look to provide some follow-up blog posts about the techniques and tools being used in this early experiment including:

We thought we’d up the game on the last demo and show a bit of lighting and animation.

Again this is purely HTML5 using the WebGL canvas and JavaScript; no plugins were used. We use mrdoob’s excellent Three.js library:

[youtube=https://www.youtube.com/watch?v=0bZmt3yQzmE]

We’ve included the full browser window in this recording to show that it is in fact running in a Chrome browser window. While it happily runs at 1080p, we’ve had to record at a much lower resolution as the screen capture utility slows everything down… Alas.

If you create canvas elements dynamically either for off-screen composition or sprite manipulation, when this is in an animation loop running hundreds of times a second it can start to leak memory.

I’m not entirely sure why this is; but it’s quite widespread in many browsers including Webkit. Perhaps the animation causes the browser to register as continually active which prevents the garbage collector from running – as it will clean it up if you stop the animation and change DOM elements e.g. from an Ajax update. But that’s just wild speculation…

What we are interested in is how to fix it, and also achieve some performance gains from not continuously creating DOM elements.

The trick is to create a function which caches these canvas elements and reuses them. The easiest way is to have a create canvas function that handles all this for you. To keep a handle on the different canvases in use in Illyriad, we use a named canvas element to ensure the correct one is returned. You can call them whatever is relevant:

I’ll do another post on some of the wonderful composition operations you can do using canvas; when this technique will become very important.

If you are updating your page using Ajax and programmatically adding and removing the display of canvas elements you will want to clear the canvas cache when you change page by adding a cache clean up to your ajax page unload function.

Animating HTML5 Canvas be it 2d or WebGl can put a strain on the end user’s browser.

Trying to draw more animation frames than the browser can handle can lock it up or cause it to terminate your script. Also if the user is viewing a different tab or is in another window; burning the user’s CPU on animation they don’t even see is just squandering their resources.

Luckily, much like debouncing and throttling for user input and ajax calls, there is something we can do. HTML5 provides a neat solution for animation. The requestAnimationFrame function allows you to put in a call-back which gets executed when the browser is ready to draw another frame. This allows the browser to throttle the animation based on the current CPU load and whether the canvas is on screen or not, resulting in a far better performance and a much lower resource usage.

However, some older browsers don’t support requestAnimationFrame, but we can use the following code to detect if they do, and if they don’t, run though some earlier implementations and finally a fall-back method to ensure we can still use the function:

To use this function we need to create an animation loop. Also if we separate our animated elements from our static elements (e.g. background) into different canvases, we only need to redraw the static elements when they change which is far less frequently.

For this we create two functions renderStatic and renderAnim; which are outside the scope of this article, but there are plenty of canvas examples to draw on. When the static elements require redrawing we just set the requiresRedraw variable to true.

Putting this altogether we end up with a animation loop looking similar to the following:

[code lang=”javascript” title=”Setting up animation callback”]
var isAnimating = false; // Is animation on or off?
var animateRunning = false; // Are we in the animation loop?
var requiresRedraw = true; // Do the static elements need to be redrawn?