Saturday, 23 February 2013

The latest Awesomenauts patch increased the framerate a lot for players with older videocards, especially during fierce battles. We managed to optimise our special effects without making them look noticeably different. Today I would like to explain how we did that!

Before I dive into the details, let me first give some background. In a 2D game like Awesomenauts, most objects are just a square with a texture on it. The texture contains an image, and you only see that image, not the entire square. However, the videocard renders the entire square. So from a performance perspective, it doesn't matter how much of the texture is actually visible. The entire square is rendered and the every pixel uses performance!

Our artists know this, so they try to crop the image to have as few transparent pixels as possible (without changing the actual looks of the end result, of course).

Since objects in Awesomenauts contain so many (partially) transparent pixels, we cannot easily detect whether an object is hidden entirely behind another object or not. So we always render every object that is on the screen, from back to front, regardless of whether an object is actually visible or not. All objects are rendered on top of each other until the image is complete.

This introduces a big performance issue: massive overdraw. Overdraw is when a single pixel needs to be drawn several times each frame. On 1920x1080, the screen contains over 2 million pixels. If we draw every pixel 10 times, that is already 20 million pixels per frame. Now do that 60 times per second, and we are rendering a whopping 1.2 billion pixels per second. That's a mindbogglingly large number of pixels per frame!

The fun thing, however, is that modern videocards are totally okay with that. However, for slightly older ones, it might become a problem to pull this off fluently...

In a previous blogpost I explained how I abused my depth of field blur as an excuse to render the backgrounds in Awesomenauts on a much lower resolution. This greatly reduces the number of pixels that need to be rendered every frame, but it only helps for the background, since the rest of the scene needs to be sharp.

Since the background are optimised this way already, the biggest remaining overdraw in Awesomenauts happens during special effects, mostly those of nauts' special skills. This is because they have a lot of overlapping particles for smoke, fire, dust and debris. Having several of those special skills on screen at the same time decreased the framerate on older and mobile videocards too much. This is extra problematic because during fierce battles, high framerate is needed most!

Once I figured out that overdraw was the cause of the framedrops we were seeing on older videocards, the solution seemed simple: decrease the number of particles on screen. The difficulty, however, is that it is difficult for artists to optimise things without being able to see what they need to optimise. Having a ton of large particles for thin smoke looks very subtle, but eats performance like crazy. And if an effect needs to be optimised, what part of that effect is the problem exactly?

So I made a small addition to our engine to visualise overdraw. Internally we have a shortcut now to enable this, and when this is pressed, the screen shows how often each pixel is rendered. White means a pixel is rendered 85 times (!), while black means it is rendered 0 times. Technically, this is a very simple thing to make: all objects are simply rendered on a very dark additive blend mode. So each rendered object adds a little bit of brightness.

This is not just informative, but also looks pretty awesome! Have a look at this video of going through the menus and then some gameplay with Skølldir to see how weird and cool this looks:

As you can see, the character itself is not every expensive to render, since it is only one square. The problem comes from the tons of particles.

Using the overdraw visualisation, our artists (in this case mostly Koen, Ralph and Gijs) were able to quickly identify which parts of which special effect needed to be changed. These changes are usually pretty simple: 40 smoke particles that are 10% opaque, look nearly exactly the same as 20 smoke particles that are 20% opaque. So in most cases the trick was to drastically decrease the number of particles and then just modify some colours to compensate for that.

The video above shows what the overdraw looks like after these optimisations were already done, so here are some comparison shots that show how massively the overdraw was decreased while hardly changing the looks of these effects:

We tested the results on our Mac (even fast, expensive Macs usually have mobile videocards, which are not very fast and far from ideal for gaming) and indeed the game runs a lot smoother now. Several users reported that while they could previously only run Awesomenauts smoothly on Normal graphics quality, they can now run on High!

These improvements were released on Steam in patch 1.14 two weeks ago, so I hope players with older videocards are enjoying the smoother framerate they are getting!

On one of the platforms Awesomenauts launched on, we had a lot of trouble getting the internet connection between players to remain stable. After a while the ping would always start slowly increasing, until in the end it got too high and the game disconnected. Sometimes this took a couple of minutes to start, sometimes half an hour, but in the end this always happened.

We contacted the support team for the platform-specific networking library that we were using, and their answer was that we used too much bandwidth and sent too many packages. So we spent a lot of time optimising, and we managed to half the number of packages and half the bandwidth. However, the problem remained, and they again told us we used too much bandwidth. Again we halved the bandwidth and the number of packages, but the problem remained. At this point we were well below what they said was the ideal bandwidth usage and we couldn't optimise much more, so we were getting pretty desperate and contacted them again.

And then it happened...

Their answer was that their bandwidth throttling code was quite buggy, so there was a hidden enum that we could use to turn that code off. We used that and... the problem was instantly fixed! So they knew they had a bug, they even had an option to turn that bug off, but they didn't tell us for two months! I spent all that time doing extra bandwidth optimisations and it wasn't even necessary! Blargh!

Of course, using less bandwidth is always an important improvement for a multiplayer game, but we were already enormously behind on schedule at that point and this took a lot of time for a small indie studio...

2. Lying videocards

This has been a personal gripe of mine for years. Some videocards simply lie about their specs. Your game asks the videocard what it can do, and it will proudly brag about features it doesn't actually have!

I have not yet used any of the newer videocard features like geometry shaders, hull shaders and compute shaders, so I have not encountered any lying videocards recently, but I would be surprised if this doesn't still happen when you try to use state-of-the-art videocard features. It sure did a lot around the appearance of shader models 2 and 3. (Note that I haven't used the new shader types because I think they are uninteresting, but that's a long story that I will not go into today. Instead, I will just leave it at the short and controversial statement that they are irrelevant.)

I had to work around such lying videocards in both Proun and De Blob. What happens is that I make special versions of my shaders for different shader models, so that older videocards can still run the game, but with less special effects. The Ogre-engine has a very elegant system to handle this, and thus Proun features proper materials for shader models 1, 2.0, 2.x and 3.0.

The core of this solution, however, is that you ask the videocard which shader models it supports, and then pick the highest allowed for the best quality. This works very well, unless the videocard lies. It might claim to support 3.0, but doesn't really work with it. In the worst case, the videocard doesn't even give a compile error when being fed a 3.0 shader and simply outputs black pixels!

Several older Ati videocards turned out to do this in a horrible way. The solution ended up to hardcode the names of such videocards and feed them different shaders based on their name. If you look in the Proun folder structure, you can see folders with the same materials, but for different videocards, with beautiful names like "NotX8orX9". That last one contains materials that cannot be used on Ati X8** and X9** cards. I even have another set of materials for Ati X1*** and X2*** cards, because they lie in a different way...

1. "If I buy the game, it isn't the demo any more"

Yes, you read that title correctly. This is by far the most hilarious bug report I have ever seen, and easily claims the number 1 spot in this list, despite not even being a real bug. It was reported to us through the bug database though, so it qualifies for this list!

A professional QA testing company that was testing one of our games for us, at some point reported to us that if they bought the full game from the demo, then when they came back to the game, it wasn't the demo any more. It was instead... the full game!

Oh really?

That happens to be the point of buying the game, now isn't it?

When I replied in the bug database that either I didn't understand what they meant, or this bug report was a slight mistake from the tester, a producer quickly removed the bug from the database, so I never received an actual answer to that.

I think the reason they reported this, is that that particular shop (outside the game, not made by us) concluded the buying process with a question like "Do you want to go back to the game?" Whether you chose "Okay" or "Cancel", the shop always brought you back to the game, and apparently the tester concluded from the fact that there were two options that one of them ought to bring you back to the demo, even though the game had just been bought. This is some pretty broken reasoning, but I can imagine where it came from.

(As crazy as this bug report may be, though, I would like to emphasize that this is the only silly report this testing company wrote to us. The rest of the reports made perfect sense, so this one mistake really shouldn't be held against them! That doesn't make it any less hilarious, though...)

That's it, folks! The 7 weirdest bugs I have encountered! Come back next week when I will discussion the Awesomenauts animation pipeline or visual effects performance (I haven't really made up my mind yet which it is going to be...)