Friday, 23 March 2012

In today's blogpost, I'd like to do something radically different: I'm going to ask you a question. I know that quite a few of the visitors of my blog are hardcore programmers, and since I have a complex situation for which I would like to improve the performance, I figured I could ask for advice here! :)

The problem is pretty complex, though, so I figured a short blogpost would be needed to really explain it. In short, this is the question:

What is the most efficient way to synchronise renderable objects between the game thread and the render thread?

There are a couple of subtleties you need to know to answer this for our situation, though. Let me start with a scheme that shows the approach to threading in the Ronitech (our in-house multiplatform 2D engine), as we use it in Awesomenauts on the Xbox 360 and Playstation 3:

This works fine and we managed to get 60fps on console this way. However, the copying phase takes relatively long, so I am still wasting a lot of performance there. Since we will also use the Ronitech in all our future games, improving here will pay off greatly in our future projects.

There are a couple of specific requirements that I have to make this work with the rest of our engine:

The gameplay thread must be able to modify (and even delete!) all renderable objects. At the same time, the renderer must still be able to render each object how it looked during the previous frame.

Within a single frame, the same object might be rendered to different viewports in different ways. For example, a character might be partially transparent in one viewport and fully visible in another. The game can register itself as a PreViewportRenderListener and thus change objects in between the rendering of two viewports.

The copying phase as I have it right now, works as follows: for every renderable object I have a clone for every viewport it might be shown on. If there are lots of viewports, this can mean lots of clones. I don't have any memory problems, so I don't mind the clones. Now every frame during the copying phase, for all the objects that are actually visible in a viewport, all the properties of the original object are copied to the clone. This is the step that costs the performance, since in three player splitscreen this can mean copying all the properties of 2000 objects every frame, and each object has dozens of properties! I currently just call the PreViewportRenderListeners during the copying phase, in between copying data for one viewport and copying data for the next viewport.

So, what alternatives are there to just copying all the relevant data for all visible objects every frame? Or should rendering in a separate thread be set up differently all together? Please comment below if you have interesting ideas for this!

Thursday, 15 March 2012

When we started development of Awesomenauts, one of the first issues we had a good look at was how to create the graphics for our world. In Swords & Soldiers all the levels were mainly randomly generated from a construction set. This looks good, but the result is rather repetitive and it is difficult to really make unique places. Also, we had quite a few people who saw screenshots of Swords & Soldiers and concluded it should be a free Flash game. Some gamers totally ignored the size and quality of the game, just because the graphical style could also be achieved in Flash. So for Awesomenauts, we wanted to step up our game and create something more painterly.

But how to go about that? Most 2D games build their levels out of square construction elements. This is very fast, but it results in rather blocky graphics, no matter how good the construction set. The ultimate example of this is of course the classic Mario games. Square construction sets can of course be a lot more free-form than this, but they remain limited nevertheless.

Two games that we thought did their level graphics really well, were Wario Land: Shake It and Braid. Especially Braid achieves graphics that don't really show any patters and look gorgeous. It turned out the artist of Braid had actually written several really interesting blogpost about how the art was made, like this one and this one.

Even though Wario Land: Shake It has straighter outlines, the insides still don't really show any seams. Somehow Wario Land: Shake It uses a construction set in which repetitive elements can be spotted, but there seams cannot. At least not in the images below.

We didn't want to paint entire levels by hand, so we concluded that our ideal workflow would be to still have a construction set, but with completely free shapes in it. These shapes can then be put next to each other and on top of each other to together form solid shapes that make up the levels. So our artists set out creating lots of basic shapes of various sizes:

The big question, though, is how to put these together. Our artists immediately knew that they wanted to be able to paint this over. Since the construction set elements need to be rotatable, there can hardly be any lighting in them. This would make the levels really bland, so they wanted to paint the lighting and shadows and maybe extra details on top of that.

The most obvious method to do this seems to be to just use Photoshop to piece together the elements, and then paint over that. In theory, that might work, but it has several major drawbacks. For of all, each element would become a layer, making Photoshop unworkable with thousands of layers per level. Also, Photoshop is not fast enough to handle textures with resolutions like 20,000 by 40,000 pixels. Since the levels contain really long and winding pieces of continuous art, splitting them in separate Photoshop file would make the seams between files difficult to handle.

So we concluded that Photoshop was not the right tool for this. Now since I was already building a level editor for the Ronitech (our multi-platform 2D engine), we decided to add some tools of our own to this. The scheme below shows the complete workflow that we used in the end. Note that this was used only for the static geometry that the player walks over, not for the backgrounds, the foregrounds or the animations in the levels.

How important lighting is, becomes obvious when looking at these comparisons of just the construction set elements and the painted over versions. At their core, these images might look the same, but the painted over versions just work so much more beautifully:

The method we used has two big drawbacks. The first is of course that it takes an enormous amount of handwork. We have all the tools to make it work from a technical standpoint, but our artists still need to paint over the entire level. This can be done relatively quickly, because the construction set elements are already there and mostly only the lighting needs to be added. Also, to save time we did mirror most of the graphics for the left and right half of the levels. But creating new levels does take way longer this way than it would have with other methods.

The other big drawback of this method is that once the level art has been finished, it is relatively cumbersome to move things around. With a design team that was constantly tweaking details in the levels really late in production, fixing the art every time cost quite a bit of time.

So for future games, we are going to use faster methods and just accept that we cannot always have the incredible hand-painted look of Awesomenauts everywhere. Making games efficiently requires having some repetition in the artwork. However, for Awesomenauts I definitely don't regret the way we approached this. I think the game looks absolutely gorgeous and part of that is because our art team painted all these beautiful lighting effects over the art, and did this uniquely everywhere in the levels. I think the end-result speaks for itself and makes the effort totally worth it!

Sunday, 11 March 2012

Last week we released a patch on Steam that finally fixed the networking issues some people were having with the PC version of Swords & Soldiers. I'm afraid that because of the work on Awesomenauts, we didn't come to it any earlier. But we finally fixed it now! :)

Now this bug we fixed was a pretty interesting one. Not as interesting as the bug I posted about last December that got my blog a whopping 40,000 viewers in a couple of days, but still interesting enough to share it with you. This one is another nice example of how bug fixing sometimes requires thinking far outside the box.

We were getting two bug reports from users:

Some users always got a Network Error at the start of a match.

Some users encountered cheaters who introduced so much lag, that the game became unplayable for the player on the right side of the map.

We initially just assumed the Network Errors were being caused by firewalls or bad router settings. But these users were not having trouble in other games, and it even didn't work when they turned off their firewalls. We also had a user who told us it worked on his laptop and not on his PC, while both were on the same network. That pretty much ruled out router issues as well.

As for the 'cheaters': we immediately had the hunch that this might as well be a network bug in our code somehow, and not really someone cheating.

We had looked at this bug at various moments in the past year without success, and wondered whether it was still a firewall, or something in the Steam networking libraries. We even sent the Steam support team questions about this one. They didn't know any bugs in their system that could cause this, which was of course right, since this turned out to be a bug in our own code...

So, what was up? In a very bright moment, my colleague Maarten, who does a lot of network programming at Ronimo Games, suddenly realised what was happening: to check what the ping is and to keep the connection alive, we regularly send ping messages over the network. Accidentally, we did that every frame. Now sending 60 extra messages per second over the network is a really bad idea, but does not usually kill a connection. However, and this is where we are getting way out of the box: some users had forced vsync to be turned off in their drivers. For those who don't know: vsync makes sure a game runs at the same framerate as your screen, usually 60 frames per second. Forcing vsync off means Swords & Soldiers might be running at hundreds of frames per second on their computers! Sending that many ping messages instantly kills most connections.

This also explains the 'cheaters': if the framerate is not high enough to actually kill the connection, it can still be high enough to make the connection really bad. The player whose computer is server in a match won't notice this, but the other player (whose computer is the client) gets so much lag he can hardly play anymore.

So, the solution was really simple: instead of sending the ping message every frame, we now send a small, fixed number of them every second. Which is actually what we had intended in the first place, had a bug not thwarted our plans!

So, another nice example of how bug fixing requires letting go of all assumptions, and researching with an open mind what is actually happening. Until we found this bug, I never would have connected vsync with networking errors!

I would like to thank the Steam users who helped us with testing to find this issue! I would also like to thank those who offered help that we ended up not using. It is great to see that gamers are so willing to help when we are stuck on a bug! Thank you very much, folks! :D

PS. If anyone encounters any new or further issues, then please don't hesitate to post them at the Steam forum or our own forum, and we will look into them as soon as we can!

Friday, 2 March 2012

Today's post is a guest post by Sonic Picnic, the team that did the incredibly awesome music and sound design for both Awesomenauts and Swords & Soldiers. Especially the Awesomenauts theme song is mindblowingly awesome and got our game a lot of love from gamers and the press. Sonic Picnic is a small team of audio designers who create audio for all kinds of media, not just games. For example, they also worked for television shows, documentaries, and the games Rocket Riot and Toki Tori. So, here is their Dev Diary for Awesomenauts. Enjoy!

The intro sequence to Awesomenauts

Music

The development of the music for Awesomenauts started halfway through 2009, at a listening session with the whole Ronimo team, the four members of SonicPicnic, and a few beers. Based on the first sketches and early prototypes of the game, everyone brought along a couple of existing music tracks which they found particularly fitting to the theme and style of the game. Everyone at the table had a different idea about what would be appropriate, so we ended up with a broad spectrum of musical styles and tastes - ranging from metal to hillbilly western tunes.

Luckily, there were some aspects to the music that everyone seemed to agree upon. Most of the tracks were fundamentally electronic or had synths, samplers and drumcomputers playing an important role, whether they were techno, glampop, or electrometal. There was also a strong preference for music that could be heard in games like Streets of Rage, Alien Breed, Project-X, and other games that we used to play on the Sega Megadrive/Genesis and Commodore Amiga. No doubt the graphical style of the game had something to do with that, as it strongly refers to games from that era. Finally, as the game oozed chaos, mayhem and wackiness - the character backstories, the gameplay, the colors, everything – we felt like we had to capture that in our music too, perhaps more than anything else.

Even though we were sure we didn’t want to tie ourselves down on one very specific style, we all agreed that the music should have a strong electronic component. If anything, we wanted it to be wacky, and have some historical reference to the early 90’s.

There was still plenty of room to differentiate within these boundaries, and we really stretched these to their limits when it came to creating the character-specific jingles. Froggy G, for example, has a smooth, somewhat oldschool-hiphop theme, and the jetpack-wearing monkey known as Yuri has a revolutionary Soviet march with a drunk Kozakian Choir.

Apart from their jingles, each of the six characters was also linked to one of the ingame tracks. When the player starts a game, they will be hearing any one of the ingame tracks chosen at random. It kind of feels like a jukebox that has been set to shuffle. Each track lasts for about 3,5 minutes, and the game just continues playing random tracks as it progresses. The bond between the character and its ingame track becomes apparent when the player achieves a ‘killing spree’. At that moment, a beefed-up version of the ingame music starts playing - which is really a mash-up of the recognizable character jingle and a regular ingame track.

Lonestar's Jingle and Killing Spree examples

With this new song pouring out of their speakers, the player gets the feeling of being an unstoppable, unbeatable superhero for a while, until, after a specific amount of time, the killing spree track flows back into the regular ingame music. After a few matches, players will be able to recognize their characters’ tracks from amongst the other tracks, and their ‘moment of fame’ will become greater still.

For the menu, we decided to compose a retro-futuristic synthesizer tune (like the synthesizer music of the seventies and eighties) to really get that space/sci-fi feel. We also composed the ‘Victory’ and ‘Defeat’ jingles that play at the end of each match in the same style, to make sure the melody is nicely interwoven throughout the game.

The title music was a special case: Ronimo wanted to have a hand drawn animation film introducing all the characters and explain the (rather shallow) storyline in a typical 80's cartoon style. The great inspiration for this sequence came from series like Thundercats, M.A.S.K. and Galaxy Rangers. From the start, it was perfectly clear that this animation just had to be supported by a glam-muscular-power-rock theme.

The opening sequence of M.A.S.K.

This was absolutely great fun to compose and write, but we originally limited ourselves to a 30 second intro to fit the length of the animation. But when the theme was released as part of several trailers that Ronimo made for the game, lots of people became excited about the song, and asked us where they could hear the complete version of the song... Which doesn’t exist... yet. So yes, we’re creating a longer version of the intro music and will turn it into a complete hit-length track. Tell your friends!

Recording vocals for the Main Theme

Aside from the electro sounds, the music in Awesomenauts can also be a bit cheesy at times. This reaches its zenith in what finally ended up as the credits music. One of the tracks that we came up with during our initial brainstorm was a song by Demis Roussos (Forever and Ever). We all agreed that there had to be something like that in the game somewhere, because ‘cheesiness’ and ‘wackiness’ are terms both Ronimo and SonicPicnic keep in very high regard. So we came up with a song called “You will ever always be my only Awesomenaut.” We originally intended it for the grand moment of victory in a match, but when it was finished, Ronimo found it to be a little bit (well, quite a little bit) too much when compared to the cheesiness of the rest of the music. So it got banished to the credits. That means it’s still in the game though, so reason enough to check out those credits!

Sound design

At the very first stages of our work on the game, we put a lot of placeholder sounds in the game. We gathered a lot of different sounds from our own library and we also created a bunch of simple ‘soundsketches’, and implemented them into a very early build of the game. This gave us an idea of how the game would work with sound, and acted as a starting point from which we could experiment with different types of sounds.

There are actually a few placeholder sounds that made it in the final build of the game (after some minor editing), for the simple reason that they just worked very well. As with every development process, there were a few features and objects that got axed as development progressed. Some of these had very cool sound effects, so we tried to find a new use for them in the game wherever possible. Sometimes we had to kill our darlings though... but that’s part of the job.

Because Awesomenauts is set in a sci-fi setting, we decided to focus on creating many of the sounds with synthesizers. This fits the graphical style very well, so all the interface sounds and many of the in-game sound effects have been created with a synthesizer. For example, the ‘powerswing’ from the sword of Leon Chameleon, or the charging of the droppod before it's being launched onto the battlefield, and the time bubble skill of Yuri.

Besides the many synth-created sounds, we actually used a lot of real life recordings. The alarm that sounds when a turret is under attack is a good example of this. One day we were working in our studio, and it was 12:00 on the first Monday of the month. That's the time when the air raid sirens are being tested in the Netherlands. We thought it would be nice to use that alarm so one of us quickly put a microphone out of the window of our studio in Utrecht and recorded the alarm. We edited it, put some processing on it like EQ and some reverb, and put it in the game to test it. It was spot-on, and it’s been in the game like that ever since! Another example of sounds we recorded are the footsteps for Froggy G. It is a very simple sound actually, which was created by one of us stomping his hand against his bare leg (not too soft). Yes, sound design is a painful job sometimes...

Voices

We recorded five voice actors for the voices of all the characters. Almost all of these recordings were processed afterwards. For the voice of Clunk, we used a vocoder and we pitched the sound a little to create his distinct robot sound. For the voice of Voltar the Omniscient, we used a lot of signal processing to get his voice to sound like it's coming from a bulb through a little speaker.

For Froggy G we layered two recordings of the same sentence, to give his voice a little multi membrane, froggy sound.

There’s actually lots of fun to be had during the recording of weird and funny character voices.

Recording and tweaking Voltar's voice

Implementation and Mix

We have worked together with Ronimo Games on Swords & Soldiers, their previous game. Compared to that project, we were much more in control of the audio implementing with Awesomenauts, all thanks to the new Ronitech engine. We were able to put sound cues on almost every animation ourselves, instead of asking the programmers for cues. This way we were able to set parameters for timing, loops, volume, whether the sound should always finish, etc. Furthermore, we had control over the distance, pitch and volume variation and instance limiting (time between triggers and quantity of sounds played in a group). It was great to have more control over the audio, and it helped us work more efficiently.

The mix of all the different audio elements is very important, and at first we made a mix in which all sounds were quite loud. This didn’t feel quite right, as a loud explosion has to stand out in order to be impressive. It was quite a challenge to get the mix right, as we wanted to keep a dynamic feel to the sounds (so when you have a higher skill e.g. a bigger bomb you can drop, it would sound louder), but also wanted the music to stand its ground and be constantly strong and bold.

Gameplay example from Awesomenauts

That left the smaller, more subtle sound effects, like footsteps, that enrich the overall experience, in need of a place somewhere in the mix. As always with mixing, it was all about balance. And especially in this game, where a lot is happening at once, the mix was a pretty difficult puzzle.

In the end, when all sound and music had been implemented, combined, and tested, we were quite happy with the awesome result. We hope you will like it as much as we do!