In this post I will talk to you about the lessons learnt during the development of ViziCities. From application architecture to fine-detailed WebGL rendering improvements, we learnt a lot in the past year and we hope that by sharing our experiences we can help others avoid the same mistakes.

What is ViziCities?

In a rather geeky nutshell, ViziCities is a WebGL application that allows you to visualise anywhere in the world in 3D. It’s primary purpose is to look at urban areas, though it’ll work perfectly fine in other places if they have decent OpenStreetMap coverage.

Demo

The best way to explain what ViziCities does is to try it out yourself. You’ll need a WebGL-enabled browser and an awareness that you’re using pre-alpha quality software. Click and drag your way around, zoom in using the mouse wheel, and rotate the camera by clicking the mouse wheel or holding down shift while clicking and dragging.

You can always take a look at this short video if you’re unable to try the demo right now:

What’s the point of it?

We started the project for multiple reasons. One of those reasons is that it’s an exciting technical and design challenge for us – both Peter and I thrive on pushing ourselves to the limits by exploring something unknown to us.

Another reason is that we were inspired by the latest SimCity game and how they visualise data about the performance of your city – in fact, Maxis, the developers behind SimCity reached out to us to tell us how much they like the project!

There’s something exciting about creating a way to do that for real-world cities rather than fictional ones. The idea of visualising a city in 3D with up-to-date data about that city overlaid is an appealing one. Imagine if you could see census data about your area, education data, health data, crime data, property information, live transport (trains, buses, traffic, etc), you’d be able to learn and understand so much more about the place you live.

This is really just the beginning – the possibilities are endless.

Why 3D?

A common question we get is “Why 3D?” – the short answer, beyond “because it’s a visually interesting way of looking at a city”, is that 3D allows you to do things and analyse data in ways that you can’t do in a 2D map. For example by using 3D you can take height and depth into consideration, so you can better visualise the sheer volume of stuff that lies above and below you in a city – things like pipes and underground tunnels, or bridges, overpasses, tall buildings, the weather, and planes! On a 2D map, looking at all of this would be a confusing mess, in 3D you get to see it exactly how it would look in the real world – you can easily see how objects within a city relate to each other.

Core technology

At the most basic level ViziCities is built using Three.js, a WebGL library that abstracts all the complexity of 3D rendering in the browser. We use a whole range of other technologies too, like Web Workers, each of which serves a specific purpose or solves a specific problem that we’ve encountered along the way.

Let’s take a look at some of those now.

Lessons learnt

Over the past year we’ve come from knowing practically nothing about 3D rendering and geographic data visualisation, to knowing at least enough about each of them to be dangerous. Along the way we’ve hit many roadblocks and have had to spend a significant amount of time working out what’s wrong and coming up with solutions to get around them.

The process of problem solving is one I absolutely thrive on, but it’s not for everybody and I hope that the lessons I’m about to share will help you avoid these same problems, allowing you to save time and do more important things with your life.

These lessons are in no particular order.

Using a modular, decoupled application architecture pays off in the long run

We originally started out with hacky, prototypal experiments that were dependency heavy and couldn’t easily be pulled apart and used in other experiments. Although it allowed us to learn how everything worked, it was a mess and caused a world of pain when it came to building out a proper application.

In the end we re-wrote everything based on a simple approach using the Constructor Pattern and the prototype property. Using this allowed us to separate out logic into decoupled modules, making everything a bit more understandable whilst also allowing us to extend and replace functionality without breaking anything else (we use the Underscore _.extend method to extend objects).

I’d argue that these 2 patterns are the most useful aspects of the new ViziCities application architecture – they have allowed us to iterate quickly without fear of breaking everything.

Using promises instead of wrestling with callbacks

Early on in the project I was talking to my friend Hannah Wolfe (Ghost’s CTO) about how annoying callbacks are, particularly when you want to load a bunch of stuff in order. It didn’t take Hannah long to point out how stupid I was being (thanks Hannah) and that I should be using promises instead of wrestling with callbacks. At the time I brushed them off as another hipster fad but in the end she was right (as always) and from that point onwards I used promises wherever possible to take back control of application flow.

For ViziCities we ended up using the Q library, though there are plenty others to choose from (Hannah uses when.js for Ghost).

The general usage is the same whatever library you choose – you set up promises and you deal with them at a later date. However, the beauty comes when you want to queue up a bunch of tasks and either handle them in order, or do something when they’re all complete. We use this in a variety of places, most noticeably when loading ViziCities for the first time (also allowing us to output a progress bar).

I won’t lie, promises take a little while to get your head around but once you do you’ll never look back. I promise (sorry, couldn’t resist).

Using a consistent build process with basic tests

I’ve never been one to care too much about process, code quality, testing, or even making sure things are Done Right™. I’m a tinkerer and I much prefer learning and seeing results than spending what feels like wasted time on building a solid process. It turns out my tinkering approach doesn’t work too well for a large Web application which requires consistency and robustness. Who knew?

The first step for code consistency and quality was to enable strict mode and linting. This meant that the most glaring of errors and inconsistencies were flagged up early on. As an aside, due to our use of the Constructor Pattern we wrapped each module in an anonymous function so we could enable strict mode per module without necessarily enabling it globally.

At this point it was still a faff to use a manual process for creating new builds (generating a single JavaScript file with all the modules and external dependencies) and to serve the examples. The break-through was adopting a proper build system using Grunt, thanks mostly to a chat I had with Jack Franklin about my woes at an event last year (he subsequently gave me his cold which took 8 weeks to get rid of, but it was worth it).

Grunt allows us to run a simple command in the terminal to do things like automatically test, concatenate and minify files ready for release. We also use it to serve the local build and auto-refresh examples if they’re open in a browser. You can look at our Grunt setup to see how we set everything up.

Travis CI is used to make sure we don’t break anything when pushing changes to GitHub. It automatically performs linting and runs our tests via Grunt when changes are pushed, including pull requests from other contributors (a life saver). Plus it lets you have a cool badge to put on your GitHub readme that shows everyone whether the current version is building successfully.

Together, these solutions have made ViziCities much more reliable than it has ever been. They also mean that we can move rapidly by building automatically, and they allow us to not have to worry so much about accidentally breaking something. The peace of mind is priceless.

Monitoring performance to measure improvements

General performance in frames-per-second can be monitored using FPSMeter. It’s useful for debugging parts of the application that are locking up the browser or preventing the rendering loop from running at a fast pace.

It’s worth keeping an eye on this to make sure objects are not being rendered when they move out of the current viewport. Early on in ViziCities we had a lot of issues with this not happening and the only way to be sure we to monitor these values.

Turning geographic coordinates into 2D pixel coordinates using D3.js

One of the very first problems we encountered was how to turn geographic coordinates (latitude and longitude) into pixel-based coordinates. The math involved to achieve this isn’t simple and it gets even more complicated if you want to consider different geographic projections (trust me, it gets confusing fast).

Fortunately, the D3.js library has already solved these problems for you, specifically within its geo module. Assuming you’ve included D3.js, you can convert coordinates like so:

The scale() value is the hardest part of the process to understand. It basically changes the pixel value that’s returned based on how zoomed in you want to be (imagine zooming in on Google Maps). It took me a very long time to understand so I detailed how scale works in the ViziCities source code for others to learn from (and so I can remember!). Once you nail the scaling then you will be in full control of the geographic-to-pixel conversion process.

Extruding 2D building outlines into 3D objects on-the-fly

While 2D building outlines are easy to find, turning them into 3D objects turned out to be not quite as easy as we imagined. There’s currently no public dataset containing 3D buildings, which is a shame though it makes it more fun to do it yourself.

What turned out interesting was how it actually turned out quicker to generate 3D objects on-the-fly than to pre-render them and load them in. This was mostly due to the fact it would take longer to download a pre-rendered 3D object than downloading the 2D coordinates string and generating it at runtime.

One thing we did notice with the generation of 3D objects was that it locked up the browser, particularly when processing a large number of shapes at the same time (you know, like an entire city). To work around this we delved into the magical world of Web Workers.

Web Workers allow you to run parts of your application in a completely separate processor thread to the browser renderer, meaning anything that happens in the Web Worker thread won’t slow down the browser renderer (ie. it won’t lock up). It’s incredibly powerful but it can also be incredibly complicated to get working as you want it to.

We ended up using the Catiline.js Web Worker library to abstract some of the complexity and allow us to focus on using Web Workers to our advantage, rather than fighting against them. The result is a Web Worker processing script that’s passed 2D coordinate arrays and returns generated 3D objects.

After getting this working we noticed that while the majority of browser lock-ups were eliminated, there were two new lock-ups introduced. Specifically, there was a lock-up when the 2D coordinates were passed into the Web Worker scripts, and another lock-up when the 3D objects were returned back to the main application.

The solution to this problem came from the inspiring mind of Vladimir Agafonkin (of LeafletJS fame). He helped me understand that to avoid the latter of the lock-ups (passing the 3D objects back to the application) I needed to use transferrable objects), namely ArrayBuffer objects. Doing this allows you to effectively transfer ownership of objects created within a Web Worker thread to the main application thread, rather than copying them. We implemented this to great effect, eliminating the second lock-up entirely.

To eliminate the first lock-up (passing 2D coordinates into the Web Worker) we need to take a different approach. The problem lies again with the copying of the data, though in this case you can’t use transferrable objects. The solution instead lies in loading the data into the Web Worker script using the importScripts method. Unfortunately, I’ve not yet worked out a way to do this with dynamic data sourced from XHR requests. Still, this is definitely a solution that would work.

Using simplify.js to reduce the complexity of 2D shapes before rendering

Something we found early on was that complex 2D shapes caused a lot of strain when rendered as 3D objects en-masse. To get around this we use Vladimir Agafonkin’s simplify.js library to reduce the quality of 2D shapes before rendering.

It’s a great little tool that allows you to keep the general shape while dramatically reducing the number of points used, thus reducing its complexity and render cost. By using this method we could render many more objects with little to no change in how the objects look.

Getting accurate heights for buildings is really difficult

One problem we never imagined encountering was getting accurate height information for buildings within a city. While the data does exist, it’s usually unfathomably expensive or requires you to be in education to get discounted access.

The approach we went for uses accurate height data from OpenStreetMap (if available), falling back to a best-guess that uses the building type combined with 2D footprint area. In most cases this will give a far more accurate guess at the height than simply going for a random height (which is how we originally did it).

Restricting camera movement to control performance

The original dream with ViziCities was to visualise an entire city in one go, flying around, looking down on the city from the clouds like some kind of God. We fast learnt that this came at a price, a performance price, and a data-size price. Neither of which we were able to afford.

When we realised this wasn’t going to be possible we looked at how to approach things from a different angle. How can you feel like you’re looking at an entire city without rendering an entire city? The solution was deceptively simple.

By restricting camera movement to only view a small area at a time (limiting zoom and rotation) you’re able to have much more control over how many objects can possibly be in view at one time. For example, if you prevent someone from being able to tilt a camera to look at the horizon then you’ll never need to render every single object between the camera and the edge of your scene.

This simple approach means that you can go absolutely anywhere in the world within ViziCities, whether a thriving metropolis or a country-side retreat, and not have to change the way you handle performance. Every situation is predictable and therefore optimisable.

Tile-based batching of objects to improve loading and rendering performance

Another approach we took to improve performance was by splitting the entire world into a grid system, exactly like how Google and other map providers do things. This allows you to load data in small chunks that eventually build up to a complete image.

In the case of ViziCities, we use the tiles to only request JSON data for the geographic area visible to you. This means that you can start outputting 3D objects as each tile loads rather than waiting for everything to load.

A by-product of this approach is that you get to benefit from frustum culling, which is when objects not within your view are not rendered, dramatically increasing performance.

Caching of loaded data to save on time and resources when viewing the same location

Coupled with the tile-based loading is a caching system that means that you don’t request the same data twice, instead pulling the data from a local store. This saves bandwidth but also saves time as it can take a while to download each JSON tile.

We currently use a dumb local approach that resets the cache on each refresh, but we plan to implement something like localForage to have the cache persist between browser sessions.

Using the OpenStreetMap Overpass API rather than rolling your own PostGIS database

Late into the development of ViziCities we realised that it was unfeasible to continue using our own PostGIS database to store and manipulate geographic data. For one, it would require a huge server just to store the entirety of OpenStreetMap in a database, but really it was just a pain to set up and manage and an external approach was required.

The solution came in the shape of the Overpass API, an external JSON and XML endpoint to OpenStreetMap data. Overpass allows you to send a request for specific OpenStreetMap tags within a bounding box (in our case, a map tile):

The by-product of this was that you get worldwide support out of the box and benefit from minutely OpenSteetMap updates. Seriously, if you edit or add something to OpenStreetMap (please do) it will show up in ViziCities within minutes.

Limiting the number of concurrent XHR requests

Something we learnt very recently was that spamming the Overpass API endpoint with a tonne of XHR requests at the same time wasn’t particularly good for us nor for Overpass. It generally caused delays as Overpass rate-limited us so data took a long time to make its way back to the browser. The great thing was that by already using promises to manage the XHR requests we were half-way ready to solve the problem.

The final piece of the puzzle is to use throat.js to limit the number of concurrent XHR requests so we can take control and load resources without abusing external APIs. It’s beautifully simple and worked perfectly. No more loading delays!

Using ViziCities in your own project

I hope that these lessons and tips have helped in some way, and I hope that it encourages you to try out ViziCities for yourself. Getting set up is easy and well documented, just head to the ViziCities GitHub repo and you’ll find everything you need.

Contributing to ViziCities

Part of the reason why we opened up ViziCities was to encourage other people to help build it and make it even more awesome than Peter and I could ever make it. Since launch, we’ve had over 1,000 people favourite the project on GitHub, as well as nearly 100 forks. More importantly, we’ve had 9 Pull Requests from members of the community who we didn’t previously know and who we’ve not asked to help. It’s such an amazing feeling to see people help out like that.

If we were to pick a favourite contribution so far, it would be adding the ability to load anywhere in the world by putting coordinates in the URL. Such a cool feature and one that has made the project much more usable for everyone.

What’s next?

It’s been a crazy year and an even crazier fortnight since we launched the project. We never imagined it would excite people in the way it has, it’s blown us away.

The next steps are slowly going through the issues and getting ready for the 0.1.0 release, which will still be alpha quality but will be sort of stable. Aside from that we’ll continue experimenting with exciting new technologies like the Oculus Rift (yes, that’s me with one strapped to my face)…

We support mozGetUserMedia, mozRTCPeerConnection and DataChannels. We have a basic UI for mozGetUserMedia which we expect to be updating in the
coming weeks.

Enabling WebRTC in Firefox

The code is behind a pref for now, pending more testing. To enable our WebRTC code in Firefox’s Nightly desktop build, browse to about:config
and change the media.peerconnection.enabled preference to true.

Here are the 3 relevant prefs to getUserMedia() and mozRTCPeerConnection():

media.navigator.enabled enables calls to mozGetUserMedia() only

media.navigator.permission.disabled automatically gives permission to access the camera/microphone and bypasses the permission/selection dialog

media.peerconnection.enabled enables use of mozRTCPeerConnection()

Note: media.peerconnection.enabled implies media.navigator.enabled has been set to true.

Demos & upcoming changes

There’s a lot you can do with these APIs, even today. For examples, check out our test landing page on GitHub. We’ll try and put up notices if you’re running an out-of-date browser — as well as news updates about important bug fixes and API changes there!

To give you an idea of the power of these APIs, in a couple of days our team whipped up a Social API integration demo that allows you to video +
text chat with your friends, drag-and-drop files to each other, drop links, tabs, etc, all making simple use of the DataChannel API.

The DataChannel API is quite simple on the surface, and has an API very similar to WebSockets. A quick example:

/**
* Assume we've connected a PeerConnection with a friend - usually with audio
* and/or video. For the time being, always at least include a 'fake' audio
* stream - this will be fixed soon.
*
* connectDataConnection is a temporary function that will soon disappear.
* The two sides need to use inverted copies of the two numbers (eg. 5000, 5001
* on one side, 5001, 5000 on the other)
*/
pc.connectDataConnection(5001,5000);function handle_new(channel){
channel.binaryType="blob";
channel.onmessage=function(evt){if(evt.datainstanceof Blob){
console.log("I received a blob");// assign data to an image, save in a file, etc}else{
console.log("I got a message: "+ evt.data);}};
channel.onopen=function(){// We can now send, like WebSockets
channel.send("The channel is open!");};
channel.onclose=function(){
console.log("pc1 onclose fired");};};/* For when the other side creates a channel */
pc.onDataChannel= handle_new;
channel = pc.createDataChannel("My Datastream",{});if(channel){
handle_new(channel);}

Filing bugs & moving forward

Progress on WebRTC (and bug-fixing) is rapid, and we encourage you to try it out and submit bugs. (We have plenty! But we’re nailing them as fast as we can, so make sure you’re on nightly and update regularly.)

The team is both excited by all the progress, and exhausted. The work so far represents tons of hours of work from so many people on the Firefox team (too many people to name — especially because we don’t want to forget
someone — but you know who you are!). Thank you to everyone who helped us land this “747″ on the flight deck.

We’ll continue to blog regularly on our progress as we work to make this a great product feature for Firefox and the web.

Just under a month ago I wrote a personal post about my thoughts on Firefox OS and why I think there is something ‘magical’ about what it stands for and the possibilities it brings to the table. This post is a follow-up that aims to cover much of the same ground but with extra detail and more of a technical focus.

What is Firefox OS?

In short, Firefox OS is about taking the technologies behind the Web, like JavaScript, and using them to produce an entire mobile operating system. It’s effectively a mobile OS powered by JavaScript!

You might be wondering what’s running Gecko, as a phone can’t naturally boot directly into Gecko. To do that, the phone boots into a very lightweight Linux kernel that, in turn, boots the Gecko process. The process is a little more involved than that and much more detail can be found in the B2G Architecture documentation, including how Gecko accesses the radio hardware and other phone-specific functionality.

The Firefox OS project also aims to combine many of the other projects at Mozilla into a single vision, what we refer to as the Web as the platform. These projects include the Open Web Apps initiative and Persona, our solution to identity and logins on the Web (formally known as BrowserID). It’s the combination of these various technologies that completes Firefox OS.

If you want to find out more technical information about the OS then definitely check out the Firefox OS pages on MDN.

Why Firefox OS?

A couple of common questions that come up are, “Why Firefox OS?” and more specifically, “Why build a mobile OS using JavaScript?” These are incredibly important questions so let’s take a moment to delve into them in a little detail.

Why build a mobile OS using JavaScript?

Answering this question can quite simply be boiled down to one sentence; because it’s possible. It’s not the one and only answer but it succinctly handles most of the arguments against JavaScript being used in this way.

A longer answer is that a JavaScript-powered OS unlocks a whole range of possibilities that aren’t normally or easily available to developers and users with existing operating systems.

The most obvious of the possibilities is the ability to build applications using the technologies that we already use to build websites; namely JavaScript, CSS, and HTML. While not a truly unique feature of Firefox OS — projects like PhoneGap have done this for years on ‘native’ platforms — it allows developers everywhere to create mobile applications without having to learn native languages and APIs.

Another draw of JavaScript is that it’s both extremely well documented and free to develop with. Anyone could sit down for a weekend and put together an application without having to pay for a single thing. Obviously that’s not true in the majority of cases, as people tend to buy their own hosting or tooling, but theoretically there is nothing to stop you building with these technologies for free.

What’s arguably most interesting about JavaScript being used in this way is that it inherently enables physical devices to communicate using the same APIs that we already use on websites. In effect, instead of accessing the Web through a mobile browser the entire phone is now capable of accessing and communicating with any Web API. For example, there is nothing to stop you building an application for Firefox OS that uses WebRTC (once added) to create Skype-like P2P video communication between phones, desktop computers, or anything else that supports WebRTC.

This really only scrapes the surface of “Why JavaScript?” but it certainly gives you a feel of how this is both interesting and important, beyond the tired debate of ‘native’ vs. Web. If you’re still not convinced, just think for a moment about how you can now customise an entire mobile OS using nothing by JavaScript. You’d be hard pushed to deny that it’s pretty darn interesting!

OK, but why Firefox OS?

Effectively, Firefox OS has been built to put our money where our mouth is (so to speak) and prove that JavaScript is capable of doing what we say it can do. However, there is much more to the project than just proving the the technology is fast enough.

The first reason ‘Why Firefox OS’ is that the mobile ecosystem is overrun with proprietary platforms, most of which prevent you from easily moving between various platforms. What Firefox OS aims to achieve is a truly ‘open’ platform that doesn’t lock you in and inherently makes it as easy and possible to move between devices as and when you choose.

Mozilla is effectively replicating its success with Firefox, in which it stormed the browser market and showed users that there is an alternative, one that lets them be in control of how they use the Web. In this case, it’s less about browsers and more about mobile platforms and application portability.

Another reason is that Firefox OS is an attempt to push the Web forward into the world of physical devices. One direct benefit of this is the addition of brand new Web standards and APIs that allow for things like hardware access using JavaScript.

Plenty of challenges

It’s fair to say that the Firefox OS journey will contain a number of technical challenges along the way, however that’s part of the fun and the reasons why we’re working on it.

One of those challenges is how to manage an apps ecosystem that is open and distributed. This is something that we are tackling with the Open Web Apps initiative and the Mozilla Marketplace. It’s a challenge that we are dealing with as things progress and as we learn more about how things work best, as is the nature with new ways of thinking.

Another of the challenges is making sure that the phone runs as fast as possible, creating the best experience possible. This also relates to questions raised within the developer community around the performance capabilities of JavaScript, particularly when it is used to do things that are perceived to be complex, or when it is compared against ‘native’ technologies. This is a challenge that we are taking very seriously and one which we feel we can overcome. In fact, it’s a challenge that I believe we have already overcome.

One prime example of how capable JavaScript has become is seeing beautiful JavaScript games running in Firefox OS at near-enough 60 frames per-second, on a low-end, cheap phone.

Beyond the mobile phone

While the phone aspect of Firefox OS is immediately interesting, you should consider the wider implications of a JavaScript OS and what possibilities it unlocks. For example, what other devices could benefit from being powered by JavaScript? And, what would a network of JavaScript-powered devices allow us to do — things like Ubiquitous Computing, perhaps?

These aren’t things that we are exploring directly at Mozilla, but they are things that are now inherently possible as a result of the work that we’re doing. There is nothing to stop you taking the Firefox OS source code from GitHub and porting it to a device that we’ve never even considered.

What about a game console powered by Firefox OS? A TV, or set-top box? What about a fridge? Individually, these are all interesting projects, but together they offer something we don’t really have at the moment, a network of different devices powered by the same, open technologies and able to access and communicate across the Web with the same APIs.

We are a long way away from that kind of world but it is projects like Firefox OS that may pave the way for it to happen. You could even be a part of it!

Getting started with Firefox OS

The hope is that by now you’re sufficiently interested in Firefox OS to begin exploring, experimenting and playing with it. The good news is that there are a whole host of ways that you can do that.

If you’re more interested with the inner-workings of the platform then you’ll want to cast an eye over the B2G wiki, which outlines much of the internals in plenty of detail.

Source code

If you’re keen to get to grips with the source code of Firefox OS then you’ll want to head over to GitHub and check it out. The two main repositories that you want are ‘b2g’ (the underlying Gecko engine) and ‘gaia’ (everything you can see, the OS).

If you just want to ask a few immediate questions then try out the #b2g and #gaia rooms on irc.mozilla.org. We’re all pretty friendly!

Development options

If you just want to dig in and make some applications, or perhaps customise the OS, then you’ll need to know about the various development options available to you. They are covered in some detail on MDN but here is a brief overview.

Slightly more involved than Nightly is using the desktop B2G client. This is effectively a chromeless build of Firefox that looks phone-like has some added APIs that aren’t normally available in standard Firefox. This doesn’t replicate phone hardware but it’s the next best thing before starting to develop on an actual device.

Setting up the desktop B2G client isn’t too hard, but it could be made easier. In the meantime, projects like r2d2b2g aim to make the process super simple. Definitely worth checking out.

The last method, and arguably the most important one, is developing on an actual Firefox OS device. This is the only method that will give you a true representation of how your application will perform. It is also the only method that will give you access to the all the new APIs that come with Firefox OS.

Go forth and be part of something big

My hope is that by now you should now have enough inspiration and information to go forth and begin building for this new platform, powered by the technologies you already use. We hope you do and we’d love to see what you come up with.

It’s not every day that you get the opportunity to be a part of something that could quite literally change the way we do things.

Last week was GDC, one of the largest game-related events in the world! We made sure a handful of game-heads from Mozilla were in attendance to take everything in and represent game development with JavaScript and HTML5. This post is a round-up of the week from a personal perspective, much like the recent post about the Mozilla HTML5 games work week.

Experimenting with modern web technologies is always fun. For my latest project, I came up with the following requirements:

Not a complex game, rather a proof-of-concept

Space shooter theme

Canvas-based rendering, but no WebGL

Re-use of existing sprites (I am no artist)

Rudimentary audio support

AI/Bots

Working network multiplayer

I worked on this project only in my spare time; after approx. two weeks, the work was done: Just Spaceships! This article describes some decisions and approaches I took during the development. If you are interested in the code, you can check the relevant Google Code project.

Animation & timing

There are two basic ways to maintain an animation: requestAnimationFrame and setTimeout/setInterval. What are their strengths and weaknesses?

requestAnimationFrame instructs the browser to execute a callback function when it is a good time (for animation). In most browsers this means 60fps, but other values are used as well (30fps on certain tablets). Deciding the correct timing is up to the browser; the developer has no control over it. When the page is in background (user switches to another tab), animation can be automatically slowed down or even completely stopped. This approach is well suited for fluent animation tasks.

setTimeout instructs the browser to execute your next (animation) step after a given time has passed; the browser will try to fulfill this request as precisely as possible. No slowdowns are performed when the page is in background, which means that this approach is well suited for physics simulation.

To solve animation-related stuff, I created a HTML5 Animation Framework (HAF), which combines both approaches together. Two independent loops are used: one for simulation, the second one for rendering. Our objects (actors in HAF terminology) must implement two basic methods:

/* simulation: this is called in a setTimeout-based loop */
Ship.prototype.tick=function(dt){var oldPosition =this._position;this._position += dt *this._velocity;return(oldPosition !=this._position);}/* animation: this is called in a requestAnimationFrame loop */
Ship.prototype.draw=function(context){
context.drawImage(this._image,this._position);}

Note that the tick method returns a boolean value: it corresponds to the fact that a (visual) position of an object might (or might not) change during the simulation. HAF takes care of this – when it comes to rendering, it redraws only those actors that returned true in their tick method.

Rendering

Just Spaceships makes extensive use of sprites; pre-rendered images which are drawn to canvas using its drawImage method. I created a benchmarking page which you can try online; it turns out that using sprites is way faster than drawing stuff via beginPath or putImageData.

Most sprites are animated: their source images contain all animation frames (such as in CSS sprites) and only one frame (rectangular part of the full source image) is drawn at a time. Therefore, the core drawing part for an animated sprite looks like this:

By the way: the sprites for Just Spaceships! have been taken from Space Rangers 2, my favourite game.

The golden rule of rendering is intuitive: redraw only those parts of the screen which actually changed. While it sounds pretty simple, following it might turn out to be rather challenging. In Just Spaceships, I took two different approaches for re-drawing sprites:

Ships, lasers and explosions use technique known as “dirty rectangles”; when an object changes (moves, animates, …), we redraw (clearRect + drawImage) only the area covered by its bounding box. Some deeper bounding box analysis must be performed, because bounding boxes of multiple objects can overlap; in this case, all overlapping objects must be redrawn.

Starfield background has its own layer (canvas), positioned below other objects. The full background image is first pre-rendered into a large (3000×3000px) hidden canvas; when the viewport changes, a subset of this large canvas is drawn into background’s layer.

Sound

Using HTML5 audio is a piece of cake – at least for desktop browsers. There were only two issues that needed to be taken care of:

File format – the format war is far from being over; the most universal approach is to offer both MP3 and OGG versions.

Performance issues on certain slower configurations when playing multiple sounds simultaneously. More particularly, my Linux box exhibited some slowdowns; it would be best to offer an option to turn the whole audio off (not implemented yet).

Multiplayer & networking model

Choosing a proper networking model is crucial for user experience. For a realtime game, I decided to go with a client-server architecture. Every client maintains a WebSocket connection to central server, which controls the game flow.

In an ideal world, clients would send just keystrokes and the server would repeat them to other clients. Unfortunately, this is not possible (due to latency); to maintain consistency and synchronicity between players, it is necessary to run the full simulation at server and periodically notify clients about various physical attributes of ships and other entities. This approach is known as an authoritative server.

Finally, clients cannot just wait for server packets to update their state; players need the game to work even between periodical server messages. This means that browsers run their version of the simulation as well – and correct their internal state by data sent by server. This is known as a client-side prediction. A sample implementation of these principles looks like this:

Multiplayer & server

Choosing a server-side solution was very easy: I decided to go with v8cgi, a multi-purpose server-side javascripting environment, based on V8. Not only it is older than Node, but (most importantly) it was created and maintained by myself ;-).

The advantage of using server-side JavaScript is obvious: game’s server runs the identical code that is executed in browser. Even HAF works server-side; I just turned off its rendering loop and the simulation works as expected. This is a cool demonstration of client-server code sharing; something we will probably see more and more in the upcoming years.

Modulus

In order to make the game more interesting and challenging, I decided that the whole playing area should wrap around – think of the game universe as of a large toroidal surface. When a spaceship flies far to the left, it will appear from the right; the same holds for other directions as well. How do we implement this? Let’s have a look at a typical simulation time step:

However, there is a glitch. To see it, we have to solve this (elementary school) formula:

(-7)%5=?

JavaScript’s answer is -2; Python says 3. Who is the winner? The correctness of the result depends on the definition, but for my purpose, the positive value is certainly more useful. A simple trick was necessary to correct JavaScript’s behavior:

Lessons learned

Here is a short summary of general tips & hints I gathered when developing JS games in general, but mostly during Just Spaceships development:

Know the language! It is very difficult to create anything without properly understanding the programming language.

Lack of art (sprites, music, sfx, …) should not stop you from developing. There are tons of resources for getting these assets; the need for original art is relevant only it later stages of the project.

Use the paper, Luke! You know what my favorite development tools are? A pen and a sheet of squared paper.

If you are not 100% sure about the whole game architecture, start with smaller (working) parts. Refactoring them later to form a larger project is natural, useful and easy.

Collect feedback as soon as possible – at the end of the day, it is the users’ opinion that matters the most.

TODO

As stated before, Just Spaceships is not a complete project. There is definitely room for improvements, most notably:

Before I start I should point out that I won’t be making any dramatic announcements from Mozilla or going into too much detail about some of the decisions that were made during the work week. While some important discussions and decisions did occur, some aren’t quite ready to be made public yet. However, we will be posting specifically about some of these things in the coming weeks so keep an eye open. The purpose of this post is to give an overview of the week’s events and some of the discussions that were made.

So with that out of the way, let’s crack on.

Why a HTML5 games work week?

With the amount of buzz happening around HTML5 and JavaScript, it could be seen as surprising that we hadn’t organised a get-together of all the interested parties within Mozilla sooner. The long and short of it is that it has only been recently that we have begun to construct a consistent vision and commitment to HTML5 games.

To create a vision we need to find out who within the Mozilla community is actually interested in games and how we can work together to lay the path towards the future. The HTML5 games work week was set up to start the ball rolling. It’s safe to say that Mozilla cares a lot about games and this work week is just the beginning.

Aside from the internal interest and passion to build games, there are an increasing number of HTML5 games being released in the Web community each and every week. This shows that there is desire for these types of games and for browsers to work out how to support them.

It’s also clear that there are areas of improvement that need to be addressed within all browsers so that HTML5 games can flourish in the future. The point of the work week has been to bring all interested parties together in one place to make stuff happen and actually make decisions about how to address these issues.

From another perspective, the work week gave Mozilla the chance to listen to external developers and hear about the issues they are experiencing in the real world of HTML5 game development. We don’t develop games every day, so it is of absolute importance for us to fully understand the concerns of the people who do. Without this type of feedback we simply won’t be able to make our improvements as useful as they could be.

Who was there?

The work week was split into 2 parts: the first was attended by Mozilla staff and key Mozilla contributors, and the second included some invited external guests.

It was clear from the beginning that we had a whole bunch of people within Mozilla who were passionate about games and wanted to see us commit to them. I already knew a handful of these people, but I had absolutely no idea that so many Mozillians from so many different areas of the organisation were so interested. It great to see all those people in one (big) room working together to make awesome stuff happen.

The invited guests for the second part of the work week included developers from game studios like EA, as well as individual developers who have proven themselves experts in specific technologies, like WebGL. We worked with these guests to hear about issues from the guys dealing with this stuff every single day. It was absolutely fascinating to hear their concerns and work with them to find ways of improving the situation for games on the Web.

What was discussed?

Throughout the week we discussed a whole variety of topics ranging from how to better engage with the game developer community all the way to narrowing down fine-grained performance improvements to be made in JavaScript.

Overall, both the internal Mozilla staff and invited guests covered the following areas and many more (in no particular order):

Looking into the issue of mobile and how its requirements differ from those on the desktop

Learning about WebRTC and how it will provide the functionality for peer-to-peer communication

Looking at how games tie into the Mozilla Marketplace and the questions that Mozilla needs to answer in this area

All in all we covered a huge amount of stuff in the space of just a few days. I won’t list each and every decision that was made but rest assured that you’ll be hearing about them in the near-future as they get finalised and released.

In the meantime, please check out the Are We Fun Yet site as it’s where you’ll be able to keep tabs on the status of game-related technology within Mozilla. There are also some links to the detailed feedback and notes taken during this work week.

External demos and code review

During the final 2 days of the work week we saw demos from our external guests and worked with them to review specific issues that they have been experiencing in Firefox.

These demos proved invaluable for those within Mozilla to get a better idea of how professional game developers are using the technologies on the Web. We were inspired and amazed by the quality of these demos and the kind of games that could be created by pushing JavaScript right to the very limits.

As a result of these demos and code reviews, issues have been pinned down, test-cases written, and in some cases even fixes produced there and then. It was really a very useful aspect of the work week that we clearly need to explore further in the future.

What’s next?

So, what does that future look like for Mozilla and game development? Frankly, very bright indeed. There is so much interest and desire within Mozilla to further the Web as a platform for games. It’s clear that we have the knowledge and resources, we just need to work out how best to apply them.

While we work on refining our position on games and exactly how we’re going to tackle them, there are a few things that I can say for sure about the future.

The first is that we’re dedicated to helping developers learn how to use Web technologies, that’s the reason why MDN and this blog exist. We’ve started to work on ways to help developers learn how to use Web technologies to create games. We’re not quite there yet but it’s a priority.

Aside from MDN, we are going to be reaching out to developers at game-related events. At some events we’ll be even bringing along hardware for you to play with and hack with JavaScript APIs, like gamepads for the Gamepad API. We’ll also be present at some of the larger events like GDC so we can keep up to speed with the game world and talk to developers about the issues they’re facing.

Documenting the game-related decisions and discussions within Mozilla is key, and it is something that we’re working on improving. For now the best place to keep up to date is this blog, the #games channel on irc.mozilla.org, and the HTML5 games page on the Mozilla wiki.

With recent announcements it’s clear that Boot to Gecko (B2G) and games will be an area that Mozilla and the developer community will explore in the near future. It will be interesting to see what people do with a mobile platform as open and hackable as B2G.

I could go on but it’s safe to say that there is a whole bunch of stuff happening around games at Mozilla, it’s an exciting time indeed. No doubt we’ll have another work week in the coming months to take stock and take more steps in the right direction.