Are We Fast Yet?

Mozilla constantly asks this question, but their answer really doesn't tell us all that much. Yes, JavaScript has been getting faster and faster and the v8bench, kraken and sunspider benchmarks are a good indicator for that. Yet, they all miss some issues that are quite important for HTML5 Gaming.

All JavaScript engines periodically run a Garbage Collector to release unused memory. In some cases this GC step can take several hundred milliseconds, in which the game comes to a complete halt. Even if that only happens every 600 frames, or 10 seconds, it's far more annoying than a game that runs at a lower, but constant framerate.

GC pauses have been a huge problem with Firefox 3.6 and Firefox 6 is still the worst offender here. Other browsers have shorter GC pauses, but there's still room for improvement. To gain some more insight, I constructed a new benchmark that essentially simulates a playthrough for a game level.

Now, to be clear, this benchmark is very “opinionated”. It's by no means a complete benchmark for all HTML5 features, but only tests one very specific use case: smooth running games rendered with the <canvas> element. It's a very real use case though; not an average for hundreds or thousands of small tests.

The score for the HTML5-Benchmark takes the total time the browser spent rendering frames and a big penalty for pauses into account (the exact formula is 1000000/(sqrt(totalTime) + lagTime * 0.1)). The benchmark automatically runs at a “reasonable” screen size - i.e. a screen size that makes sense for a game on the particular device. Thus, the final score is a real indicator for the browser's ability to smoothly run HTML5 games.

A score of about 1500 means the game is playable. Note that the benchmark actually runs a bit slower on mobile devices than the game itself. This is because the game utilizes a special pre-rendered background mode on mobile devices, but doesn't do so in the benchmark to keep the scores comparable.

It's almost comical how bad the Android's Browser performed in my tests: the Samsung Galaxy Tab 10.1 is actually slower than a four year old 1st gen iPod Touch. Firefox 4 on the same tablet performs a lot better though.

The desktop version of Firefox 6 still suffers a lot from the occasional GC pauses. This makes Firefox the last choice if you want to enjoy HTML5 games. The performance decrease from the stable Chrome 13 to the dev version of Chrome 15 is quite strange too. It may have to do something with the new hardware acceleration for Canvas' 2d context.

There are some things that the benchmark can't measure. Opera for instance has a very steady timing and never misses a frame. On my machine however, the benchmark still looks choppy - probably because the benchmark's timer isn't synchronized with the refresh rate of the display. Opera essentially renders frames that it never displays. The new requestAnimationFrame() could fix this, but it hasn't arrived in all browsers yet.

IE9's smoothness on the other hand is remarkable. Of all browsers and systems I tested, IE9 subjectively produced the best results. It seems to precisely synchronize JavaScript execution with the refresh rate: the benchmark runs at exactly 60 FPS on my machine, even though the timer is scheduled every 16 milliseconds, or 62.5 FPS.

This leads me to ask a few questions: shouldn't browser vendors concentrate on fixing setInterval() first, before implementing requestAnimationFrame()? And shouldn't it behave more like setInterval() instead of setTimeout()? That is – I don't want to call a function to schedule each and every frame I want to draw, but just call it once and let the browser schedule the frames.

requestAnimationFrame() looks good on paper, but it doesn't actually produce visually better results than setInterval() in those browsers that implement it.

On my machine, Chrome 13 and Firefox 6 awkwardly jump between 120 and 60 FPS when using it (it's disabled in the benchmark) and even without GC pauses there are noticeable hiccups, while IE9 is able to produce perfectly smooth animations without it. At the moment requestAnimationFrame() is truly worthless for games.

Another much needed rant! I've come to optimize everything in my Javascript programs for results largely slower than a 486 from 1995. I'm tired of pooling, hoisting variable out of loops, hoisting array lengths and breaking encapsulation, without getting the nice stuff in Javascript (if something like even exists).

What version of Opera were you testing using? Every version since 10 has claimed to be 9.8 in the UA string (with the addition of Version/11.50 or whatever as the real version). I presume you're testing 11.50 — the latest stable release. Would be interesting to see how recent snapshots, available from <my.opera.com/desktopteam/blog/> do.

You should call this Canvas Benchmark or similar. The media is already in a muddle with what the hell HTML5 is and this'll only confuse matter more. According to this site on my machine IE9 is the greatest HTML5 browser on the planet.

@Geoffrey Sneddon: Yes, you are correct; I fell for that trap. I actually tested Opera 11.50, not 9.8.

@Ric: I thought a while about the name, but in the end decided for the catchy, "incorrect" one. And IE9 is indeed one of the best browsers to enjoy Canvas games on - it's one aspect of HTML5 that I care about.

Thank you so much! This will save me plenty hours of debugging time for the reason why my own little canvas game feels so "stuttering" and your benchmark shows those "render lag spikes" in a absolutely obvious way.

This is amazing. Its funny I've always been running the dev version of chrome (6398 on MBP 8,2), but I have to remember that not everyone is a crazy browserphile like I am. Actually there aren't and developers that are...

Mozilla is working on reducing GC pauses.
The first thing they seem to want to do is turn their GC into an incremental GC. This means they will set a limit on how long they can GC (say 1ms), but GC more often to compensate.bugzilla.mozilla.org/show_bug.cgi?id=641025

FYI: Firefox 7 (which launches tomorrow, September 26th) invalidates much of this article. Chrome CGs *VERY* frequently, Firefox 7 does CG very intelligently, and much more often, to avoid the negative performance outcomes stated in this article.

Sorry to crash the party, but the tales of CG performance issues in Firefox are apparently only going to be valid for another few hours ;)

strange... Debian here. My HTML5 game (in dev) runs around 60 fps in Chrome, and this is very stable (no noticeable GC locks). But FF7 is worse than ever, not slow, but too unstable (from 50 to 90 fps). I normally don't care much as I fix game dynamics depending on current FPS. What really suck is that FF misses some mouse move events due to this (VM too busy at 90Hz). Before FF7, it was a bit better, but now, it's just not acceptable. For the time being, I decided to switch back to an old setTimeout() method which is a bit better than mozRequestAnimationFrame() !! What the hell are they doing?! I used to love FF, but now, it is really getting on my nerves.

I hope they will fix that soon, I'm very close to recommand Chrome over FF for using my stuffs...

I had some strange results on my machine. IE9 scored about the same as it did in your test case (roughly 6400) and performed exceedingly smoothly. Chrome looked visually choppy yet scored around 8100. It almost appeared as though that it counted every tiny bit of stutter on IE9 as lag (which totaled ~330ms) but neglected to do the same for Chrome (which gave me a result of ~77ms, which seemed very low for how it looked).

Firefox 7 still doesn't perform particularly well. It generally holds 50-60fps, but it lags significantly more. I think it finished with about 1200ms lags total, but I stopped paying attention after it crossed the 1 second threshold. Its final score was around Firefox 6's -- roughly 4500 on my machine.

For any curious parties, my 4th generation iPod Touch scores 1895, putting it in like with the iPad. Sadly, I was unable to get the test to start on my Mango-equipped Windows Phone (the Impact logo appears and appears to go through, but after that goes away the canvas stays black; this happens when I use either the desktop or the mobile page), so I can't provide any mobile IE9 results.

Are you going to release this benchmark on github for example for open source? if not, i'm suggesting you a partnership in which we will create an app for social networks like facebook and vkontakte in which users can run this test inside app window ...

we will get a big amount of data for analyze on hardware + earn some money on advertising,

Request: Please add a "%" score of test-time over lag-time, with 4 or 5 decimal places, to your html5-benchmark. Eg, if the test took 500.3 s with 51234ms lag (51.234s), a value of "9.765% lag" would be displayed.

It seems that in the 9 months since you released this benchmark, some of the major browsers have done a lot to improve their scores. Is there any chance you could test them again & upload a newer set of scores so we can see where things currently stand? :)

Hay, really great blog. I think it is more informatics and useful site. All visitors are benefited to read this sit’s article. All article is Unique and relevant. I like this kind of blog. So I am searching by <a href=”bouncycastlehireliverpool.pen.io/”> inflatable hire</a> and I find this site. Thanks for admin who share that’s.