JavaScript Engine Speeds

Recently, I’ve been spending a lot of time analyzing the speed of pure JavaScript engines, looking at how well they perform and what their particular strengths and weaknesses are. To start with, I analyzed the bleeding-edge code from:

Right now I’m only looking at pure, JavaScript-only, tests (no tests of DOM or other APIs) and am NOT looking at the speed of the browsers’ native JavaScript engine implementations. (So, even though you may see a speed for a particular engine, that does not directly correlate to the speed of the JavaScript running within the browser itself. There’s always a significant amount of overhead required to run JavaScript code seurely within a browser, thus the efficiency of that security layer will frequently become a deciding factor in the results.

The four engines that I picked all had complete JavaScript implementations and usable JavaScript shells (that way I could feed my tests in and have them cleanly run).

To browse the results I’ve pulled together a simple application that can be used to view a representation of the data from all the major JavaScript engines paired with the code from the tests which run them.

Right now the browser works fine in Firefox, is quirky in Opera and Safari, and explodes in IE (it requires canvas support). I’ll finesse it into shape when I have a little more time this week.

Note: This demo uses a bunch of functionality from the new jQuery UI library, including themes, tabs, accordion, and resizables.

that’s a great ! I have started to implement a JS compiler for the Erlang virtual machine (right now it just compiles/runs a very small subset of Ecma 262, and haven’t released the code yet). So if things go as planned (but they usually don’t) you can add another engine to your charts in a few weeks …

If I read your graphs right, it looks like on most tests you run, Spidermonkey and Tamarin are roughly on par (with a few notable exceptions), while Rhino and JSCore tend to lag well behind, with Rhino worse than JSCore. Is that a fair summary?

What versions of the various JavaScript engines are being tested here? Looking at the testkjs executable in the SVN repository you reference, it is linked against /Users/jresig/Code/WebKit/WebKitBuild/Release/JavaScriptCore.framework/Versions/A/JavaScriptCore, which clearly will not exist for anyone interested in reproducing your results. The dynamic linker will fall back to whatever version of JavaScriptCore.framework people happen to have on their system which would lead to wildly different results.

> Thereâ€™s always a significant amount of overhead required to run JavaScript code seurely within a browser, thus the efficiency of that security layer will frequently become a deciding factor in the results.

… and Firefox is pretty bad at it. And the more extensions you install, the slower it gets…

Which is fastest seems to depend on which benchmark you look at. JSC seems to do pretty well on array and string operations (with a few exceptions), excellent on regexp operations, and not so good on things that are math-heavy (which most of the “real world” tests are.

John, I assume you’ll be filing bugs on cases when Tamarin is slower than SpiderMonkey, right? Array.unshift comes to mind, as do most of the regexp tests, some of the String tests, the partial-sums test, the Fib test.

It’s probably not really the case, but the time taken by Spidermonkey seems to increase linearly, while the time taken by Tamarin seems to increase exponentially. Do you fancy running a larger number of iterations? :-)

@Peter: It all depends on the test. I’m trying to build some better aggregate views of the data, but I’m wary of presenting false assumptions. I’ll figure something out and get it up online.

@Mark: Ah, damn! Yeah, I built a copy of JavaScriptCore from the WebKit SVN just before I ran these tests, yesterday. Hmm… I’ll have to try and think of another strategy for distributing that binary. Any thoughts are appreciated.

John, would you be interested in contribution of more tests for this benchmark? I have a huge pile of JavaScript benchmarks from the web and I think some of them cover interesting things that aren’t covered by these tests (which nontheless are pretty awesome).

Also, would you be interested in contribution of a harness for browser-based testing? I don’t think there will be a big difference in relative performance from the raw engine numbers, but it would allow fair comparisons to the JS engines in IE, Opera and other browsers.

@steve: I’m not sure offhand, that’s my area of expertise, but I suspect that there must be something (or, at least, a way to build a simple CLI to JSript). Anyone with the knowhow is welcome to pick this up!

@Asd: Don’t get it wrong – Rhino is quite speedy, it’s just that these other engines are speedier. :-)

John, you should be able to “fix” testkjs by including the built JavaScriptCore.framework along side it in the dep directory and then running the following command to update the linker search path for the framework:

An easy way to verify that this has worked would be to attempt to run testkjs on a Mac OS X machine without the Safari 3 beta installed. Prior to this command, or if the command does not do what is intended, testkjs will refuse to run with an error from dyld.

John – those tests are interesting and quite useful, though there were some surprises: Tamarin’s regex engine appears to be unusually slow! However, I’d have two requests for the tool itself (when you get the chance):

1. Allow bookmarks to particular views of the tool, so I can link to eg. the graph for Array constructors.

2. Have some sort of way to get at the specific value of each bar in the graph, whether it’s by mouseover or what have you. It’s somewhat frustrating to look at a graph like that for “Array Construction, new Array()” and not be able to have any idea about the values for the rest of the graph because one engine (in this case, Rhino) is such an outlier.

Anyway, I know these features are both somewhat complicated to build, but I thought I’d get that in there for the record.

Sun’s JVM’s use a period of profiling code-use before they perform optimizations. The -server mode you used is intended for long-running server applications and sacrifices initial speeds for greater eventual improvements after a long time. (I read once that the server VM needs 10000 executions before it will optimize a given section of code).

@BC: I haven’t seen the speed costs that you mention – the server flag seemed to help, when I tried it. If you’re interested, you could run the tests to see if that flag really is a significant factor. Let me know if it is!

This is really cool John. I especially like the way you show the js source of the tests. However, it seems the js shown is not always complete or correct. For example, the code for the n-body test ends with “for (i = 0; i”.

In Rhino, i just noticed that String.replace becomes slow (and I mean _pathetically_ slow) when the string you’re performing the replacements on is large(ish). I tried s = s.replace(sFrom, sTo, 'g'); where s.length was 964949, sFrom.length == 3, sTo.length == 4 — and it turned out that var a = s.split(sFrom); s = a.join(sTo); was faster by a factor of over 1600 (!!). There is clearly some sort of anomaly here.
Btw. s = ( new java.lang.String(s) ).replace( sFrom, sTo ); did reasonably well, but still took about 70 percent more time than using Array.split. (There were about 9000 instances of s.From in the original string s).

The reason that made me test this was that in one script, I had tried to perform a replace on a string of length about 3MB. Several _hours_ later it still hadn’t completed (and was hogging the cpu 100%), so I killed the jvm and was left wondering whether there might be some infinite loop bug there. But no, it is just that ridiculously slow.

(Actually, it ought to work even without the //E flag; cscript.exe should recognize the .js extension. But google tells me there are applications that clobber the relevant registry entries, so explicitly saying //E:jscript is safer.)

Comments are closed.
Comments are automatically turned off two weeks after the original post. If you have a question
concerning the content of this post, please feel free to contact me.