Safari 5 tested: Chrome, Opera still have JavaScript edge

Apple released a new version of Safari on both Mac OS X and Windows, …

Apple released the latest major version of its Web browser, Safari 5, earlier this week. Along with several new features—one of them somewhat controversial—the update was said to pack a number of performance improvements, including DNS pre-fetching and optimizations to Safari's Nitro JavaScript engine. "Safari continues to lead the pack in performance," Apple SVP of worldwide marketing, Phil Schiller, said in a statement. We decided to put those claims to the test, pitting Safari against leading browsers on both Mac OS X and Windows 7.

Included in our tests are a number of JavaScript benchmarks, including the WebKit team's SunSpider, Google's V8 Benchmark, and Mozilla's Dromaeo. We also took a look at graphics acceleration performance using Microsoft's HTML5 "Flying Images" speed demo. The tests were run on the latest stable versions of Safari, Chrome, Firefox, and Opera on Mac OS X 10.6.3 running on a 3.0GHz, 8-core Mac Pro with 10GB RAM and two ATI Radeon 2600XT GPUs. The tests were also run on Windows 7 on a 2.67 GHz Core 2 Duo PC with 4 GB of RAM and an ATI 4830 GPU. On our Windows 7 rig we had a recent development build of Chrome (as opposed to the latest stable version), and we also ran the tests using IE8 and a developer preview of IE9.

Right off bat, our tests more or less confirmed Apple's JavaScript speed claims on Mac OS X. "Powered by the Nitro JavaScript engine, Safari 5 on the Mac runs JavaScript 30 percent faster than Safari 4, three percent faster than Chrome 5.0, and over twice as fast as Firefox 3.6," according to a statement released by the company. We found Safari to be about four percent faster than Chrome, and slightly faster than Opera. However, Firefox continues to lag significantly behind the other three browsers on raw JavaScript performance—Safari was nearly three times faster on our testbed.

However, looking at the other common JavaScript benchmarks, Safari didn't fare so well. Chrome was nearly twice as fast as Safari running the V8 Benchmark suite, which attempts to simulate the kind of code that large-scale, next-generation Web apps would use. Even Opera bested Safari on these tests by a significant margin. Unsurprisingly, Firefox trailed much farther behind the other three browsers—Safari is 6x faster, Opera is 8x faster, and Chrome is nearly 12x faster on these benchmarks.

The most surprising results, however, come from the Dromaeo JavaScript tests. Safari was more than twice as fast as Firefox, but Opera was nearly 11 times faster than Safari, and Chrome was a whopping 23 times faster. Looking closer at the individual tests, Both Chrome and Opera performed far, far better on regular expressions tests than Safari. Optimizing regular expressions was something that the WebKit team had done two years ago to push the performance of earlier versions of its Nitro (née SquirrelFish Extreme) engine, but both Opera and especially Chrome have made significant improvements since then.

Our own Ryan Paul explained that V8's approach to regular expressions spends more time on regex compilation to extract faster performance. Regexes aren't necessarily that common on all websites, but are commonly used for form input validation. Google's implementation provides the biggest boost to regexes that are iterated repeatedly over larger data sets—quite similar to those used in benchmarking. The benchmarks may weigh the results a little too heavily on regex performance, but from the looks of it, the WebKit team definitely has its work cut out for it to catch up on this particular optimization.

Another surprising result came from running one of the speed demos from Microsoft's Internet Explorer 9 Test Drive page. The Flying Images demo uses a number of browser icons arranged in three-dimensional space, and spins the arrangement around. Pointing the cursor in the middle of the browser window forces the perspective to exaggerate, taxing the graphics drawing code. Chrome only managed to animate five frames per second. Safari was twice as fast at 11fps, but Firefox was faster still at 17fps. Opera was the dark horse in this test, turning in a staggering 51fps. To Safari's credit, however, it rendered the demo far better than Opera. On the other hand, Firefox's rendering was just as good with much smoother animation. Given Apple's focus on accelerating 3D CSS Transforms and otherwise leveraging the GPU for more performance, and its push to replace Flash with HTML5 and related standards, it seems unusual that Safari didn't perform better on this test.

On Windows, Safari ran just slightly slower than the development build of Chrome 6 installed when running the SunSpider benchmarks—a sign that Chrome's fast pace of development is retuning dividends. No surprise that IE8 just plain sucked, but the preview of IE9 shows promise. The patterns on V8 Bench and Dromaeo were repeated on Windows, with Chrome having a massive lead on these tests. IE9 again shows some promise of decent performance, besting Firefox, but it's still not on par with either Safari or Opera.

The Flying Images demo results again surprised us. Apple specifically mentioned limited GPU acceleration as one of the features for the Windows version of Safari 5. However, Safari tied for dead last place on this test with Chrome—which hasn't done much if anything to leverage GPU acceleration—at a measly 3fps. Firefox was far faster at 12fps, and Opera was far faster yet again with 28fps. However, IE9 turned in a mind-blowing 64fps. Again, given Apple's promotion of newer Web standards for animation, the fact that this demo runs 6 times slower on Mac OS X on much more capable hardware has to be embarrassing.

Overall, Safari 5 is quite snappy, and does feel faster than Safari 4—especially on Windows. Though Safari's Nitro JavaScript engine may have a very slight raw performance advantage, the benchmarks we ran show that Apple and the WebKit team certainly have some areas where it can improve—both in JavaScript performance characteristics as well as optimizing performance for animation and other graphics rendering. All this fierce competition among Apple, Microsoft, Google, Mozilla, and Opera for "fastest" browser, though, ultimately benefits all users in the long run.

That Flying Images benchmark is very inconsistent. I found that if I moved the mouse so that the icons were at the top of the screen and then moved it back down, I could get 60 fps in Safari 5 on my Mac for 5 or 10 seconds and then the fps would drop back down.

How did you do the flying images test? I get 60 FPS with Firefox 3.6.6 on Linux.

From the article:

Quote:

The Flying Images demo uses a number of browser icons arranged in three-dimensional space, and spins the arrangement around. Pointing the cursor in the middle of the browser window forces the perspective to exaggerate, taxing the graphics drawing code.

So move the cursor to the middle of the browser window to move the shape to the middle of the screen; I verified that the fps slowed way down on Mac OS X by testing on a different machine that the Mac Pro used for benchmarking.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

Huh every variation I have tried with the flying images demo I have been able to get a nearly consistent 60 FPS on the various Mac OS X systems I have (Mac Pro, iMac, MacBook Pro). 60 is frame rate for the display, OS limits update rate to that unless disabled.

Update...

Looks like bumping up to larger images and/or more animated elements gets the frame rate to drop radically on my Mac Pro will a rather low end video adapter. Don't have the time to profile things with developer tools to see what may be the limiting factor.

So move the cursor to the middle of the browser window to move the shape to the middle of the screen; I verified that the fps slowed way down on Mac OS X by testing on a different machine that the Mac Pro used for benchmarking.

Well that exactly what I did, wonder why it's so fast on my computer. :-) Got bleeding edge open-source graphics drivers for my Radeon 4770 and run under Compiz. (Ubuntu 10.04 64-bit, Firefox 3.6) I noticed that it's actually capped at 60 fps, probably vsync. If I bump the test to 256 icons I get 40 fps.

It's good to see this fight, although I can't really understand how Opera is still around.

Quote:

All this is about JS performance, which doesn't really effect me that much.

Sure it does. Your average news website will have at least a few hundred lines of javascript, and web applications like Facebook and GMail have thousands. Rendering speed of CSS and HTML reached the point of diminishing returns on most browsers a while ago.

How did you do the flying images test? I get 60 FPS with Firefox 3.6.6 on Linux.

From the article:

Quote:

The Flying Images demo uses a number of browser icons arranged in three-dimensional space, and spins the arrangement around. Pointing the cursor in the middle of the browser window forces the perspective to exaggerate, taxing the graphics drawing code.

So move the cursor to the middle of the browser window to move the shape to the middle of the screen; I verified that the fps slowed way down on Mac OS X by testing on a different machine that the Mac Pro used for benchmarking.

Did you change the number of images in the animation?with the default it is 60fps on my macbook pro but increasing it to 100 cuts is down to about 7fps

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

That was only for Chrome and only for the Win7 test. Chrome on OS X and Opera on both platforms still beat Safari comparing the current releases. To bring it back to your analogy it's comparing the 1978 Ferrari Formula 1 car to 9 different 2010 models.

On that note, why did Ars select the nightly dev build instead of latest stable for Chrome on Win7?

When the Flying Images first loaded for me (OS X 10.6.3, 2.8GHz C2D Late '08 Unibody MBP with Safari 5.0 (6533.16, r60868)), the frame rate was terrible, but once it had fully loaded the images and started spinning them, I was exceeding 50fps.

Even moving the set of icons around I never dropped below 40fps. I don't understand why your frame rate was so low in the testing you did. Granted, I'm using a WebKit nighly and not a vanilla Safari 5.

So move the cursor to the middle of the browser window to move the shape to the middle of the screen; I verified that the fps slowed way down on Mac OS X by testing on a different machine that the Mac Pro used for benchmarking.

Well that exactly what I did, wonder why it's so fast on my computer. :-) Got bleeding edge open-source graphics drivers for my Radeon 4770 and run under Compiz. (Ubuntu 10.04 64-bit, Firefox 3.6) I noticed that it's actually capped at 60 fps, probably vsync. If I bump the test to 256 icons I get 40 fps.

I'll have to test it on my 10.04 64-bit when I go home for lunch. I have a GeForce 9800GTX running the proprietary drivers though.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

That was only for Chrome and only for the Win7 test. Chrome on OS X and Opera on both platforms still beat Safari comparing the current releases. To bring it back to your analogy it's comparing the 1978 Ferrari Formula 1 car to 9 different 2010 models.

On that note, why did Ars select the nightly dev build instead of latest stable for Chrome on Win7?

Plain silly to compare devs with stable releases. I guess Ars wanted to hurt Apple a bit, even if only by words "the other is better, try again" because they REALLY don't like the Reader feature of Safari 5 (according to the article, and especially Ars's staff comments in the discussion: http://arstechnica.com/apple/news/2010/ ... e-ipad.ars). Guess low blows are good for everyone huh? Please, stop that kind of practice, Ars'worth better than that.

However, I agree with the conclusion of the article: "All this fierce competition among Apple, Microsoft, Google, Mozilla, and Opera for "fastest" browser, though, ultimately benefits all users in the long run".

.... the fact that this demo runs 6 times slower on Mac OS X on much more capable hardware has to be embarrassing.

I disagree. All of these browsers are in flux. One month Chrome might lead in speed, but be lax in certain other areas by comparison to Safari or Firefox. The next month Firefox might star in certain areas and not in others, and then the next month, Safari.

Speed is important, but with such a diverse user base for the Internet, and with things constantly improving, a star one month is not so the next. Support for current standards and emerging standards for example, is key for me, with speed being important, but I know that Firefox will get another spike soon, and then Safari, and then Chrome.

All this is about JS performance, which doesn't really effect me that much. General browsing does seem faster on 5. Seems faster than FF, about the same as Chrome.

Maury

These days the other statistics are basically not important. The WebKit engine in Google Chrome can render something like 2700 web pages a second. The performance of a web browser these days is mostly bottlenecked by JavaScript performance and network latency, and JavaScript is the only one that we can get much of a useful metric on.

Firefox 3.7 nightly 64-bit with Direct2D enabled on Win7 can also do 60fps even if you move your mouse like crazy.With 256 images it does 30fps on my really average machine (q6600 ,nvidia 7600gs).And it rocks SVG demos too.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

That was only for Chrome and only for the Win7 test. Chrome on OS X and Opera on both platforms still beat Safari comparing the current releases. To bring it back to your analogy it's comparing the 1978 Ferrari Formula 1 car to 9 different 2010 models.

On that note, why did Ars select the nightly dev build instead of latest stable for Chrome on Win7?

Plain silly to compare devs with stable releases. I guess Ars wanted to hurt Apple a bit, even if only by words "the other is better, try again" because they REALLY don't like the Reader feature of Safari 5 (according to the article, and especially Ars's staff comments in the discussion: http://arstechnica.com/apple/news/2010/ ... e-ipad.ars). Guess low blows are good for everyone huh? Please, stop that kind of practice, Ars'worth better than that.

However, I agree with the conclusion of the article: "All this fierce competition among Apple, Microsoft, Google, Mozilla, and Opera for "fastest" browser, though, ultimately benefits all users in the long run".

What's being lost in the complaints about the Chrome dev build is that it's only one of the tested browsers. Ultimately, Ars tested 10 browsers, one of them was a dev build. Let's say we cut the data on that one browser out, you're still left with Opera OS X, Opera Win7, and Chrome OS X beating Safari in the testing. 7 competitors to Safari and 1 dev build of a competitor were tested, 3 competitors and 1 dev build could be said to have "beaten" Safari. With those results, I don't understand how anyone could agree that Safari "leads the pack."

Like most benchmarks, these are rather meaningless for real-world performance. Can't somebody put together a benchmark of common real-world user tasks and measure those? Like applying a bunch of tags in gmail and archiving 100 messages, etc. It can't be that difficult to do that. This would make a lot more sense than running 100,000 iterations of some regexp matching.

Firefox 3.7 nightly 64-bit with Direct2D enabled on Win7 can also do 60fps even if you move your mouse like crazy.With 256 images it does 30fps on my really average machine (q6600 ,nvidia 7600gs).And it rocks SVG demos too.

Like most benchmarks, these are rather meaningless for real-world performance. Can't somebody put together a benchmark of common real-world user tasks and measure those? Like applying a bunch of tags in gmail and archiving 100 messages, etc. It can't be that difficult to do that. This would make a lot more sense than running 100,000 iterations of some regexp matching.

These benchmarks are supposed to simulate the kind of hammering future web applications will do to browsers. It's not just RegExp matching, but lots of tests on basic JavaScript operations that are used in everything. In my experience as a JavaScript developer, the real world performance I see out of these browsers is about the same ratios you'll see here--Chrome with a slight edge, Safari 5 not far behind, Firefox is much slower than the rest, and IE8 is just hilariously slow.

These tests are supposed to simulate future demands for a reason--IE and its slow ass JavaScript engine are real bottlenecks for developers right now, including IE8. I've spent many hours toiling away profiling and rewriting code because IE was slow to the point where I couldn't show it to my boss as-is, while the other browsers were just fine. It's very important to prepare these browsers for growth and as such they should be held to this arbitrarily high standard. IE6 is still kicking after all these years, but had browsers been held to such a high standard way back when we might be a lot more advanced than we are.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

You Apple fanboys need to take a chill pill. I've got the current release version of Chrome on my PC (XP) and it's at version 5.0.375.70

I have a 2.93Ghz Intel Core 2 Duo MacBook Pro with 8GB of ram and running with the mere integrated graphics. (The discreet graphics that comes with this machine is disabled.)

The Flying Images benchmark always performed like a champ in Safari 5, at around 60 FPS - sometimes a tad lower, sometimes a tad higher. The only way I could slow it down was to move my mouse pointer to the bottom of my screen to have my dock pop up over the browser window, and even then it only dipped down to between 39 to 44 FPS.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

You Apple fanboys need to take a chill pill. I've got the current release version of Chrome on my PC (XP) and it's at version 5.0.375.70

I like Apple products but deplore a lot of Apple decisions so do quit with the 'fanboy' bollocks. I have the latest release version of Chrome on OSX and the version number is what I posted above. But Ars aren't comparing stable releases and that's the point.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

You Apple fanboys need to take a chill pill. I've got the current release version of Chrome on my PC (XP) and it's at version 5.0.375.70

I like Apple products but deplore a lot of Apple decisions so do quit with the 'fanboy' bollocks. I have the latest release version of Chrome on OSX and the version number is what I posted above. But Ars aren't comparing stable releases and that's the point.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

Before you make a comment like this, you should at least check your facts; 5.0.375.70 is the latest release build of Chrome. The lousy analogy just makes you sound like a jerk--next time, try omitting it.