Infinite Loop —

Safari 5 tested: Chrome, Opera still have JavaScript edge

Apple released a new version of Safari on both Mac OS X and Windows, …

Apple released the latest major version of its Web browser, Safari 5, earlier this week. Along with several new features—one of them somewhat controversial—the update was said to pack a number of performance improvements, including DNS pre-fetching and optimizations to Safari's Nitro JavaScript engine. "Safari continues to lead the pack in performance," Apple SVP of worldwide marketing, Phil Schiller, said in a statement. We decided to put those claims to the test, pitting Safari against leading browsers on both Mac OS X and Windows 7.

Included in our tests are a number of JavaScript benchmarks, including the WebKit team's SunSpider, Google's V8 Benchmark, and Mozilla's Dromaeo. We also took a look at graphics acceleration performance using Microsoft's HTML5 "Flying Images" speed demo. The tests were run on the latest stable versions of Safari, Chrome, Firefox, and Opera on Mac OS X 10.6.3 running on a 3.0GHz, 8-core Mac Pro with 10GB RAM and two ATI Radeon 2600XT GPUs. The tests were also run on Windows 7 on a 2.67 GHz Core 2 Duo PC with 4 GB of RAM and an ATI 4830 GPU. On our Windows 7 rig we had a recent development build of Chrome (as opposed to the latest stable version), and we also ran the tests using IE8 and a developer preview of IE9.

Right off bat, our tests more or less confirmed Apple's JavaScript speed claims on Mac OS X. "Powered by the Nitro JavaScript engine, Safari 5 on the Mac runs JavaScript 30 percent faster than Safari 4, three percent faster than Chrome 5.0, and over twice as fast as Firefox 3.6," according to a statement released by the company. We found Safari to be about four percent faster than Chrome, and slightly faster than Opera. However, Firefox continues to lag significantly behind the other three browsers on raw JavaScript performance—Safari was nearly three times faster on our testbed.

However, looking at the other common JavaScript benchmarks, Safari didn't fare so well. Chrome was nearly twice as fast as Safari running the V8 Benchmark suite, which attempts to simulate the kind of code that large-scale, next-generation Web apps would use. Even Opera bested Safari on these tests by a significant margin. Unsurprisingly, Firefox trailed much farther behind the other three browsers—Safari is 6x faster, Opera is 8x faster, and Chrome is nearly 12x faster on these benchmarks.

The most surprising results, however, come from the Dromaeo JavaScript tests. Safari was more than twice as fast as Firefox, but Opera was nearly 11 times faster than Safari, and Chrome was a whopping 23 times faster. Looking closer at the individual tests, Both Chrome and Opera performed far, far better on regular expressions tests than Safari. Optimizing regular expressions was something that the WebKit team had done two years ago to push the performance of earlier versions of its Nitro (née SquirrelFish Extreme) engine, but both Opera and especially Chrome have made significant improvements since then.

Our own Ryan Paul explained that V8's approach to regular expressions spends more time on regex compilation to extract faster performance. Regexes aren't necessarily that common on all websites, but are commonly used for form input validation. Google's implementation provides the biggest boost to regexes that are iterated repeatedly over larger data sets—quite similar to those used in benchmarking. The benchmarks may weigh the results a little too heavily on regex performance, but from the looks of it, the WebKit team definitely has its work cut out for it to catch up on this particular optimization.

Another surprising result came from running one of the speed demos from Microsoft's Internet Explorer 9 Test Drive page. The Flying Images demo uses a number of browser icons arranged in three-dimensional space, and spins the arrangement around. Pointing the cursor in the middle of the browser window forces the perspective to exaggerate, taxing the graphics drawing code. Chrome only managed to animate five frames per second. Safari was twice as fast at 11fps, but Firefox was faster still at 17fps. Opera was the dark horse in this test, turning in a staggering 51fps. To Safari's credit, however, it rendered the demo far better than Opera. On the other hand, Firefox's rendering was just as good with much smoother animation. Given Apple's focus on accelerating 3D CSS Transforms and otherwise leveraging the GPU for more performance, and its push to replace Flash with HTML5 and related standards, it seems unusual that Safari didn't perform better on this test.

On Windows, Safari ran just slightly slower than the development build of Chrome 6 installed when running the SunSpider benchmarks—a sign that Chrome's fast pace of development is retuning dividends. No surprise that IE8 just plain sucked, but the preview of IE9 shows promise. The patterns on V8 Bench and Dromaeo were repeated on Windows, with Chrome having a massive lead on these tests. IE9 again shows some promise of decent performance, besting Firefox, but it's still not on par with either Safari or Opera.

The Flying Images demo results again surprised us. Apple specifically mentioned limited GPU acceleration as one of the features for the Windows version of Safari 5. However, Safari tied for dead last place on this test with Chrome—which hasn't done much if anything to leverage GPU acceleration—at a measly 3fps. Firefox was far faster at 12fps, and Opera was far faster yet again with 28fps. However, IE9 turned in a mind-blowing 64fps. Again, given Apple's promotion of newer Web standards for animation, the fact that this demo runs 6 times slower on Mac OS X on much more capable hardware has to be embarrassing.

Overall, Safari 5 is quite snappy, and does feel faster than Safari 4—especially on Windows. Though Safari's Nitro JavaScript engine may have a very slight raw performance advantage, the benchmarks we ran show that Apple and the WebKit team certainly have some areas where it can improve—both in JavaScript performance characteristics as well as optimizing performance for animation and other graphics rendering. All this fierce competition among Apple, Microsoft, Google, Mozilla, and Opera for "fastest" browser, though, ultimately benefits all users in the long run.

158 Reader Comments

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

That was only for Chrome and only for the Win7 test. Chrome on OS X and Opera on both platforms still beat Safari comparing the current releases. To bring it back to your analogy it's comparing the 1978 Ferrari Formula 1 car to 9 different 2010 models.

On that note, why did Ars select the nightly dev build instead of latest stable for Chrome on Win7?

Basically because that's what Peter had installed on his machine, and I couldn't convince him to install the stable release.

That dev build isn't OMG amazingly faster than the stable release of Chrome, though. IT came out very slightly ahead of Safari 5 on SunSpider, but like squidz says, on V8 Bench and Dromaeo the stable version murdered Safari 5 on Mac OS X running on some serious hardware.

The js speed race is awesome for developers. As a user, FF, Safari & Chrome are all plenty fast in loading pages. My new benchmark is CPU intensity. I hate coming back to my sleeping Mac just to find it hot, low on battery, because browser has been sucking up the CPU. Safari 5 has lighter CPU footprint than Chrome. With Flash, the plugin will use 2X more CPU on Chrome than Safari while viewing the same page. Scrolling also takes less CPU on Safari than Chrome.

It would be useful to start profiling CPU intensity too. I fear that if the arms race for speed continues, browser will start draining my battery faster.

Why wasnt the earlier version of Safari included in the tests? As a Safari 4 user, am more interested in the delta performance improvement that safari 5 offers.. If the it was tested in an earlier article, can it please be linked?

One of the techniques that Firefox uses to achieve this performance is to slightly degrade the quality of the images during scaling, while the other browsers attempt to maintain full image quality. One of the challenges with not being hardware accelerated is that you’re often forced to make tradeoffs around things like performance and quality.

The js speed race is awesome for developers. As a user, FF, Safari & Chrome are all plenty fast in loading pages. My new benchmark is CPU intensity. I hate coming back to my sleeping Mac just to find it hot, low on battery, because browser has been sucking up the CPU. Safari 5 has lighter CPU footprint than Chrome. With Flash, the plugin will use 2X more CPU on Chrome than Safari while viewing the same page. Scrolling also takes less CPU on Safari than Chrome.

It would be useful to start profiling CPU intensity too. I fear that if the arms race for speed continues, browser will start draining my battery faster.

Very good point. I did some tries on both my 2006 1.83ghz MB and my 2008 2.4ghz iMac, both 10.6.3, one 32 bit the other 64 bit. I got the same results you did. Also, I've found Safari to be more reactive while working and loading pages than Chrome, FF or Opera (last stable version of each), especially on my MB, but it was very noticeable on the iMac. Speed isn't everything, even if it matters a lot.

edit: forgot to write that Safari and Chrome had very similar loading times, Opera and FF were a bit slower but nothing really bad (on all sort of webpages). Nothing scientific, just some tests.

Strange. MacWorld's benchmark showed Safari 5 ahead of Chrome 5 by 3.5%, close to Apple's published number as well as beating all the other browsers.

This isn't the first time I've seen a Safari benchmark compare a release version of Safari to a development version of Chrome with no acknowledgement in the article. I'm not clear how much I should trust Ars right now for objective Apple coverage given their working relationship with Gizmodo, Aurich's constant defense of their behavior, and the reaction of Ars staff and contributors to Safari Reader, which contained statements like "Jobs can go fuck himself." I have a low image of the site right now.

I'm not sure what happened to your Flying Images test for Safari 5, I'm using basically the same rig ("Early 2008" 2x 3.2ghz Quad-Core Xeons Mac Pro, 10gb RAM, NVidia 8800 GT 512mb, running OS X.6.3), and the performance is pretty steady at 60fps or higher (except right at the start when the images are still loading).

No matter what I do with it I can't get it anywhere near the 11fps you got!

Before you make a comment like this, you should at least check your facts; 5.0.375.70 is the latest release build of Chrome. The lousy analogy just makes you sound like a jerk--next time, try omitting it.

This was a typo, I'm astonished they don't happen in your world. It occurred because I cut 'n' paste the version number from the Safari 'About' box into the reply window but you can't cut 'n' paste the same version number from the 'About' box in Chrome.

Basically because that's what Peter had installed on his machine, and I couldn't convince him to install the stable release.

That dev build isn't OMG amazingly faster than the stable release of Chrome, though. IT came out very slightly ahead of Safari 5 on SunSpider, but like squidz says, on V8 Bench and Dromaeo the stable version murdered Safari 5 on Mac OS X running on some serious hardware.

Fair dues - thanks for the response. I was wasn't, at least wasn't intending, to suggest any impropriety but as I'm sure you guys are aware an awful lot of experimentation occurs in dev/nightly builds and some builds try out performance-enhancing code which is then later disregarded for various reasons.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

That was only for Chrome and only for the Win7 test. Chrome on OS X and Opera on both platforms still beat Safari comparing the current releases. To bring it back to your analogy it's comparing the 1978 Ferrari Formula 1 car to 9 different 2010 models.

On that note, why did Ars select the nightly dev build instead of latest stable for Chrome on Win7?

Basically because that's what Peter had installed on his machine, and I couldn't convince him to install the stable release.

That dev build isn't OMG amazingly faster than the stable release of Chrome, though. IT came out very slightly ahead of Safari 5 on SunSpider, but like squidz says, on V8 Bench and Dromaeo the stable version murdered Safari 5 on Mac OS X running on some serious hardware.

I expect better of Ars. You should drop the Flying Images out the window as it is exceedingly unreliable (as shown by all the comments about it performing significantly better for everybody except Ars).

And you shouldn't be settling on versions because so-and-so didn't want to install an appropriate version.

Firefox is slower but its the only browser where you can really block everything. I run firefox 3.6.4 next to chromium 6 and Firefox is just as quick because everything is blocked. Check out the FF add-on RequestPolicy and look at how many sites try to pull information from you when you visit them. Until the Webkit browsers let you block things like this speed is pointless and the browser is dangerous.

Strange. MacWorld's benchmark showed Safari 5 ahead of Chrome 5 by 3.5%, close to Apple's published number as well as beating all the other browsers.

The resolution one is running certain tests (certainly the Flying Images test) is also a factor. Firefox at full resolution on my 30" (2560 x 1600) drops to a crawl similar to that in Chrome in any resolution. When I reduce the size of the browser window the speed increase dramatically.

Benchmarks do give some indications but as mentioned before, all of these browsers are under constant updates. These speeds won't be particularly relevant, or a particular factor in your browsing for much longer.

bonchbonch wrote:

This isn't the first time I've seen a Safari benchmark compare a release version of Safari to a development version of Chrome with no acknowledgement in the article.

It was acknowledged. Chrome stable on the Mac, Chrome development on Windows.

bonchbonch wrote:

I'm not clear how much I should trust Ars right now for objective Apple coverage given their working relationship with Gizmodo, Aurich's constant defense of their behavior, and the reaction of Ars staff and contributors to Safari Reader, which contained statements like "Jobs can go fuck himself." I have a low image of the site right now.

Just because others have a different viewpoint then yours doesn't make them less objective. This is akin to calling them haterz! The funny thing the Ars staff is so regularly cast as Fanbois or Haterz that such "arguments' are shown for what they are, weak and absolute nonsense, not to mention insulting.

What's being lost in the complaints about the Chrome dev build is that it's only one of the tested browsers. Ultimately, Ars tested 10 browsers, one of them was a dev build. Let's say we cut the data on that one browser out, you're still left with Opera OS X, Opera Win7, and Chrome OS X beating Safari in the testing. 7 competitors to Safari and 1 dev build of a competitor were tested, 3 competitors and 1 dev build could be said to have "beaten" Safari. With those results, I don't understand how anyone could agree that Safari "leads the pack."

Strange. MacWorld's benchmark showed Safari 5 ahead of Chrome 5 by 3.5%, close to Apple's published number as well as beating all the other browsers.

This isn't the first time I've seen a Safari benchmark compare a release version of Safari to a development version of Chrome with no acknowledgement in the article. I'm not clear how much I should trust Ars right now for objective Apple coverage given their working relationship with Gizmodo, Aurich's constant defense of their behavior, and the reaction of Ars staff and contributors to Safari Reader, which contained statements like "Jobs can go fuck himself." I have a low image of the site right now.

Ars came to that same conclusion (4% mentioned in the article) for one test in one environment. For Sunspider on OS X, Safari beat Chrome by about 4%. For all other tests, Chrome and Opera thoroughly dominated Safari. Claiming that "Safari leads the pack" is as intellectually honest as Microsoft's claim that "IE9 is the fastest and best browser out there." They both have exactly one target benchmark that they "win" (Microsoft's "win" is by a much larger margin) but are easily beaten by both Chrome and Opera for other tests.

One of the techniques that Firefox uses to achieve this performance is to slightly degrade the quality of the images during scaling, while the other browsers attempt to maintain full image quality. One of the challenges with not being hardware accelerated is that you’re often forced to make tradeoffs around things like performance and quality.

I didn't notice that in FF3.6.3 on Mac OS X, but it's quite obvious that is what Opera is doing—the graphics looked like blocky crap compared to Safari, Chrome, and FF on my machine.

Sure it does. Your average news website will have at least a few hundred lines of javascript, and web applications like Facebook and GMail have thousands.

But I haven't seen any real-world difference in rendering performance on those pages that isn't the same in pages without those elements. IE, FF is about as much slower rendering GMail as it is a Wordpress site. The overall rendering speed in the pages I use, which include GMail and such, appears to be data limited, not JS.

And Safari 5 does _seem_ to be a bit faster than 4. At least it feels faster, or less stuttery anyway.

What's being lost in the complaints about the Chrome dev build is that it's only one of the tested browsers. Ultimately, Ars tested 10 browsers, one of them was a dev build. Let's say we cut the data on that one browser out, you're still left with Opera OS X, Opera Win7, and Chrome OS X beating Safari in the testing. 7 competitors to Safari and 1 dev build of a competitor were tested, 3 competitors and 1 dev build could be said to have "beaten" Safari. With those results, I don't understand how anyone could agree that Safari "leads the pack."

Thank you for taking the time to keep a level head about this.

Safari beats Chrome on test A, Chrome beats Safari on test B. Both are way faster than any browser in 2008-2009, so I'm happy. It's just that IE9 and Chrome 6 tests have nothing to do here, except if the article was about a Webkit nightly (they are at version 604 or something right? While Safari is 533.16). Chrome will be the fastest when 6 comes out, then IE, maybe FF4, Opera 11, Safari 5.1 or 6, etc. It's just too easy to do this.

The problem is not that Safari gets "beaten", it's that the author's methodology leaves to be desired. And anyway, as stated before, each of these browser, except IE8, give good speed in real world perfs. I've tried them on many sites (from Ars, to Gmail, Facebook, NYTimes and others) and none "kills" the others in raw rendering speed. However, as I stated before, Safari was noticeably more reactive on my Macs than all the others. Has someone tested this on Windows 7? For me, Safari is a lot more enjoyable to use because of this, as it never feels sluggish. Can't say the same about Chrome (even if not that bad), FF or Opera.

Just kidding. I do care, just not nearly as much as I used to. Now i'm more interested in stability, and to some degree, compatibility.

But you know what really matters the most to me? How pages look when rendered to my preference. With a 27" screen, I sit farther away and scale web content accordingly. I would readily cut my browser performance in 1/2 in exchange for better scalability. It seems that all browsers are lagging horribly in this regard.

Flying Windows on my early 2008 Penryn MBP with a window that's "maximized" (so somewhere close to 1440x900) with 10.6 showed performance that's all over the place; NVIDIA 8400M, I think? Averaging maybe 20, spiking to 30 or 40. My (much slower) Lenovo X300 (GMA3000 I think?) at the same resolution using IE9 fared much better. Using Safari 5, much worse.

The reason I used Chrome 6-dev is because that's what I use, day in, day out. Its performance seemed to be inline with Chrome 5 in most areas, so I'm not sure it's a big deal.

It sucks that Firefox is the sore thumb on those Mac charts. On the Windows side that goes to IE8 which is just hilariously bad, but when it's Firefox it's not so funny.

Firefox is honestly only a tad bit better than IE.

There is much faster better browsers than Firefox but everyone has their blinders on.

It depends on what you're looking for. My Firefox renders things more than quickly enough for my tastes. A tiny speed boost means almost nothing to me today. What does matter to me are my extensions and the appearance of the browser. Chrome is quite possibly the ugliest piece of software that I have ever run, that's including all those games for my 286 when I had to select CGA or Tandy for them to display properly sometimes.

To me personally, Firefox is the best browser right now. Honestly, I prefer the IE 7 I have to use at work to Chrome.

It depends on what you're looking for. My Firefox renders things more than quickly enough for my tastes. A tiny speed boost means almost nothing to me today. What does matter to me are my extensions and the appearance of the browser.

The flying image benchmark is not very consistent. I tried it with Safari 5.0 and Firefox 3.5.9. I could get the rates to go from 2 fps to 60 fps depending on how large I made the browser window. Small windows very high frame rates large windows very low frame rates. I have a 30-inch screen so I could make them very large. Having both Safari and Firefox running at the same time seemed to have little effect, I could get both up to 60 fps with modest sized windows (634x507 pixels).

Add mine to the "Flying Images" results being f*ed up. I'm on a 13" MacBook Pro, 2.53 GHz Core 2 Duo, 4GB RAM with a NVIDIA GeForce 9400M running the built-in monitor and a 24" Apple LED Display, and Safari 5.0 is getting a consistent 59-61 fps no matter where I put the cursor. Although, quite honestly, how is anyone supposed to even take Ars' "testing" seriously anymore? Especially with this:

foresmac108 wrote:

Basically because that's what Peter had installed on his machine, and I couldn't convince him to install the stable release.

You're kidding, right? Is this what Ars has become...??!!??!! Although, I imagine Ars is quite familiar with the number of posts (read: page views) "controversial" testing generates. At one time, Ars was an interesting technology source. Recently, it's barely scratching above the level of DailyTech. So sad. I guess who cares anymore when you've got bank from a sugar daddy named Condé Nast?

Given the number of variables involved with the Flying Images test (number of icons on screen, size of the window, location of the cursor, rotation speed of the icons, quality of the rendering etc.), I wonder if providing more details for that test would have been helpful in raising confidence? Or possibly going with another test entirely. A rigorous methodology is vital for this kind of test.

I had a go on my own machine (Macbook Pro, 2.4 GHz, 4GB, NVidia 8400M). My methodology was to create a window in each browser with the same sized-recndering area then conduct three tests, each with 64 images. The first involved centring the mouse to get a straight-on view of the text, with zero rotation. The second involved the mouse to the bottom-right corner, creating a rotation with the image half off the bottom of the screen. The third involved placing the mouse over a fixed point roughly half-way up the left-hand side of the screen (where I keep the Dock, so the point was consistent), which caused the image to rotate in the centre of the screen. For each test, I left the image for a few seconds to allow the fps to stabilise then recorded the result:

Opera didn't render with the same quality as the other three, of which Safari was the fastest and Chrome comfortably the slowest. Contrary to the Ars findings, Safari was faster than Firefox. In case Safari degrades worse than Firefox, I bumped the images to 144, but it seems that Firefox actually suffers worse:

Am I the only one who has actually looked at the actual source code of the Flying Images demo?That demo does not use any CSS3 transforms, transitions or animations. It has a draw function which is called by Javascript periodically with a setTimeout. The positioning and size are calculated by Javascript.

That is absolutely not what Apple is promoting. CSS3 is where it's at!

That said, it does indeed show that Safari doesn't handle that type of rendering all that well. But that is not an efficient way of doing it!

CSS3 animations are a bad design. You don't blend a declarative system and state transitions, it is bound to produce problems. What the IE9 team did is much more impressive. They actually accelerated the way web apps are built in the real world, instead of forcing developers to change their practices. The IE team have a talent for optimizing "the real world". IE8 outright fails on benchmarks, but it's not _that_ slow when you actually use it to surf the web.

What I find particularly interesting about these benchmarks is that nobody has the absolute performance crown. Nobody is fast at everything. That demonstrates that there's a lot of room left for improvement, on all sides.

So we're not comparing the nightly build of Safari (r60920) to the latest dev build of Chrome (6.0.422.0 dev). Nor are we comparing the last release version of Safari (v5.0 6533.16) to the release version of Chrome (5.0.373.709).

We're comparing the latest dev build of Chrome against the release version of Safari, which although only only released 2 days ago, is still a huge amount of builds behind the nightly builds.

For our next test, we'll compare the 1978 Ferrari Formula 1 car to the 2010 season car. I wonder how that will pan out. Nice work, Ars.

Give them a couple of weeks to get the 1978 Ferrari 312 T3 tuned a bit, adding new tires and the like, and I think it would do quite good in the comparison, at least in a straight line.At least looking at F1 regulations for 2010 gives no doubt that the the 312 T3 weighs less, around 100 kg, and has a bigger engine compared to the 2010 car. Specs found at http://www.ultimatecarpage.com/car/162/ ... 12-T3.html

---

Formula 1 aside I agree this article is worth near nothing when the writer decides some browsers are represented with their newest builds(Chrome 6 and Internet Explorer 9) and others are left with their stable releases(Firefox, Safari and Opera).

At least for Opera I know the differences would have mattered, as the 10.5x series was the introduction of a lot of new technology and the Opera team has done an amazing job of tweaking it. We're talking improvements that will put it in the lead over even the latest Chrome build(just check #1 in Peacekeeper).

CSS3 animations are a bad design. You don't blend a declarative system and state transitions, it is bound to produce problems...

CSS3 animations are a bad design?! It's the way it should have been done in the first place!!

Having a Javascript timer loop desperately trying to update stuff that is happening on screen is not efficient. Having CSS3 rendering it directly in the HTML renderer is much more efficient. It is less CPU intensive since the GPU can fully take care of every thing (ie. Safari) and not just certain aspects of updating the screen (ie IE9).

I am currently heavily developing HTML5/CSS3 web apps and all those CSS3 goodies are a god sent! It enables me to write much less code since all the presentational stuff (animations included) is handled in CSS. All that is left is the core logic that is handled in Javascript.