Microsoft’s own speed tests show IE beating Chrome, Firefox (Updated)

Microsoft has released its own tests that show IE8 can load many websites …

Just before announcing that Internet Explorer 8 has been finalized, Microsoft has released a new report titled "Measuring Browser Performance: Understanding issues in benchmarking and performance analysis." The document explains the various browser and network components and how each piece can impact performance when benchmarking, capabilities and limitations of various benchmarking tools, as well as ways to design tests to avoid these issues. What makes this report extremely dubious is the chart that is buried at the bottom. It shows IE8 outperforming Firefox 3.05 and Chrome 1.0, something that no other test on the Internet has ever shown before.

Before I start to rant and rave, I want to explain that I am an IE8 user (in fact this post was written using IE8 on Windows 7), though I do use other browsers from time to time. IE8 is a vast improvement over IE7, especially in performance, but I will be the first to admit that it still does not compete with third-party browsers when it comes to speed. However, the table Microsoft provides shows the load times for the top 25 websites according to comScore, and IE8 does quite well. The report notes that Microsoft "used the browser 'Done' indicator for timing when the page when the page is completely loaded at that point. For pages which continue to load and change after the 'one' indication we have used common visual cues to generate the timings. Timing is started when the Go button is pressed. These timings were captured in January 2009; because Internet content is always changing you may get different timings when you run these tests."

As you can see, IE8 outperforms Firefox 3.05 and Chrome 1.0 in loading 12 websites, Chrome 1.0 places second by loading nine sites first, and Firefox brings up the rear by loading four sites faster than the other two browsers. Also, in case you missed it, IE loads mozilla.com faster than Firefox, and Firefox loads microsoft.com faster than IE, just for kicks. This report is bound to stir up a lot of controversy, especially since Microsoft outlines its exact methodology. The report is available publicly at the Microsoft Download Center in PDF and XPS formats, so check it out if you have the time for reading through 14 pages. Microsoft chooses approximately 25 websites for daily testing, and tens of thousands on a monthly basis. If you're going to do your own tests, Microsoft emphasizes that "any list of websites to be used for benchmarking must contain a variety of websites, including international websites, to help ensure a complete picture of performance as users would experience on the Internet."

Microsoft obviously wants to get users who are using alternative browser to come and try out IE8, which is why this report is quietly being released just before the new browser becomes available this month. Is this the best way to go about it? The fact that this is the first test (at least that I'm aware of) to show IE8 loading multiple websites faster than Firefox and Chrome, and the top 25 websites no less, is very supicious. What do you think?

Update

Microsoft has a promotional video up on microsoft.com that shows off these speed tests. Looks like the report may have been released quietly, but Microsoft is promoting the data from it quite strongly.

My suspicion is raised by the fact that no comparison of Javascript speed is done; something which Chrome, FF3.1 and Safari4 are heavily competing over at the moment (not to mention Opera).

Does the full report mention anything of this? (I didn't read it, I admit). How quickly would all browsers load GMail for instance, from the moment that you click on the login-button to the moment your inbox is displayed?

Microsoft maintains that for most browsing, Javascript accounts for 3.48% of loading time. And for sites like GMail, it rises to a modest 13.59%. Consequently, even doubling Javascript speed would only make GMail 6.8% faster.

I don't know how much faster the IE8 Javascript engine is; they do claim speed boosts.

However, given that most "speed tests" I've seen reported are for Javascript performance, and not for browsing performance, it's not inconcievable that (a) IE8 is much, much slower than any other browser at Javascript, and (b) overally, IE8 is faster.

I'm not saying I agree with that - I'm quite skeptical - but Microsoft's justification for measuring speed (as described in the blog post I linked to) seems pretty reasonable.

I'd like to see the tests replicated, preferably using a better methodology that just looking at the "done" status.

Huh? Seriously, who cares if a browser loads web page 0.2s faster than another browser? It's the javascript speed and rendering speed that matters. And that's where IE8 is way beyond it's competition. No wonder. While the guys at Microsoft are struggling with getting the rendering right, gecko and webkit people can optimize the hell out of their javascript and rendering engines or implement stuff from HTML5.

Still, it's nice to say that situation has changed and someone at Microsoft has finally realized that their marketshare is way low to be able to dictate proprietary "standards" and need to start playing nicely and support what other browsers do.

The flaw in this rational is that javascript is going to be used more and for more sophisticated functionality in the near future. That is the reason why the competing browsers are emphasizing javascript rendering.

quote:

Microsoft maintains that for most browsing, Javascript accounts for 3.48% of loading time. And for sites like GMail, it rises to a modest 13.59%. Consequently, even doubling Javascript speed would only make GMail 6.8% faster.

Lets talk about another kind of performance, like when tabs go crazy. I used to like IE but in 7.0 and now the 8.0 beta/rc I've been noticing major performance issues when a tab is doing something really busy it won't let me switch to another one or do anything else. This is on a dual-core system with 2GB ram. Chrome doesn't have this problem - even on my super old 500mhz laptop I can switch tabs and read what is already there if another tab is busy. It happens a lot on this old thing. ;-) Even if the cpu is at 100% it still switches instantly and smoothly from tab to tab - it still impresses me when it happens. The performance difference is magnified when it is run on old hardware but it is a noticeable difference even on my fast system.

Does anyone remember the days when it didn't matter because our connections were so slow our browsers had ample time to render?

Regardless, I think at this point browsers are running out of things to use for convincing users to switch.

My question however is, why does Microsoft care if people use a product that they aren't getting paid for? Is a browser that large of a selling point in an OS? Really? Do people buy a Mac because it has Safari? Whatever, competition is good. If Microsoft can give the other browser developers something to constantly stay ahead of, I guess we should thank them for it.

Kudos to Microsoft for an impressive spin on the data in the table of browser load times. Looking at the averages of each column, however, IE8 comes in at 3.5320 seconds, an insignificant 800 microseconds ahead of Chrome's average of 3.5328. Hardly a statistically conclusive result.

thewooly: strangely 90% of the time, that behavior your describing is because of Flash, not the browser itself. If you want to see it in action, disable Flash for awhile and see how differently the browser behaves, and how much more responsive and reliable it is...

The flaw in this rational is that javascript is going to be used more and for more sophisticated functionality in the near future. That is the reason why the competing browsers are emphasizing javascript rendering.

quote:

Microsoft maintains that for most browsing, Javascript accounts for 3.48% of loading time. And for sites like GMail, it rises to a modest 13.59%. Consequently, even doubling Javascript speed would only make GMail 6.8% faster.

The flaw in your rational is that JavaScript is not the primary rendering mechanism *now*. Nor will it be in the near future.

If/when search engines are able to evaluate JavaScript efficiently during normal indexing passes, your argument might hold some water. Until then, fast markup rendering is the key to a fast browsing experience. (And frankly, this is the reason why Chrome and IE8 *feel* fast---not because of how they execute JS.)

Originally posted by 45736F7465726963:Kudos to Microsoft for an impressive spin on the data in the table of browser load times. Looking at the averages of each column, however, IE8 comes in at 3.5320 seconds, an insignificant 800 microseconds ahead of Chrome's average of 3.5328. Hardly a statistically conclusive result.

Fair point. But you do have to agree that both are indeed fast.

I know it's painful to say good things about IE. Versions 3-7 gave us so much to hate.

Originally posted by zonk3r:Next Microsoft will be claiming IE makes your websites whites whiter and your colors more vibrant...

You're thinking of Shitfari there

quote:

Originally posted by zonk3r:How about making IE less bloated, faster overall (not just in loading), more standards compliant and kill your scary plugin support and move to a more secure model.

They just tend to prefer a delayed rendering technique before displaying shit in IE, realistically you get to teh end point at teh same speed as any other browser, they have been increasing standards support ever since ie7, the plugin support isnt realisitcally any different to firefox's and IE7 in vista/win7 secure mode is the most secure browser there is.IE has never been "bloated", hell it uses less resources than FF and a shitload less than shitfari.I'm tired of that generic anti-M$ rant about 'bloated'ness, an app uses slightly more resources than an app written half a decade ago? who woulda thought! lets fucking stick to nice DOS where you have to fit it in a tiny bit of ram. Other than outlook (for some fucked up reason) M$ apps tend to get faster over time on the same hardware.

Notice that they compare a beta browser against a stable. Unfortunately for MS their glory is short lasting, if they played fair and compared a beta against a beta, namely IE8 vs. Firefox 3.5, they would be quite dissociated to find that Firefox blows it out of the water.

IE8 has never felt slow compared with Firefox so I'm not terribly surprised, unfortunately IE8 tends to hang a little too much on my rig. Funny thing is that when IE8 hangs I can kill a random process until I hit the hung one, then I can continue browsing without loosing a single tab.

When Firefox crash it's less of an issue since it always comes back without me having to do much. No killing random processes and nothing lost either.

If theres one good thing to take from this article, its that Microsoft, of all companies, is trying to convince people to forgive it for past IE browsers and try out IE8. Its trying to convince people that from now on it will be standards compliant, and offer all of the features we are used to in FireFox/Chrome/Safari.

Its not enough to make me change browsers, but I am glad to see that the competition between the browsers benefits us all.

This is good that MS is finally looking at their own products seriously and making serious efforts to improving them. Sure they have a ways to go but its good to see MS taking things seriously. First with Vista vs W7 and now IE 7 vs IE8.

drag...that's compared to two other browsers, so if you want to be faster for the other 60%, you'd have to use Firefox AND Chrome...not a great solution.

As for "loading time," this is pretty stupid. I mean, load time is an important benchmark, no question, but what about page responsiveness (i.e., javascript re-rendering speed)? This is what slows usability on gmail and other web 2.0 sites. I want my load time as low as possible, but I would easily wait twice as long for gmail to load if using it was only 20% faster...because I load it once per day and use it all day, sometimes hundreds of times.

Microsoft's BS about "javascript doesn't account for much load time" is accurate, but it (probably deliberately) misses the point that we don't care about *downloading* the js files, but parsing and processing them in real time. Duh!

No javascript does not determine how fast a page loads, which is why MS did this test. But in the functionality and responsiveness of using websites are becoming more based on javascript. Performance will lag on browsers that don't have good JS rendering engines.

quote:

The flaw in your rational is that JavaScript is not the primary rendering mechanism *now*. Nor will it be in the near future.

If/when search engines are able to evaluate JavaScript efficiently during normal indexing passes, your argument might hold some water. Until then, fast markup rendering is the key to a fast browsing experience. (And frankly, this is the reason why Chrome and IE8 *feel* fast---not because of how they execute JS.)

You make a good point, responsiveness is king. I think that load time would actually be a good indicator of responsiveness. when you interact with a control on a web page and cause some javascript code to execute that code will often change the layout of the page in some way that triggers a rerender. That render will often be a subset of the load path. So saying load is overall faster is a imperfect but reasonable measure of perf. My problem with straight javascript speed tests on a desktop browser is that javascript is by no means going to be the bottleneck even on the slowest of them all.

It's also just plain hard to test responsiveness on real world sites. This is especially difficult for a thrid party. To test post load responsiveness you A need to know where to fire meaningful events to measure the time between them. You need to have a intimate understanding of the webpage to know what interactions to measure. You need to know how to isolate differences in runtime. Websites these days are complicated pieces of software.

I'd actually be more sceptical of a "responsiveness" test suite becuase it's so much easier to game the system. In a responsiveness test you can only really test a subset of functionality. Inevitably some browsers will be faster or slower on certain actions. If a software company wrote up such a test I'd assume that they just picked the items they executed well on. Picking load eliminates this potential source of bias.

Measuring software performance is something I have to think about a lot. It's a non-trivial problem when doing it just for internal consumption. When you're doing it as a cross product study it gets even trickier.

Yeah, like firefox where plugins are natively executing code that hook the browser and have full access to its contents, as well as native system access. Or like Chrome where plugins are natively executing code that hook the browser and have full access to its contents, as well as native system access. Or... You get the point. There is effectively no security model for ANY of the current major plugin engines- they aren't sandboxed and users of any of the browsers can be tricked into installing them (granted the approach varies based on the install paradigm). The only thing firefox has going for it is the couple second countdown for the user to change their mind.

Google *may* figure out how to effectively sandbox them with their nativecode project, but I am skeptical that they can blacklist everything bad that can be done. Regardless, right now if you install a plugin for ANY browser you are simply trusting the benevolence of the coders behind the plugin- the mozilla website has had to pull malicious plugins in the past and will have to do so in the future. Sure, it may be marginally easier to invoke an activex object on a third party malicious website (in IE 8 actually not so much if activex developers update their objects to use new features), but why bother with flaws in IE only activex objects when you can just post a malformed swf, pdf, or mov file and exploit the MYRIAD of vulnerabilities in flash, acrobat, and quicktime plugins in every browser.

Do not assume that this is an IE only problem and that you have nothing to worry about with the competitors. You are much better off being skeptical ALL the time rather than assuming that all security problems stem from MS products.

I think these sats say it all. It really does not matter whar browser you use if all you are interested in is speed. They all have their better sites but most are very close in comparison. Anymore, you might as well use what you like because even security issues are common with all of them from time to time.

snuffub, you've got a point there, but what I notice is that the web-sites that use javascript engines aren't included in the test! Microsoft is only testing sites at the main page, which is great for when you first browse to a site like google, or something, but what about when you actually start using it?

I agree that part of the load time of gmail includes the javascript processing. That would make gmail.com (after being logged in, of course) a decent test to run. But they didn't run that test, or at least they didn't give us the results.

So, while it may be a *decent* test of load+javascript processing+render time to just open sites and wait for them to load, how come Microsoft is omitting those sites that actually use a heavy JS engine...?

Once again, their excuse that it only accounts for ~13% or something is meaningless. If it's so small, include it in the results...and prove it doesn't matter.

I know some sites show a "simplified" version of the page for IE's gimpy rendering. I wonder if any of the tested sites do something like that. I also wonder if some of the difference might in plugins like Flash. I suppose I should just grab the PDF and see if I can answer my own questions :P