Microsoft has released a seventh test version of Internet Explorer 9 the company says is better at "real-world" Web-based JavaScript programs. But with it has come a rival's accusation that Microsoft essentially engaged in the storied computer industry practice of benchmark engineering--designing technology to be fast on an artificial speed test.

The seventh IE9 platform preview comes with new improvements to its Chakra JavaScript engine. And Microsoft asserted Chakra is engineered to do better on actual Web site tasks rather than narrow benchmarks.

"Over the last few weeks, we've been tuning the JavaScript engine for more of the patterns we've found in real-world sites. Based on the progress since the last platform preview, we're releasing an updated platform preview build," said Dean Hachamovitch, Microsoft's IE general manager, in a blog post yesterday.

The company has been creating a range of Web applications to let people test all the browsers that figure in today's fierce competition. And with the new version, Microsoft signaled its growing impatience with one test of JavaScript, the SunSpider benchmark--both with how basic its tests are and how limited it is compared to the full range of browser performance challenges.

"Microbenchmarks and real-world Web patterns have little in common," Hachamovitch said. "We think people should evaluate browser performance with real-world scenarios. Real-world scenarios involve using all the subsystems in the browser together rather than looking at single subsystems in isolation...real-world Web sites use JavaScript to respond to user input, manipulate strings, and move objects around the screen, and much more."

But also this week, Mozilla took issue with one aspect of IE and JavaScript: "dead code elimination."

This idea involves scrutinizing software before it runs and eliminating parts that are mere busywork--flawed code programmers don't always realize they use. One aspect of SunSpider, a test called math-cordic, "is written exactly in a way that triggers this general optimization," Hachamovitch said.

His analysis showed that IE9 is using dead-code elimination for some tasks involving a number of mathematical operators. "Curiously, the IE9 team has chosen not to eliminate more expensive operations such as multiplication or division. The set of analyzed operators do happen to be the ones that appear in the math-cordic benchmark," Sayre said.

Then came his damning conclusion, essentially that Microsoft had tuned Chakra just for a good SunSpider result.

"I don't think it's possible to consider IE9's implementation of dead code elimination to be a serious general-purpose optimization. It seems clear that it hasn't been tested on anything other than SunSpider," Sayre said.

Benchmark engineering is an ages-old practice in the computer industry. For example, with prominent server database test, TPC-C, server makers would attach absurdly large numbers of hard drives in an effort to achieve high scores. The design represent real-world constraints customers faced.

Wilson "would rather see collaboration across vendors and developers on suite of tests that span DOM [the document object model interface browsers use to interact with Web pages] as well as pure JS [JavaScript]," he said in a tweet. He
added that he prefer people "back away" from a single numeric JavaScript benchmark score as a performance claim, "frankly, because it has little to do with real-world user performance."

About the author

Stephen Shankland has been a reporter at CNET since 1998 and covers browsers, Web development, digital photography and new technology. In the past he has been CNET's beat reporter for Google, Yahoo, Linux, open-source software, servers and supercomputers. He has a soft spot in his heart for standards groups and I/O interfaces.
See full bio