VMware benchmark tool raises fairness questions

VMware is making a free benchmarking tool available to IT professionals to evaluate the performance of virtualisation technology in their datacentres. At the same time, it is seeking to assure competitors that the benchmarking standards will be fair to all of them.

VMmark is available for a [free download] and can be used to determine how well virtualisation software works with any of six typical data center workloads: file server, email server, Web server, a standby server, an online transaction processing (OLTP) database and a Java order entry system. VMmark runs on either Linux or Windows.

VMware shared a prototype of VMmark in October with members of the Standard Performance Evaluation Corp. (SPEC), an industry group that establishes performance benchmarks so customers can evaluate products. At that time, SPEC had just formed a working group to develop benchmarks for virtualisation software. It is still working on those benchmarks.

Though helpful for comparing various products, benchmarks are controversial because of concerns that the results can be manipulated to make one vendor look better than another. SPEC and other industry benchmarking groups are organized to create fair benchmarks that will yield impartial results.

VMware acknowledged the skepticism about its benchmarking tool being a fair way to compare VMware with other brands, so it strove to be impartial, senior director of enterprise and technical marketing for VMware, Andrea Eubanks, said.

VMware is represented on the 13-member SPEC work group that is developing the virtualisation benchmark along with competitors such as SWsoft and Trigence. Major technology companies including HP, Microsoft and Intel are also represented.

"We have created VMmark to be an open standard," Eubanks said. "In order to prevent misconceptions of what our intentions are ... we presented our technology preview to SPEC so they could use it as the basis of the open-standards benchmark. We want to improve our product, we don't want to cook our benchmarks."

VMware is introducing VMmark while the SPEC panel is still developing its standards. "The process to becoming a standard is a quite long one," Eubanks said.

But VMware has not always been as impartial as it claims to be, vice-president of marketing for XenSource, a VMware competitor, John Bara, said.

Bara said VMware released a white paper earlier this year comparing VMware's ESX Server hypervisor to XenSource's Open Source Xen hypervisor on a Windows operating system and the results showed VMware performed better. Bara said the comparison was not fair because Open Source Xen was optimised to run best on Linux, not Windows. XenSource's Xen Enterprise product is optimised to run Windows and would be a more balanced comparison to VMware's ESX.

"We cooperate with VMware in a lot of areas, but we went back to them and said 'foul'," Bara said. To VMware's credit, he said, it redid the test and accepted the revised benchmark that showed VMware ESX and Xen Enterprise with nearly comparable performance.

The making of an MSSP: a blueprint for growth in NZ

Partners are actively building out security practices and services to match, yet remain challenged by a lack of guidance in the market. This exclusive Reseller News Roundtable - in association with Sophos - assessed the making of an MSSP, outlining the blueprint for growth and how partners can differentiate in New Zealand.

Reseller News Platinum Club celebrates leading partners in 2018

The leading players of the New Zealand channel came together to celebrate a year of achievement at the inaugural Reseller News Platinum Club lunch in Auckland. Following the Reseller News Innovation Awards, Platinum Club provides a platform to showcase the top performing partners and start-ups of the past 12 months, with more than ​​50 organisations in the spotlight.​​​

Copyright 2019 IDG Communications. ABN 14 001 592 650. All rights reserved. Reproduction in whole or in part in any form or medium without express written permission of IDG Communications is prohibited.