Joe Jakubowski, Virtualization Performance Lead Engineer, and Nancy Reaves, Offering and Alliance Manager, recently gave an excellent talk on understanding virtualization benchmarks. Here is what you need to know:

As we know, benchmarks are only one of many ways to evaluate systems. Performance is one component - don't forget to consider reliability features, memory capacity/scalability, I/O, and manageability when selecting a virtualization solution.

Virtualization performance (and performance in general) is not just a function of transaction throughput. It is also important to analyze response time.

When comparing virtualization benchmark results, it is important to compare like builds and like configurations to make an apples-to-apples comparison. Often a gap in results can be attributed to the hypervisor build. The Build Number can be found in the benchmark disclosure report.

SPECvirt_sc2010 is the industry-standard single-host virtualization benchmark for all systems, including x86 servers. This benchmark has many enhancements including:

On something called the Coast to Coast trail, that goes from the Irish Sea to the North Sea in northern England. It's also called the C to C, the C2C, or whatever else that allows the name to fit on the few trail signs that exist.

After you've hiked the 190 miles or so through coastal cliffs, rocky mountains, moors, and many pastures full of sheep, and you're sitting at "The End" in Wainwright's pub, you can't help but feel truly special.

IBM just published an outstanding result on SPECjEnterprise2010, SPEC's new full system end-to-end Java benchmark. This first Power Systems result, using POWER7 systems, adds to the portfolio of excellent IBM results on this benchmark, all using WebSphere and DB2.

IBM is #1 in both of these benchmarks.

I wasn't first and I surely wasn't the fastest to walk across England. But it was by all means truly special.