We introduced our new vApus FOS (For Open Source) server workloads in our review of the Facebook "Open Compute" servers. In a nutshell, it a mix of four VMs with open source workloads: two PhpBB websites (Apache2, MySQL), one OLAP MySQL "Community server 5.1.37" database, and one VM with VMware's open source groupware Zimbra 7.1.0. Zimbra is quite a complex application as it contains the following components:

Jetty, the web application server

Postfix, an open source mail transfer agent

OpenLDAP software, user authentication

MySQL is the database

Lucene full-featured text and search engine

ClamAV, an anti-virus scanner

SpamAssassin, a mail filter

James/Sieve filtering (mail)

All VMs are based on a minimal CentOS 5.6 setup with VMware Tools installed. All our current virtualization testing is on top of the hypervisor which we know best: ESXi (5.0). CentOS 5.6 is not ideal for the Interlagos Opteron, but we designed the benchmark a few months ago. It took us weeks to get this benchmark working and repeatable (especially the latter is hard). For example it was not easy to get Zimbra fully configured and properly benchmarked due to the complex usage patterns and high I/O usage. Besides, the reality is that VMs often contain older operating systems. We hope to show some benchmarks based on Linux kernel version 3.0 or later in our next article.

We tested with five tiles (one tile = four VMs). Each tile needs seven vCPUs, so the test requires 35 vCPUs.

The Opteron 6276 stays close to the more expensive Xeons. That makes the Opteron server the one with the best performance per dollar. Still, we feel a bit underwhelmed as the Opteron 6276 fails to outperform the previous Opteron by a tangible margin.

The benchmark above measures throughput. Response times are even more important. Let us take a look at the table below, which gives you the average response time per VM:

vApus FOS Average Response Times (ms), lower is better!

CPU

PhpBB1

PHPBB2

MySQL OLAP

Zimbra

AMD Opteron 6276

737

587

170

567

AMD Opteron 6174

707

574

118

630

Intel Xeon X5670

645

550

63

593

Intel Xeon X5650

678

566

102

655

The Xeon X5670 wins a landslide victory in MySQL. MySQL has always scaled better with clock speed than with cores, so we expect that clock speed played a major role here. The same is true for our first VM: this VM gets only one CPU and as result runs quicker on the Xeon. In the other applications, the Opteron's higher (integer) core count starts to show. However, AMD cannot really be satisfied with the fact that the old Opteron 6174 delivers much better MySQL performance. We suspect that the high latency L2 cache and higher branch misprediction penalty (20 vs 12) is to blame. MySQL performance is characterized by a relatively high amount of branches and a lot of accesses to the L2. The Bulldozer server does manage to get the best response time on our Zimbra VM, however, so it's not a complete loss.

Performance per watt remains the most important metric for a large part of the server market. So let us check out the power consumption that we measured while we ran vApus FOS.

The power consumption numbers are surprising to say the least. The Opteron 6174 needs quite a bit less energy than the two other contenders. That is bad news for the newest Opteron. We found out later that some tinkering could improve the situation, as we will see further.

1. For example, a website that scales to 32 cores easily: most people will be amazed how many websites have trouble scaling beyond 8 cores.

2. Getting an OLTP database to scale to 32 cores is nothing to sneeze at. If your database is small and you run most of it in memory, chances are that you'll get a lot of locks and that it won't scale anyway. If not, you'll need several parallel RAID cards which have a lot of SSDs. We might pull that one off (the SSDs), but placing several RAID cards inside a server is most of the time not possible. once you solve the storage bottleneck, other ones will show up again. Or you need an expensive SAN... which we don't have.

We had an OLAP/ OLTP and Java benchmarks. And they were excellent benchmarks, but between 8 and 16 cores, they started to show decreasing CPU utilization despite using SSDs, tweaking etc.

Now puts yourself in our place. We can either spend weeks/months getting a database/website to scale (and we are not even sure it will make a real repeatable benchmark) or we can build upon our virtualization knowledge knowing that most people can't make good use of a native 32 core database anyway (or are bottlenecked by I/O and don't care anyway), and buy their servers to virtualize.

At a certain point, we can not justify to invest loads of time in a benchmark that only interest a few people. Unless you want to pay those people :-). Noticed that some of the publications out there use geekbench (!) to evaluate a server? Noticed how many publication run virtualization benchmarks?

"That doesn't make sense either - how many enterprises you know run database servers on VM?"

Lots of people. Actually besides a few massive Oracle OLTP databases, there is no reason any more not to virtualized your databases. SQL server and MySQL are virtualized a lot. Just googling you can find plenty of reports of MySQL and SQL server on top of ESX 4. Since vSphere 4 this has been common practice.

"etc. tppc, tpce, tpch would be a good place to start "

No not really. None of the professional server buyers I know cares about TPC benches. The only people that mentione them are the marketing people and hardware enthusiast that like to discuss high-end hardware.

So you prefer software that requires 300.000$ of storage hardware over a very realistic virtualization benchmarks which are benchmarked with real logs of real people?

Your "poor benchmark choice" title is disappoing after all the time that my fine colleagues and me have spend on getting a nice website + groupware virtualization benchmark running which is stresstested by vApus which uses real logs of real people. IMHO, the latter is much more interesting than some inflated TPC benchmarks with storage hardware that only the fortune 500 can afford. Just HMO.Reply

While scaling to 32 cores can be problematic for some software, it's worth keeping in mind that the vast majority of dual-socket servers don't have 32 cores.

In fact, a dual-CPU Intel server only has *at most* 12 cores, that's a far cry from 32-cores. Postgresql & MySQL has no problem at all to scale to 12 cores and beyond.

Now if AMD decided to make a CPU with crappy per-core performance but has so many cores that most software can't take full advantage of, that's their own fault. It's not like they haven't been warned. Sun tried and failed with the same approach with T2. If AMD is hellbent on making the same mistake, they only have themselves to blame.

My post title is a bit harsh. But it is disappointing to see a review that devotes FOUR separate benchmarks to 3D rendering, an application that the vast majority of enterprises have no use for at all. Meanwhile, the workhorse applications for most enterprises, OLTP, OLAP, and such, received far too little attention.Reply

I have trouble understanding why people think a review should include research into every other similar product that might be used for the same purpose.

I mean, I can understand ASKING for a review of another specific product, particularly if you've actually done some research on your own and haven't found the information you want, but to imply a review isn't complete because it didn't mention or test another piece of hardware is a bit - unrealistic.

Frankly I can't imagine a situation where someone would have trouble to decide between a Westmere-EX and an AMD CPU. Most people checking out the Westmere-EX go for the RAS features (dual) or RAS + ultimate high thread performance (Quad). In all other cases dual Xeon EP or Opterons make more sense power and pricewise. Reply

Really? Is it that much trouble to understand that people want to see the latest AMD cpu's compared to the most current generation of Intel hardware? Especially when the previous Intel processor review posted on this site reported on Westmere-EX performance? I have trouble understanding why people wouldn't expect it.Reply