IaaS Performance Benchmarks Part 2: AWS

I conducted my own benchmarking project to compare IaaS cloud providers and their services. I tested Amazon Web Services first, and here are the results.

This is the second part in a series of articles about creating my own IaaS performance benchmarking project. In the first part, I explained the methodology I'm using to I'm test instance types across major IaaS providers. In this part, I post the results from my tests of Amazon Web Services instance types.

I launched more than 175 AWS VMs across current instance types, in all three U.S. regions (East, West and Oregon) and in three availability zones (AZs) per region. (The West and Oregon regions have only three AZs; East--for me--has only three AZs that allowed me to launch all of the instance types.) I ran instance-store-backed instances for every instance type I could; I ran EBS-backed for the t1.micro, and HVM-EBS-backed for the m3s and high-computing instance types.

In addition, I ran EBS-backed for a few of the instances that you can run either instance-store-backed or EBS-backed to see if there was a performance difference. The high-computing types aren’t allowed in US-West, and the cg type is only available in US-East. I was blocked from running a few instances in one of the US-West AZs. I ran a mix of spot and on-demand, and couldn’t find any performance difference between them.

I had to patch UnixBench to get it to run properly on the cg1 and cr1 instance types, as both have more than 16 cores. In my charts, I used on-demand pricing to compare relative differences in pricing. When I compare AWS to other providers in later parts of this series, I will discuss both reserved and on-demand pricing for AWS.

I also found that, in at least some circumstances, Ubuntu outperformed CentOS in benchmarks. However, I am sticking with CentOS for all benchmarks because there is more support for it across IaaS providers. (Specific circumstance: m3.2xlarge with HVM EBS AMIs had a single-core UnixBench score that was more than 20% higher on Ubuntu 12.04 than on CentOS 6.4).

Instance-Type Bake Off

Here are the main takeaways that I have from comparing instance types across the benchmarks:

• m1.small and t1.micro are both probably the best instance types as far as price/performance (at least if UnixBench is a reasonable measure of performance for you), but they are poor as far as performance-per-machine goes as compared with other options. If you’re using either, just know that if you pay more, you’ll get much better performance.

• The m3 family did really well in these benchmarks, but be warned that this is performance on HVM EBS AMIs. If you run m3s on regular EBS AMIs (which AWS will let you do, perhaps because HVM AMIs don’t appear to be available in US-West), the benchmark scores are significantly lower (well less than half in my benchmarks). Also, I would recommend testing Ubuntu (and perhaps other flavors of Linux) over CentOS for these, as I saw significantly better scores under Ubuntu with m3 instances in particular.

• Looking at the $1/hour or less (on-demand pricing) options: The m3.2xlarge ($1/hour), c1.xlarge ($0.68/hour) and m3.xlarge ($0.50/hour) stand out with significantly higher scores than the rest. One reason for favoring the c1.xlarge: You can run it backed with Amazon’s instance-store, instead of EBS, which means that you will not have the two single points of failure that exist with EBS-backed instances (the physical machine that the instance is on and the EBS volume).

• The cheapest instance that does fairly well in these benchmarks is the c1.medium ($0.145/hour). You have to pay more than five times as much to get double the performance in many of the benchmarks I ran, and the performance is close to much more expensive instance types for many uses.

• I think it would be hard to justify using the m1.xlarge in almost any circumstance. The m3.xlarge is only 2 cents/hour more than the m1.xlarge (on demand), has the same or better specifications, and gets significantly higher scores in these benchmarks.

• As you spend more money on AWS instances, you get generally better performance in these benchmarks. For example, the cg1.4xlarge costs about 3.5 times more than the c1.xlarge, and produced a multicore UnixBench score of roughly 3.5 times as much as the c1.xlarge.

• Amazon provides a good descriptive guide to picking instance types for particular uses, and these results should not be considered in any way a better general guide to selecting an appropriate instance-type family.

That said, I have met many AWS users who seem to select instance types based on either a “Hey, this one looks good and cheap” or a “Let’s spend as much money as possible so it goes fast” basis. If you know (or suspect) that your AWS deployment choices are based on this kind of criteria, these benchmarks should push you toward re-evaluating your selections.

Note that in both SysBench charts above, I cut off the top for both the m1.small and t1.micro; their results were literally "off the chart."

Next Page: AWS Region Bake Off

Joe began his career by winning the 1996 Weird Software Contest with the Mutant Chicken Races and creating the first Windows-based iPod application. Over the past ten years, Joe transitioned from development to systems design and data analysis, creating the first BuildFax ... View Full Bio

Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% ­extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.

Most -- 77% -- of respondents to our 2014 Private Cloud Survey, all from companies with 50 or more employees, have clouds in place now or are piloting. Those without plans may be left at a disadvantage.