Microsoft’s Windows Azure beat all competitors in a year’s worth of cloud …

Microsoft’s Windows Azure has beaten all competitors in a year’s worth of cloud speed tests, coming out ahead of Amazon EC2, Google App Engine, Rackspace and a dozen others.

The independent tests were conducted by application performance management vendor Compuware using its own testing tool CloudSleuth which debuted last year. Anyone can get results from the past 30 days for free by going to the CloudSleuth website, but this is the first time Compuware has released results for an entire 12-month period.

Compuware uses 30 testing nodes spread around the globe to gauge performance of the cloud services once every 15 minutes. The company performed 515,000 tests overall for a year’s worth of data covering August 2010 to July 2011, which Compuware released today. Each test requires the loading of a simulated retail shopping site consisting of two pages, one page containing 40 item descriptions and small JPEG images, and the second page containing a single, larger image of 1.75MB.

The Windows Azure data center in Chicago completed the test in an average time of 6,072 milliseconds (a little over six seconds), compared to 6.45 seconds for second-place Google App Engine. Both improved steadily throughout the year, with Azure dipping to 5.52 seconds in July and Google to 5.97 seconds. Also scoring below 7 seconds for the whole year were the Virginia locations of OpSource and GoGrid along with BlueLock in Indiana. Rackspace in Texas posted an average time of 7.19 seconds, while Amazon EC2 in Virginia posted a nearly identical 7.20. Amazon’s California location scored 8.11 seconds on average.

“The choice of a web site as the initial target application should be seen as a first step to understanding the availability, responsiveness and consistency of cloud service providers,” the company explains. “While admittedly monochromatic (especially in light of the richness of services provided by cloud providers), the choice reflects the observation that the majority of modern applications rely on the Internet protocols as their transport mechanism. It enables us to create a relatively small and simple application that still gives us great insight into the core performance of cloud service providers. Just as importantly, it can be easily implemented on both PaaS and IaaS cloud providers.”

Amazon EC2 has proven its worth in many real-world scenarios, including the building of a 30,000-core HPC cluster on EC2 and a separate EC2-based cluster that ranked in the world's Top 500 fastest supercomputers.

But within the limits of the Compuware test, Azure has improved in relation to its competitors. In July 2010, Network World conducted a test using the CloudSleuth tool and found that both Google App Engine and Amazon EC2’s Virginia location were faster than Windows Azure over the course of a month. (Google App Engine, perhaps because of its distributed nature, is tested by CloudSleuth as a whole rather than from specific locations.)

Although Compuware tries to make the tests expansive by spreading nodes throughout the world, the results are still highly affected by location. For example, both Azure and Amazon posted poor scores in their Singapore data centers (16.10 seconds for Azure and 20.96 seconds for Amazon, the worst time in the survey) but the discrepancies between North America and Asia are due in large part to limitations in the Compuware testing network. “Within Asia, the performance is generally abysmal by North American standards,” says CloudSleuth product manager Lloyd Bloom. But the measurements are skewed because “most of our measurement points are not in Asia.”

These times we've been discussing so far have been worldwide averages. North American-only times are generally about twice as fast. Azure continued to be the fastest in the past 30 days, both in North America only and in a worldwide average, according to results pulled from the CloudSleuth website this week. Azure also did well in availability, with its Chicago facility hitting 99.93 percent uptime over the past month, significantly better than the 97.69 percent score posted by Google App Engine, and among the highest overall. Rackspace in Texas achieved 99.96 percent uptime, while Amazon EC2 in California scored 99.75 percent and EC2 in Virginia was up 99.39 percent of the time.

While Compuware’s results may be a good starting point for customers trying to decide between various cloud services, they’re not perfect. For example, Salesforce’s Force.com cloud isn’t tested, even though it may be the most widely used platform-as-a-service cloud.

Compuware’s testing isn’t sponsored by any vendor, but several of the vendors in the testing are either Compuware partners or customers, including SoftLayer and OpSource. OpSource chief marketing officer Keao Caindec called the CloudSleuth test easy to use and among the most extensive in terms of the number of clouds tested, but the website loading test doesn’t include everything a potential customer might care about. “There are several ways of looking at load testing,” he noted. Another test is to simulate credit card transactions, which requires several steps. Compuware does provide that sort of testing, Caindec said, but you have to pay for it. Compuware is only willing to give away so much for free, but CloudSleuth does provide some interesting information.

“It’s been difficult in the past to actually come up with an objective way for our clients to compare us to other clouds,” Caindec said.

Compuware designed the webpage loading test to be fairly generic. “We wanted to make sure we didn’t play to the strengths of any one specific provider and their ability to accelerate pages,” said Compuware product marketing manager Ryan Bateman.

While Bateman noted that “some providers are going to fare much better because they’re geographically closer to the locations in which we’re testing from,” CloudSleuth provides enough granular data to help customers figure out the differences. For example, on CloudSleuth.net, you can click the measurement location in, say, New Jersey or Argentina and see a ranking of all the cloud providers from that location. Depending on the application or services you want to host with a cloud provider, the location-specific data may be more important than the global average.

43 Reader Comments

Be consistent. Don't compare milliseconds with seconds ( "average time of 6,072 milliseconds (a little over six seconds), compared to 6.45 seconds") - there is no need for this source of confusion - which is extra confusing for those who use comma instead of the decimal point.

Get the units right. A second is a second and a millisecond is 1000 times less. Maybe if you weren't confusing yourself by mixing s and ms and/or if you used the metric system every day you would notice it quicker ("16.10 seconds for Azure and 20.96 milliseconds for Amazon").

Clicking the various nodes, the spread of access times is significant. I'd have a hard time making reasonable sense out of the data, and would probably need a better breakdown of my customers weighted by location to tell whether any of it matters.

6.1 vs 6.5 for second place ? I do not know what the error spread is on these measuements but I would bet on more than 6%.Also, the only thing that is tested here is the web server serving HTML and Jpegs.There is more to these services than web servers !

This test is really only calculating http response speed and is not reflective of the overal cloud infrastructure speeds. One of the big features of both Rackspace Cloud and Amazon infrastructure is they bundle a CDN service with their offerings. This test had assets stored in the AMI/Machine Image (https://www.cloudsleuth.net/web/guest/m ... #TargetApp). In the "real world" a retail site would typcially store images, css, js, etc. in a CDN. If that had been the case, I suspect the response speeds would have been significantly faster for the cloud offerings using CDN.

Looks to me like this is extremely misleading. Remove Beijing and Google becomes by far the winner (instead windows being a marginal winner it is now). As we all know Google does not play nice with China these days. Also, the average for Amazon is better than windows without Beijing. So, if you don't use the service from China why use Windows?

I agree with above. It is pretty lame to give a straight average as the metric to compute a winner. Especially one that includes clear issues in the data. At the very least, the times should be given with and without Beijing.

I am really intersted in these services but I do not fully understand what they offer and how they differ from each other.

I want something like EC2's virtual machines running Linux but it is my understanding that Azure and App Engine does not offer this. What exactly do they offer? Wikipedia and their respective promotional websites doesn't really give me a complete picture.

Maybe an idea for an article on Ars? Maybe with a guide on how to move from a current dedicated server into a cloud service complete with some way to calculate how much CPU you will use?

I am really intersted in these services but I do not fully understand what they offer and how they differ from each other.

I want something like EC2's virtual machines running Linux but it is my understanding that Azure and App Engine does not offer this. What exactly do they offer? Wikipedia and their respective promotional websites doesn't really give me a complete picture.

Maybe an idea for an article on Ars? Maybe with a guide on how to move from a current dedicated server into a cloud service complete with some way to calculate how much CPU you will use?

Azure and AppEngine are about scalable development platforms you can deploy applications to. You design and deploy an app for their clouds, click deploy, and their infrastructure handles things like redundancy, scale, failover, etc. The EC2 concepts you're talking about are more about hosted infrastructure. You deploy the entire OS to someone else's servers, and then you're responsible for making it work right in your environment. It is a very different creature.

I am really intersted in these services but I do not fully understand what they offer and how they differ from each other.

I want something like EC2's virtual machines running Linux but it is my understanding that Azure and App Engine does not offer this. What exactly do they offer? Wikipedia and their respective promotional websites doesn't really give me a complete picture.

Maybe an idea for an article on Ars? Maybe with a guide on how to move from a current dedicated server into a cloud service complete with some way to calculate how much CPU you will use?

Azure and AppEngine are about scalable development platforms you can deploy applications to. You design and deploy an app for their clouds, click deploy, and their infrastructure handles things like redundancy, scale, failover, etc. The EC2 concepts you're talking about are more about hosted infrastructure. You deploy the entire OS to someone else's servers, and then you're responsible for making it work right in your environment. It is a very different creature.

Yes if you want a cloud-based VM running Linux, Azure or App Engine won't offer that as far as I know. You can, however, get that from Amazon, Rackspace or GoGrid, and probably others.

I am really intersted in these services but I do not fully understand what they offer and how they differ from each other.

I want something like EC2's virtual machines running Linux but it is my understanding that Azure and App Engine does not offer this. What exactly do they offer? Wikipedia and their respective promotional websites doesn't really give me a complete picture.

Maybe an idea for an article on Ars? Maybe with a guide on how to move from a current dedicated server into a cloud service complete with some way to calculate how much CPU you will use?

Azure and AppEngine are about scalable development platforms you can deploy applications to. You design and deploy an app for their clouds, click deploy, and their infrastructure handles things like redundancy, scale, failover, etc. The EC2 concepts you're talking about are more about hosted infrastructure. You deploy the entire OS to someone else's servers, and then you're responsible for making it work right in your environment. It is a very different creature.

What exactly does "deploying an app" mean?Hosting an application for download and upgrade? Hosting a complete web application? Does it support storage for offline desktop apps?

As I see it all these offer various services and some are overlapping. I would really like to see a write-up on this. It's an interesting field, especially for smaller startups.

This article will likely lead many people to make bad business decisions. I recently had to choose a cloud hosting provider for a startup and I can say that Windows Azure is second only to Amazon as being one of the worst options you can choose.

Amazon and Microsoft's core businesses will continue operating no matter what outage occurs. In Amazon's case, its business operations should continue even if it losses an entire datacenter. Microsoft really doesn't make money as a result of its website, but I imagine that its website is designed similarly. As a consequence, they have much less reason to ensure that their hosting services maintain 100% reliability and their SLAs show it.

If you contrast that to a company like Rackspace, which ranked #2 in the 30-day rankings, there is a remarkable difference in quality. Before I made my recommendations, I looked at information about past outages. In the case of Rackspace, I could find no outage lasting more than an hour. In the case of Amazon, I found outages spanning entire days. Microsoft is no different:

This article is misleading in many ways and it will likely lead to some very bad business decisions. With that said, I chose Rackspace before I knew that these benchmarks even existed and the benchmarks seem to validate my choice.

I really don't understand this test at all. What kind of asstastic application are they using that has mean response times of 5-6 seconds in the best case? Where I work we measure frontend latency in the tens of milliseconds.

I really don't understand this test at all. What kind of asstastic application are they using that has mean response times of 5-6 seconds in the best case? Where I work we measure frontend latency in the tens of milliseconds.

The Methodology is published on the site and answers your question. Those response times are a collective 2 page load, one of which contains a large image.

Global Provider View “Target” ApplicationIn creating a target application for the Global Provider View, we wanted to ensure the test application could be deployed to each provider without modification. It also needed to be a representative proxy for a type of application very commonly deployed to cloud service providers. Finally, the test application had to be relatively small, yet still give us sufficient feedback to make monitoring practical.

We decided to begin by instantiating a very simple simulated retail shopping site as the target application. The “site” consists of two pages. The first page is a list of 40 item descriptions and associated images. Each image is a small (approximately 4K) JPEG file. The second page contains a single large (1.75MByte) JPEG image. The test script directly navigates between the two pages of the site, rendering each page in full. The test is intended to simulate a user browsing a product catalog and viewing a single product image in detail.

The choice of a web site as the initial target application should be seen as a first step to understanding the availability, responsiveness and consistency of cloud service providers. While admittedly monochromatic (especially in light of the richness of services provided by cloud providers), the choice reflects the observation that the majority of modern applications rely on the Internet protocols as their transport mechanism. It enables us to create a relatively small and simple application that still gives us great insight into the core performance of cloud service providers. Just as importantly, it can be easily implemented on both PaaS and IaaS cloud providers.

Our approach to implementing the target application stressed parity. Where absolute parity was not possible, because of the inherent differences between IaaS and PaaS service providers, we chose the implementation practice recommended by the service provider’s publicly available documentation. The content of the web site is identical for all implementations.

Since when is 1.75 MB a large file? That is very small and a small hiccup anywhere along the route can cause a large spike in time. Averaged out over a 10 or 100 meg file, those hiccups would most likely disappear, or become so exaggerated as to indicate a real issue.

I really don't understand this test at all. What kind of asstastic application are they using that has mean response times of 5-6 seconds in the best case? Where I work we measure frontend latency in the tens of milliseconds.

The Methodology is published on the site and answers your question. Those response times are a collective 2 page load, one of which contains a large image.

Yes, thanks for cuntpasting the whole article I already read. That doesn't explain anything. It's obvious that their application is dominated by HORRIBLE PROGRAMMING. Moving it from one cloud to another doesn't help anything, it's just deck chairs on the Titanic. If your page loads in 6.13 seconds on App Engine, you aren't going to bother moving it to Azure to save 17 millis. You're going to fire all the programmers and re-write it so it loads in < 500ms. Then you can probably start thinking about platform variance.

In creating a target application for the Global Provider View, we wanted to ensure the test application could be deployed to each provider without modification. It also needed to be a representative proxy for a type of application very commonly deployed to cloud service providers. Finally, the test application had to be relatively small, yet still give us sufficient feedback to make monitoring practical.

This is where they utterly failed. Who would deal with the headache of GAE, Azure or even EC2 for a site that just serves static HTML?

Yes, thanks for cuntpasting the whole article I already read. That doesn't explain anything. It's obvious that their application is dominated by HORRIBLE PROGRAMMING. Moving it from one cloud to another doesn't help anything, it's just deck chairs on the Titanic. If your page loads in 6.13 seconds on App Engine, you aren't going to bother moving it to Azure to save 17 millis. You're going to fire all the programmers and re-write it so it loads in < 500ms. Then you can probably start thinking about platform variance.

Horrible programming, or it's a benchmark.They most likely had it do some nontrivial amount of bullshit work in order to highlight differences in execution speed between the different providers. If all the dynamic requests came back in 100ms and each platform's score was almost entirely dictated by network latency and bandwidth, it wouldn't be very interesting.

Yes, thanks for cuntpasting the whole article I already read. That doesn't explain anything. It's obvious that their application is dominated by HORRIBLE PROGRAMMING. Moving it from one cloud to another doesn't help anything, it's just deck chairs on the Titanic. If your page loads in 6.13 seconds on App Engine, you aren't going to bother moving it to Azure to save 17 millis. You're going to fire all the programmers and re-write it so it loads in < 500ms. Then you can probably start thinking about platform variance.

Horrible programming, or it's a benchmark.They most likely had it do some nontrivial amount of bullshit work in order to highlight differences in execution speed between the different providers. If all the dynamic requests came back in 100ms and each platform's score was almost entirely dictated by network latency and bandwidth, it wouldn't be very interesting.

A much better benchmark would be a sane application with fast load times, loaded 1000 times, and measuring the median and 99th percentile performance.

Anyone paid just a little attention to the subject kows that Microsoft online service track records are the worst of all and Azure is catching up fast with the rest of MS online offerings in the race to the bottom: http://www.windows7news.com/2011/06/22/ ... ure-cloud/ (first link from Google BTW.)And that's after we didn't mention the day-long outage in 2009 and back...

I'm working with Azure right now, so that's what I'm familiar with. I just wanted to point out a few misleading points in the comments here about it. I'm not trying to be biased, Azure is just what I know more about.

The day long outage was in 2009. Azure was not officially released until 2010 (to paying customers), so it's hard to count an outage in a beta system against its production state. That's why betas exist.

The several outages listed this year were not with Azure, they were with the various Microsoft Online Services, like BPOS, Office 365, Hosted Exchange/Sharepoint, etc. Complaining about that is relevant if you use those services, but it's not the same as having a problem with Azure itself.

I thought it was strange that they tested two North American EC2 facilities (Virginia, East and California, West) but only one Azure facility (Chicago, North) when there is another Azure data center in Texas (San Antonio, South).

Azure has a CDN, too, but I don't think that's what they were testing. If you look at some of the specific locations, there are some that come back in well under a second. The only reason any provider would return that quick sometimes, and more than ten times that other times, is network latency. That appears to be what this is testing. I don't think they can test compute, as the definition of an instance differs between providers and it would be hard to set up a level playing field.

When it comes to describing the differences between the various cloud service providers, it's probably good to start with understanding the difference between IaaS, PaaS, and SaaS, which Ars covered a few years ago. Each provider covers subsets of one or more. I'm using Azure's PaaS offering, though they have an IaaS option, too.

This article will likely lead many people to make bad business decisions. I recently had to choose a cloud hosting provider for a startup and I can say that Windows Azure is second only to Amazon as being one of the worst options you can choose.

Amazon and Microsoft's core businesses will continue operating no matter what outage occurs. In Amazon's case, its business operations should continue even if it losses an entire datacenter. Microsoft really doesn't make money as a result of its website, but I imagine that its website is designed similarly. As a consequence, they have much less reason to ensure that their hosting services maintain 100% reliability and their SLAs show it.

If you contrast that to a company like Rackspace, which ranked #2 in the 30-day rankings, there is a remarkable difference in quality. Before I made my recommendations, I looked at information about past outages. In the case of Rackspace, I could find no outage lasting more than an hour. In the case of Amazon, I found outages spanning entire days. Microsoft is no different:

This article is misleading in many ways and it will likely lead to some very bad business decisions. With that said, I chose Rackspace before I knew that these benchmarks even existed and the benchmarks seem to validate my choice.

Your article from eweek is very misleading too. You quote an outage of a test release that happened back in 2009 when they were getting the service ready. Do you have any data to back your claim up of extended outages after they RTM'ed? And you mentioned "several days" when in the test release it was 22 hours. In the test release.For your "business descision" based on their SLA they state 99.95 external connectivity which for your 30 day period works out to a downtime of 0.36 hours. Not sure where you're getting your data from but it definitely looks inaccurate.

As I see it all these offer various services and some are overlapping. I would really like to see a write-up on this. It's an interesting field, especially for smaller startups.

What I did is I literally went to Google's App Engine site and found they have the equivalent of Amazon's Micro (free) instance. They don't charge under a certain amount of traffic and space used. I then followed their Python tutorial which makes a basic message posting service site and added an RSS parser/reader part in order to get a feel for the whole thing. Even just doing this gave me a pretty decent flavor of what to expect. It only took me probably 2 hours to follow their tutorial and the rest was just for fun

I really don't understand this test at all. What kind of asstastic application are they using that has mean response times of 5-6 seconds in the best case? Where I work we measure frontend latency in the tens of milliseconds.

The Methodology is published on the site and answers your question. Those response times are a collective 2 page load, one of which contains a large image.

Yes, thanks for cuntpasting the whole article I already read.

I too am interested in this cuntpasting, can we have an article on that? Kidding aside, is this test simply testing the network infrastructure of the different cloud providers? Static html + "big" picture is probably not testing much else.

I had the chance to work on Amazon EC2 and OpSource for my internship to study the network performances and the migration of VMs and unless you specify what instance type (http://aws.amazon.com/ec2/instance-types/) you are dealing with on Amazon EC2, you are talking in the void.

What is astonishing, is that during the whole explanation of the tests, they mention nowhere the type of instances used on Amazon! Whether they don't want to say that they use the free micro instances (that get almost no CPU time at all and are not suited for computational tasks) or they have no idea what are these, and thinking that Amazon provides one type of computing power in all the instance types (which is unlikely).

I had the chance to work on Amazon EC2 and OpSource for my internship to study the network performances and the migration of VMs and unless you specify what instance type (http://aws.amazon.com/ec2/instance-types/) you are dealing with on Amazon EC2, you are talking in the void.

What is astonishing, is that during the whole explanation of the tests, they mention nowhere the type of instances used on Amazon! Whether they don't want to say that they use the free micro instances (that get almost no CPU time at all and are not suited for computational tasks) or they have no idea what are these, and thinking that Amazon provides one type of computing power in all the instance types (which is unlikely).

Tl;dr A void and groundless study.

All of the major cloud service providers offer instance types like that. That's not unique to Amazon.

From the study (linked above): "We deploy an identical target application (a cute/fake e-commerce site that sells boots) to each cloud service provider. The Gomez Performance Network (GPN) is used to run test transactions on the deployed target applications and monitor the response time and availability from multiple points around the globe."

From the sound of it, instance types would most likely be accounted for in some fashion. I'd suggest you contact the authors to find out more information on this facet before you debase everything on a whim.