Speed of the Clouds

In Price of the Clouds we compared prices of running servers in various clouds. In this article we will compare performance. We will compare the same server configurations we compared prices for in the previous article, i.e. as clos as we can get to 2 vCPU’s, 4 GB RAM and SSD-disks. We’ll measure performance on processors, RAM, storage and networking.

We have chosen not to include measurements for a private VMware installation. This is due to the fact, that we would mostly just be measuring performance on the hardware purchased for the occasion, which could vary very much, depending on our more or less smart choice of hardware. We also will not include measurements for an Origo Private cloud, due to the fact that our public cloud Origo Cloud runs on exactly the same hardware, and therefore would be expected to give the exact same results.

As background information, it is worth noting that almost all servers today are run on hardware which is very much identical, with only small variations over the basic theme of Intel processors, Intel chipsets, standard ECC-RAM, SSD’s and network interfaces. Going into this test we would therefore expect only minimal variations in basic performance, but let’s jump right into it.

We have chosen to perform 5 measurements of:

CPU (max. prime calculations)

Disk IOPS (disk I/O operations per second)

Disk I/O (read/write performance)

RAM (read/write performance)

Networking (we measure bandwith to a test server i Schwitzerland)

We have chosen to use commonly tried and tested Linux tools, which we run in a virtual server with Ubuntu 16.04 installed in 3 different clouds. We have also added a less technical measurement, which is counting the number of clicks it takes to start the chosen virtual server in each cloud. The raw measurement numbers are included at the bottom of this article.

We have to stress that this is neither a scientific nor an authoritative analysis. A more thorough analysis would demand longer and more varied measurements on different times of the day. Under the chosen limitations it is however our opinion, that the results are reliable within a reasonable margin of error of say +/- 10%.

For the technically interested we are listing the incantations we have performed below:

CPU

All the analyzed servers are configured with 2 vCPU’s. Single-threaded x86 CPU-performance has in fact not improved much for a number of years. Our measurements are double-threaded, miroring the chosen number of vCPU’s, and we would expect reasonably close results. In reality we are mostly analyzing if, and if so, how much the vendor is over-commiting the physical CPU’s. The chosen server in AWS is of the type “CPU optimized”, which obviously had us expecting it to deliver great CPU performance. As shown, AWS does in fact deliver a bit better performance than Digital Ocan, but a bit poorer than Origo Cloud. Digital Ocean is a clear looser in this test, but the differences are really too small in our opinion to have any impact on your choice of cloud.

IOPS

Many if not most real-world performance-related issues boil down to bottlenecks relared to reading and writing to and from disk, especially when databases are involved. The number of available I/O operations per second is therefore an important factor, if you have to choose where and how to run a demanding application. As can be seen from the measurements, AWS is a very clear looser in this test.

However the measurements are not entirely fair to AWS, as they basically just show that in AWS you get exactly what you pay for, which all things considered, in this case is not very much. The measurements show that AWS delivers very close to the promised 400 IOPS, which was also the maximum we could configure for our server, as we only chose a 8 GB disk. AWS has two storage models: “Baseline”, where you 3 IOPS available per GB (with a minimum of 100), and “provisioned”, where you choose (and pay for) up to 50 IOPS per GB. Amazon indicates that baseline storage is able to “burst” up to 3.000 IOPS. Our measurements show that evidently this is not the case for provisioned storage.

The test results mirror the storage infrastructure of each cloud vendor. This is also the area where we would expect the largest technological difference between the clouds, since there really is no widely used, scalable standard technology available. The rather large difference between read and write performance in Digital Ocean is a little bit unusual. Read performance is very good, but write performance is under 1/4 of this! Origo Cloud delivers consistently good both read and write performance.

I/O

For this test we have chosen to consolidate measurements of reading/writing data to network, disk and RAM in one diagram. Besides illustrating the differences between the selectec clouds, it also illustrates pretty well the speed differences between the different layers our data pass through.

Origo Cloud delivers the fastest disk I/O, which is in line with the fact that this test measures both reading and writing, and therefore is affected by Digital Oceans limited write performance. AWS is of course affected by the 400 IOPS limitation. Again – you can buy better performance, but it will cost you.

Standard networking I/O with AWS and Digital Ocean is in the 50 Mbit/s neighborhood, which is hardly impressive. Origo Cloud delivers around 100 Mbit/s, which is a bit more useable. As with storage you may also here purchase better performance with AWS.

With arguments along the same line as for CPU performance, we had expected RAM performance to be closer than they turned out to be. Digital Ocean comes out as the looser in this test.

Conclusion

We are obviously not neutral, but according to our measurements, Origo Cloud looks like the all-round best performing cloud when comparing standard configurations. Among the clouds compared, AWS is, according to our previous article by far the most expensive cloud. At the same time it is clearly the worst performing cloud in this test, mostly because of their storage model, where disk performance either depends on the amount of storage you purchase or on supplemental purchases. To be clear, you can purchase better storage performance in AWS, but of course this does not exactly lower the price. The difference for the other measurements is not enormous, but large enough to surprise us a little. We do not see any clear performance-related benefits to choosing the market-leader AWS for regular server operations – on the contrary, and we do not have any reason to expect things to be much different with e.g. Microsoft Azure. When discussing the relatively high premium you are paying as an AWS customer, you are often presented with the argument that AWS has so many other great features besides IaaS. We don’t disagree with that, but we have to point out the disadvantages that come with too much vendor lock-in. AWS already looks very much like a one-way street. For most businesses by far the most relevant other feature AWS offers is their CDN, called CloudFront. As pointed out before there exists a great free alternative to this called CloudFlare.

In short – it looks like the smaller IaaS clouds, including our own, deliver better performance at a lower price compared to the market-leaders. This of course is not entirely unexpected – it’s called competition. What’s not expected is the size of the performance- and price-gaps, which did surprise us. So we would like to encourage more businesses to expand their horizon a bit. When it comes to extra features, there is a thriving eco-system out there – it’s more affordable and you avoid vendor lock-in.

If you found this analysis interesting, you can look forward to the follow-up, where we will test user interface and API’s in various clouds hands-on. Also remember that you can subscribe to our news letter, if you would like articles from us directly in your inbox.