This is the fourth post in our series on cloud server performance benchmarking. In this post, we'll examine memory IO performance using a compilation of 7 different benchmarks. This is the last of the performance posts to focus on synthetic benchmarks. Memory IO performance is particularly important for applications that read and write from memory heavily like caching systems and memory-based data stores such as memcache and redis.

Benchmark Setup

All benchmarked cloud servers were configured almost identically using CentOS 64-bit (or 32-bit in the case of EC2 m1.small, c1.medium, Gandi, and IBM cloud servers).

Benchmark Methodology

We chose to use a dedicated, bare-metal cloud server as the performance baseline for this post. This will provide a more readily comparable reference to non-cloud configurations. The server we chose as the baseline is the NewServers Jumbo server configured with dual Intel E5504 quad core 2.00 GHz processors and 48GB DDR3 ECC ram. We chose NewServers because they are the only IaaS cloud that does not utilize a virtualization layer that could adversely affect the benchmark results. All NewServers servers run on top of physical hardware. We assigned the baseline server a score of 100. All other servers were assigned a score proportional to the performance of that server, where greater than 100 represents better results and less than 100 represents poorer results. For example, a server with a score of 50 scored 50% lower than the baseline server overall, while a server with a score of 125, scored 25% higher.

To compute the score, the results from each of the 7 benchmarks on the baseline server are compared to the same benchmark results for a cloud server. The baseline server benchmark score represents 100% for each benchmark. If a cloud server scores higher than the baseline it receives a score higher than 100% (based on how much higher the score is) and vise-versa for a lower score.

Benchmarks

The following benchmarks were used to calculate the aggregate memory IO performance (MIOP) score displayed in the results tables below. Geekbench and Unixbench measure both CPU and memory IO performance and are included in the aggregate MIOP metric with a weight of 25 points each. To view the raw Geekbench and Unixbench scores for each of the servers in this post see the post What is an ECU? CPU Benchmarking in the Cloud.

CacheBench [weight=100]: This is a performance test of CacheBench, which is part of LLCbench. CacheBench is designed to test the memory and cache bandwidth performance. 50 points are assigned to each of the benchmarks read and write.

Geekbench [weight=25]: Geekbench provides a comprehensive set of benchmarks engineered to quickly and accurately measure processor and memory performance. Designed to make benchmarks easy to run and easy to understand, Geekbench takes the guesswork out of producing robust and reliable benchmark results.

hdparm cached reads [weight=50]: Determines the speed of reading directly from the Linux buffer cache without disk access. This measurement is essentially an indication of the throughput of the processor, cache, and memory of the system under test.

RAMspeed [weight=100]: This benchmark tests the system memory (RAM) performance. 33 points are assigned to each of the benchmarks add, copy and scale.

Redis Benchmark [weight=50]: Redis is an in-memory key-value store. It includes the redis-benchmark utility that simulates SETs/GETs/INCRs/LPUSHs/LPOPs/PINGs/LRANGEs done by N clients at the same time sending M total queries (it is similar to the Apache's ab utility). Our benchmark is performed with 50 simultaneous clients performing 100000 requests (./redis-benchmark -n 100000). The result is the average requests per second for all benchmark actions (gets, sets, etc.). For specific results, download the benchmark output source.

Stream [weight=100]: This benchmark tests the system memory (RAM) performance. 25 points are assigned to each of the benchmarks add, copy, scale and triad.

Unixbench [weight=25]: UnixBench provides a basic indicator of the performance of a Unix-like system. Multiple tests are used to test various aspects of the system's performance. These test results are then compared to the scores from a baseline system to produce an index value, which is generally easier to handle than the raw scores. The entire set of index values is then combined to make an overall index for the system.

Results

The results are divided into tables separated by provider. If the provider has more than one data center location, multiple tables are included for each location. Each table shows the server identifier, CPU architecture, memory (GB), Redis Benchmark score (linked to the complete Redis benchmark results), and the aggregate baseline relative score (as described above). The top performers in this benchmark category were EC2, Storm on Demand (6 servers with scores 116 and above), BlueLock, and IBM Developer Cloud.