CloudHarmony

Sign In

Cloud Benchmarking

One of the greatest benefits of cloud computing is in allowing organizations to focus more on their core competencies and less on managing IT infrastructure. Organizations can save time, resources, costs, and generally end up with a better IT infrastructure by allowing dedicated cloud vendors deal with the hassles of researching, purchasing, implementing and maintaining computer hardware and networks.

Because cloud vendors often share compute and networking resources between multiple clients, and generally do not disclose the technical details of the infrastructure (i.e. CPUs, hard drives, storage architecture, etc.), it can be very difficult for organizations to anticipate cloud performance. To make matters worse, cloud vendors often use very different terminology when referring to resource allocation. This in our opinion is a significant barrier to entry to the cloud. One of our primary goals is to eliminate this pain. To do so, we have developed, and continue to improve upon a set of benchmarks we use to provide objective, independent performance comparisons between different cloud providers. Using this data, organizations can more quickly compare providers and adopt cloud services with reasonable expectations for performance. The benchmarks we provide fall into 3 categories:

Performance is an important factor in evaluating cloud services, particularly for common cloud applications like web and database servers. Our CloudBench benchmark suites and test software fascilitates such testing and analysis of associated metrics. To date we have run tens of thousands of benchmarks in all of the major public server clouds. In doing so, we have found a very broad range of performance characteristics. These characteristics include CPU, disk IO, memory, and other performance factors, measured using about 100 synthetic and real-world benchmarks like Unixbench, IOzone, SPECjvm2008 and many others. Organizations should consider these characteristics when selecting a cloud service in order to establish reasonable expectations and growth planning.

Please contact us for more information about CloudBench, our benchmarking services or to schedule a demo or walkthrough. Click here to purchase reports containing CloudBench performance data.

Network throughput and latency is another important cloud performance characteristic. Our CloudProbe network testing softlayer fascilitates such testing and analysis of associated metrics. Performance measurements include both network performance within the cloud as well as out of the cloud. Organizations should choose a cloud service that will provide good throughput and low latency to themselves and their potential users or customers. For example, a European based organization will most likely observe better network performance utilizing a service that is also physically located within Europe.

To measure network performance, we monitor latency and throughput using a couple of techniques. For cloud-to-cloud performance we utilize a network of about 50 servers located throughout the world to periodically measure and record latency and throughput measurements to other clouds. For cloud-to-consumer performance we host a cloud speedtest which measures latency and throughput using a browser-based application. This data is published periodically on our blog and also available by subscribing to one of our our network performance reports.

Please contact us for more information about CloudProbe, our network testing services or to schedule a demo or walkthrough. Click here to purchase reports containing CloudProbe performance data.

Service Monitoring

Service availability or uptime, is a critical cloud performance characteristic. It is often assumed that cloud providers have measures in place to provide very high availability. To measure uptime, we have setup services with most public cloud vendors and have monitors in place to track any outages of those services. We typically follow up with the vendor after an extended outage and append supporting comments to the uptime report. We also exclude scheduled maintenance. We have found very little correlation between SLAs and actual uptime. Since we began monitoring services around January 2010, we have observed significant variances between the availability of different cloud services. If uptime is an absolute critical requirement for your organization, certain cloud providers may be a better fit than others based on historical performance. The Cloud Status section of our website allows users to view and filter historical uptime statistics.

Consulting

We are available for cloud consulting engagements related to benchmarking and service selection assistance. If you are interested in discussing how we might assist in your next cloud project, please contact us. Services we can provide include:

Cloud Benchmarking Consultation

We are available for both public and private cloud benchmarking consulting engagements. Benchmarks can be kept confidential and shared only with the client. Benchmarking consultation includes access to raw benchmark result data and high level reports including graphs and comparative data.

Cloud Selection Assistance

If you are in an organization currently looking to migrate to the cloud, we can help. Our knowledge of the different cloud services available, features, pricing, and access to extensive performance metrics allow us to recommend the services that will best fit your project needs.

CloudHarmony is independently owned and operated. We do not accept kickbacks or commissions for promoting one cloud service over another. Simply put, our goal is to provide objective, measurable, and unbiased analysis on cloud services.