How to Test & Benchmark CDNs?

I recently came across a very interesting post about CDN testing by Jonathan Klein, someone I respect tremendously in the Web Performance world. During his tests with Webpagetest he was unable to “see” the value provided by the CDN on his site.

I wanted to follow-up on his blog and share some of the knowledge we have acquired both from our experiences at DoubleClick (building our own CDN in 1997, then using various commercial CDNs like Speedera, Akamai, then using a combo Internal + Akamai until Google’s acquisition and switching to the Google stack) and now with Catchpoint constantly monitoring CDNs.

They have more engineers dedicated to optimization of various stacks to deliver things faster…

They can deliver content close to the end-user because they have servers in major cities.

They have teams monitoring this stuff 24/7.

They can handle huge spikes.

In other words they leverage economies of scale to provide you with a service that is faster, better, and more cost-effective than what it would be if you relied on your own infrastructure.

Testing Methodology

So how do you test and benchmark CDNs to make sure they deliver on their promise? This is not so much about the toolset, some will say that Backbone monitoring is not accurate; it’s about the methodology. A benchmark is a benchmark, if there are issues just adjust the lens & zoom in you will see the differences and problems.

The testing period is very important, make sure you test for a long period of at least 1 to 2 weeks! And test as frequently as possible! (I was recently involved in multi CDN benchmark where a company used 1 Million + data points from Catchpoint alone, other tools were used).

Before you start your tests, please make sure the DNS TTL for all URL benchmarked are the same. For example you cannot have a TTL of 60 seconds for cdn.test.com and a TTL of 3,600 seconds for origin.test.com and a TTL of 300 seconds for cdn2.test.com.

Please make sure you are comparing the same file size.

The testing methodology I have used and have seen others use relies on a 2-phase approach:

Phase1: Static Content Performance Testing

Use an external performance tool (Synthetic Backbone or Last Mile, RUM, WPT…) to load 2 pair of files of various sizes (5kb, 10kb, 50kb, 100kb, 500kb…) from both the CDN URL and the Origin URL.

Phase 1 Key Take Aways:

Your want to make sure that the CDN is faster than your Origin.

You want the CDN to be good at delivering a 5kb file but also a larger payload.

One of the goals of CDNs is to minimize the geographical impact by reducing the latency to fetch all the bits necessary to make your page load faster. So make sure to understand the performance impact by geography before and after or origin vs. cdn url.

Phase 2: Web Page Performance Test

Create a custom html page based on your existing webpages. Place CSS files, JS files and Images but NO third parties (ads, widgets, tracking, etc), something that matches your setup (CDNs cannot do anything about 3rd parties). Same as the previous phase, Page A will hit the CDN and Page B will hit your Origin. Your CDN will make your site faster but cannot make the ads load any faster. Monitor and Measure both pages using the same tools as in phase 1.

Phase 2 Key Take Aways:

Page with CDN is faster, look at render start time, document complete, total web page response.

Geographical performance improvements.

Metrics & Data to look for

– DNS time. Some CDNs have more complex DNS setup than others and can slow things down. What I have seen is the time gained in Wait time was diluted by slower DNS response time.

Keep in mind that DNS performance from the last mile, end user, is quite different from the tests run in the backbone. End users rely on DNS resolvers of their ISPs or Public Resolvers. Backbone monitoring relies on resolvers that very close to the machine running the tests.

DNS Lookup time of Various CDNs (US only)

– Connect time: This is to make sure your CDN has great network connectivity, low latency and no packet loss. Additionally you want to make sure it does not get slower during peak hours and they are routing you to the right network peering. Example if an end-user is on Verizon FIOS there is no reason to go through 5 different backbone networks because that CDN does not have a direct peering with Verizon.

Connect Time of Various CDNs (US Only)

– Wait time: This metrics is important when looking at various CDNs, it helps you see if your content is hot on the edge or that does edge needs to fetch it from the Origin servers. The Wait time is also an indicator of potential capacity issues or bad configuration at the CDN level or Origin server (for example setting cache expiration in the past). A CDN will deliver different performance if an asset is hot, requested 100,000+ times in the past hour vs. a few times an hour. A CDN is a shared environment where more popular items are faster to deliver than others, if something is in memory it’s fast, if it has to hit a spinning disk it’s a different story. Thus I would personally consider having Solid State Drives as criteria in my CDN selection.

Wait Time of Various CDNs (US Only)

– Throughput: Make sure that the throughput of the CDN test is higher than the origin no matter what the file size is!

– Traceroutes! You need to run traceroutes from where you are monitoring to make sure you are not mapped to the wrong place. Many CDNs use commercial geo-mapping databases and the data for the IP could be wrong. From my Time Warner home connection in Los Angeles, some CDNs sent requests to UK (at times).

Other things to keep an eye on

– Most CDNs will give you access to a control panel so make sure you monitor your Cache Hit / Miss ratio. How often do they have to come back to the Origin! A good CDN architecture should not come often to the Origin. We disqualified various CDNs at DoubleClick because they would not agree to our miss ratio SLAs. You have to also ask questions about what happen when an edge server does not have that content? How long does it take to purge a file? How long does it take to load a file to the edge? How long before a Cname is active?

– How well the CDN handles public Name Resolvers such as OpenDNS, Dyn, Google? These companies are carrying more and more of the DNS traffic and this could impact certain CDNs geo-load balancing algorithms.

– Are the metrics from the CDN consistent? DNS, Connect, Wait and Response (Please do not just look at averages), remember Great performance is more than speed, its reliability or ability to deliver a consistent experience.

Major CDN vendor’s Response Time by Hour of Day (US Only)Major CDN’s Response time – Different Statistical Models (US Only)

Conclusion:

So after doing all these tests, the simple questions that must be answered are:

Did the CDN improve my performance in key markets where we lack physical server presence?

By how much? Is it a 5% improvement, 20% or 200%?

Will this translate into revenue increase? Put in place a way to monitor the long-term impact of CDN on your revenue. The reason I used long terms is because you have probably lost those users due to slow performance, so you have to give them time to come back! (Are users in California placing more orders more since a CDN was introduced and reduced response time by 2 seconds in that market?).

Now beside speed, CDNs do bring other benefits that are not measured in seconds:

By offloading your static files you have more servers, more bandwidth and personnel to deal with more important things than serving static files.

They can help you handle DDOS attacks.

They are better prepared to handle seasonal traffic.

And once you have selected a CDN, and are up and running on a CDN platform, keep an eye on them, always monitor a file from the CDN and the Origin at all times. Another observation I can share with you is a CDN is not a fire & forget technology, you have to stay on top of them, make sure the configuration is up to date, that the GZIP is always on… I have been on many interesting calls where I unfortunately hear a Sales Engineer from a CDN company say “oops, we forgot to turn that on after our last release” or we need to tweak your “map”…”.

I welcome comments, suggestions and tips to help create a common knowledge base about CDN benchmarking. I am also looking forward to the RUM data that Jonathan is going to publish!

You Might Also Like

Comments (7)

This is a wonderful article, great job writing it.
We at Turbobytes (www.turbobytes.com) know for a fact that not all CDNs are equal when it comes to performance. We collect RUM data from all over the world all the time and this shows big differences on a daily basis.
They takeaways I like most from your article are:

1. “CDN is not a fire & forget technology, you have to stay on top of them”.
So true. The best way to do this is to continuously monitor CDN performance and *use* the data to improve.

2. Test with a full page, with no 3rd party stuff on it.
Spot on. Measuring how much faster a single object loads is good, but it’s definitely best to look at a full page too.

3. “make sure to understand the performance impact by geography before and after or origin vs. cdn url”.
Yes, you need to have data in place for all important geographical markets.
If US is you key market, get data in place for the top 10 states. You want awesome performance in California, but may care less about North-Dakota.

You pretty much cover it in the discussion after the testing section but even with the 2-phase testing there are things you need to be really careful with:

Hot/Cold resources. If you are repeatedly testing the same file (or set of files in a contrived page) then you are probably hitting those files often enough on a given edge that it will always be considered hot during your period of testing. To get more realistic results you need to include a mix of frequently requested resources (js/css) and long-tail resources (product images or some such) and preferably move the testing around a lot so it doesn’t give a lot of benefit to a small set of edge servers. Even the cache miss reports from a provider could be giving an artificially rosy picture if they had to fill the resource from disk or from another server but not go all the way back to the origin.

In phase 2, be really careful with the type of connectivity you do your testing on. Backbone testing near a CDN edge can hugely over-emphasize the benefit of latency reduction compared to what actual users will see.

Eliminating 3rd party resources from the test will give you an interesting data point but unless you actually serve pages to your users without 3rd-party content you still need to understand the real-world impact. Segmenting traffic and doing a real live test is the ultimate test (once you have narrowed things down).

I’m a huge believer in CDN’s (heck, that’s why it is one of the top-line checks on webpagetest). Measuring the impact on end-user performance is just complex (and as you mention, not nearly the only reason for using one).

The CDN solution for one market may not be the most appropriate one for another. This adds complexity to vendor selection, but if a CDN is aware that you are making a choice based on the strength of coverage in a number of global regions, this may make them pay more attention to how you’re testing.

I agree with Patrick (as you probably guessed I would): while datacenter measurements give you a good sense of the consistency of delivery, measurements from the edge – real-user and synthetic – will be the litmus test of truly effective coverage and service.

I agree with you Stephen on the geography matters issue. For example very few CDNS have a real solution for China. Delivering into China from HK or Japan is not China. I have been burned by our CDN at DoubleClick during the Olympics 2000 in Australia where the Gomez and Keynote agents were served out of Australia but users were not because BW is expensive in AU, still is! I will never forget the nasty call I had with the CMO of IBM.
One thing I forgot to add to my methodology is: Change the user agents and add Random numbers to the tests.