CDN Debugging Tips: Part 1

Published on Aug 11 2017

This article is a first in a series of three on the topic of CDN debugging.
Here we will share three tips for collecting useful data, which really is the first thing you want to do when you believe something is wrong with your CDN.

curl -svo /dev/null

Curl (or cURL) is a command line tool widely used for testing and troubleshooting client-server data transfer over HTTP.
The free tool comes preinstalled on Linux and Mac OS X machines and curl for Windows is available too.

When using curl to debug CDN behavior, don't ever use curl -I.
Using the I flag results in sending a HEAD request and that is often pointless and unintended.
Your users will send GET requests not HEAD requests and the CDN may treat HEAD requests very differently from GET requests.
So remember: don't use curl -I, ever.

Your curl commands should start with curl -svo /dev/null.
The -svo /dev/null is a series of flags:

-s enables silent mode; don't show progress meter or error messages.

-v enables verbose output

-o /dev/null sends output to a file, but not really because the /dev/null item does not save anything (note: on Windows, use -o NUL

So, -svo /dev/null gives you a nice and clean display of information, most importantly the request and response headers.

Don't test against just one POP

When debugging your CDN, make sure you get don't fall in the trap of looking into the behavior of just one POP.
It's important you find out if the problem is POP specific or not and the only way to do that is to test against several/many POPs.
But how do you do that?

Services like Pingdom or Catchpoint have test machines across the globe, and this is great because you can perform tests against POPs in many locations.
However, the visibility you can get using such services is suboptimal.
For example: in France, Level 3 CDN has POPs in Marseille and Paris, but Pingdom only has servers in a datacenter in Strasbourg.
When testing from the Pingdom servers in Strasbourg, you'll likely hit the same one of two Level 3 POPs every time and consequently know nothing about the other POP.

Ideally, your CDN provides you a way to send a request to a specific POP, so you can run curl tests against each and every POP comfortably from your office computer. Unfortunately, not many CDNs provide this feature.
As far as we know, EdgeCast and Highwinds (part of StackPath) enable you to debug a specific POP.
Fastly enables sending requests to a specific cache node (e.g. cache-lax1427) which is nice but only useful if you know the cache node that is serving your content. In our opinion, it's more useful to be able to send a request to 'the Amsterdam POP'.

To send a request to a specific target endpoint (e.g. a CDNetworks server in Amsterdam), use the -H flag to add a Host header with your domain:

Test from real user networks, not datacenters

Even if you can run curl tests against specific CDN POPs from your office location, or from datacenters across the globe, you're still somewhat in the dark.
Real users don't live in your office or in datacenters. They are at home, at work and on the go and connected to last-mile networks.

It's not uncommon for a CDN's performance to go bad only for users on one particular last-mile network.
The problem may be a congested network path, suboptimal routing (e.g. users on AS9143 (Ziggo) in NL are routed to New York instead of Amsterdam) or the ISP's DNS resolvers can't reach the CDN's DNS nameservers.
The only way to spot these kind of issues is by testing from devices connected to the last-mile (or: eyeball) networks.

The free tool TurboBytes Pulse enables you to quickly collect DNS, HTTP and Traceroute responses from many computers around the world. Most of these 'agents' are connected to consumer ISP networks. Pulse currently has ~70 agents online at any time, including in the United States, Canada, many countries in Europe, Brazil, Australia, India, Thailand, Taiwan and Vietnam.

About us

CDN Planet is built and operated by Aaron Peters and Sajal Kayan.
As former web performance consultants and the current co-founders of TurboBytes (Multi-CDN),
we have 6+ years experience with all things CDN, including DNS, performance monitoring and networking.