Blogs on CDN Planethttps://www.cdnplanet.com/blog/feed/
Recent content in Blogs on CDN PlanetHugo -- gohugo.ioen-usMon, 12 Mar 2018 00:00:00 +0000Introducing the CDN Performance Checker toolhttps://www.cdnplanet.com/blog/introducing-cdn-performance-checker-tool/
Mon, 12 Mar 2018 00:00:00 +0000https://www.cdnplanet.com/blog/introducing-cdn-performance-checker-tool/<p>
Getting complaints about your site being slow or your app not loading?<br>
Validate your CDN's performance with our new <a href="https://www.cdnplanet.com/tools/cdnperfcheck/">CDN Performance Checker</a> tool!
</p>
<p>
The <a href="https://www.cdnplanet.com/tools/cdnperfcheck/">CDN Performance Checker tool</a> instantly checks if an object will load fast from your CDN.
The tests run from 10 residential locations across the globe: US (Cleveland and Seattle), Canada, Ireland, France, The Netherlands, India, Philippines, Vietnam and Australia.
</p>
<h2>Eyeballs, not datacenters</h2>
<p>
Many performance monitoring services and tools use test machines in datacenters. This is not great.
Your CDN may have great connectivity to some datacenter, but poor performance getting your content to your actual users (often referred to as 'eyeballs').
The best insight into CDN performance is gained from testing from the perspective of real users, who use the Internet at home, at work and on the go.
This is what our <a href="https://www.cdnplanet.com/tools/cdnperfcheck/">CDN Performance Checker</a> tool does, powered by TurboBytes Pulse.
</p>
<p>
<a href="https://pulse.turbobytes.com/" class="external-link">TurboBytes Pulse</a> (built and operated by us!) is a free online tool for diagnosing CDN and DNS performance. Pulse has ~60 locations around the globe currently and most of these are in people's homes. We handpicked 10 locations for use on the CDN Planet site in the CDN Performance Checker tool.
</p>
CDN Debugging Tips: Part 1https://www.cdnplanet.com/blog/cdn-debugging-tips-part-1/
Fri, 11 Aug 2017 00:00:00 +0000https://www.cdnplanet.com/blog/cdn-debugging-tips-part-1/<p>
This article is a first in a series of three on the topic of CDN debugging.
Here we will share three tips for collecting useful data, which really is the first thing you want to do when you believe something is wrong with your CDN.
</p>
<h2 id="1">curl -svo /dev/null</h2>
<p>
Curl (or cURL) is a command line tool widely used for testing and troubleshooting client-server data transfer over HTTP.
The free tool comes preinstalled on Linux and Mac OS X machines and <a href="https://curl.haxx.se/download.html" class="external-link">curl for Windows</a> is available too.
</p>
<p>
When using curl to debug CDN behavior, don't ever use <code>curl -I</code>.
Using the <code>I</code> flag results in sending a HEAD request and that is often pointless and unintended.
Your users will send GET requests not HEAD requests and the CDN may treat HEAD requests very differently from GET requests.
So remember: don't use <code>curl -I</code>, ever.
</p>
<p>
Your curl commands should start with <code>curl -svo /dev/null</code>.
The <code>-svo /dev/null</code> is a series of flags:
</p>
<p><code>-s</code> enables silent mode; don't show progress meter or error messages.</p>
<p><code>-v</code> enables verbose output</p>
<p><code>-o /dev/null</code> sends output to a file, but not really because the /dev/null item does not save anything (note: on Windows, use <code>-o NUL</code></p>
<p>
So, <code>-svo /dev/null</code> gives you a nice and clean display of information, most importantly the request and response headers.
</p>
<p>
The <code>-H</code> flag can be used to send an arbitrary request header, for example a specific User-Agent string: <code>-H 'User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25</code>
</p>
<p>
Below you see an example curl command for fetching the (gzip) compressed version of <code>http://edition.cnn.com/css/2.33.0/global.css</code>.
</p>
<pre>
curl -svo /dev/null --compressed 'http://edition.cnn.com/css/2.33.0/global.css'
</pre>
<!-- <p><a href="https://www.cdnplanet.com/blog/cdn-debugging-tips-part-2/">CDN Debugging Tips: Part 2</a> shows how the <code>-H</code> flag comes in handy for getting extra debug info back from the CDN.
</p> -->
<p>
Learn more about curl and all the flags by reading the <a href="https://curl.haxx.se/docs/manpage.html" class="external-link">curl manual page</a>.
</p>
<h2 id="2">Don't test against just one POP</h2>
<p>
When debugging your CDN, make sure you get don't fall in the trap of looking into the behavior of just one POP.
It's important you find out if the problem is POP specific or not and the only way to do that is to test against several/many POPs.
But how do you do that?
</p>
<p>
Services like Pingdom or Catchpoint have test machines across the globe, and this is great because you can perform tests against POPs in many locations.
However, the visibility you can get using such services is suboptimal.
For example: in France, Level 3 CDN has POPs in <a href="https://www.cdnplanet.com/cdns/level3/#network">Marseille and Paris</a>, but Pingdom only has servers in a datacenter in Strasbourg.
When testing from the Pingdom servers in Strasbourg, you'll likely hit the same one of two Level 3 POPs every time and consequently know nothing about the other POP.
</p>
<p>
Ideally, your CDN provides you a way to send a request to a specific POP, so you can run curl tests against each and every POP comfortably from your office computer. Unfortunately, not many CDNs provide this feature.<br>
As far as we know, EdgeCast and Highwinds (part of StackPath) enable you to debug a specific POP.
Fastly enables <a href="https://community.fastly.com/t/ways-to-test-your-caching-with-curl/480/4" class="external-link">sending requests to a specific cache node</a> (e.g. cache-lax1427) which is nice but only useful if you know the cache node that is serving your content. In our opinion, it's more useful to be able to send a request to 'the Amsterdam POP'.
</p>
<p>
To send a request to a specific target endpoint (e.g. a CDNetworks server in Amsterdam), use the <code>-H</code> flag to add a Host header with your domain:
</p>
<pre>
curl -svo /dev/null -H 'Host: cdn.yourdomain.com' 'http://91.194.205.21/path/to/file'
</pre>
<h2 id="3">Test from real user networks, not datacenters</h2>
<p>
Even if you can run curl tests against specific CDN POPs from your office location, or from datacenters across the globe, you're still somewhat in the dark.
Real users don't live in your office or in datacenters. They are at home, at work and on the go and connected to last-mile networks.
</p>
<p>
It's not uncommon for a CDN's performance to go bad only for users on one particular last-mile network.
The problem may be a congested network path, suboptimal routing (e.g. users on AS9143 (Ziggo) in NL are routed to New York instead of Amsterdam) or the ISP's DNS resolvers can't reach the CDN's DNS nameservers.
The only way to spot these kind of issues is by testing from devices connected to the last-mile (or: eyeball) networks.
</p>
<p>
The free tool <a href="https://pulse.turbobytes.com/" class="external-link">TurboBytes Pulse</a> enables you to quickly collect DNS, HTTP and Traceroute responses from many computers around the world. Most of these 'agents' are connected to consumer ISP networks. Pulse currently has <a href="https://pulse.turbobytes.com/agents/" class="external-link">~70 agents</a> online at any time, including in the United States, Canada, many countries in Europe, Brazil, Australia, India, Thailand, Taiwan and Vietnam.
</p>
<!--<p>
Read <a href="https://www.cdnplanet.com/blog/cdn-debugging-tips-part-2/">CDN Debugging Tips: Part 2</a> ... <strong>3 tips to <i>prevent</i> problems with your CDN.</strong>
</p>-->CDN market news and announcements in June 2017https://www.cdnplanet.com/blog/cdn-market-news-announcements-june-2017/
Wed, 05 Jul 2017 00:00:00 +0000https://www.cdnplanet.com/blog/cdn-market-news-announcements-june-2017/<p>
Learn what happened in the CDN market in June 2017: a new product offering by CDNetworks, new price plans at CDN77 and the all new Cloudflare Apps.
</p>
<h2>CDNetworks launches Cloud Security 2.0</h2>
<small class="pub-date">June 13 2017</small>
<p>
CDNetworks has announced the launch of Cloud Security 2.0. The newly released application includes next generation behavioral web application firewall (WAF) technology integrated into CDNetworks' global content delivery network.
</p>
<p>
Read the <a href="https://www.cdnetworks.com/en/news/cdnetworks-launches-cloud-security-20/3834" class="external-link" onclick="_gaLt(event);">full press release</a>.
</p>
<h2>CDN77 offers new monthly plans</h2>
<small class="pub-date">June 6 2017</small>
<p>
The new monthly plans are month-to-month based plans that come with PoPs in North America, Europe (incl. Russia and Ukraine), Istanbul, Hong Kong, Singapore and Seoul. Customers can cancel any time and switch back and forth between Pay-As-You-Go and mont-to-month.
</p>
<p>
<a href="https://www.cdn77.com/pricing/medium-volumes/monthly-plans" class="external-link" onclick="_gaLt(event);">CDN77 monthly plans</a>
</p>
<h2>Cloudflare announces the next generation Apps</h2>
<small class="pub-date">June 27 2017</small>
<blockquote>
Cloudflare Apps is an open platform of tools to build a high quality website. It's a place where every website owner can select from a vast catalog of Apps which can improve their websites and internet properties in every way imaginable. Selected apps can be previewed and installed instantly with just a few clicks, giving every website owner the power of technical expertise, and every developer the platform only Cloudflare can provide.
</blockquote>
<p>
Read the <a href="https://blog.cloudflare.com/cloudflare-apps-2/" class="external-link" onclick="_gaLt(event);">announcement on Cloudflare blog</a>
</p>
CDN market news and announcements in May 2017https://www.cdnplanet.com/blog/cdn-market-news-announcements-may-2017/
Thu, 15 Jun 2017 00:00:00 +0000https://www.cdnplanet.com/blog/cdn-market-news-announcements-may-2017/<p>
Learn what happened in the CDN market in May 2017, including a new product offering by Cloudflare, network expansion by Cloudfront and a big funding round for Fastly.
</p>
<h2>Limelight Networks promises to reduce video rebuffer rates by 10% or more</h2>
<small class="pub-date">May 16 2017</small>
<p>
Limelight Networks announced new performance and functionality advancements to the Limelight Orchestrate Platform, resulting in significant reduction in video sessions experiencing rebuffers. Limelight also introduced a money-back guarantee to reduce online video rebuffer rates for new customers.
</p>
<p>
Read the <a href="http://investors.limelight.com/file/Index?KeyFile=2000602234" class="external-link" onclick="_gaLt(event);">full press release</a> or visit the
<a href="https://www.limelight.com/video-rebuffer-promo/" class="external-link" onclick="_gaLt(event);">video rebuffer promo page</a>.
</p>
<h2>Cloudflare launches Argo, a "virtual backbone" for the modern Internet</h2>
<small class="pub-date">May 18 2017</small>
<p>
Argo analyzes and optimizes routing decisions across the global Internet in real-time.
Cloudflare claims Argo can "deliver content across our network with dramatically reduced latency, increased reliability, heightened encryption, and reduced cost vs. an equivalent path across the open Internet".<br>
Argo is priced at $5/domain monthly, plus $0.10 per GB of transfer from Cloudflare to end users.
</p>
<p>
Read the <a href="https://blog.cloudflare.com/argo/" class="external-link" onclick="_gaLt(event);">press release</a>.
</p>
<h2>Fastly raises $50 million</h2>
<small class="pub-date">May 23 2017</small>
<p>
Fastly has announced $50 million in funding led by Sorenson Capital with participation from additional new investor Sapphire Ventures. The funding will be used to meet growing demand for cloud services at the edge and to ensure long-term independence.
</p>
<p>
Read the press release <a href="https://www.fastly.com/press/press-releases/fastly-closes-50-million-meet-demand-edge-cloud-services" class="external-link" onclick="_gaLt(event);">here</a>.
</p>
<h2>CloudFront adds edge locations in Seattle, Dallas/Fort Worth and Tokyo.</h2>
<small class="pub-date">May 23 2017</small>
<p>
Amazon CloudFront has expanded its network with new edge locations in two cities in the United States (Seattle and Dallas/Fort Worth) and in Japan (Tokyo).
This brings the total number of CloudFront locations to 88 (including 77 points of presence and 11 regional edge cache locations).
</p>
<p>
Read the announcements <a href="https://aws.amazon.com/about-aws/whats-new/2017/05/announcing-second-edge-location-in-seattle-washington-for-amazon-cloudfront/" class="external-link" onclick="_gaLt(event);">here</a> and <a href="https://aws.amazon.com/about-aws/whats-new/2017/05/amazon-cloudfront-adds-our-fourth-edge-location-in-tokyo-japan-and-our-third-in-dallas-fort-worth-texas/" class="external-link" onclick="_gaLt(event);">here</a>.
</p>
CDN market news and announcements in April 2017https://www.cdnplanet.com/blog/cdn-market-news-announcements-april-2017/
Mon, 01 May 2017 00:00:00 +0000https://www.cdnplanet.com/blog/cdn-market-news-announcements-april-2017/<p>
Learn what happened in the CDN market in April 2017, including new product offerings, network expansions and an acquisition.
</p>
<h2>Fastly unveils edge cloud platform, introduces three new products</h2>
<small class="pub-date">Apr 18 2017</small>
<blockquote>
Today, Fastly publicly unveiled its edge cloud platform, which empowers the world’s most popular businesses to deliver consistently secure, fast and personalized digital experiences. Fastly is also debuting three new services, available today on its edge cloud platform: the Fastly Web Application Firewall (WAF), Image Optimizer, and Load Balancer.
</blockquote>
<p>
Read the full press release: <a href="https://www.fastly.com/press/press-releases/fastly-builds-content-delivery-network-heritage-unveils-edge-cloud-platform" class="external-link" onclick="_gaLt(event);">Fastly Builds on Content Delivery Network Heritage, Unveils Edge Cloud Platform</a>
</p>
<h2>Limelight introduces Security Alert and WAF Express</h2>
<small class="pub-date">Apr 18 2017</small>
<p>
<a href="https://www.cdnplanet.com/cdns/limelight/">Limelight Networks</a> launches two new additions to its Cloud Security Services that enhance protection against attacks on websites and unauthorized access or theft of content.
<br>
Read the full press release: <a href="https://www.limelight.com/news/cloud-security-enhancements/" class="external-link" onclick="_gaLt(event);">New Limelight cloud security services offer scalable protection to safeguard websites and apps from online attacks</a>
</p>
<h2>Akamai acquires SOASTA</h2>
<small class="pub-date">Apr 7 2017</small>
<p>
SOASTA is a well-known player in the Digital Performance Management market, offering web performance load testing, performance monitoring (RUM) and analytics.
<br>
Read the full press release: <a href="https://www.akamai.com/us/en/about/news/press/2017-press/akamai-completes-acquisition-of-soasta.jsp" class="external-link" onclick="_gaLt(event);">Akamai Completes Acquisition Of SOASTA</a>
</p>
<h2>Cloudflare launches 6 new POPs</h2>
<small class="pub-date">Apr 2017</small>
<p>
<a href="https://www.cdnplanet.com/cdns/cloudflare/">Cloudflare</a> expanded its global network with new POPs in Kansas City (US), Curaçao, Djibouti, Belgrade (Serbia), Munich (Germany) and Budapest (Hungary). The Cloudflare network now has 111 datacenters.
<br>
View the <a href="https://www.cloudflare.com/network/" class="external-link" onclick="_gaLt(event);">Cloudflare network map</a> or read the new POP announcements on the <a href="https://blog.cloudflare.com/" class="external-link" onclick="_gaLt(event);">Cloudflare blog</a>
</p>
<h2>Verizon / EdgeCast adds a POP in Auckland, New Zealand</h2>
<small class="pub-date">Apr 4 2017</small>
<p>
Verizon Digital Media Services has collaborated with Megaport (Australia) Pty Ltd to expand the <a href="https://www.cdnplanet.com/cdns/edgecast/">Verizon / Edgecast CDN</a> with a new POP in Auckland, New Zealand.
<br>
Read the full press release: <a href="https://www.verizondigitalmedia.com/blog/2017/04/megaport-collaboration-expand-edgecast-content-delivery-network-into-new-zealand/" class="external-link" onclick="_gaLt(event);">Verizon Digital Media Services collaborates with Megaport to expand the Edgecast Content Delivery Network into New Zealand</a>
</p>
10 Tips to Optimize CDN Performancehttps://www.cdnplanet.com/blog/10-tips-optimize-cdn-performance/
Wed, 26 Apr 2017 00:00:00 +0000https://www.cdnplanet.com/blog/10-tips-optimize-cdn-performance/<div class="toc">
<strong>Table of contents</strong>
<ul>
<li><a href="#1">Use high performance DNS</a></li>
<li><a href="#2">Move your origin close to your CDN</a></li>
<li><a href="#3">Have IPv6 connectivity</a></li>
<li><a href="#4">Tune your initcwnd</a></li>
<li><a href="#5">Keep connections alive forever</a></li>
<li><a href="#6">Reduce TLS connection time</a></li>
<li><a href="#7">Minimize byte size</a></li>
<li><a href="#8">Be a Cache-Control master</a></li>
<li><a href="#9">Enable conditional requests</a></li>
<li><a href="#10">Be careful with the Vary header</a></li>
</ul>
</div>
<p>
For the most part, your CDN is responsible for the performance of your content delivery.
The CDN controls its network, load balancers, caching servers and all else involved in delivering your content from the edge to your users.
</p>
<p>
So what can <i>you</i> do to contribute to excellent content delivery performance?
</p>
<p>
Read our <strong>10 tips to optimize CDN performance</strong>.
</p>
<h2 id="1">1. Use high performance DNS</h2>
<p>
Imagine you're using <code>static.example.org</code> on the CDN and <code>origin.example.org</code> as your origin.
If the authoritative DNS for <code>example.org</code> suffers from low availability and high latency, this surely has a bad impact on CDN performance.
</p>
<p>
Don't use the DNS service of your hosting provider just because it is included in the deal.
It's easy and cheap to use high performance, global anycast DNS from providers like AWS, NS1, Cloudflare, Google or Azure.
</p>
<p>
Also, make sure to use high TTLs (Time-To-Live) on your DNS records, so resolvers can cache the records for a long time.
</p>
<h2 id="2">2. Move your origin close to your CDN</h2>
<p>
If most of your users are in Asia, do not place your origin in a far-away location like Amsterdam or New York but in ... Asia.
</p>
<p>
Keeping the latency low between CDN and origin is an effective way to optimize CDN performance for cache miss responses.<br>
</p>
<p>
If you can't host your origin close to the CDN, consider using an Origin shield.
Origin shield is extra caching layer between the CDN edge servers and your origin.
The shield helps offload your origin and speed up cache miss responses.
Read our <a href="https://www.cdnplanet.com/guides/origin-shield/">CDN Guide: Origin shield</a> to learn more and find out which CDNs offer origin shielding.
</p>
<h2 id="3">3. Have IPv6 connectivity</h2>
<p>
Facebook has done a lot of research into the impact of IPv6 on performance and concluded the effects are positive:
</p>
<blockquote>
We've observed that accessing Facebook can be 10-15 percent faster over IPv6.
</blockquote>
<p>
Can your CDN connect to your origin over IPv6? If yes, consider moving your origin to an IPv6-enabled hosting environment.<br>
<a href="https://aws.amazon.com/blogs/aws/now-available-ipv6-support-for-amazon-s3/" class="external-link">AWS S3 supports IPv6</a>. Unfortunately, <a href="https://cloud.google.com/compute/docs/networks-and-firewalls" class="external-link">Google Compute Engine currently does not support IPv6</a>.
</p>
Further reading:
<ul>
<li><a href="https://code.facebook.com/posts/1192894270727351/ipv6-it-s-time-to-get-on-board/" class="external-link">IPv6: It's time to get on board (Facebook)</a></li>
<li><a href="http://www.internetsociety.org/deploy360/blog/2015/04/facebook-news-feeds-load-20-40-faster-over-ipv6/" class="external-link">Facebook News Feeds Load 20-40% Faster Over IPv6</a></li>
</ul>
<h2 id="4">4. Tune your initcwnd</h2>
<p>
The initial congestion window parameter (initcwnd) on your origin server likely has a value of 10.
This means the server sends out 10 packets in the first round trip over a fresh connection.
</p>
<p>
A value of 10 is not bad, but a higher initcwnd likely has a significant positive effect on TCP performance, resulting in faster content transfer between origin and CDN.
</p>
<p>
Some CDNs have a initcwnd of 10, other CDNs have a (much) higher value. Read more: <a href="https://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-providers/">Initcwnd settings of major CDN providers</a>.
</p>
<p>
Use our <a href="https://www.cdnplanet.com/tools/initcwndcheck/">Initcwnd Checker tool</a> to check your current initcwnd.
</p>
<p>
<span class="danger">Important:</span> do not 'simply' increase the initcwnd on your server(s) and assume that will work out well. It may not. A good starting point is the blog article <a href="https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/">Tuning initcwnd for optimum performance</a>.
</p>
<h2 id="5">5. Keep connections alive forever</h2>
<p>
When the CDN needs to pull content from your origin, a TCP connection must exist between the two servers.
Ideally, that connection is already there and can be reused, saving roundtrips and precious milliseconds to establish a fresh connection.
</p>
<p>
The CDN or the origin may terminate the connection.
You have no control over how long the CDN keeps a connection alive, but you <i>do</i> control the keep-alive behavior on your origin.
</p>
<p>
Don't close the connection on the origin side, ever.<br>
Worried about the many CDN servers establishing TCP connections with your origin and the origin not being able to handle that?
Set up an <a href="https://www.cdnplanet.com/guides/origin-shield/">Origin shield</a>.
</p>
<h2 id="6">6. Reduce TLS connection time</h2>
<p>
Do you have a secure HTTPS origin? If yes, there are several optimizations you can do to improve CDN performance.
To name a few: TLS False Start, TLS session resumption and TLS record size optimization.
</p>
<p>
Before getting to work with these TLS optimizations, ask your CDN if anything is needed from their side to make these optimizations have effect.
</p>
Futher reading:
<ul>
<li><a href="https://istlsfastyet.com/" class="external-link">Is TLS Fast Yet?</a></li>
<li><a href="https://hpbn.co/transport-layer-security-tls/#optimizing-for-tls" class="external-link">Optimizing for TLS</a></li>
</ul>
<h2 id="7">7. Minimize byte size</h2>
<p>
Reducing the byte size or 'weight' of your content is very effective in speeding up content delivery performance.
The less bytes over the wire, the faster your content arrives at your users.
</p>
<p>
There are many ways you can minimize byte size to enhance CDN performance
Compression is the most effective and often easiest to implement.
Read our <a href="https://www.cdnplanet.com/guides/compression/">CDN Guide: Compression</a> to learn more about Gzip, Brotli and which content types should be served compressed.
The Google Developers site has a good in-depth article about <a href="https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/optimize-encoding-and-transfer#text_compression_with_gzip" class="external-link">Text compression with GZIP</a>.
</p>
<p>
Two other techniques to reduce byte size are <a href="https://blog.stackpath.com/glossary/minification/" class="external-link" onclick="_gaLt(event);">minification</a> and <a href="https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/image-optimization" class="external-link">image optimization</a>.
</p>
<h2 id="8">8. Be a Cache-Control master</h2>
<p>
How can you make the CDN cache your objects for as long as you want and maximize the cache hit ratio?<br>
Out of the box, most or all CDNs will follow the caching instructions your origin server sends by way of the <code>Cache-Control</code> header.
It's a very effective tool to improve CDN performance.
</p>
<p>
Some examples:
</p>
<p>
<code>Cache-Control: max-age=86400</code> tells the CDN and browsers the object may be cached for 86400 seconds.
</p>
<p>
<code>Cache-Control: max-age=3600, s-maxage=86400</code> informs the CDN it may cache the object for 24 hours and browsers should cache the object no longer than 1 hour. Note: not all CDNs support <code>s-maxage</code> by default.
</p>
<p>
<code>Cache-Control: max-age=86400, stale-while-revalidate=300</code> instructs the CDN and browsers to cache the object for 24 hours and, at the end of those 24 hours, the CDN may serve the stale response for up to 300 seconds while new content is being fetched from origin.
</p>
Learn more about Cache-Control:
<ul>
<li><a href="https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/http-caching" class="external-link">HTTP Caching | Web | Google Developers</a></li>
<li><a href="https://www.incapsula.com/cdn-guide/cdn-caching.html" class="external-link" onclick="_gaLt(event);">The essential guide to CDN: CDN Caching (Incapsula)</a></li>
<li><a href="https://www.cdnplanet.com/guides/serve-stale/">CDN Guide: Serve stale</a></li>
</ul>
<h2 id="9">9. Enable conditional requests</h2>
<p>
When a CDN receives a request for a stale object (in cache, but expired), the CDN must first contact the origin before sending a response to the user.
If the cached object has a <code>Last-Modified</code> and/or <code>ETag</code> header, the CDN can make the request to origin conditional by adding <code>If-Modified-Since</code> and/or <code>If-None-Match</code> headers. The origin can decide to respond with a lightweight <code>304 Not Modified</code> response (headers only) or send a heavy <code>200 OK</code> response (headers and body).
</p>
<p>
Obviously, the <code>304 Not Modified</code> response is better for performance than the <code>200 OK</code> response because the size of the response is much smaller and so less roundtrips are needed between CDN and origin.
</p>
<p>
Configure your origin server to always send a <code>Last-Modified</code> and/or <code>ETag</code> header, as this greatly helps improve cache miss performance.
</p>
<h2 id="10">10. Be careful with the Vary header</h2>
<p>
Your origin should not serve objects to the CDN with a header like <code>Vary: Referer</code>, <code>Vary: User-Agent</code>, <code>Vary: Cookie</code> or <code>Vary: User-Agent, Cookie</code>. These Vary headers have a big negative effect on cache hit ratio and CDN performance. <!-- because they instruct the CDN to store a separate copy of the object for each unique value of the Referer, User-Agent and/or Cookie in the client-to-CDN request-->
</p>
<p>
Is your origin sending these Vary headers for no purpose? Remove them!
</p>
<strong>Important:</strong>
<ul>
<li>Don't ever use <code>Vary: *</code>. An object with that header will never be stored in a CDN cache</li>
<li>Always send the <code>Vary: Accept-Encoding</code> header with (Gzip) compressed content</li>
</ul>
<p>
Further reading: <a href="https://www.fastly.com/blog/best-practices-for-using-the-vary-header" class="external-link" onclick="_gaLt(event);">Best Practices for Using the Vary Header</a>
</p>Faster Google Fonts with Preconnecthttps://www.cdnplanet.com/blog/faster-google-webfonts-preconnect/
Mon, 24 Apr 2017 00:00:00 +0000https://www.cdnplanet.com/blog/faster-google-webfonts-preconnect/<h2>Summary</h2>
<ul>
<li>Load the Google font files faster by adding the <code>preconnect</code> hint</li>
<li>The <code>preconnect</code> hint is supported by Chrome, Opera, Firefox and Android browsers</li>
<li>Don't forget to add the <code>crossorigin</code> attribute!</li>
</ul>
<h2>The Google Fonts performance problem</h2>
<p>
The CDN Planet website uses the Roboto font, powered by the free and easy to use Google Fonts.
</p>
<p>
Google Fonts is a very reliable service and network performance is generally great (the service lives on Google's global CDN).
But yet, there is one problem with Google Fonts performance: the font files start downloading late.
</p>
<p>
The waterfall chart below shows the performance problem in action:
</p>
<p>
<a href="https://www.webpagetest.org/result/170421_44_NQC/3/details/#waterfall_view_step1"><img class="img-no-max-width" src="https://www.cdnplanet.com/images/waterfall-cdnplanet-before-preconnect.png?9bc01e4e01660b5c713f2173a816d6ca" width="600" height="350" alt="waterfall cdnplanet before preconnect"></a>
</p>
<p>
The CSS loads first, then the font files.
This behavior is not so great for performance but quite logical: the URLs of the font files are in the CSS file, so the browser must first download and parse the CSS.
</p>
<p>
Luckily, Google always serves the fonts from the same domain, <code>fonts.gstatic.com</code>, and you can instruct the browser to 'preconnect' to that domain while it is loading the CSS!
</p>
<h3>The preconnect hint</h3>
<p>
Preconnect is one of the Resource Hints as defined in the <a href="https://www.w3.org/TR/resource-hints/#preconnect" class="external-link">W3C Working Draft</a>:
</p>
<blockquote>
The preconnect link relation type is used to indicate an origin that will be used to fetch required resources. Initiating an early connection, which includes the DNS lookup, TCP handshake, and optional TLS negotiation, allows the user agent to mask the high latency costs of establishing a connection.
</blockquote>
<p>
Preconnect is <a href="http://caniuse.com/#feat=link-rel-preconnect" class="external-link">supported by most browsers</a> and improves Google Fonts performance.
</p>
<h2>Preconnect done wrong</h2>
<p>
Implementing preconnect is simple: add the <code>preconnect</code> hint to the <code>head</code> section of the HTML:
</p>
<pre>
&lt;link rel="preconnect" href="https://fonts.gstatic.com/"&gt;
</pre>
<p>
<a href="https://www.webpagetest.org/result/170421_VW_Q8N/1/details/#waterfall_view_step1">
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/waterfall-cdnplanet-after-preconnect-no-crossorigin.png?aad2d4424b8efb7b9975655b1ec6cddf" width="600" height="350" alt="waterfall cdnplanet after preconnect no crossorigin">
</a>
</p>
<p>
Hmmm, Chrome performs the DNS lookup only. Why does the browser not also establish the (secure) connection?
</p>
<p>
The answer is simple: I did not add the <code>crossorigin</code> attribute to the <code>link</code> element.
</p>
<p>
Ilya Grigorik explains why this attribute is needed in his excellent article <a href="https://www.igvita.com/2015/08/17/eliminating-roundtrips-with-preconnect/" class="external-link">Eliminating Roundtrips with Preconnect</a>
</p>
<blockquote>The <a href="http://www.w3.org/TR/css3-fonts/#font-fetching-requirements" class="external-link">font-face specification requires</a> that fonts are loaded in "anonymous mode", which is why we must provide the crossorigin attribute on the preconnect hint: the browser maintains a separate pool of sockets for this mode.</blockquote>
<h2>Preconnect done right!</h2>
<pre>
&lt;link rel="preconnect" href="https://fonts.gstatic.com/" crossorigin&gt;
</pre>
<p>
<a href="https://www.webpagetest.org/result/170421_E0_24V0/3/details/#waterfall_view_step1"><img class="img-no-max-width" src="https://www.cdnplanet.com/images/waterfall-cdnplanet-after-preconnect-with-crossorigin.png?31a65f083ae6b5c73f578a5abe3c54f7" width="600" height="350" alt="waterfall cdnplanet after preconnect with crossorigin"></a><br><br>
</p>
<p>
Excellent!<br>
DNS, TCP and TLS happens in parallel with the loading of the Google Fonts CSS file.
We effectively improved Google Fonts performance by reducing the load time by ~ 100 milliseconds.<br>
</p>
<h2>Learn more about webfont performance</h2>
<ul>
<li><a href="https://calendar.perfplanet.com/2016/loading-web-fonts-asynchronously/" class="external-link" onclick="_gaLt(event);">Loading web fonts asynchronously</a></li>
<li><a href="https://www.filamentgroup.com/lab/font-events.html" class="external-link" onclick="_gaLt(event);">How We Load Web Fonts Progressively</a></li>
<li><a href="https://www.zachleat.com/web/comprehensive-webfonts/" class="external-link" onclick="_gaLt(event);">A Comprehensive Guide to Font Loading Strategies</a></li>
</ul>
CDN behavior testing with Dummy Originhttps://www.cdnplanet.com/blog/cdn-behavior-testing-dummy-origin/
Wed, 05 Apr 2017 00:00:00 +0000https://www.cdnplanet.com/blog/cdn-behavior-testing-dummy-origin/<div class="toc">
<strong>Table of contents</strong>
<ul>
<li><a href="#intro">Introduction</a></li>
<li><a href="#features">Features of Dummy Origin</a></li>
<li><a href="#install">Installation & configuration</a></li>
<li><a href="#testing">Testing CDN behavior</a></li>
<li><a href="#github">It's open source!</a></li>
</ul>
</div>
<a name="intro"></a>
<p>
Before you sign the contract with your new CDN you want to know <i>for a fact</i> the CDN can do what you want and behave as you need it to behave. Right?
Sure, you can read the CDN documentation and chat with support, but if you're like us, that is not good enough.
You want to test rigorously, <i>observe</i> the CDN's behavior and match it to your functional requirements.
Dummy Origin helps you do this better and with greater ease.
</p>
<p>
Dummy Origin is a simple yet special server, written in Go. It was built specifically for the purpose of evaluating CDNs in an origin pull setup.
Dummy Origin is easy to install, lightweight, stable and fast. And <a href="#github">open source</a>!
</p>
<p>
Why use Dummy Origin and not simply put the CDN in front of your <i>real</i> origin with your <i>real</i> content?<br>
The short answer: flexibility and ease-of-use.<br>
</p>
<ul>
<li>No changes to your stack</li>
<li>Ultimate flexibility in which response headers the origin sends and seeing how the CDN then behaves</li>
<li>Easy to test a scenario like 'my origin is serving 502 responses, does the CDN serve stale?'</li>
</ul>
<a name="features"></a><h2>Features of Dummy Origin</h2>
<p>
The key feature of Dummy Origin is arbitrary response header injection.
In short: whatever you send as a query string parameter in the request will be served as a header with the response.
</p>
<p>
Send <code>?ETag="41a0544696e19971383984fdaaeb5aed"</code> in the request and get the <code>ETag:"41a0544696e19971383984fdaaeb5aed"</code> header back.
</p>
<p>
Sending a request to get back multiple headers is just as easy:<br>
<code>?Cache-Control=no-cache&Referer=https://www.cdnplanet.com/</code>
</p>
<p>
The arbitrary response header injection gives you great flexibility in testing the CDN's behavior and there is no need to constantly change the server config and restart it.
</p>
<p>
The other features of Dummy Origin:
</p>
<table class="table-striped table-three-col-333333">
<thead>
<tr>
<th>Feature</th>
<th>Description</th>
<th>Benefits</th>
</tr>
</thead>
<tbody>
<tr>
<td>Gzip</td>
<td>Compress the response with default GZip compression level. Not based on <code>Content-Type</code> but file extension. Currently supports <code>.html</code>, <code>.css</code> <code>.js</code> and <code>.json</code> extensions only but very easy to add more</td>
<td>You want to know if the CDN requests files from origin compressed or uncompressed</td>
</tr>
<tr>
<td>Conditional requests</td>
<td>Server always sends <code>Last-Modified</code> header with 200 responses and sends a 304 or 200 response if it gets a <code>If-Modified-Since</code> request (as per <a href="https://tools.ietf.org/html/rfc7234#section-4.3" class="external-link">RFC7234 section 4.3</a>)</td>
<td>Enables testing if the CDN sends conditional requests after a file in cache has expired</td>
</tr>
<tr>
<td>Range requests</td>
<td>Send the requested byte range in a 206 partial response + the appropriate <code>Content-Length</code> header</td>
<td>Essential for serving large files through the CDN, for example MP4 videos</td>
</tr>
<tr>
<td>404 and 301</td>
<td>Serve a 404 if resource does not exist, serve a 301 if request is for <code>/index.html</code></td>
<td>Does the CDN cache 404 responses? Does the CDN forward the 301 or does it follow the redirect?</td>
</tr>
<tr>
<td>Special error generator</td>
<td>When the server receives request in the format <code>GET /err/&lt;code&gt;</code> it returns with http status <code>&lt;code&gt;</code>. <code>&lt;code&gt;</code> must be between 400 and 599</td>
<td>Handy for seeing how the CDN behaves in case of variour origin error responses. E.g. does the CDN cache 404 responses?</td>
</tr>
<tr>
<td>X-Tb-Time</td>
<td>Server always sends current time in the <code>X-Tb-Time</code> header</td>
<td>Helps determine if CDN served the response from cache and how long it has been in CDN cache</td>
</tr>
<tr>
<td>Loggly support</td>
<td>The request and response details are written to Loggly if <code>LOGGLY_TOKEN</code> environment variable is set (optional)</td>
<td><a href="https://www.loggly.com/" class="external-link">Loggly</a> gives you a near real-time view on the origin logs. This is very handy for seeing the CDN to origin requests</td>
</tr>
</tbody>
</table>
<h3>Missing in Dummy Origin</h3>
<ul>
<li>HTTPS</li>
<li>Brotli</li>
<li>Can't serve different content/headers based on the value of <code>User-Agent</code> or <code>Accept-Language</code> requests headers</li>
</ul>
<a name="install"></a><h2>Installation & configuration</h2>
<p>
Install the server: <a href="https://github.com/turbobytes/dummyorigin/blob/master/README.md" class="external-link">README.md on Github</a>.
</p>
<p>Put some files on the server. These should be similar to what you want to serve through the CDN.
You may want to use dummy files, for example
<a href="https://en.wikipedia.org/wiki/File:Joseph_Dudley.jpg?cdnp" class="external-link">15kb.jpg</a>,
<a href="https://en.wikipedia.org/wiki/File:Procnias_tricarunculata.jpg?cdnp" class="external-link">100kb.jpg</a>,
<a href="https://en.wikipedia.org/wiki/File:BlueBellsOfScotland.PNG?cdnp" class="external-link">15kb.png</a>,
<a href="https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js?cdnp" class="external-link">13kb.js</a>,
<a href="https://ajax.googleapis.com/ajax/libs/angularjs/1.6.1/angular.min.js?cdnp" class="external-link">160kb.js</a>,
<a href="https://archive.org/download/UCONN_2014_Parade_in_Hartford_CT/UCONN_2014_Parade_in_Hartford_CT.mp4" class="external-link">108mb.avi</a>.
</p>
<div class="notice-msg info">
Do NOT put an <code>index.html</code> on the server so you can always see what files are on the origin by simply visiting the origin domain in your browser.
</div>
<p>Set things up on the CDN.
Configure the CDN to treat query strings as unique (<code>file.jpg?123</code> is not the same as <code>file.jpg?456</code>). This is often the default behavior of a CDN but make sure to check.
Also, make sure the CDN forwards all headers from origin to client including the <code>X-Tb-Time</code> header.
</p>
<p>
Configure your DNS. Use a low TTL for the origin host (e.g. 300 seconds) as this will come in handy later when you want to find out how the CDN behaves in case the origin is down (more on this later).
</p>
<a name="testing"></a><h2>Testing CDN behavior: example use cases</h2>
<p>
In these examples, <code>dummy.mydomain.com</code> is the CDN host and <code>dummy.origin.com</code> is the origin host.<br>
Our tool of choice for testing is cURL. Our cURL commands always start with <code>curl -vo /dev/null</code> so we send a GET request and can see the request and response headers, but not be bothered with the response body.
</p>
<p>
Do not use the <code>-I</code> flag to send a HEAD request to the CDN. The CDN may treat HEAD requests very differently from GET requests (and remember: your users will send GET requests not HEAD requests).
</p>
<p>
Learn more about cURL by reading <a href="https://support.cloudflare.com/hc/en-us/articles/222971907-Using-cURL-when-Troubleshooting-with-Cloudflare" class="external-link" onclick="_gaLt(event);">Using cURL when Troubleshooting with Cloudflare</a> or <a href="https://curl.haxx.se/docs/manpage.html" class="external-link">the cURL manual</a>.
</p>
<h3>Don't cache and always fetch full file from origin</h3>
<p>
Not the typical CDN use case, but if this is what you need for your 'uncacheable' content, it needs to work well.
The question here is: which Cache-Control directive(s) must be used to make the CDN behave as desired?
Based on what is in <a href="https://tools.ietf.org/html/rfc7234#section-5.2.2">RFC 7234</a> it seems logical to use <code>Cache-Control:no-store</code>.
The RFC states:
<cite>The "no-store" response directive indicates that a cache MUST NOT store any part of either the immediate request or response. This directive applies to both private and shared caches.</cite>
Ok, let's try that
</p>
<pre>curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=no-store'</pre>
<p>
Below you see the response headers from our tests with <a href="https://www.cdnplanet.com/cdns/fastly/">Fastly</a>. Surprisingly, Fastly does not honor the <code>no-store</code> and serves the file from cache after a first initial cache miss.
</p>
<pre>
HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Cache-Control: no-store
Content-Type: image/jpeg
Last-Modified: Tue, 21 Mar 2017 10:13:25 GMT
X-Tb-Time: 2017-03-30 14:39:11.358951299 +0000 UTC
Content-Length: 15887
Accept-Ranges: bytes
Date: Thu, 30 Mar 2017 14:39:23 GMT
Via: 1.1 varnish
Age: 12
X-Served-By: cache-ams4136-AMS
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1490884763.605627,VS0,VE0
Vary: Accept-Encoding
</pre>
<p>
Should we use <code>no-cache</code> or <code>max-age=0</code>? We don't want the CDN to cache the response and send conditional requests to the origin. It is important the CDN does not store the response in cache.
</p>
<p>
Testing shows <code>no-cache</code> does not work (again, Fastly caches the responses) but <code>max-age=0</code> does work.
</p>
<h3>Cache for 300 seconds, then do origin validation</h3>
<p>
Getting a CDN to cache a file for 300 seconds is easy: <code>Cache-Control=max-age=300</code>.
The key question here is: after the file expires in cache, will the CDN send a conditional or unconditional request to origin?
One can argue the conditional request is optimal, because if the file on the origin has not changed, the origin can send a lightweight 304 response ("No, the file has not changed, it is fine to serve the cached object!") and the CDN can quickly respond to the client. This is faster than when the origin sends the full file to CDN.
</p>
<p>
RFC 7234 does <i>not</i> state a cache <i>must</i> send a conditional request for cache validation. Therefore it should be not a big surprise if the CDN sends unconditional requests (although not great for performance).
</p>
<pre>curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300'</pre>
<p>
<a href="https://www.cdnplanet.com/cdns/fastly/">Fastly</a> and <a href="https://www.cdnplanet.com/cdns/cdnetworks/">CDNetworks</a> play nice and send conditional requests. <a href="https://www.cdnplanet.com/cdns/stackpath/">StackPath</a> however sends unconditional requests to origin, meaning their caches always fetch the full file. We could see this in the origin logs and by inpecting the value of the <code>X-Tb-Time</code> header in the CDN to client response. We also tried with <code>Cache-Control=max-age=300,must-revalidate</code> but no luck.
</p>
<pre>
HTTP/1.1 200 OK
<strong>Date: Fri, 31 Mar 2017 11:20:16 GMT</strong>
Content-Type: image/jpeg
Content-Length: 33956
Access-Control-Allow-Origin: *
Cache-Control: max-age=20,must-revalidate
Last-Modified: Fri, 31 Mar 2017 09:49:42 GMT
<strong>X-Tb-Time: 2017-03-31 11:20:16.091275242 +0000 UTC</strong>
Server: NetDNA-cache/2.2
Accept-Ranges: bytes
<strong>X-Cache: EXPIRED</strong>
Connection: keep-alive
</pre>
<p>
We contacted StackPath about this behavior and they quickly responded to say the CDN can be configured to send conditional requests if the customer wants that.
</p>
<h3>Cache for 5 minutes, may serve stale for 15 minutes</h3>
<pre>curl -vo /dev/null 'http://dummy.mydomain.com/15kb.jpg?Cache-Control=max-age=300&stale-while-revalidate=900&stale-if-error=900'</pre>
<p>
As far as we know, <a href="https://www.cdnplanet.com/cdns/fastly/">Fastly</a> and <a href="https://www.cdnplanet.com/cdns/cdn77/">CDN77</a> are the only CDNs that support the <code>stale-while-revalidate</code> and <code>stale-if-error</code> Cache-Control extensions. We tested the behavior of Fastly in three situations:
</p>
<p>
1. Origin host does not resolve in DNS (NXDOMAIN)<br>
In summary: long after the DNS TTL of our origin must have expired in Fastly caches/systems, Fastly still served stale content. That's nice.
</p>
<p>
More interestingly, Fastly also kept fetching fresh content from our origin that had not yet made it to their caches.
Does Fastly remember the last known IP address of the origin and use that in case of origin resolution failure? Or was this behavior due to Fastly keeping a long-lived connection with the origin? We're not sure and need to dig in deeper.
</p>
<p>
2. Origin serves 400 response<br>
Something goes wrong at the origin and it starts serving <code>400 Bad request</code> responses.
We expected the <code>stale-if-error</code> to kick in here but Fastly simply forwarded the 400 response to the client.
As it turns out, our expectation was wrong because <a href="https://tools.ietf.org/html/rfc5861" class="external-link">RFC 5861: HTTP Cache-Control Extensions for Stale Content</a> states
<cite>an error is any situation that would result in a 500, 502, 503, or 504 HTTP response status code being returned</cite>
</p>
<p>
3. Origin drops all incoming traffic<br>
In our third scenario we changed our origin DNS record to point to 72.66.115.13. This is the IP address behind blackhole.webpagetest.org.
<a href="https://twitter.com/patmeenan" class="external-link">Pat Meenan</a>, creator of <a href="http://www.webpagtest.org/" class="external-link">WebPageTest.org</a> set this up back in 2011 to enable <a href="http://blog.patrickmeenan.com/2011/10/testing-for-frontend-spof.html" class="external-link">testing for frontend SPOFs in WPT</a>. The WPT blackhole will not return any response to any request.
</p>
<p>
Did Fastly serve stale? No. Fastly served 503 responses after ~ 1 second and these 503 responses were served with the <code>Retry-After: 0</code> header. This is actually to be expected with Fastly out of the box. Their <a href="https://docs.fastly.com/guides/performance-tuning/serving-stale-content#serving-stale-content-on-errors" class="external-link" onclick="_gaLt(event);">Serving stale content</a> document describes the default behavior:
<cite>If Varnish cannot contact the origin for any reason, a 503 error will be generated</cite>
The doc then shows and explains the custom VCL code to have Fastly serve stale in this case (and two other cases).
</p>
<p>
FYI: StackPath can serve stale content based on customer configured status codes (404, 502, 503 etc). This works as expected but StackPath does not serve stale in case of NXDOMAIN or blackhole.<br>
Read more about serve stale in our <a href="https://www.cdnplanet.com/guides/serve-stale/" class="external-link">Serve stale CDN Guide</a>.
</p>
<a name="github"></a><h2>It's open source!</h2>
<p>
<a href="https://github.com/turbobytes/dummyorigin" class="external-link">Dummy Origin is on Github</a> with MIT license. Fork away!
</p>
Introducing CDN Guideshttps://www.cdnplanet.com/blog/introducing-cdn-guides/
Sun, 12 Mar 2017 00:12:00 +0000https://www.cdnplanet.com/blog/introducing-cdn-guides/<p>
As part of the relaunch of CDN Planet, we're introducing a new content section: CDN Guides.<br>
A CDN Guide is an article packed with information about one content delivery network feature. The first three CDN Guides are about Compression, Purge and Serve stale.
</p>
<p>
The CDN Guide will help you get a solid understanding of the feature and - perhaps most importantly - show the differences between the various CDNs. You will also find detailed specifics per CDN and links to relevant docs/content on the CDN websites and support portals.
</p>
<h2>Compression</h2>
<p>
Do all CDNs pull compressed from the origin? Which CDNs can do gzipping on the edge? What are content types that should be served compressed? What is Brotli and does your CDN support it?
<br>View the <a href="https://www.cdnplanet.com/guides/compression/">CDN Guide: Compression</a>
</p>
<h2>Purge</h2>
<p>
What is purging? Which CDNs invalidate objects and which CDNs delete objects? Can all CDNs purge by directory or content type? Is purging free and without rate limits?
<br>View the <a href="https://www.cdnplanet.com/guides/purge/">CDN Guide: Purge</a>
</p>
<h2>Serve stale</h2>
<p>What is stale content? Do all CDNs serve stale content in case your origin is down or serving errors? Is 'serve stale' a free feature or do CDNs charge extra for it? Do any CDNs support the HTTP Cache-Control Extensions for Stale Content?
<br>View the <a href="https://www.cdnplanet.com/guides/serve-stale/">CDN Guide: Serve stale</a>
</p>
<div class="notice-msg info">
Let us know in the comments or on <a href="https://twitter.com/cdnplanet" class="external-link">Twitter</a> what feature should be in our next CDN Guide!
</div>Root domain on CDN and performancehttps://www.cdnplanet.com/blog/root-domain-on-cdn-performance/
Sat, 11 Mar 2017 00:12:00 +0000https://www.cdnplanet.com/blog/root-domain-on-cdn-performance/<p>
This is a repost of the article I wrote for the 2016 Performance Calendar on PerfPlanet.
The original article is here: <a href="https://calendar.perfplanet.com/2016/root-domain-cdn-performance/" class="external-link">https://calendar.perfplanet.com/2016/root-domain-cdn-performance/</a>.
</p>
<hr>
<p>Nowadays it is not uncommon to publish your site on the root domain (aka 'naked domain' or 'domain apex'). Instead of using <code>www.awesomesite.com</code> you go for <code>awesomesite.com</code>. Looks good, right? Yeah.</p>
<p>Running the whole site through a CDN is also the modern thing to do and can help give your users an excellent experience. The CDN will give you an endpoint to point to in DNS and now you run into a roadblock: you can't use a CNAME on the root domain (per <a href="https://tools.ietf.org/html/rfc2181" class="external-link">RFC2181</a>). Now what?</p>
<p>Luckily for you, there are DNS providers that have a solution, usually called <code>ANAME</code> or <code>ALIAS</code>, allowing you to point to the CDN endpoint on the root domain. <!-- The 'magic' in these solutions is that the DNS provider resolves the CDN endpoint on their nameservers and then responds to the ISP or open resolver with an A record. --></p>
<p><a href="https://ns1.com/articles/cname-alias-and-linked-records" class="external-link">NS1</a>, <a href="http://www.dnsmadeeasy.com/services/anamerecords/" class="external-link">DNS Made Easy</a>, <a href="https://constellix.com/dns/aname/" class="external-link">Constellix</a>, <a href="https://support.dnsimple.com/articles/alias-record/" class="external-link">DNSimple</a>, <a href="http://dyn.com/managed-dns/alias/" class="external-link">Dyn</a> and <a href="https://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root/" class="external-link" onclick="_gaLt(event);">Cloudflare</a> have this <code>ANAME</code>/<code>ALIAS</code> solution. <a href="http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-cloudfront-distribution.html" class="external-link" onclick="_gaLt(event);">AWS Route 53</a> has it too, but you can only use it with their own CDN (Cloudfront).</p>
<p>So you pick one of these DNS providers, set things up and be done with it, right? Not so fast.</p>
<h2>The problem</h2>
<p>The <code>ANAME</code>/<code>ALIAS</code> is resolved by your DNS provider's nameserver instead of by a recursive resolver (ISP, Google Public DNS, OpenDNS or other) and this may lead to end users being routed to a far-away CDN node and consequently getting a poor experience.</p>
<h3>Let's take a look at how ANAME/ALIAS resolution works:</h3>
<p>Imagine Thailand is an important market for your company, <code>awesomesite.com</code>. Logically, you work with a CDN that has a POP in Thailand. Your authoritative DNS provider has POPs in Hong Kong and Singapore, but not in Thailand.</p>
<p>The DNS server of an ISP in Thailand sends a query for <code>awesomesite.com</code> to your DNS provider and that query is routed to their Hong Kong POP. The authority's nameserver in Hong Kong now resolves the ANAME/ALIAS, by sending a query to the CDN's DNS server.</p>
<p>You can probably guess what happens: the CDN's DNS server sees a query coming in from Hong Kong and it hands out the IPs for the CDN POP in Hong Kong. As a result, people in Thailand visiting <code>awesomesite.com</code> get the content from the CDN in Hong Kong (100 ms RTT) instead of from Thailand (30 ms RTT).</p>
<p>So far, this is just theory. Let's see this in action in the real world!</p>
<p><a href="http://www.swiftserve.com/en/" class="external-link">SwiftServe CDN</a> has POPs in Thailand (TH) and Hong Kong (HK). NS1 has POPs in Asia, including in HK and SG, but not in TH. For this test, I set up our domain <code>startrender.com</code> in NS1, pointing to the SwiftServe endpoint (<code>edge.conversant.swiftserve.com</code>) with an ALIAS record on the root domain and with a normal CNAME on <code>www.startrender.com</code>.</p>
<p>What happens for users in HK and TH on Google Public DNS and OpenDNS? Both open resolvers support <a href="https://tools.ietf.org/html/draft-ietf-dnsop-edns-client-subnet-03" class="external-link">EDNS-Client-Subnet</a> which means they send part of the client IP in the query to the CDN's DNS server, so that server can make a more informed decision and hand out the IP addresses for the best POP.</p>
<p>We use our free tool <a href="https://pulse.turbobytes.com/" class="external-link">TurboBytes Pulse</a> (DNS, HTTP and Traceroute diagnostics from 80+ machines around the globe) to first see what the IPs are of the SwiftServe POPs (<a href="https://pulse.turbobytes.com/results/58593761ecbe407b4f00716d/" class="external-link">link</a>). We need this later as a reference.</p>
<p>Next, we run tests on the Pulse agents located in HK and TH to find out what happens in both cases: <code>www.startrender.com</code> (CNAME) and <code>startrender.com</code> (ALIAS). Guess what? </p>
<table class="table-striped table-responsive margin-bottom-small">
<thead>
<tr>
<th>Domain</th>
<th>User location</th>
<th>CDN POP location</th>
</tr>
</thead>
<tbody>
<tr>
<td>www.startrender.com</td>
<td>HK</td>
<td>HK</td>
</tr>
<tr>
<td>www.startrender.com</td>
<td>TH</td>
<td>TH</td>
</tr>
<tr>
<td>startrender.com</td>
<td>HK</td>
<td>SG</td>
</tr>
<tr>
<td>startrender.com</td>
<td>TH</td>
<td>GB</td>
</tr>
</tbody>
</table>
<p>As the table shows, using <a href="https://pulse.turbobytes.com/results/585d2d23ecbe407b4f008ca8/" class="external-link">Google Public DNS</a> and <a href="https://pulse.turbobytes.com/results/585d2d54ecbe407b4f008cab/" class="external-link">OpenDNS</a> from HK and TH, <code>www.startrender.com</code> resolves to the desired CDN POPs. However, for <code>startrender.com</code>, users on <a href="https://pulse.turbobytes.com/results/585936efecbe407b4f007169/" class="external-link">Google Public DNS</a> and <a href="https://pulse.turbobytes.com/results/58593829ecbe407b4f00716e/" class="external-link">OpenDNS</a> in HK will end up hitting the SwiftServe POP in SG (not so bad) and users on those resolvers in TH are routed to the SwiftServe POP in London (ugh!)</p>
<p>The impact of this problem really depends on where your users are and where the POPs of your DNS and CDN providers are. If you only care about US, you're probably in the clear. Asia, LATAM, Middle-East, Russia? Assume this problem exists for you.</p>
<p>Any solutions to this problem? Yes!</p>
<h2>Solution A: DNS providers do a better job</h2>
<p>The authoritative DNS provider for your domain should be smarter when resolving the ANAME/ALIAS from their nameservers, by 'doing the EDNS-Client-Subnet thing': send part of the IP address of the originating query source in its query to the CDN's DNS server.</p>
<p>If your authoritative DNS provider received the query from OpenDNS or Google Public DNS, it should pass along the client IP address subnet. If the query came from an ISP resolver, the authority server should send the /24 subnet of that resolver's IP in the query to the CDN's DNS server.</p>
<p>I'm quite sure NS1, DNS Made Easy/Constellix and DNS Simple don't send EDNS-Client-Subnet queries when resolving <code>ANAME</code>/<code>ALIAS</code>. I don't know why these DNS providers don't do this. Maybe it's because this would require caching much more responses, as responses can now be different for different recursive/client IP subnets.</p>
<h2>Solution B: get integrated authoritative DNS and CDN</h2>
<p>AWS, Akamai and CDNetworks are companies that operate a global CDN <em>and</em> provide authoritative DNS services. They can be smart and have their authoritative DNS and CDN work together, meaning the auth DNS always knows what the best CDN IPs are for whatever query comes in. Bye bye root domain problem.</p>
<p><a href="http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-cloudfront-distribution.html" class="external-link" onclick="_gaLt(event);">AWS Route 53/Cloudfront</a> is doing this smart thing today. <a href="https://www.cdnplanet.com/cdns/akamai/">Akamai</a> and <a href="https://www.cdnplanet.com/cdns/cdnetworks/">CDNetworks</a>: I'm not sure but likely.</p>
<p> Note: <a href="https://www.cdnplanet.com/cdns/fastly/">Fastly</a> and <a href="https://ns1.com/" class="external-link">NS1</a> have a special integration so that's an option too.</p>
<h2>Solution C: use an Anycast CDN</h2>
<p>
Highwinds. <a href="https://www.cdnplanet.com/cdns/tata-communications/">Tata</a>. <a href="https://www.cdnplanet.com/cdns/cachefly/">CacheFly</a>. MaxCDN (without their Flex POPs). <a href="https://www.cdnplanet.com/cdns/cloudflare/">Cloudflare</a>. These are CDNs that use <a href="https://en.wikipedia.org/wiki/Anycast" class="external-link">Anycast</a> for routing users to the best POP. All POPs globally have the same IP address and users are 'magically' routed to the POP that is closest from a network topology perspective (shortest network path). For example, <code>www.cloudflare.com</code> points to <code>198.41.{214,215}.162</code> (<a href="https://pulse.turbobytes.com/results/585d17b7ecbe407b4f008c13/" class="external-link">everywhere</a>)</p>
<p>Since all POPs have the same IP address, the root domain problem does not exist.</p>
<p>Note: <a href="https://docs.fastly.com/guides/basic-configuration/using-fastly-with-apex-domains" class="external-link" onclick="_gaLt(event);">Fastly</a> does not use Anycast for most customers, but can provide Anycast IP addresses if you must use their service on the root domain.</p>
<p>Note: EdgeCast (now <a href="https://www.cdnplanet.com/cdns/edgecast/">Verizon Digital Media Services</a>) uses Anycast too but per region (Americas, EMEA, APAC) so there still is a chance, although small, of suboptimal routing.</p>
Initcwnd settings of major CDN providershttps://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-providers/
Mon, 13 Feb 2017 00:12:00 +0000https://www.cdnplanet.com/blog/initcwnd-settings-major-cdn-providers/<p>
In November 2011, we first published the initcwnd values of CDNs, following our blog post <a href="https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/">Tuning initcwnd for optimum performance</a> that showed how tuning the initial congestion window parameter (initcwnd) on the server can have a significant improvement in TCP performance.
</p>
<p>
CDNs constantly tune their platform to improve performance. When we updated this article on August 27 2014, we measured higher initcwnd values for many CDNs (compared to the 2011 measurements). The most recent measurements (February 13 2017) again show some CDNs have increased the initcwnd size but interestingly for other CDNs we measured a lower value than in 2014. Also, quite a few CDNs have still not changed to e higher than the default 10 in Linux kernel.<br>
First we show you the data, then some <a href="#con">conclusions</a> and finally we describe our <a href="#method">test methodology</a>.
</p>
<h2>CDN initcwnd values</h2>
<table class="table-condensed table-striped">
<thead>
<tr>
<th>CDN</th>
<th>Initcwnd value (Feb 2017)</th>
<th>Initcwnd value (Aug 2014)</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/cachefly/">Cachefly</a></td>
<td><img alt="down" src="https://www.cdnplanet.com/images/arrow-down-bold-circle.svg?50e889f5af7eea74add9289fff7d8665" width="24" height="24" class="icon-in-table-cell">46</td>
<td>66</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/level3/">Level3</a></td>
<td><img alt="up" src="https://www.cdnplanet.com/images/arrow-up-bold-circle.svg?d6400a2e308d980a6cbb805ba74e5e4b" width="24" height="24" class="icon-in-table-cell">46</td>
<td>12</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/highwinds/">Highwinds</a></td>
<td><img alt="up" src="https://www.cdnplanet.com/images/arrow-up-bold-circle.svg?d6400a2e308d980a6cbb805ba74e5e4b" width="24" height="24" class="icon-in-table-cell">44</td>
<td>36</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/akamai/">Akamai</a></td>
<td><img alt="up" src="https://www.cdnplanet.com/images/arrow-up-bold-circle.svg?d6400a2e308d980a6cbb805ba74e5e4b" width="24" height="24" class="icon-in-table-cell">32</td>
<td>16</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/quantil/">QUANTIL</a></td>
<td><img alt="unknown" src="https://www.cdnplanet.com/images/help-circle.svg?b71e803c19f1149bd6e4edbacfe26ac6" width="24" height="24" class="icon-in-table-cell">30</td>
<td>-</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/edgecast/">EdgeCast / Verizon</a></td>
<td><img alt="up" src="https://www.cdnplanet.com/images/arrow-up-bold-circle.svg?d6400a2e308d980a6cbb805ba74e5e4b" width="24" height="24" class="icon-in-table-cell">30</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/cloudfront/">Cloudfront</a></td>
<td><img alt="up" src="https://www.cdnplanet.com/images/arrow-up-bold-circle.svg?d6400a2e308d980a6cbb805ba74e5e4b" width="24" height="24" class="icon-in-table-cell">25</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/cdnetworks/">CDNetworks</a></td>
<td><img alt="up" src="https://www.cdnplanet.com/images/arrow-up-bold-circle.svg?d6400a2e308d980a6cbb805ba74e5e4b" width="24" height="24" class="icon-in-table-cell">22</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/limelight/">Limelight</a></td>
<td><img alt="up" src="https://www.cdnplanet.com/images/arrow-up-bold-circle.svg?d6400a2e308d980a6cbb805ba74e5e4b" width="24" height="24" class="icon-in-table-cell">14</td>
<td>12</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/tata-communications/">Tata Communinications</a></td>
<td><img alt="same" src="https://www.cdnplanet.com/images/minus-circle.svg?d6c40b9a978656b15e84627cde76d0ef" width="24" height="24" class="icon-in-table-cell">10</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/maxcdn/">MaxCDN</a></td>
<td><img alt="down" src="https://www.cdnplanet.com/images/arrow-down-bold-circle.svg?50e889f5af7eea74add9289fff7d8665" width="24" height="24" class="icon-in-table-cell">10</td>
<td>32</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/fastly/">Fastly</a></td>
<td><img alt="same" src="https://www.cdnplanet.com/images/minus-circle.svg?d6c40b9a978656b15e84627cde76d0ef" width="24" height="24" class="icon-in-table-cell">10</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/cloudflare/">Cloudflare</a></td>
<td><img alt="same" src="https://www.cdnplanet.com/images/minus-circle.svg?d6c40b9a978656b15e84627cde76d0ef" width="24" height="24" class="icon-in-table-cell">10</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/cdn77/">CDN77</a></td>
<td><img alt="same" src="https://www.cdnplanet.com/images/minus-circle.svg?d6c40b9a978656b15e84627cde76d0ef" width="24" height="24" class="icon-in-table-cell">10</td>
<td>10</td>
</tr>
<tr>
<td><a href="https://www.cdnplanet.com/cdns/leaseweb/">Leaseweb</a></td>
<td><img alt="same" src="https://www.cdnplanet.com/images/minus-circle.svg?d6c40b9a978656b15e84627cde76d0ef" width="24" height="24" class="icon-in-table-cell">10</td>
<td>10</td>
</tr>
</tbody>
</table>
<div class="notice-msg info">
The initcwnd value is not the only parameter that determines CDN performance. Don't think CacheFly is the fastest CDN globally because their initcwnd is highest. The initcwnd value is just *one* of the performance parameters.
</div>
<a name="con"></a><h2>Conclusions</h2>
<p>Six out of the measured 15 CDNs have a initcwnd of 10. This is the default value in Linux kernel and apparently many these CDNs have found this to work well. Most the others set the initcwnd to be higher than 20 except for Limelight which is at 14. CacheFly still is at top with a value of 46. This is 20 lower than the 2014 measured value but very likely this is a direct result of how we measured initcwnd this time which is different from last time. Read on ...
</p>
<a name="method"></a><h2>Test Methodology</h2>
<p>During the 2014 measurements, the tests were conducted on a Macbook Air in The Netherlands with an initial receive window (initrwnd) set to a high 262144.
For each CDN, we made requests to some far-away POP (US-West or APAC) to ensure a high RTT to make it easier reading tcpdumps. For each test, we made few hits to prime the cache at the edge servers. We then studied the tcpdumps, ran the entire process several times for sanity check. For the global IP Anycast CDNs like Cachefly and Cloudflare We added an extra 500ms latency using <a href="http://www.joemiller.me/2010/08/31/simulate-network-latency-packet-loss-and-bandwidth-on-mac-osx/" class="external-link">Dummynet</a>. No dummy packet loss was added.<br>
We used <a href="https://www.cdnplanet.com/uploads/probe.py">this python script</a> to run tests and capture results using tcpdump. The dumps were manually analyzed in wireshark as <a href="https://www.cdnplanet.com/blog/tune-tcp-initcwnd-for-optimum-performance/#howtotest" title="How to test initcwnd">described here</a>
</p>
<p>That was all in 2014. In 2017, we got lazy and simply used the <a href="https://www.cdnplanet.com/tools/initcwndcheck/">Initcwnd Checker tool</a> on this CDN Planet website. We measured the initcwnd of every CDN a few times. The tool works just fine. The only differences with the 2014 testing methodology are:</p>
<ol>
<li>the initial receive windown was 65000</li>
<li>the tests ran from a AWS EC2 instance in US-East</li>
<li>we did not take the trouble to figure out what the IPs are of far-away POPs and/or add latency to have higher RTTs</li>
</ol>
<p>
Differences 2 and 3 are not so relevant but the lower initial receive window is important and explains why we measured 'just' a initcwnd of 46 for Cachefly.
46 packets at ~1400 bytes each is ~65000 bytes and that is the initrwnd on the test machine. In other words, the test machine could not receive more than 65000 bytes in the first roundtrip and so Cachefly served no more than 46 packets. Had we advertised a higher initrwnd, maybe Cachefly would have sent out more packets.
</p>
How to protect your CDN origin serverhttps://www.cdnplanet.com/blog/how-protect-your-cdn-origin-server/
Mon, 18 Mar 2013 00:12:00 +0000https://www.cdnplanet.com/blog/how-protect-your-cdn-origin-server/<p>
A website needs to be up &amp; running at all times. In today's world, this means you need to be prepared for malicious attacks, possibly a <a href="http://en.wikipedia.org/wiki/Distributed_denial-of-service" class="external-link">DDoS attack</a>, which is <cite>"an attempt to make a machine or network resource unavailable to its intended users ... One common method of attack involves saturating the target machine with external communications requests, so much so that it cannot respond to legitimate traffic, or responds so slowly as to be rendered essentially unavailable"</cite>
</p>
<p>
Using a CDN helps combat a DDoS attack. CDNs have many high-capacity servers, so they can handle the peak in requests much better than your servers could. Also, CDNs tend to be better prepared for DDoS attack mitigation, with experienced staff, technology and processes. Nevertheless, behind the CDN sits your own server(s), commonly known as the CDN origin, that serves content to the CDN when needed. The attackers might go after the origin server and if they bring that down the CDN will not be able to download content anymore and as a result not serve the content to your end users: your site is broken.
</p>
<p>
In this post I outline two options for protecting your CDN origin.
Your goal is to only receive (and allow) requests from the CDN. Requests from everybody else must be prevented. Not countered, prevented.
</p>
<h2>IP rate limiting</h2>
<p>
One way to protect a server is to do IP rate limiting: only allow X number of requests from an IP address in a given timeframe. This will not work with a CDN because a CDN will do many valid requests from a small number of IP addresses and you want all of these to go through.
</p>
<h2>Whitelisting</h2>
<p>
Another possible method for allowing only requests from your CDN is to whitelist the CDN. This should work well in theory, but in practice it is difficult to do effectively. You can whitelist IP addresses or some unique identifier in a request header.
</p>
<h3>Option A: whitelist IP addresses</h3>
<p>
The challenge with whitelisting on IP address is that you need to always have the IPs of all the CDN edge servers that may hit your origin. This is bound to fail. Many CDNs will not give you the list of IPs and if they do, it will surely happen that they add an IP address and forget to tell you.
</p>
<h3>Whitelist a unique identifier in a request header</h3>
<p>
The idea is simple: the CDN sends something unique in the requests to the origin that you can use on the origin to identify the CDN and allow the requests. You would have to ask your CDN provider about the possibilities in order to find out if this is a viable option for you. But, even if they support this, it is not bulletproof. Request headers can be freely set by attackers. If they know you use a certain CDN and they know how that CDN identifies itselt on the origin, they can easily spoof this.
</p>
<h2>Option B: unguessable origin hostname</h2>
<p>
This is a simple trick and it is also the best solution.
Create some random, long set of alphanumeric characters and use that as the subdomain.
Example: <code>205ck07023nckhfsh92485.example.com</code>.
This hostname will then be only known to the CDN, the owner of the origin and the origin's DNS provider(s). Can it be guessed? Yes, but highly unlikely. Can it leak? Yes, but again: highly unlikely.
</p>
<p>
Simply whitelist for requests that have this hostname in the <code>Host</code> header and you're done.
</p>Real-world CDN performance for Google DNS and OpenDNS usershttps://www.cdnplanet.com/blog/real-world-cdn-performance-googledns-opendns-users/
Thu, 18 Oct 2012 00:12:00 +0000https://www.cdnplanet.com/blog/real-world-cdn-performance-googledns-opendns-users/<p><strong>This is article 3 in a series of three blog posts.</strong></p>
<p>
Two days ago we published <a href="https://www.cdnplanet.com/blog/google-dns-opendns-and-cdn-performance/">Google DNS and OpenDNS usage stats</a> and in yesterday's article we explained edns-client-subnet, gave an overview of <a href="https://www.cdnplanet.com/blog/which-cdns-support-edns-client-subnet/">which CDNs support edns-client-subnet</a> and stated that the impact of <i>not</i> supporting edns-client-subnet on CDN performance is high for several CDNs, including Akamai. Today we present real-world performance data that backs this up and gives good insight in how CDN performance for Google DNS and OpenDNS users compares against all users.
</p>
<h2>Performance impact for small file static object</h2>
<p>
As mentioned before, we collect performance data at <a href="http://www.turbobytes.com/" class="external-link">TurboBytes</a> for many CDNs using a custom RUM (Real User Monitoring) solution. In a nutshell: JavaScript code in webpages, executes after page finishes loading and then fetches a 15 KB file from one of the CDNs. The timing data gets beaconed back to our server. Our DNS tests are initiated from the same JavaScript code and that enables us to gain insight in CDN performance for users on Google DNS or OpenDNS, versus all users.
</p>
<p>
Below you see four charts, one for each of these countries: United States, Italy, Indonesia and Brazil. Each chart shows, for several CDNs, by how much median total time to download that 15 KB file improved or got worse for users on Google DNS, OpenDNS or either one of these public DNS services, versus all users.<br>
The charts can help answer questions like:
<ul>
<li>In the US, do OpenDNS users get better or worse performance on Akamai?
<li>Is CDN performance better on Google DNS or OpenDNS in Italy?
</ul>
<p>Note: a positive percentage means performance is worse, negative means performance is better.
</p>
<h3>United States</h3>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/CDN-loadtime-median-delta-publicdns-vs-other-US-Oct2012.png?abaa6945626b6729cd2a26340e7a1441" width="598" height="589" alt="Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - US - Oct 2012"><br><br>
Akamai performance degrades by 20% - 40% on both public DNS services and by that is the CDN that is most negatively impacted.
Performance of Internap, EdgeCast and CloudFront for Google DNS users is much worse than for all users, while it actually improves a bit for OpenDNS users.
ChinaCache needs to look into how they route OpenDNS users. Something is going horribly wrong there.
With the exception of the EdgeCast/Google DNS combination, for all CDNs that do anycast for HTTP (and Fastly), performance is slightly better for the users of the two public DNS services versus all users.
<br>
We analyzed ~280,000 measurements per CDN, except for Akamai (~80,000).The lower count for Akamai does not impact the data shown here. We validated that by looking at different 80,000 sized samples for other CDNs.
</p>
<h3>Italy</h3>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/CDN-loadtime-median-delta-publicdns-vs-other-IT-Oct2012.png?23dffb265fde030e75b1ca33ee78df3c" width="594" height="588" alt="Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - Italy - Oct 2012"><br><br>
For most CDNs, performance improves a bit, on average circa 10%. But there are three CDNs who give Google DNS and OpenDNS users a relatively bad performance: Internap, Level3 and ChinaCache. Internap shows the same picture as for US: performance is much worse for Google DNS users, while slightly better for OpenDNS users. Level3 has no issues in the US, but in Italy OpenDNS users get almost a 100% worse performance compared to all users. ChinaCache seems to have the same issue(s) as in the US: fine for Google DNS users but a whopping ~170% worse performance for OpenDNS users.
<br>
Measurements per CDN is ~30,000 (Akamai ~20,000).
</p>
<h3>Indonesia</h3>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/CDN-loadtime-median-delta-publicdns-vs-other-ID-Oct2012.png?651f23d195953d9e7607ec5de938ec09" width="592" height="593" alt="Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - Indonesia - Oct 2012"><br><br>
Google DNS and OpenDNS are very popular in Indonesia, so CDNs that want to deliver good performance there should make sure all is good for users of those public DNS services. The chart clearly shows that is far from true. Let's start positive: CacheFly and Bitgravity are the only CDNs that delivers better performance to users of both public DNS services. Performance of Highwinds, CloudFront, NetDNA and Fastly is only impacted a little bit and EdgeCast and CDN77 are not so bad either. Internap, CDNetworks, Level3, ChinaCache and Akamai are the ones that need to take action, and especially Akamai and Internap. They have the biggest, negative performance delta for Google DNS users (30%+) and Google DNS is far more popular than OpenDNS in Indonesia.
<br>
Measurements per CDN is ~80,000 (Akamai ~45,000).
</p>
<h3>Brazil</h3>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/CDN-loadtime-median-delta-publicdns-vs-other-BR-Oct2012.png?2614fb74900977be8bf30b60478d1ac9" width="590" height="593" alt="Delta in median total loadtime for 15 kb object - Public vs other DNS resolvers - Brazil - Oct 2012"><br><br>
The situation in Brazil is ... different.
There is really no CDN that delivers significantly better performance to Google DNS or OpenDNS users.
Only Highwinds delivers a mere 4% better performance to Google DNS users.
Highwinds has a local POP in Brazil and Google DNS has resolvers in Brazil and apparently this works out.
CloudFront, Akamai, CDNetworks and CDN77 also have POP(s) in Brazil, but they deliver relatively bad performance to Google DNS users, with Akamai performance being impacted most (+57% !).
EdgeCast stands out too with a ~40% worse performance for Google DNS users. That is odd, because EdgeCast does not have a POP in Brazil or in any other country in South-America. They always serve into Brazil from the US. So why is performance 40% worse for Google DNS users from Brazil? Are they serving from a different POP in the US? We don't know.
CloudFront is also likely serving from US instead of from their POP in Brazil.
<br>
Measurements per CDN is ~80,000 (Akamai ~45,000).
</p>
<h3>The big table</h3>
<p>
Select a CDN from the dropdown menu to view a table with the total time median delta for Google DNS, OpenDNS or the aggregate of these public DNS services, per country.
The column 'Pct' shows what % of users in that country used that DNS service and the column 'Delta' shows by how much CDN performance differs from performance all users get. By default the table is sorted on percentage of users on a public DNS service, from high to low, but you can sort on any column by clicking one of the small arrows.
</p>
<div data-ng-controller="BlogPostCtrl">
<select data-ng-model="cdn" class="ng-cloak">
<option value="">.</option>
<option data-ng-repeat='c in cdns' data-ng-selected='c == cdn' value="{{c}}">{{c}}</option>
</select>
<h4 data-ng-hide='true'>Akamai</h4>
<p>A positive percentage means performance is worse, negative means performance is better.</p>
<div data-angulartable='' data-ng-model='tbl_by_cdn' class="datatable">
<table class="angulartable" style="border:solid"><thead><tr><th colspan='1'>Region</th><th colspan='1'>Count</th><th colspan='2'>OpenDNS</th><th colspan='2'>Google DNS</th><th colspan='2'>All Public DNS</th></tr><tr><th colspan='1'></th><th colspan='1'></th><th colspan='1'>Pct</th><th colspan='1'>Delta</th><th colspan='1'>Pct</th><th colspan='1'>Delta</th><th colspan='1'>Pct</th><th colspan='1'>Delta</th></tr></thead><tbody><tr class="odd"><td>Turkey</td><td>11,445</td><td>11.94 %</td><td>4.67 %</td><td>26.10 %</td><td>13.27 %</td><td>38.03 %</td><td>10.65 %</td></tr><tr class="even"><td>Viet Nam</td><td>7,833</td><td>5.34 %</td><td>85.70 %</td><td>27.75 %</td><td>6.01 %</td><td>33.09 %</td><td>16.35 %</td></tr><tr class="odd"><td>Indonesia</td><td>46,051</td><td>5.28 %</td><td>27.91 %</td><td>16.41 %</td><td>23.84 %</td><td>21.69 %</td><td>24.97 %</td></tr><tr class="even"><td>Malaysia</td><td>38,382</td><td>2.06 %</td><td>144.82 %</td><td>10.07 %</td><td>-32.96 %</td><td>12.13 %</td><td>-6.59 %</td></tr><tr class="odd"><td>Italy</td><td>20,715</td><td>2.92 %</td><td>7.35 %</td><td>7.21 %</td><td>-2.13 %</td><td>10.13 %</td><td>0.24 %</td></tr><tr class="even"><td>Thailand</td><td>9,963</td><td>1.89 %</td><td>218.66 %</td><td>7.50 %</td><td>16.13 %</td><td>9.38 %</td><td>32.49 %</td></tr><tr class="odd"><td>Brazil</td><td>46,212</td><td>1.67 %</td><td>61.25 %</td><td>7.65 %</td><td>57.23 %</td><td>9.32 %</td><td>58.12 %</td></tr><tr class="even"><td>India</td><td>31,979</td><td>2.65 %</td><td>96.11 %</td><td>6.19 %</td><td>76.42 %</td><td>8.84 %</td><td>81.87 %</td></tr><tr class="odd"><td>Russian Federation</td><td>12,003</td><td>0.90 %</td><td>32.83 %</td><td>7.76 %</td><td>-14.13 %</td><td>8.66 %</td><td>-12.46 %</td></tr><tr class="even"><td>Poland</td><td>28,595</td><td>1.06 %</td><td>-7.44 %</td><td>7.39 %</td><td>9.93 %</td><td>8.46 %</td><td>7.44 %</td></tr><tr class="odd"><td>Czech Republic</td><td>6,498</td><td>0.54 %</td><td>-11.68 %</td><td>7.88 %</td><td>0.34 %</td><td>8.42 %</td><td>0.34 %</td></tr><tr class="even"><td>Spain</td><td>20,286</td><td>0.88 %</td><td>28.87 %</td><td>6.66 %</td><td>21.78 %</td><td>7.54 %</td><td>22.31 %</td></tr><tr class="odd"><td>Mexico</td><td>45,615</td><td>0.67 %</td><td>40.18 %</td><td>6.42 %</td><td>83.71 %</td><td>7.10 %</td><td>77.01 %</td></tr><tr class="even"><td>Netherlands</td><td>14,347</td><td>1.98 %</td><td>-3.21 %</td><td>5.04 %</td><td>20.18 %</td><td>7.02 %</td><td>13.30 %</td></tr><tr class="odd"><td>United States (Illinois)</td><td>9,969</td><td>2.73 %</td><td>56.39 %</td><td>4.21 %</td><td>129.07 %</td><td>6.94 %</td><td>105.29 %</td></tr><tr class="even"><td>United States (Michigan)</td><td>6,040</td><td>3.56 %</td><td>41.94 %</td><td>3.33 %</td><td>104.84 %</td><td>6.89 %</td><td>80.65 %</td></tr><tr class="odd"><td>Canada (Quebec)</td><td>11,585</td><td>3.79 %</td><td>90.91 %</td><td>3.05 %</td><td>42.15 %</td><td>6.84 %</td><td>75.62 %</td></tr><tr class="even"><td>Argentina</td><td>11,708</td><td>1.39 %</td><td>56.50 %</td><td>5.36 %</td><td>48.23 %</td><td>6.75 %</td><td>49.41 %</td></tr><tr class="odd"><td>Canada (Ontario)</td><td>29,566</td><td>3.73 %</td><td>114.95 %</td><td>2.95 %</td><td>97.20 %</td><td>6.68 %</td><td>107.48 %</td></tr><tr class="even"><td>Morocco</td><td>8,048</td><td>0.48 %</td><td>-1.78 %</td><td>6.10 %</td><td>46.96 %</td><td>6.59 %</td><td>44.44 %</td></tr></tbody></table></div>
</div>
<h2>In summary</h2>
<p>
Many people use the Google DNS or OpenDNS services. From our data the combined 'market share' is ~8% globally, and in some (big) countries <a href="https://www.cdnplanet.com/blog/google-dns-opendns-and-cdn-performance/#ms">much higher</a>. Both public DNS services allow CDNs to have the edns-client-subnet be passed on to them, so they can determine with more accuracy what the geolocation of the end-user is. In consequence, the CDNs can provide a better user experience by serving content from a POP closeby over a low-latency connection. We listed <a href="https://www.cdnplanet.com/blog/which-cdns-support-edns-client-subnet/#cdns">9 CDNs that would benefit from supporting/using edns-client-subnet</a>, and only two actually support edns-client-subnet: CDN77 and ChinaCache. Others, including Akamai, Internap and CDNetworks, do not currently. This really is too bad, because from the performance data we collected, it is clear these CDNs deliver (much) worse performance currently in many countries to Google DNS and OpenDNS users. We believe performance can greatly improve if they would support edns-client-subnet. But edns-client-subnet is not a magic bullet. The performance data clearly shows that some non-anycast CDNs (ChinaCache) need to do more and in some countries the anycast CDNs are not servicing the public DNS users as well as they should (EdgeCast).
Our final message is to Google DNS and OpenDNS: please make an effort to publish an uptodate map/list of the resolver locations, so CDNs can use that to keep their systems uptodate. Even better would be to proactively inform them of location changes. Maybe you do this already, but ... please crank it up a notch. Txs.
</p>
<script type="text/javascript" >
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = '/scripts/compagg.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>Which CDNs support edns-client-subnet?https://www.cdnplanet.com/blog/which-cdns-support-edns-client-subnet/
Wed, 17 Oct 2012 00:12:00 +0000https://www.cdnplanet.com/blog/which-cdns-support-edns-client-subnet/<p><strong>This is article two in a series of three blog posts.</strong></p>
<p>
Yesterday we published <a href="https://www.cdnplanet.com/blog/google-dns-opendns-and-cdn-performance/">Google DNS and OpenDNS usage stats</a> that we generated from the data of millions of end-user tests. Interesting as this is, the real goals of those tests was to a) find out which CDNs support edns-client-subnet and b) gain insight in the impact of edns-client-subnet on CDN performance. In this article, we explain what edns-client-subnet is, its relevance to CDN performance and show which CDNs currently support it. In article 3, to be published tomorrow, we will show real-world performance data and make clear how edns-client-subnet (significantly) impacts CDN performance.
</p>
<ul>
<li><a href="#p">The problem</a>
<li><a href="#s">Solutions</a>
<li><a href="#cdns">Which CDNs support edns-client-subnet</a>
<li><a href="#testing">Test methodology</a>
</ul>
<a name="p"></a><h2>The problem</h2>
<p>
Some CDNs use DNS to determine the geographical location of the user. They can't use the IP address of the client for this, because it is masked by the DNS resolver, and so the CDNs use the IP address of the DNS resolver instead. In case of the Google DNS or OpenDNS servers, for many end users those servers are not close to them, simply because these providers don't have servers in every country and every ISP's network. For example, OpenDNS does not have DNS servers in South-America (<a href="http://www.opendns.com/technology/network-map/" class="external-link">network map</a>). Someone in Brazil using OpenDNS will likely hit their resolver in Florida. The CDN will then think the user is in Florida and as a result it will serve content to the user from a server far away (Florida, not Brazil) resulting in a slow experience.
</p>
<p>
The two diagrams below illustrate how things work for a user in Thailand who wants to connect to a hostname on Akamai, using either the DNS resolver of his ISP or the resolver of OpenDNS.
</p>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/cdn-without-edns-isp-resolver.png?e569aea23f63a7cd5a5ea8f2ddb5122c" width="577" height="309" alt="CDN without edns-client-subnet support - normal ISP resolver"><br><br>
The user ends up hitting a low-latency Akamai edge server in Thailand because Akamai's DNS server can accurately detect the user's geolocation. This is good.
</p>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/cdn-without-edns.png?d16af458461fe796c2b78ddaf279ab31" width="601" height="355" alt="CDN without edns-client-subnet support"><br><br>
Now Akamai cannot accurately detect the user's geolocation and believes the user is in Singapore. The user will connect to a high-latency Akamai edge server in Singapore. This is bad.
</p>
<a name="s"></a><h2>Solutions</h2>
<p>
There are two solutions a CDN can implement to counter this problem. They can either use anycast for HTTP or support edns-client-subnet. We'll take a quick look at anycast for HTTP but focus on edns-client-subnet.
</p>
<h3>Anycast for HTTP</h3>
<p>
CDNs that use anycast for HTTP are not affected by the problems with geo targetting using DNS since they resolve to the same IP for all users (or all users in one region; EdgeCast). They rely on BGP to direct users to the closest server based on preferred path. Using anycast for HTTP may come with its own set of problems, but that is beyond the scope of this post.
</p>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/cdn-with-anycast.png?f7a3d2a6f3c345b989627e539ea76a51" width="598" height="339" alt="CDN with Anycast for HTTP"><br><br>
The CDN's authoritative name server (ADNS) will respond with an IP address regardless of the user's location and that will lead the user to a nearby edge server wherever the user is.
</p>
<h3>Support edns-client-subnet</h3>
<p>
To mitigate the problem of DNS based geo-targetting, Google proposed a technical solution to the issue in an IETF draft <a href="http://tools.ietf.org/html/draft-vandergaast-edns-client-subnet-01" class="external-link">Client subnet information in DNS requests</a>. This is an experimental DNS extension that allows DNS resolvers to pass the client's IP address (or part of) to compatible authoritative DNS servers. The CDN's DNS server can then use this information to better determine where the end user is. Google DNS and OpenDNS implemented this solution as part of the <a href="http://afasterinternet.com/" class="external-link">Global Internet Speedup</a> initiative in August 2011.
</p>
<p>
The drawback is the experimental nature of the spec and limited support in existing DNS server software. Only OpenDNS and Google Public DNS seems to support it on the resolver side. With both these providers, one must apply to be "whitelisted" in order to receive client's subnet with the query. The whitelisting procedure is <a href="http://tools.ietf.org/html/draft-vandergaast-edns-client-subnet-01#page-22" class="external-link">pretty straightforward</a>. Contact OpenDNS or Google, tell them your hostnames, nameservers IPs and they will probably whitelist you within a couple of days without any fuss.
</p>
<p>
Our test DNS server supported edns-client-subnet since the start of this month, and no resolver apart from Google and OpenDNS sent us queries with client subnet information.
</p>
<p>
The following illustration shows how supporting edns-client-subnet should result in getting a low-latency connection to the CDN.
</p>
<p>
<img class="img-no-max-width" src="https://www.cdnplanet.com/images/cdn-with-edns.png?aaa98cfebe5ff8d3a4ab505a73979c24" width="590" height="359" alt="CDN with edns-client-subnet support">
</p>
<a name="cdns"></a><h2>Which CDNs support edns-client-subnet</h2>
<table class="table-striped table-two-col-5050">
<thead>
<tr>
<th>CDN</th>
<th>Supports edns-client-subnet?</th>
</tr>
</thead>
<tbody>
<tr>
<td>Akamai</td>
<td><img src="https://www.cdnplanet.com/images/close-circle.svg?b74dae73aeb9a11192044c6105f9a4ce" width="24" height="24" class="icon-in-table-cell" alt="High impact"> High impact</td>
</tr>
<tr>
<td>Bitgravity</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Anycast"> Anycast, so no problem</td>
</tr>
<tr>
<td>CacheFly</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Anycast"> Anycast, so no problem</td>
</tr>
<tr>
<td>CDN77</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Excellent"> Excellent!</td>
</tr>
<tr>
<td>CDNetworks</td>
<td><img src="https://www.cdnplanet.com/images/close-circle.svg?b74dae73aeb9a11192044c6105f9a4ce" width="24" height="24" class="icon-in-table-cell" alt="High impact"> High impact</td>
</tr>
<tr>
<td>ChinaCache</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Excellent"> Excellent!</td>
</tr>
<tr>
<td>CloudFlare</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Anycast"> Anycast, so no problem</td>
</tr>
<tr>
<td>CloudFront</td>
<td><img src="https://www.cdnplanet.com/images/close-circle.svg?b74dae73aeb9a11192044c6105f9a4ce" width="24" height="24" class="icon-in-table-cell" alt="High impact"> High impact</td>
</tr>
<tr>
<td>EdgeCast</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Anycast"> Anycast, so no problem</td>
</tr>
<tr>
<td>Fastly</td>
<td><img src="https://www.cdnplanet.com/images/check-circle-partially.svg?e132e979d65286fe6626fb86eedb75ee" width="24" height="24" class="icon-in-table-cell" alt="Not big problem now"> Not big problem now</td>
</tr>
<tr>
<td>Highwinds</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Anycast"> Anycast, so no problem</td>
</tr>
<tr>
<td>Internap</td>
<td><img src="https://www.cdnplanet.com/images/close-circle.svg?b74dae73aeb9a11192044c6105f9a4ce" width="24" height="24" class="icon-in-table-cell" alt="High impact"> High impact</td>
</tr>
<tr>
<td>Level 3</td>
<td><img src="https://www.cdnplanet.com/images/close-circle.svg?b74dae73aeb9a11192044c6105f9a4ce" width="24" height="24" class="icon-in-table-cell" alt="High impact"> High impact</td>
</tr>
<tr>
<td>Limelight</td>
<td><img src="https://www.cdnplanet.com/images/close-circle.svg?b74dae73aeb9a11192044c6105f9a4ce" width="24" height="24" class="icon-in-table-cell" alt="High impact"> High impact</td>
</tr>
<tr>
<td>NetDNA</td>
<td><img src="https://www.cdnplanet.com/images/check-circle.svg?862be7b7c6a071622928a4a0b261b26a" width="24" height="24" class="icon-in-table-cell" alt="Anycast"> Anycast, so no problem</td>
</tr>
</tbody>
</table>
<h3>CDNs that need to take action</h3>
<p>
Quite a few CDN providers don't do anycast for HTTP and don't support edns-client-subnet: Akamai, CDNetworks, CloudFront, Fastly, Internap, Level3 and Limelight.
These are the CDNs that provide worse performance to some Google DNS and OpenDNS users than possible. But the size of the problem is not equal for all these CDNs.
Fastly is the newcomer in the CDN market and currently has a limited number of POPs (4 in the US and 3 in Europe). Supporting edns-client-subnet will not improve performance all that much for them.
Performance of the bigger CDNs however, like Akamai and Limelight, could improve significantly for many end-users if they would support edns-client-subnet. They have thousands of servers in many, many locations and quite a few Google DNS and OpenDNS users currently don't connect to an edge server inside the network of their ISP, but rather to a server in some other country or even continent.
</p>
<h3>A few more things to say</h3>
<p>
CDNetworks is listed as a <a href="http://www.afasterinternet.com/participants.htm" class="external-link">participant</a> in the Global Internet Speedup initiative, but they actually don't support edns-client-subnet. That is odd. The same goes for Bitgravity and CloudFlare, but since these CDNs do anycast, it is a none-issue.<br>
EdgeCast is the only CDN with anycast for HTTP that supports edns-client-subnet, but it shouldn't really matter much because they do anycast (per region actually: North-America, Europe, APAC, ...). Perhaps EdgeCast can improve the effectiveness of their regional anycast architecture just a little bit with edns-client-subnet.
</p>
<a name="testing"></a><h2>Test methodology</h2>
<p>This is what I did to test support for edns-client-subnet by CDNs:
</p>
<p>
I downloaded the patch to dig from <a href="http://wilmer.gaa.st/edns-client-subnet/" class="external-link">http://wilmer.gaa.st/edns-client-subnet/</a> and installed it. Actually, I copied the patched dig as dig-client into my path for easy access.
</p>
<pre>
sajal@sajal-laptop:~$ dig-client gp1.wac.v2cdn.net @ns1.edgecastcdn.net +client=58.8.96.26
; &lt;&lt;&gt;&gt; DiG 9.9.1-P3 &lt;&lt;&gt;&gt; gp1.wac.v2cdn.net @ns1.edgecastcdn.net +client=58.8.96.26
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 8715
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; CLIENT-SUBNET: 58.8.96.26/32/14
;; QUESTION SECTION:
;gp1.wac.v2cdn.net. IN A
;; ANSWER SECTION:
gp1.wac.v2cdn.net. 3600 IN A 117.18.237.1
;; Query time: 56 msec
;; SERVER: 72.21.80.5#53(72.21.80.5)
;; WHEN: Tue Oct 16 17:08:39 2012
;; MSG SIZE rcvd: 91
sajal@sajal-laptop:~$
</pre>
<p>
The response from EdgeCast shown above means that EdgeCast is saying that the response is only valid for 58.8.96.26/14 (i.e. 58.8.0.1 - 58.11.255.254). It may be the case that the CDN has only whitelisted Google/OpenDNS to receive client-subnet. In that case we can query Google. Google passes along subnet information to everyone.
</p>
<pre>
sajal@sajal-laptop:~$ dig-client gp1.wac.v2cdn.net +client=58.8.96.26 @8.8.8.8
; &lt;&lt;&gt;&gt; DiG 9.9.1-P3 &lt;&lt;&gt;&gt; gp1.wac.v2cdn.net +client=58.8.96.26 @8.8.8.8
;; global options: +cmd
;; Got answer:
;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NOERROR, id: 5615
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; CLIENT-SUBNET: 58.8.96.26/32/14
;; QUESTION SECTION:
;gp1.wac.v2cdn.net. IN A
;; ANSWER SECTION:
gp1.wac.v2cdn.net. 2192 IN A 117.18.237.1
;; Query time: 60 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Oct 16 17:13:56 2012
;; MSG SIZE rcvd: 74
sajal@sajal-laptop:~$
</pre>
<!-- <p>
My bash script is <a href="http://pastie.org/private/voudrszepsldypeepbilta" class="external-link">here</a>
</p> -->A better CDN Finderhttps://www.cdnplanet.com/blog/better-cdn-finder/
Tue, 29 May 2012 00:12:00 +0000https://www.cdnplanet.com/blog/better-cdn-finder/<p>
The <a href="https://www.cdnplanet.com/tools/cdnfinder/">CDN Finder tool</a> has been very popular since the day we launched it 6 months ago. It enables you to easily find out what CDN(s) a site is using or what CDN is behind a hostname. CDN Finder is easy to use, pretty fast and often gives correct results, but definitely not always. We have always been aware of the tool's imperfections and Sajal recently decided to spend a Saturday on building a new, better CDN Finder from the ground up, with the primary goal to return better results (less false negatives: our tool says 'no CDN' while the hostname <i>does</i> point to a CDN).
</p>
<p>
Mission accomplished: the <a href="https://www.cdnplanet.com/tools/cdnfinder/">new CDN Finder</a> is done, deployed and really is a lot better.
</p>
<h2>What we did to improve CDN Finder</h2>
<p>
In the first version of the tool, CDN Finder fetched the HTML and parsed it with very simple logic to get to a list of hostnames. For each found hostname CDN Finder would then do a DNS lookup, look at the last CNAME only and regex match that hostname to our list of CDN hostnames.
</p>
<p>There are three problems with this method:
<ol>
<li>Hostnames serving content on the page linked to from within CSS or injected via JavaScript are missed
<li>Using just the last CNAME in the chain is error prone as some CDNs have many different hostnames
<li>Our list of CDN hostnames was not complete
</ol>
<p>We tackled all three problems and the results are now much better.
</p>
<h3>1. Full page rendering</h3>
<p>
CDN Finder (full site lookup) now loads and renders the full page in Webkit using <a href="http://phantomjs.org/" class="external-link">PhantomJS</a>.
The 'browser' will parse the HTML, CSS and JavaScript, execute the JavaScript, etc.
As a result, CDN Finder has much better list of hostnames that served at least one object on the page.
And there is another reason to render the page with PhantomJS: we get the response headers for each resource.
This allows catching CDNs even if no CNAMEs are given. For example, if the <code>Server</code> response header contains <code>cloudflare-nginx</code>, the resource is served by CloudFlare. We use this only as a fallback, if the CNAME chain processing has 'no CDN' as a result.
</p>
<h3>2. Full CNAME chain processing</h3>
<p>
Taking that better list of hostnames as input, for each hostname the CDN Finder analyzes the full CNAME chain and by that can better decide which CDN is behind it.
Example:
<pre>
cdnfinder@cdnplanet:~$ host ec.cdnplanet.com
ec.cdnplanet.com is an alias for wac.6e8d.edgecastcdn.net.
wac.6e8d.edgecastcdn.net is an alias for gp1.wac.edgecastcdn.net.
gp1.wac.edgecastcdn.net has address 117.18.237.250
</pre>
<p>Previously the CDN Finder would only use <code>gp1.wac.edgecastcdn.net</code> to detect the CDN. Now it uses <code>ec.cdnplanet.com</code>, <code>wac.6e8d.edgecastcdn.net</code> and <code>gp1.wac.edgecastcdn.net</code>. In this example, the <code>wac.6e8d.edgecastcdn.net</code> part will be stable, as that is what the customer CNAMEs to. <code>gp1.wac.edgecastcdn.net</code> can be changed at any time by the CDN, into for example <code>gp1.wac.eccdn.net</code> and that would then fail as <code>eccdn.net</code> is not in our whitelist.
</p>
<h3>3. More CDNs on our whitelist</h3>
<p>
CDN Finder now also catches the following CDNs: Bitgravity, ChinaCache, CDN77, OnApp and Turbobytes.
</p>
<h2>Benefits to the user</h2>
<p>Simply put: the results are more correct/reliable. If the result for a hostname is 'no CDN', then this really should be the case. We feel confident about the methodology and the tool runs smoothly. We hope you enjoy using the <a href="https://www.cdnplanet.com/tools/cdnfinder/">CDN Finder</a> and please share your thoughts in the comments!
</p>
<h2>Open-source</h2>
<p>
CDN Finder is open-source. You can find it on Github <a href="https://github.com/turbobytes/cdnfinder" class="external-link">here</a>.
</p>