SamKnows Critical Services Report: CDN Performance (UK)

Introduction

The COVID-19 pandemic has led to the world’s largest ever working-from-home experiment. This has inevitably led to questions about the ability of broadband networks to cope with the additional traffic. In an effort to separate fact from rumour, we are prototyping a tracking report that studies key quality indicators for broadband. More specifically, we will be tracking the degradation across measures like speed and latency to a variety of applications, and will be comparing data from early March 2020 (before everyone started working remotely) and more recent dates.

In this first blog post we will be looking at a very high-level download speed measure, and latency measurements to a variety of destinations, including two major CDNs.

Download Speed (multiple TCP connections)

Download speed is measured nine times per day per Whitebox over three parallel TCP connections to multiple test servers in London, each of which are equipped with 10Gbit/s network interfaces.

All ISPs are seeing a slight degradation in download speed over the past couple of weeks. However, this change really is minimal, with no ISP seeing more than a 2% decline since the start of March. The sharper drop on March 10th for two ISPs can be attributed to the Call of Duty Warzone release, which caused record traffic levels at LINX.

We note that others in the industry (such as CDNs) are reporting much larger drops amongst ISP's download speeds around the world. This is likely due to them measuring a small object over a single TCP connection. The performance of this will likely be dominated by latency rather than capacity, and we'll see more on this below. We perform both single-TCP and multiple-TCP measurements, and will be exploring the differences between these in a future blog post.

Percentage change in download speed since 1st March 2020. Higher values are better. Results are derived from tests to the off-net London servers, using three parallel TCP sessions for 10 seconds. Results from all hours of the day are used (not just peak).

Latency

Our generic latency metric is measured hourly from every Whitebox to our test servers in London. When there is some congestion in the path, latency is one of the first metrics where this will become visible. We present two different views on latency in this blog post.

Firstly, we present the change in latency per day as a percentage. This allows us to see trends over the course of the past month. The chart below shows that latency is generally trending upwards slightly since mid-March. No ISP sees more than a 3.3% increase in latency on average, which is a negligible amount in real terms. However, these averages mask edge cases, so it is likely that some users are seeing much larger increases in latency.

Percentage change in latency since 1st March 2020. Lower values are better. Results are derived from tests to the off-net London servers, using the hourly UDP latency test. Results from all hours of the day are used (not just peak).

Secondly, we show the hourly change in latency between the weeks of 2020-03-02 - 2020-03-09 and 2020-03-16 - 2020-03-23. The line for each ISP shows the percentage change between the two weeks, split by hour of day. This will help us to see if there's a correlation in between increasing latency and the hours of the day when people are more likely to be working at home.

As we can see from the chart below, all ISPs are seeing a small increase in latency during the daylight hours. ISP D see the most sizeable increase in latency at around 3% between 9am and 7pm. However, in real world terms, the additional latency presented here would be unnoticeable to users.

Percentage change in latency for each hour of the day, split by week. The reference week is the week commencing 2020-03-02. Lower values are better. Results are derived from tests to the off-net London servers, using the hourly UDP latency test. Results from all hours of the day are used (not just peak).

Cloudflare's CDN

The following chart presents the percentage change in latency to Cloudflare's CDN between the first week of March and the week of 16th-22nd March. As with previous latency measures, most ISPs see almost no change in latency. ISP D is the only small exception, with latency rising by 5% from 9am onwards.

Latency to Cloudflare, split by hour of day and week. Lower values are better. Results from all hours of the day are used (not just peak).

Akamai's CDN

Measurements to Akamai again show minimal change in latency for most ISPs between the first week of March and the week of 16th-22nd March. The one exception is ISP D, but this time it's an improvement for them. We understand they have recently upgraded their capacity with Akamai, which has resulted in the 12% improvement in latency at certain times.

Latency to Akamai, split by hour of day and week. Lower values are better. Results from all hours of the day are used (not just peak).

Conclusions and next steps

The UK's major ISPs are coping well with the increased demand from home workers during the COVID-19 pandemic. ISPs design and build their networks to cater for the busiest periods, which have traditionally been the evening hours. This means that their networks have been much more idle during the daylight hours, and are therefore able to absorb the additional usage that's a result of all of the new home workers.

That said, there are some early warning signs that some networks are seeing slight effects of congestion. ISP D, for example, sees an increase of 3-5% in latency measurements here. In real terms this is fractions of a millisecond, so is unnoticeable to users, but it is worth keeping an eye on.

Methodology

The charts presented below are taken from the SamKnows UK panel of Whiteboxes, between 2020-03-01 and 2020-03-24. The Whitebox is a small hardware measurement device installed in a user's home, which carries out automated performance tests over their broadband connection many times per day. The results here are derived from a sample of approximately 4,000 Whiteboxes in the UK.

Crucially, all results are presented as the percentage from the beginning of March. In this report, we are not concerned with absolute speed or latency figures, we are instead concerned with whether the situation is degrading from a few weeks ago (before everyone was spending so much more time at home).

Results are split by major ISP only, and not by the individual broadband products they offer. We have also not split the results out by nation or county at this stage, but we may do so in the future.

A few notes on exclusions:

All charts use a consistent set of Whiteboxes across the entire time range used in the chart. So a Whitebox must have generated measurements for every date in the period in order for it to be included. If a Whitebox did not generate measurements even for a single day, then the Whitebox is excluded entirely.

If a Whitebox changed ISP during the period then it has been excluded entirely.

If a Whitebox changed broadband product during the period, or its measured speed changed by more than 50% (either up or down), then it is excluded entirely.

All of these mechanisms help ensure that we have a consistent base for the comparison.