Beyond speed tests: CDN performance

In this blog post, we examine the technologies used to deliver live TV streams over the internet by the BBC and ITV, the UK’s two largest terrestrial broadcasters. We also carry out performance measurements from nearly 700 homes in the UK during prime time and take a look at how the broadcasters, broadband providers, and the CDNs stack up.

What is live streaming and how is it delivered over the internet?

Before we go any further, let’s be clear about what we mean by “live streaming” and how it differs from “on-demand” or “catch-up” streaming. Live streaming involves watching live TV on the internet, as it is broadcast (often with a small delay, which itself can be controversial!). For example, going to BBC iPlayer and watching the BBC News channel live. On-demand streaming involves watching pre-recorded programmes as and when you feel like it. Watching last week’s episode of Have I Got News For You on iPlayer would be a good example of this.

At SamKnows, we already have a variety of measurements for on-demand streaming, including tests for BBC iPlayer, Netflix, YouTube, and Hulu.

But live TV streaming is quite different. The “live” aspect makes it significantly more difficult for broadcasters and CDNs — more on this later. On-demand TV providers, such as Netflix, have the luxury of being able to pre-fill their vast array of global caches overnight, ensuring content is always readily available and near to the viewer. This is not so easy with live streams!

There are a variety of mechanisms used to deliver live streams over the internet. Some broadband providers in the UK provide a dedicated TV set-top box, which uses multicast to receive live TV. This is very efficient from a network standpoint, as one stream may be shared by many users concurrently. Some TV providers use Flash plugins in web browsers still, and rely on RTMP (or one of its variants) to deliver content to users. Others use more modern protocols such as HLS, HDS, and MPEG-DASH, which are supported by modern web browsers (without needing plugins) and — most importantly — Android and Apple mobile devices. Many TV providers will use a combination of these approaches and will adapt according to the device that the user is using.

Regardless of the protocol, the “content provider” (e.g. the BBC and ITV in this case) will typically not deliver the content directly to end users. Instead, they will rely on an intermediary called a CDN (Content Delivery Network) to handle the vast bandwidth demands for them. In some cases, multiple CDNs may be used by one content provider. These CDNs will typically have excellent network connectivity to the broadband providers, as they will want to ensure that they can get content to the end users as fast as possible. In some cases, the CDNs will even place “caches” inside ISPs’ networks, which further reduces the distance content has to travel to end users and saves on bandwidth costs for the ISP and CDN.

Live streaming by the BBC and ITV

The BBC use two external CDNs called Akamai and Limelight, and have more recently built their own CDN called BIDI (“BBC Internet Distribution Infrastructure”). In our testing, we only observed Akamai and Limelight being used for BBC live streaming. The BBC use a blend of HLS, HDS, and DASH, depending upon the client type.

ITV uses a single CDN — Akamai — to deliver its content to users. It too uses a blend of protocols that depend upon the client type. Desktop web browsers still relied on Adobe Flash plugins, with content being delivered via RTMP, whilst mobile users were served content using HLS.

Akamai, used by both the BBC and ITV, is one of the largest and oldest CDNs in the world, reportedly serving between 15% and 30% of all web traffic. Their traffic volumes are so large in fact that they offer ISPs caches to install in their networks, which reduces the amount of user requests that need to leave the ISP network. This is done with the goal of improving user experience whilst simultaneously reducing costs to the ISP, CDN, and content provider. As we will see later on, the largest ISPs have all taken Akamai up on this offer and employ Akamai caches inside their networks.

Limelight is a significantly smaller CDN than Akamai. Our testing didn’t observe Limelight caches inside the networks of any of the UK’s largest ISPs, meaning that content from Limelight would be served from more distant locations — although in a country as small as the UK this is arguably not significant.

Our test setup

We carried out our measurements on the evening of 24th July 2018, between 6pm and 10pm BST (GMT+1). Measurements were carried out from 687 Whiteboxes (small, Linux-based hardware appliances installed in consumer homes), spread out across broadband connections provided by BT, Virgin, Sky, TalkTalk, Vodafone and Hyperoptic. For the VDSL/FTTC (Fiber to the Cabinet) ISPs, we only conducted measurements on their fastest products — those with a 76Mbps headline speed. For Virgin, the sole cable ISP, we only conducted measurements on their 200Mbps and 350Mbps products.

Our tests streamed BBC1 from the BBC and ITV1 from ITV.

For BBC1, we used the BBC’s HDS stream. Our measurement client inside the Whiteboxes connected to the BBC “Media Selector”, which provides links to the manifest files for the different bitrates of the BBC1 streams. This is also where the load balancing between Akamai and Limelight took place. Our measurement client would then randomly select between Limelight and Akamai, and would stream the highest quality BBC stream available, which was an HD stream at a target bitrate of 1.7Mbps. The client would request the successive chunks that composed the stream for a duration of 20 seconds.

For ITV1, we used the ITV’s HLS stream, as delivered to their mobile apps. Our measurement client inside the Whiteboxes would fetch their manifest file and begin downloading the segments as they were offered to us. We utilised the highest quality stream available to us, a 1.8Mbps HD stream. As with the BBC, the client would stream content for 20 seconds.

The same metrics were recorded for both the BBC and ITV. These were:

Date and time of the transaction

Hostname of the CDN server that delivered the content

IP address of the CDN server

Segment ID

Size of the segment

Download speed of the segment

Time to establish the TCP connections

The “Last Modified” HTTP header of the segment

In total, 16,553 results were collected for the BBC and 7,622 results were collected for ITV.

Analysis with SamKnows One

We developed the new tests to integrate directly into our data ingestion pipelines and were able to easily add these metrics to SamKnows One, our powerful cloud-based analytics and data visualisation platform.

When analysing the data we focused on two particular metrics, the TCP connection time (a rough proxy for round-trip latency) and the download speed (throughput) of the segment. These were the two metrics we added into SamKnows One.

SamKnows One also has functionality for prefiltering and so we pre-filtered out any failed test results or results that might otherwise skew results unfairly. We removed results where any of the following was true:

The total size of the segment was not more than 0 bytes

The HTTP response code was not 200

And the TCP connect time was more than 3000ms, acting as a timeout

Using SamKnows One we were able to aggregate and split the data in a whole variety of different ways to generate metrics, delve deeply into the data, see common patterns, identify anomalies, and produce all the visualisations you see below.

Results

Figure 1

Figure 1 above presents the average download speed from the BBC (blended across Akamai and Limelight) and ITV. At first glance, it might appear that ITV holds a significant edge. However, there are a few things to keep in mind here.

Firstly, the BBC’s average download speed of 37.7Mbps is still far above the actual content bitrate of 1.7Mbps. This would mean that users, on average, would experience no trouble streaming BBC content at this speed. Secondly, there is some bias towards ITV in the delivery method here. The average size of ITV’s segments as delivered via their CDN was 1.4MiB, whilst the BBC’s was around 800KiB. The smaller file size for the BBC means that TCP is far less likely to have time to ramp up to full speed before the 800KiB is exhausted.

Figure 2

Figure 2 above presents a scatter plot for the same time range. Performance is quite varied (as would be expected across a blend of nearly 700 different broadband connections), but it is consistent — there are no significant drops that would indicate the services are suffering from congestion during peak hours.

Limelight vs Akamai

Figure 3

The fact that the BBC delivers content over both Akamai and Limelight allows us to perform an apples-for-apples comparison of the two CDNs.

Figure 3 above compares the average download speed for Akamai and Limelight, as well as their average TCP connection establishment time (which is an approximate proxy for round-trip latency). The results show a noticeably higher download speed for Akamai and a lower TCP connection time (lower is better). This is expected given the deployment of Akamai caches inside the ISP’s networks, which is something that Limelight doesn’t appear to do.

On-net vs off-net Akamai performance

Figure 4

Figure 4 shows the presence of Akamai caches inside ISP’s networks. Virgin, BT, Vodafone, Sky and TalkTalk all deployed Akamai caches inside their networks. Of the ISPs measured, only Hyperoptic didn’t have local Akamai caches. That said, there doesn’t appear to be a noticeable penalty for content going “off-net” and being delivered from Akamai’s centralised caches.

Akamai on-net cache hit rate

Figure 5

For an ISP that has invested in deploying Akamai caches inside their network, it’s likely they’ll want to ensure that these caches are being used 100 percent of the time, and that content isn’t being delivered from off-net locations. Figure 5 above shows the volume of requests that were delivered from Akamai caches inside the ISP networks versus those outside of the ISP’s networks. Curiously, no ISP saw 100% of their requests being delivered from caches inside their networks. BT’s on-net hit rate was 97%, whilst Vodafone’s was only 78%.

We can also see the number of Akamai caches (or at least distinct IP addresses) used in each ISP. With 105 and 142 distinct IP addresses respectively, BT and Virgin have by far the largest footprint of Akamai caches inside their networks. In reality, these are almost certainly not hundreds of separate servers, but are instead multiple IP addresses and/or interfaces attached to a relatively small number of big caching servers.

Limelight on-net caches

Figure 6

Figure 6 above very simply demonstrates that 100% of requests to Limelight were served from Limelight-owned IP addresses. None of the requests were served from the address space of the ISPs, which is a very strong indicator that Limelight caches are not present in the ISPs’ networks.

IPv4 vs IPv6

Figure 7

IPv6 adoption has finally accelerated in recent years and even many of the UK’s ISPs have caught up and are already running it in production or are trialling it. However, as Figure 7 shows, neither Akamai or Limelight’s CDNs delivered content for the BBC or ITV over IPv6. This isn’t due to the Whiteboxes not having IPv6 connectivity; there are simply no AAAA records associated with the CDN endpoints advertised by the BBC and ITV manifest files. Curiously, the BBC’s manifest files itself are served over IPv6 and are also served by Akamai, but the content they reference is not.

ISP comparison

Figure 8

Figure 9

Figures 8 and 9 above plots the average download speed and TCP connection time respectively for each of the ISPs tested to both the BBC and ITV content. Note that BBC is presented as a blend of Akamai and Limelight delivered content in this chart. Whilst there may appear to be some sizeable differences in numbers in these charts, all of them are perfectly adequate for the current video streams available from the BBC and ITV. Again, the relatively small object size (800KiB for the BBC and 1.4MiB for ITV) plays a large role in why speeds are relatively low.

The one interesting outlier is Hyperoptic, which is significantly faster than all other ISPs. Whilst Hyperoptic are the only ISP included with some 1Gbps connections measured, the main reason for their outperformance is the vastly lower latency (which can be seen in the TCP connection time). This allows them to complete the TCP ramp up far quicker and spend more time at a higher rate when transferring the object, thus increasing the overall average speed.

Conclusions

Streaming live TV over the internet is a complex business. Content providers like the BBC and ITV make use of an extensive network of invisible infrastructure to ensure that we can stream content interruption free. Our testing has shown the differences in how two of the major CDNs — Akamai and Limelight — structure their networks to deliver content to end users. Regardless of the approach, performance for both BBC and ITV live streaming services appears to be good across the major UK ISPs, at least for the current HD level of video quality.

With the advent of 4K live streaming, which the BBC trialled for the world cup recently in a limited fashion, bandwidth demands will increase significantly. Recent reports talk of the BBC using approximately 35Mbps in their 4K streams. Rest assured, SamKnows will continue developing new measurements to keep track of the performance of ISPs, content providers, and CDNs.

This is the first of a series of articles where we “look beyond speed tests” and measure other things that affect the performance of an application. Follow us to keep an eye out for future updates on this topic, and other developments.