Level 3’s Lower Cost Comes From Owning The Network, Not Free Peering

One of the things I've seen some bloggers, Wall Street analysts and readers commenting on regarding the Level 3 and Comcast dispute, is the idea that Level 3 only won the Netflix business with their low price, because Level 3 thought it could get free peering from Comcast. Many also want to imply that Level 3 only won the Netflix deal because of that low price and suggest that Netflix is somehow sacrificing quality by using Level 3, in exchange for saving some money. None of this could be further from the truth.

There is this false notion amongst many that Level 3 can't make money on the Netflix deal if they have to pay Comcast for peering and that this is the reason Level 3 is so upset. Some have even suggested that Level 3 is being unfair because they don't want to pay for something the other CDNs pay for. I've also seen many suggest that Level 3 didn't know what it was getting itself into when it signed Netflix as a customer or that Level 3's approach to the CDN business is the same one they had four years ago when they first entered. Some want to use the example of how Level 3 might charge Cogent for peering and suggest Comcast is simply doing the same thing except that Cogent is not a last mile provider like Comcast. All of this has me wondering why almost no one has noticed that Level 3 owns the network and has a lower cost which means they can offer a cheaper price in the market to begin with, regardless of whether or not they have to pay Comcast.

I've seen some posts say that without free peering from Comcast it would, "erode Level 3's profitability on the Netflix business." Really? Based on what data? Because if you ask Level 3, and I have, they say they can make money on the business regardless of whether or not they have to pay Comcast simply due to the fact that Level 3 has a lower cost to distribute the content since they own the infrastructure. Not to mention, the fee that Comcast charges Level 3, which is based on each port Level 3 turns up, is not a lot of money. Level 3 is not balking about the price they have to pay, it's not a lot. They are arguing about the principle of what Comcast is doing. The rate Comcast is charging Level 3 today has no financial impact on Level 3's business and you don't see Level 3 or Comcast debating this.

The issue at hand is not about Level 3 getting something for free. It is about Level 3 using their network, which they made a huge investment in, to carry traffic further than anyone else. Any network connection is two way and a CDN connection is one way. So for some to compare Level 3's infrastructure costs to those CDNs who don't own the network is simply not an apples to apples comparison. For some to suggest that this is an "unfair" advantage that Level 3 has, they are wrong. It is an advantage, but it is not unfair, Level 3 spent a lot of money to build out the network. The idea that Level 3 should simply pay Comcast because that's what all of the other CDNs do is not a valid argument. Level 3 is not like the other CDNs.

Capex, space and power are cheaper for Level 3 and they run on a very predictable capex cost decline. Commodity servers, like the ones Level 3 users, as well as some other CDNs, decline at 20-30% per annum. The throughput is then what drives Level 3's cost of that capex on a unit basis. If Level 3 doubles the throughput of their servers they just halved that element of the cost. Since Level 3 has been very focused on a particular segment, broadcast and media and entertainment, the right sort of library and traffic actually improves Level 3's cost base the more they get, as in the case of Netflix. In parallel they do a lot of work on the software to improve the throughput with every code release they do. I'm not sure other CDNs think about this in as maniacal a way as Level 3 does. Maybe they do, but since Level 3 owns and operates a network, they have a different engineering culture.

Level 3 uses their own colocation facilities to house their own servers. They know exactly what the difference is between their cost and the price of colocation because Level 3 also sells it. It's a big gap and one that is increasing as colo rates have been going up steadily over the last few years. Yes some of the other CDNs have servers that are in "free" facilities within ISPs but not the ones that drive most of their traffic. I think we all know by now the fallacy of "thousands of locations" used to deliver large objects. Akamai, Limelight and Level 3's distribution architecture is very similar for that sort of traffic. And other CDNs are in third party colocation facilities, including Level 3's.

Last mile delivery is the same. Level 3 users their own network and they sell the same thing. While prices are not going up for this offering, Level 3 can still make a healthy margin and their cost is a lot less than what their competitors pay for IP bit delivery. And they all buy IP Transit, again some from Level 3. Here it gets a little trickier to access though because Level 3's competitors do peer and they do also get free delivery where they have persuaded an ISP to embed their servers in their network for free. But peering isn't free. You need a routing edge to deliver traffic to your peers. That's exactly the same as Level 3's, but at far less scale. No one even approaches Level 3's cost to deliver bits at the edge. For the free bits delivered inside an ISP network (without a router), you have to consider that it might be free, but Level 3 gets paid to deliver a portion of their bits. So even here free isn't low enough.

These are some of the advantages Level 3 has of network ownership. But it is also a very particular network ownership. One that owns colocation; one that operates at the greatest scale; one that sells IP transit to ISPs. There aren't too many of those. Even amongst the biggest IP networks Level 3 has the lowest cost.

As for the idea that Netflix is sacrificing quality for a cheap price, that's laughable. For those that don't know, Netflix is very particular about the quality of their suppliers and hands-down one of the most sophisticated content owners in the industry in how they determine that quality. Netflix has employees on staff called "Business Intelligence Architects" and employees in their product development group who build the analytic systems Netflix uses to measure performance and usage of their streaming service. Any Netflix member knows that Netflix routinely sends out emails asking customers to rate the quality of the video they just watched and Netflix has an incredible amount of data on what the real-world experience is for their customers at any given time.

Not to mention, Netflix now sees the delivery of video via streaming media as their core business and the DVD business second. That means in order for Netflix to grow their business, they have to focus on the quality of the video they are delivering. It's laughable to think that Netflix is going to purposely deliver a poor quality video experience and put their business at jeopardy just so they can save money. Yes, price is always an issue, but Netflix is balancing performance AND price together and they do not and cannot separate the two.

Some are also writing about the five to one ratio of the volume of traffic taking place between Comcast and Level 3 and using that as an excuse to suggest that it's unfair to Comcast. Comcast used the word "dump" on their blog saying that Level 3 was dumping traffic onto their network yet Level 3 is simply sending content to Comcast's subscribers who have requested it. Level 3 is not "dumping" anything. For those that think the five to one ratio of traffic is out of whack and not fair to Comcast, go read Rob Powell's post on TelecomRamblings.com which very clearly explains how when connecting to a last mile network on the internet, that ratio is the norm, not unique.

Some have suggested that Level 3 could just refuse to pay Comcast and buy transit from those who peer with Comcast but the way Comcast's network is setup, it keeps Level 3 from even having that as an option. Level 3 could go deeper into Comcast's network, but so far, Comcast is not looking at this as a solution, even though it would remove a lot of burden from Comcast, which is what they are complaining about. If the networks interconnect in 5 cities but there were ultimately more cities where the ISP has customers then obviously connecting in more cities is getting deeper. Comcast operates in 40 metro areas in the US for example and typically, interconnection in the US between large networks is done in less than ten cities.

Hot potato routing is used predominantly on the Internet (see here for details). What that means is that packets sent from one network to another will exit the first network as soon as they can. If network x is interconnected to network y in both LA and NYC and a packet needs to get from LA to NYC then it will be handed over in LA and carried right across the country by network y. In order for that not to happen (for network x to carry it further) you first need to connect in more cities.

Then the net receiver of traffic (the last mile network) needs to use elements of BGP like MEDs, communities and/or selective route announcements (see here for details). This means they announce details that the net sender can then use to carry the traffic further before handing it over. A CDN can do part of this but there is always a network connection from a CDN device and the consumer. So a combination of the two gives ultimate flexibility of who carries what packets how far. And of course CDNs are simply not used for all traffic exchanged by interconnecting networks.

If you cut through all of this, my opinion is that Comcast sells a competitive product to Netflix and is simply scared of over-the-top video. I think that's the real issue here with regards to Comcast and that they will do anything they can to try and combat it, including finding creative ways to try and get paid as many times as possible. But none of that has to do with how Level 3 operates their network and the cost advantage they have as a result of owning and operating it.

“Some have suggested that Level 3 could just refuse to pay Comcast and buy transit from those who peer with Comcast but the way Comcast’s network is setup, it keeps Level 3 from even having that as an option.”
This is the second time you’ve stated this, what exactly do you mean by this? Are you saying comcast is packet filtering other providers interconnects and blocking level3 ip space, making people who are multihomed out of level3 ip space unable to use comcast? AS-PATH filtering 3356 from other peers?
Or do you mean that, in order for say, NTT or global crossing to sell level3 comcast, *they’d* need to contact comcast and ask for 30 more 10G ports..at which point, those companies would have their peering re-evaluated by comcast?

Tariq Biziou

As far as most of us know, Comcast only buys from Tata. So Level3 could at best see Comcast via Tata (which anyone who has dealt with knows is not viable, congestion is rampant on the Tata/Comcast interconnects). The other option would be buying from a customer of Comcast.. but, it seems all the contracts anyone is willing to talk about point out that Comcast prohibits, in their contracts, customers from selling just Comcast routes.. And I doubt any customer is big enough to carry any reasonable portion of Level3’s traffic.

http://www.businessofvideo.com Dan Rayburn

Hey Matt, I’m afraid I am not allowed to go into further details on the technical side of how Comcast has their network setup.

Ignorant Investor

“I think we all know by now the fallacy of “thousands of locations” used to deliver large objects. Akamai, Limelight and Level 3’s distribution architecture is very similar for that sort of traffic.”
Unfortunately, we don’t all know this. Pls enlighten the masses as to why this is the case. Thx.

http://www.whatisyourvenue.com Aaron Booker

Yes, I’ve heard this from limelight but please do explain why many POPs aren’t better….

http://yahoo.com Grins

Unfortunately – it is not about owning the network… and peering is more of this than Dan is willing to admit. The peering relationships are driven from settlement and when they are out of whack – disputes like this happen. This is not all that un-common.
And if Level 3 wins this argument – which I hope Comcast concedes – this will push all networks to move from a settlement arrangement and all networks will now peer for free – which is what the outcome would be – and could be of value – but – where would the costs go for the equipment necessary for this arrangement to happen?

And why should Comcast help Level3 move deeper into their network for free? Wouldn’t everybody just ask for that? Why should comcast have to lose money on that?

Russ

Ultimately, is LVLT just blustering to improve its negotiating position prior to a private resolution with Comcast (perhaps with the FCC facilitation)?
Comcast seemed initially to have a superior negotiating position. However, when LVLT fans the flames of public sentiment for net-neutrality its position becomes vastly improved. It seems unrealistic that the FCC will implement N-N following the recent google/verizon deal and the Sprig 2010 federal court ruling noting the FCC’s very limited authority in this area without further legislation.
From a purely negotiating standpoint, it seems wise for LVLT to push for N-N.

Anon Poster

Why can’t Level 3 buy from another transit provider to send the imbalance like all their competitors that send the exact same traffic?

Stephen

To answer the CDN Architecture question,for many types of traffic, a less distributed architecture can produce performance that is indistinguishable from a highly distributed architecture, especially in the case of long form video. Once a video starts playing, as long as the CDN node is close enough to the end user that latency doesn’t affect throughput, a few less miliseconds of latency buys you nothing.
By concentrating network and server capacity in larger quantity at a smaller number of nodes, a less distributed but strategically placed architecture can even outperform a more distributed CDN because it’s less vulnerable to the saturation and “shedding” that can occur on smaller, more widely distributed CDNs. That’s why even a CDN that was massively distributed would “narrow” the number of nodes they distribute long-form video from to increase cache efficiency on the less popular content, and to avoid the unnecessary costs of replicating content at nodes that afford no customer-discernable performance increases.

Daily posts by Dan Rayburn about the online video industry, business trends & analysis, market data & research as well as the online video business models in the media & entertainment, broadcast, advertising & enterprise industries.