Net neutrality? Comcast says that its peering dispute with Level 3 is just …

Share this story

Level 3's inflammatory Monday afternoon press release, in which it basically accused Comcast of trying to whack Netflix streaming traffic and flout net neutrality principles, certainly started a fire. Within hours of issuing the press release, the head of the FCC's Wireline Competition Bureau was on the phone with Comcast, giving the company the business end of her Question Stick. What exactly was going on here, she wanted to know.

So Comcast told her, and in a follow-up letter (PDF), made its answers public for the world to read. That's because, in Comcast's view, the entire dispute is “nothing but good old-fashioned commercial peering dispute" and Level 3 is being "entirely disingenuous."

The entire blowup was extraordinary, because Level 3 has been in commercial arrangements with Comcast for years. It apparently decided to risk whatever goodwill it had built up on its attempt to drive a hard bargain that would put Level 3 in a better position than content delivery network competitors like Akamai and Limelight.

A rash bid

Why take the risk? In Comcast's view, Level 3 may be in over its head with its new agreement to deliver all of the Netflix streaming video traffic. The argument is that Level 3 won the Netflix contract away from other CDN providers by providing a surprisingly low bid, one that it now has to back up with a good deal from the major US Internet providers.

“Level 3 has low-balled its way into a new business deal that will significantly increase the amount of Level 3's traffic Comcast would carry,” says the cable giant, “and suddenly wants to seriously disrupt settled economics of Internet traffic to meet its new business plan. Its position is not based on any principles of fair play on the Internet, but instead is merely the result of its rash bid to carry Netflix traffic at radically low rates, based on the flawed assumption that it could use its Tier 1 Internet backbone status to cram its CDN traffic onto others' networks on a settlement-free basis.”

Indeed, because Level 3 is such a major Internet backbone provider and has historically run only a smallish CDN operation, Comcast has actually been the one paying Level 3 for certain interconnections in the past.

But the Netflix deal alone is so massive that Level 3 has suddenly become a major CDN player. Comcast says that it was approached by Level 3 two weeks ago, just after Level 3 signed the Netflix deal, and was asked for 27 to 30 new interconnection ports using the two companies' existing interconnection agreement (that is, Level 3 would pay nothing more for adding up to 30 direct 10GigE connections to Comcast's network).

Comcast “was able to scramble and provide Level 3 with six ports (at no charge) that were, by chance, available and not budgeted and forecasted for Comcast's wholesale commercial customers.” After providing these six additional ports, Comcast concluded that the existing settlement-free peering agreement with Level 3 was still (barely) valid, but if Level 3 really wanted another 21 to 24 ports, this was simply too much traffic. Level 3 would have to pay for those ports like any commercial paid peering customer.

Comcast also notes that nothing about any of these new ports was contingent on the type of traffic coming across it. That is, the company was concerned only about traffic volumes, not content.

Comcast's peering and transit policies are available on the company's website, and those policies make clear that settlement-free peering is only available to a company that can “maintain a traffic scale between its network and Comcast that enables a general balance of inbound versus outbound traffic.”

163 Reader Comments

Some people have used this to brand Comcast as some kind of evil entity.

I look at this and to me it looks more like the fox and the weasel are having a fight.

The previous article revealed a lot about the inner workings of traffic and routing of the Internet that I previously didn't know, so I think this is not really about consumers at all. Painting Comcast as the bad guy doesn't seem the be the full story here - from the looks of it it seems that Level 3 is trying to scrape together an intensely great deal at Comcast's expense, which of course has Comcast saying, "Wait a minute here".

It would be interesting to learn more about the exact costs that are required on the backend here - for example there's talks of ports. How much do they usually cost? What costs them money? An in-depth article about the economics of Internet infrastructure might be something I'd love to see on Ars.

The story fails to mention that the only reason Level 3 is sending Comcast this traffic is because Comcast's customers requested it. Their entire argument hinges on utilizing each other's networks to transfer bits for shared gain, it falls to pieces when you realize Level 3 is only serving what Comcast requested.

Given their peering agreements, I wonder who can then possible peer settlement free with Comcast. Seeing as they have almost no commercial hosting and are almost entirely made up of end user subscribers, their traffic must be consistently only into their network.

If no one can qualify for settlement free peering, then their policy is a sham.

If Comcast customers are requesting data from Level 3 IP addresses, then it would seem to be in Comcast's best interests to accomodate such requests. Unless of course they don't care about satisfying their customer's requests.

L3 probably has to press the issue because of some commitments they made to NetFlix. If so, then they really can't cry about it if Comcast wants to charge them for peering.

The bottom line is that you don't traditionally pay for peering unless you just want to move your traffic through the other party's network. Since the endpoint here will be Comcast, they are trying to change the way things work, and not for the better.

"In other news, Level 3 has announced that they're forming 10 subsidiary groups, each of which will handle a tithe of their duties to contractors. L3's customers are delighted by the news, Comcast is fuming as each individual subsidiary's bandwidth doesn't reach the level of qualifying for additional fees. Rumors that these might attempt to file as non-profit charities still swirl."

If Comcast customers are requesting data from Level 3 IP addresses, then it would seem to be in Comcast's best interests to accomodate such requests. Unless of course they don't care about satisfying their customer's requests.

The story fails to mention that the only reason Level 3 is sending Comcast this traffic is because Comcast's customers requested it. Their entire argument hinges on utilizing each other's networks to transfer bits for shared gain, it falls to pieces when you realize Level 3 is only serving what Comcast requested.

This was my understanding as well. My basic understanding was that peering agreements dealt with "I want to send this traffic through through your network instead of using another possible network", not "I'm sending this traffic to your network because your network requested it".

I can understand that comcast wants help with expanding its network because these costs are the direct result of Layer 3, but they should be saying "We've expanded as much as we can, buy us more equipment if you want it faster.", while instead they're saying "We're changing our arrangement, you need to pay us to send any traffic to us."

What it sounds like to me is that Level 3 is wanting to play without paying while being paid to play. I think that is the extent of it.

No, I am unfortunately a Comcast subscriber and I subscribe to Netflix and do all my television viewing that way. I do not have cable, so I am the one asking for that data to come to me. Level 3 isn't just moving it across Comcast's network for just the sake of doing so and Comcast would prefer me to subscribe to their services instead of using a more customizable service at less than a 1/3 of what Comcast charges. So their point is Comcast is attempting to double dip on fees... I pay Comcast for that data already. ;-)

this is hilarious. they are pointing to their peering agreement, when this is (I believe) NOT a peering issue. Comcast is, for the most part, simply providing the last-mile for content delivered FROM Level3. If they can prove that L3 is using their (Comcast's) networks to peer while delivering content to OTHER subscribers (i.e. those whose ISP is not Comcast), then they have the right to charge for peering. I doubt that they can prove this, since the distribution deal hasn't started yet, and L3 hasn't ACTUALLY started delivering Netflix movies to people's TVs/computers/Xboxes/PS3s etc.

Some people have used this to brand Comcast as some kind of evil entity.

I look at this and to me it looks more like the fox and the weasel are having a fight.

The previous article revealed a lot about the inner workings of traffic and routing of the Internet that I previously didn't know, so I think this is not really about consumers at all. Painting Comcast as the bad guy doesn't seem the be the full story here - from the looks of it it seems that Level 3 is trying to scrape together an intensely great deal at Comcast's expense, which of course has Comcast saying, "Wait a minute here".

It would be interesting to learn more about the exact costs that are required on the backend here - for example there's talks of ports. How much do they usually cost? What costs them money? An in-depth article about the economics of Internet infrastructure might be something I'd love to see on Ars.

This! Well put.

I hate Comcast as much as the next guy, and I think that Level 3 was banking on the US' general hatred of them in order to try and win this out. It hurts for me to say this, but I don't think Comcast was in the wrong, here.

I'm increasingly getting the feeling that Level 3 should've had previous discussions about the possibility of becoming a CDN. Though it may have been difficult for them to anticipate issues with Comcast - if Comcast were only an ISP, it sounds like there wouldn't be a problem.

When Netflix suddenly starts directing all their customers to Level3, won't Comcast suddenly have a bunch of free ports that were previously carrying Netflix traffic and are now idle? I don't see why they need to expand at all, just shift resources around. This should be an interesting story to follow.

So to all of those who think they know how the internet works...Comcast has become a slow moving corporate giant who is trying to own most of the content (note the pending NBC acquisition) it will provide for its customers. Comcast execs are probably not very happy that they were not able to hook up directly with Netflix and now have to deal with another providers network for content delivery. Basically L3 moved quicker, low-balled a bid (a separate argument entirely) and secured its ownership of the Netflix content which will in turn drive more end ISP's of the smaller DSL and Cable ISP's to become customers of L3. Sound like a good business plan? I think so...cut a cheap deal with a major content provider and then draw end users to their network with said content. That's business folks, deal with it.

That said, Comcast has built out a nationwide network to support its customer base and in doing that had a longer term plan which has been circumvented. It will happen again, and I am not sure the NBC deal should be allowed to be completed as it will make the NBC content available solely via Comcast, which is exactly what they are complaining about now, pot calling kettle black maybe?

Although Comcast has generally proved themselves to be a dickish company, I really don't think they're in the wrong here. If you read the towerpost that was in the comments on the article yesterday, it looks like there is a good chance that this is a peering issue, and Level 3 was counting on the shitstorm hurting Comcast more than themselves.

Following Comcast's logic in charging L3 for incoming traffic requested by their customers, said customers would be justified in asking payment from Comcast for every bit they send their way.Does that make sense? No. Neither does Comcast's position, even though they are already practicing it with Akamai. That's the moral point.

Let us take a look at it from an other angle: the business point. Every bit that gets transferred from L3's network to Comcast's, or the other way, transits on both networks at some point. L3 would have has much justification in charging Comcast for each of those bits than Comcast would have in charging L3 (they are both spending money carrying those bits).

Now, Comcast is riled at loosing traffic from Akamai (that Akamai is paying for) to L3 (that L3 is not paying for), L3 is riled at having to pay for traffic that it used to get for free. So... who gets to pay in the end? Whoever can least deal with loosing that traffic.

I don't see how it makes a difference that Level 3 has become a CDN. If the streams were coming from Netflix servers, they'd still need to go through Level 3, or a company like them. Would it have made a difference if Netflix contracted with another peer that was two hops away, but still made the traffic go through Level 3 as opposed to their other peers?

- NetFlix was going through Akamai to Comcast customers- Akamai was paying Comcast because it is a CDN- NetFlix is now going through Level 3- Level 3 is not paying Comcast because it is not a CDN

I don't think Comcast should get paid for delivering traffic requested by its end customers, whether it's a Wikipedia article or a competing video stream. This begs the question: What's special about a CDN? Why was Akamai paying Comcast anything?

The story fails to mention that the only reason Level 3 is sending Comcast this traffic is because Comcast's customers requested it. Their entire argument hinges on utilizing each other's networks to transfer bits for shared gain, it falls to pieces when you realize Level 3 is only serving what Comcast requested.

And Comcast received approximately $8 billion from us customers for delivering those bits. That is just their Internet revenue and it is their most profitable business. Level 3 only has $4B in revenue.

Comcast is not a backbone and their traffic is never going to be symmetrical. It is Comcast's customers asking for the asymmetrical traffic so this is Comcast's problem. If they don't like it they can tell their customers to generate symmetrical loads.

Generally no-cost peers exchange only non-transit routes with each other. Non-transit routes are routes directly owned by the peering parties and their direct attached customers, but not routes learned from other ISPs or transit providers. Therefore any traffic sent to Comcast is destined for Comcast or one of its directly attached customers. Level3 would not send traffic through Comcast that is destined for a Verizon customer for example.

Peers also agree to traffic terms. For a no-cost peering relationship, the traffic exchange has to be mutually beneficial to both parties. Depending on their business plans, this could mean many different things. A simple peering relationship could be that they send each other an approximately equal amount of traffic. When one peer starts sending more, the other will often start asking for payment to deliver it since the costs have just gone up.

I do not know the details of the Level3/Comcast peering relationship to comment whether one entity is entitled to payment or not.

A 10Gbps ethernet port can be very expensive, depending on the equipment. However, this cost is usually a drop in the bucket for what it costs to build and deploy circuits for a large-scale backbone. That being said, it is not trivial if you have not budgeted for it. No business likes unexpected expenses.

- NetFlix was going through Akamai to Comcast customers- Akamai was paying Comcast because it is a CDN- NetFlix is now going through Level 3- Level 3 is not paying Comcast because it is not a CDN

I don't think Comcast should get paid for delivering traffic requested by its end customers, whether it's a Wikipedia article or a competing video stream. This begs the question: What's special about a CDN? Why was Akamai paying Comcast anything?

Exactly!

If this not a peering dispute, is it really common practice for CDN's to pay ISP's? Wouldn't that be redundant to what the consumer pays already for his last mile connection?

and those policies make clear that settlement-free peering is only available to a company that can “maintain a traffic scale between its network and Comcast that enables a general balance of inbound versus outbound traffic.”

How can you expect to have balanced traffic ratios, when Comcast is primarily an eyes network? Typical web browsing is probably in the area of 5:1 downbound, and possibly as high as 10:1. This seems a specious position at best.

Is Comcast worried about Level 3's traffic going over Comcast's network to other ISPs? Or is Comcast worried about Level 3's traffic going over Comcast's network to Comcast ISP customers?

If the former, then I have total sympathy for Comcast.If the latter (e.g. Comcast customers accessing Netflix), then Comcast should stuff it since it's already being paid by households to deliver data to said households. The origin of the traffic shouldn't matter

People don't seem to get that Level3 is getting paid to host and pass on this data.Netflix pays Level3 to transfer the data across their part of the network. That's who is paying for data. This data then gets put onto Comcasts network. Yes, the Comcast user has requested it, but they requested it from Netflix.The user pays Comcast to get the data from its provider to their computer.Netflix pays Level3 to get the data from the host server to the users network (in this case Comcast).

Why should Comcast soak up the costs of this data data going across Level3's network towards it? Netflix is already paying for that.

The amount of data being sent by Level3 to Comcast is greater than the amount of data being sent by Comcast to Level3. Level3 is getting paid to transfer this data, Comcast is getting paid to move it further along.

Comcast has nothing to do with what or how much the user requests, they are responsible for delivering it last mile (and a bit more). The host of the data is paying Level3 to get it to Comcast. It's not like Level3 are sending all this data to people for free. Why should Comcast be paying them to get the data, when they are being paid to get the data by the people providing the data?

Bullshit. Comcast won't even bend over backwards to help their subscribing customers. Much less a potential competitor. I've received nothing but substandard service from them since I've lived in Philadelphia. Worst company ever.

I don't think Comcast should get paid for delivering traffic requested by its end customers, whether it's a Wikipedia article or a competing video stream. This begs the question: What's special about a CDN? Why was Akamai paying Comcast anything?

Because Netflix was paying Akamai to use their servers with fat pipes linked to edge nodes (that is, residential customers like you and me). Many of those customers have Comcast as an ISP, so Akamai paid Comcast for exclusive high-speed connections between its servers and Comcast's. Then Akamai could show Netflix that their servers could deliver a lot of content quickly to Netflix's subscribers, in an attempt to stop Netflix from switching CDNs.

A CDN's business model is to place servers at strategic locations in order to push a lot of data to edge nodes, and then sell their bandwidth to companies trying to reach those edge nodes. To do that, it makes sense to pay whomever provides the edge nodes with internet access for direct connections between your servers and theirs. This is because the residential customers are much less likely to switch ISPs based on CDN access than companies are likely to switch CDNs based on their access to ISPs, unless the ISP is very small and/or in a competitive market (which rarely is the case for Comcast).

Hence, the ISP has little incentive to peer with the CDN, while the CDN has a lot of incentive to peer with the ISP, so they typically end up paying money to the ISP.

I see some people siding with Comcast, which is surprising in and of itself. There are certainly some points in favor of Comcast here, but overall, they are the ones who are making the situation untenable. Let me explain why:

All traffic which much be passed to Comcast must also go over L3 lines in order to get there. If that is the case, L3 has just as much right to charge Comcast for the added bandwidth requests coming FROM Comcast customers as Comcast has to charge L3 for sending the data. The data flow is one way, but it is by request, not "forced". Were L3 sending data through Comcast lines to get to 3rd party customers, the demand for constant compensation would be valid. As it is direct L3 to Comcast data, there is little argument.

Charter is blocking ALL communications from L3 for this. If they don't want to install 21 to 24 new connections, that is understandable. It has a real-world cost for them to do so. However, they are not saying "we won't add connections unless you pay us". They are saying "since you won't pay us, we're entirely blocking your data". So rather than simply creating a bottleneck of data from L3, they are effectively using all Netflix customers (and potential customers) as bagaining chips, sayin it's all-or-nothing.

I personaly think L3 should pay for the connections, by which I mean they should be expected to cover the cost of adding these connections and offering to compensate for any work that must be done in order to get these connections working. That's fair. What Comcast is asking for, however, is a recurring fee for providing their own customers the content they want. L3 has had to expand their network greatly in order to fascilitate the increased amount of data they provide. It would not be reasonable for them to charge Comcast a monthly fee to deal with the amount of data they have to send to Comcast-- they are being paid to provide a service, it is their cost. The same is true of Comcast-- they are being paid huge sums of money to provide internet access-- including Netflix content. If they have to upgrade their infrastructure to deal with the load, that is their responsibility as part of the service agreement with their customers.

Let's hypothesize and say to set up all these extra connections it costs $5 million. I have no problem saying that L3 should cover at least half of that, if not all of that. But to expect them to pay (again, this is an example) $1 million per quarter, forever, to provide data that Comcast customers want is silly. Again, L3 has to pay for THEIR infrastructure and bandwidth to meet the demands of THEIR customers. It is silly to say that they should have to pay for COMCAST's expenses to meet the demands of COMCAST customers.

People don't seem to get that Level3 is getting paid to host and pass on this data.Netflix pays Level3 to transfer the data across their part of the network. That's who is paying for data. This data then gets put onto Comcasts network. Yes, the Comcast user has requested it, but they requested it from Netflix.The user pays Comcast to get the data from its provider to their computer.Netflix pays Level3 to get the data from the host server to the users network (in this case Comcast).

Why should Comcast pay to get the data across Level3's network? Netflix is already doing that.

The amount of data being sent by Level3 to Comcast is greater than the amount of data being sent by Comcast to Level3. Level3 is getting paid to transfer this data, Comcast is getting paid to move it further along.

Comcast has nothing to do with what or how much the user requests, they are responsible for delivering it last mile (and a bit more). The host of the data is paying Level3 to get it to Comcast. It's not like Level3 are sending all this data to people for free. Why should Comcast be paying them to get the data, when they are being paid to get the data by the people providing the data?

Huh? No-one is suggesting Comcast pay Level3. Like you said, Netflix pays Level3 to deliver the content to Comcast's network. Comcast's customers pay Comcast to complete the delivery across Comcast's network. Comcast is saying Level3 should also pay them to complete the delivery.

It's akin to a mailroom in an apartment complex charging fedex $20 for every package they deliver.

I don't understand this. In a normal CDN/ISP peering relationship (ignoring for a second that L3 is both), wouldn't it work like this:

Code:

Backbone *----* Backbone | | Comcast * * CDN |Home User *

Home user pays Comcast, Comcast pays Backbone, CDN pays Backbone (same or different). If they choose to peer, then both Comcast and the CDN would no longer have to pay money to their backbone providers but there would be additional costs of operating the peering connection and equipment. If one company is maintaining that connection, then it would make sense to ask the other to pay for the peering agreement to make up for it. But that has nothing to do with asymmetric traffic. That argument applies to peering between backbone networks, but doesn't make sense here.

Comcast is actually asking L3 to pay to get the data across Comcast's last mile.

If it is L3 that is paying to get the data across Comcast's last mile, then what are Comcast subscribers paying for?

Exactly.

There are non-trivial costs associated with adding 21-24 new connections, there is no doubt about that. The responsibility for the cost of adding them, however, falls on both parties. Comcast is being paid to provide data to their customers, L3 is being paid to send that data. If getting the data from L3 to Comcast requires more connections, both parties should be expected to pay up to improve the connections. Comcast is saying "you have to constatly pay us, or not only will we not improve connections, we will cut off the existing ones". This is why people are upset with Comcast.

Let's say Comcast and L3 are two countries that trade between each other. The United States of L3topia makes money by selling oil. One day they find a huge new oil field and want to sell more oil. The people in the Kingdom of Comcastia want this oil-- and the more there is, the more people will want to use it, which will mean more tax revenue for the Kingdom of Comcastia. Thus, it is financially benefitial to both countries to maximize the trade between them by increasing the amount of roadways between them. It is reasonable, therefore, that they split the cost of building bridges between them. However, the King of Comcastia is saying "you will pay us money every month, or we will allow no more oil shipments, even the ones we used to allow". Is this fair? No. The people of Comcastia want the oil, they pay for the Kingdom to open trading lanes with everyone. It is their right to have that oil. But the King of Comcastia is saying that the tax payers no longer have rights to the oil, even though the had it before.

Basically, Comcastia sucks as a nation, and you shouldn't live there if you have a choice.