Updated: A Play by Play on the Comcast & Level 3 Spat

Updated: A fight between Comcast, the nation’s largest ISP, and Level 3 (s lvlt), the nation’s largest broadband backbone provider, is portrayed as a simple commercial disagreement over how much traffic Level 3 plans to send Comcast (s cmcsa) after Level 3 signed an agreement to act as the content delivery network for Netflix. But many believe it has the potential to change the way the web works. Over the next few hours and days, many people will weigh in on the issue and many new insights will be shared. We’ll document them here so our readers can see the story as it emerges, so check back for updates and insights as they hit our inboxes.

Nov. 29:

Level 3 issues a statement claiming Comcast is charging it for delivering movies and video content to Comcast subscribers, a fee that Level 3 likened to placing a toll bridge on the Internet. Comcast later came out and said that Level 3 was trying to take advantage of its peering agreement with Comcast to undercut other CDNs and that the dispute was essentially a commercial one. Netflix and the Federal Communications Commission are silent on the issue.

Advertisement

Nov. 30:

Commentary on the web accelerates with some accepting the commercial disagreement argument, and others seeing in the move a chance to implement a double-sided revenue model for ISPs that would allow ISPs to collect revenue not only from the end consumer of broadband, but also the folks trying to send the traffic over the ISP’s pipes. Meanwhile, Netflix is still silent, but it likely didn’t see this fight coming.

In a press conference the FCC Chairman Julius Genachowski declined to comment on the issue but said, “The staff is looking into it.”

Level 3 Responds to Comcast’s statement with a new statement that scoffs at the idea of this as a peering dispute and portrays it again as Comcast being anticompetitive. The statement reads in part:

The fundamental issue is not whether Comcast sends more traffic to Level 3 or whether Level 3 sends more traffic to Comcast. Both Level 3 and Comcast are responding to the requests of Comcast’s subscribers, who want to be free to see and use the full suite of content and applications that are available on the Internet today and in the future. Level 3 wants to assure that freedom is preserved.

”Instead, the fundamental issue is whether Comcast, as the largest cable company in the country with absolute control over access to its cable TV and broadband access subscribers, has the right to unilaterally set a ‘price’ for that access that effectively discriminates against competitors of Comcast’s cable and Xfinity content.

Update 1: Comcast tries to help its position by posting a video on how peering works. It’s also filed a letter with the FCC defending its position that detailed its past relationship and proposed new relationship with Level 3. According to Comcast, prior to this dispute Comcast provided Level 3 with 27 interconnection ports under its peering agreement. Then, after Level 3 signed its agreement with Netflix, Level 3 asked for more. From the letter (PDF):

The parties’ current interconnection facilities[3] could not begin to support that type of traffic flow. As a result, Level 3 approached Comcast approximately two weeks ago (shortly after reaching their Netflix agreement, we later learned, although they made no mention of it at the time) and demanded 27 to 30 new interconnection ports, which would allow them to send a much greater amount of traffic onto Comcast’s network.[4] To that request, Level 3 added the following twist: it insisted that Comcast should provide it with all those new facilities – and support this vast new influx of traffic – for free.[5]

Comcast was able to scramble and provide Level 3 with six ports (at no charge) that were, by chance, available and not budgeted and forecasted for Comcast’s wholesale commercial customers. The amount of traffic that would be exchanged over these six ports (and the parties’ preexisting ports) was just barely within the generally accepted bounds of a roughly balanced traffic exchange between the parties. Level 3 accepted those ports, and they are using them right now to send traffic to Comcast’s network.

Bravado aside, Comcast is responding to the FCC’s request for information so it’s answers should be taken seriously. Comcast also brought up the point that Netflix also has a deal to deliver its content though Limelight, so even if Comcast blocks Level 3, Comcast’s subscribers will still get Netflix content from Limelight. Comcast ends the letter pointing out that this is a commercial dispute as opposed to one requiring regulatory involvement.

The National Cable and Telecommunications Association weighs in with its take reinforcing the idea that this is a commercial dispute and something unrelated to network neutrality. And for the deeply nerdy, and researchers thinking about peering and the web, there’s this article from the New York Times earlier this year on the topic of public peering stations as opposed to direct peering arrangements to provide some background on the general topic of peering. Meanwhile, Ars Technica interviews a lot of smart people and comes up with a similar conclusion to mine (although without such a dramatic headline). This is a commercial dispute, but it’s also suspiciously like the beginning of a new way for ISPs to make money off of the content flowing through their pipes.

Updated 2: Dec. 3

Level 3 took Comcast’s video post on peering and fired back with an FAQ of things people should know about the dispute, including the lack of competition in last mile services and how that affects the debate between Level 3 the cable ISP. What’s important to note here is that Level 3 is seeking to change this into a debate about interconnection as opposed to one over peering, with the idea that peering agreements are commercial in nature, while the ability to interconnect lies at the heart of how the Internet works. From the FAQ:

A: No. The dispute between Level 3 and Comcast is not a peering dispute, which relates to connection of Internet backbone networks. At issue is a fundamental interconnection disagreement between Comcast, as a provider of local high speed Internet access to consumers who pay Comcast for access to content, and Level 3, which delivers content to residential broadband access providers like Comcast in response to consumer requests. Unlike “peering” in the Internet backbone, where competition abounds and prices have been declining steadily, Internet carriers that have content requested by Comcast subscribers have no choice but to exchange traffic with Comcast. Comcast is using this dominant position to demand payment for traffic delivered at its customers’ requests. You simply cannot “route around” Comcast to provide requested content to Comcast’s subscribers.

Level 3 in its FAQ is arguing what I argued earlier– namely that Comcast is abusing it’s position as a last mile provider to may Level 3 pay Comcast money to deliver content on behalf of Level 3’s customers. Unfortunately, without more information, such as how much Level 3 is paying to Comcast or how much less Akamai (s akam) may be paying to Comcast after losing the Netflix account, it’s hard to get to the actual truth behind this battle of rhetoric. Meanwhile, Comcast, offered a much shorter reply that basically says Level 3 is still wrong and Comcast won’t cut off people’s Netflix. For more on this, check out a Daniel Golding’s guest post for us that lays out some of the issues involved in a peering dispute.

Update 3: Dec. 7
And the sniping continues with Comcast posting its own FAQ’s in response to those LEvel 3 issued on Friday, basically continuing it’s line of reasoning that Level 3 is gaming the system by calling itself a CDN for customers and then trying to arrange for peering with Comcast. The format is exhausting, with Comcast quoting Level 3’s questions and then providing a refutation. However, in this type of he-said-she-said debate it’s the things we don’t yet know that will likely decide this issue, as opposed to either side’s spin.

Check back for more.

Edit Note: This post was originally published on Nov. 30 at 12:40, and has been periodically updated since then with new information. Related GigaOM Pro Content (sub req’d):

I don’t disagree with the idea that Comcast want to earn more money from their infrastructure. They are a private company and they have a right to do that. There was a poll about this earlier, http://my-take.com/poll/Should-comcast-be-allowed-to-charge-websites-for-web-traffic The internet isn’t free and there are a lot of physical costs to make it work. Considering how popular streaming is and how much bandwidth it takes up, Level 3 should have expected to pay higher fees. Comcast has a right to single out Netflix because their site now accounts for a large percentage of the web traffic. This is not a Net Neutrality issue. It would be if a Net Neutrality case Comcast buys NBC which owns part of Hulu and decides to charge Netflix unequally. That is a conflict of interest I would fight against.

It is easy to cast the blame against Comcast because all of the negative press about horrible service. Level 3 isn’t that much of a victim either. We are all the losers because one segment of us will end up paying for an unfair system and bad service anyway.

In order to watch a YouTube video, you must pay your monthly fee for (faux)unlimited internet access, and YouTube must pay by the bit. Both sides pay; that is normal, and is what many are calling “double dipping.” The only people exempt from this are backbones due to peering agreements, but these agreements don’t extend CDNs (nor does it make sense to unless every backbone is also a CDN).

When you’ve got L3 playing both identities, they’ll naturally try to pass on the benefits of being a backbone to the more profitable and growing CDN business and that is what Comcast is crying foul about. Don’t think this has anything to do with net neutrality…

Brian,
I can only tell you how we as small providers pay for bandwidth, and it is by consumption. We pay, for every byte, in some cases we have mutual peering agreements but it still costs us to transport it out to the sites, and to the consumer. Until there is a REAL value applied to the costs of bandwidth, the market won’t change, they will never have a competitor.

We are small and service areas the big guys won’t, bringing bandwidth to areas that don’t have any, and yet we are always compared to the big guys. Consumers think bandwidth is free and unlimited, and until they get over it everyone loses.

Level 3 charges us for bandwidth (we have a cable system – 200 customers and are an ISP), under this analogy shouldn’t that be free to us, because we are giving it to their customers and transporting on our network for them? So lets see we pay level 3, you pay level 3 so talk about double dippping. Where is my free bandwidth! The quantity of available bandwidth we have is extremely expensive and limited. The cable system currently pays $3600/mo for 6Mb of data (4 T1) and there is no way to save more, it’s one telco, and a linear scale for the T1’s and no DS-3 availability, so who is going to pay for this waste of bandwidth? The consumer.

Right. The consumer will pay for this – and so will the smaller level ISPs – which you’ve brought up. I understand your “bandwidth isn’t free” stance, but you have to also see the obvious monopoly complications in this.

The key difference here is that your little ISP is not in a peering relationship with Level3, but a transit one. You aren’t just exchanging your own traffic with Level3, but asking Level3 to transport that traffic between you and ISPs in Singapore, Australia, Japan, Poland…

If you had enough traffic, Level3 would allow you ‘free peering’, where you just exchange traffic between your network and their own … but you’d have to provide your own route to every other ISP out there. That’s the bit you have to pay for: not getting to and from Level3’s own network, but using them to connect you to the rest of the planet. If you have enough traffic to justify it, you can get ‘free peering’ where you just exchange traffic with another ISP as equals without payment; or you can get partial transit (within a particular country only, for example) – or get full transit, where the other company will, for a fee, get your packets to or from anywhere from Azerbaijan to Zimbabwe as needed.

You are paying Level3 to get your packets around the world for you; Level3 aren’t paying Comcast, because Level3 transport their own packets to everywhere other than Comcast’s own network. There’s a huge difference in cost involved.

Comcast should be embarrassed by going on the record stating that their capacity is so limited so as to only provide six peering ports. Appearances of false generosity in disputes such as these reek of guilt.

This is all about the cost of bandwidth and distribution, not about net neutrality. I support net neutrality, especially with diversified behemoths like Comcast, Microsoft, and Google.

Do you all have water bills? Notice how you pay for quantity used? Why? Because it costs money to distribute that water. Guess what, Internet distribution costs money too, and the farther you are from core areas, the more it costs. So lets look at the reality here, this is wasteful use of bandwidth, for a “perceived” value. Until all providers start charging for bandwidth, consumers will continue to believe it is free.

BANDWIDTH IS NOT FREE, and this only further penalizes small communities with childish nonsense.

When you pay Comcast for the internet connection you pay for the bandwidth used. This is why they put 2 limits on you. 1. Max transfer per month (250GB per month) 2. Max transfer speed (10Mb per second)

Now on top of paying for both a max limit per month and a max limit per second they also want you to pay for content delivered to you.

This is plain and simple double dipping, and is designed to make sure that Comcast has a unfair advantage with their own video streaming services which will not be double dipped against.

Level 3 undercut its real competitors bids to NetFlix based on assumption that they could keep providing CDN service as access prices. Now Comcast calls them on it and will not provide CDN interconnect for free.

Sounds to me like the old Telephone interconnect games, and ill result in regulation of prices.

You’re on crack, Ray. When you pay a month by month bill that’s the SAME COST each month for internet bandwidth, you’re being alotted either unlimited, or in the case of Comcast, a specific amount of data transfer. Your likening to water is illogical because it makes the supposition that you would pay a flat fee per month for water service, which you do not; you pay based on your overall usage of that water. If internet service were the same, you would pay a bill at the end of the month that outlined what you’d used. This would also mean that when you signed up, you would have been told you’d be billed like this. This is also an incredibly inane way to handle how traffic on the internet is handled. Unlike water, the average person has NO idea how to track how much data they’ve downloaded over the internet in a month. Even if they did, charging people in this way is an entirely unfair and improper way of handling the internet. Considering the number of internet connecting devices available in this day and age, what you’re suggesting is that someone should be ok with there being a leaky faucet that they can’t see in the their house, and that they should be charged for that faucet. The average person has no idea how many apps connect to the Net and use it for data, nor do they have much control over it. To say that they should is simply not going to work.

Comcast isn’t trying to do this because of Bandwidth costs. They’re trying to do this because they see the possibility to make a lot more money by blocking a popular web service and dressing it up as, “Oh, poor us, our capacity is going to expand!”. It already did, jackasses. That’s how the internet works.

Cry me a river. Comcast needs to get of this, change their terms of service, or get out. I’m glad I don’t give these morons any of my money.