Level 3 outbid Akamai on Netflix by reselling stolen bandwidth

Level 3 communications raised a lot of eyebrows earlier this month when it outbid rival Content Delivery Network (CDN) provider Akamai to deliver Netflix content to large parts of the United States. The announcement was a huge win for Level 3′s CDN business and it meant leasing around 2.9 terabits of distributed capacity to Netflix. But there’s just one little problem: Level 3 won that bid because it intends to break its contractual obligations on peering with Comcast and essentially resell stolen bandwidth to Netflix. Now it makes perfect sense how Level 3 managed to outbid Akamai since no CDN provider operating legally could outbid hot goods.

So how does Level 3 intend to get away with this? CNET and other news sources reported that Level 3 is trying to put it under the “Net Neutrality” violation banner and invent a story that Comcast is blocking content when it merely wants to enforce existing contractual agreements with Level 3. Then almost predicably, we get groups like Media Access Project piling on Comcast and bloggers like Stacy Higginbotham claiming that “Comcast might break the web” despite the fact that the kind of contractual agreement between Comcast and Level 3 is typical of how the Internet has always worked. On the contrary, allowing Level 3 to violate their peering agreement under the banner of “Net Neutrality” is what would actually break the Internet and turn peering and Internet investment economics on its head.

What makes the Internet work

The Internet is comprised of privately built and privately operated networks that connect on a contractual basis. These contracts are mutually agreed upon and there are generally two types of contractual agreements.

Transit agreements

Peering agreements

A transit agreements is when a network carrier delivers data between two other networks, and both end point networks have to pay the central carrier to carry that traffic at a metered rate. So if Comcast and Level 3 communications had no direct method of connecting to each other, they must each pay a third party network carrier to connect their two networks. Transit agreements are usually more expensive than peering agreements for every packet delivered and they’re generally slower because of distance and congestion. According to DrPeering.net, typical Internet transit rates are $3 to $10 per Megabit per second (Mbps) per month for server bandwidth.

Peering agreements occur when two networks have a direct physical connection to each other. A physical connection exists when it is financially feasible and when there is a business agreement in place. Physical connections are often practical to construct since many end point networks meet at Internet Exchange Points within the same physical building and that merely involves connecting a few in-building Ethernet cables on some gigabit or multi-gigabit switched ports. The business agreement requires that both parties come to mutually agreed upon terms on a financial settlement scheme.

The settlement scheme can involve a fee based agreement or it might involve no money changing hands. When one network sends more traffic to another network than it receives, it usually has to pay a metered fee on the extra traffic that it sends. When the two networks are more or less sending the same amounts of traffic to each other, the two networks can agree on settlement free terms where no money changes hands. These terms of settlement might sound arbitrary and unfair to outside observers that smacks of “might makes right”, but it is based on sound economics that ensures continued investment in the Internet.

The network sending more traffic does so because it has cheaper-to-serve customers than the network receiving more traffic. The network receives more traffic because it has more “eyeballs” i.e., broadband consumers who are hundreds of times more expensive to connect than the sending network’s customers which are the Netflix and YouTubes of the world. Providing a gigabit per second (Gbps) of capacity to a server in a data center might cost a few hundred dollars of capital expenditures but an additional gigabit of last-mile broadband capacity could cost millions in capital expenditures. Another example of one network that might charge another network to peer is when one network operates more long distance lines which are more expensive to build and maintain, especially those going under the oceans connecting continents.

A network that didn’t build the expensive infrastructure can’t just expect to use those networks without paying to use them. The network that spent all the money up front in capital expenditures provides value to the network that spent little money building the network and it needs to recoup its costs by charging the network that didn’t build the most expensive part of the network. Settlement based peering is the mechanism than ensures that all players on the Internet pay their share of expanding the Internet. This is the only way to ensure continued private investment and growth on the Internet.

How Level 3 intends to steal bandwidth from Comcast

Before Level 3 Communications landed the Netflix CDN deal, it sent approximately the same amount of traffic to Comcast as it received. Because of that, the two networks exchanged services of roughly equal value and they agreed to a settlement free peering contract. The two companies agreed not to charge each other for peering connectivity so long as the traffic exchange levels remained more or less the same.

But with the massive new Netflix CDN deal where Netflix is currently the largest source of traffic in North America, Level 3 will likely start sending 5 times more traffic to Comcast than it receives. That would violate its current settlement free peering agreement and it would require a new fee based agreement where Level 3 has to pay Comcast for the extra traffic it sends to Comcast. That makes sense as Netflix’s old CDN provider Akamai paid to peer with Comcast, but Level 3 decided that it would simply insist on violating its existing free peering agreement with Comcast which would allow it to outbid Akamai on the lucrative Netflix CDN service. Level 3 would essentially steal bandwidth from Comcast to outbid Akamai which pays Comcast for bandwidth.

From Comcast’s perspective, this impacts them severely as they’re losing bandwidth revenue from Akamai since Akamai lost Netflix business to Level 3, but Level 3 will refuse to pay for the bandwidth they use from Comcast by free riding on their existing peering connection. Even without the bogus Net Neutrality claims, Level 3 can cause major problems for Comcast if a peering dispute turns ugly. If Comcast insists on enforcing its contractual agreement and charges Level 3 and Level 3 refuses to pay, there’s little Comcast can do about it other than to sever the peering connection and send Level 3 traffic to its Internet transit provider that will eventually reach Level 3. But with most peering disputes, the contract violator will simply refuse to return to the Internet transit path and they’ll keep sending traffic down a broken unpaid pipe to Comcast which breaks network connectivity between Level 3 and Comcast completely.

Each will point the finger while customers on the two networks are cut off from each other and it makes both providers look bad. The reputation of both companies will be harmed and it will essentially be a game of “chicken” to see who blinks first. For smaller peering disputes that don’t garner much attention, the dispute is settled in a court or arbitration while the network is cut off for days or weeks. For larger peering disputes that receive media attention and congressional attention because fuming voters are calling their representatives and senators, it turns ugly and may lead to calls for additional Internet regulations which no one on the Internet wants.

With the added confusion of “Net Neutrality”, the hopeless pro-anything-labeled-Net-Neutrality biased blogosphere and the FCC proceedings on the Comcast NBC merger, Level 3 figured that they had a good chance of getting away with bandwidth theft or at least forcing much more favorable rates from Comcast. But not only is Level 3 trying to steal bandwidth, they’re a hypocrite because they’ve fought just as vigorously against other bandwidth thieves that tried to violate Level 3′s peering agreements. Joe Waz of Comcast pointed this out by citing Level 3′s own words when it came to the sanctity of peering agreements.

“To be lasting, business relationships should be mutually beneficial. In cases where the benefit we receive is in line with the benefit we deliver, we will exchange traffic on a settlement-free basis. Contrary to [other ISPs] public statements, reasonable, balanced, and mutually beneficial agreements for the exchange of traffic do not represent a threat to the Internet. They don’t represent a threat to anyone other than those trying to get a free ride on someone else’s network.”

Of course Level 3 wasn’t actually being genuine when they said that. They only meant it when someone else tried to free ride on Level 3′s network but they think it’s just fine if they free ride and steal bandwidth from Comcast.

A possible solution to free riders

I think there might be a better solution than severing a peering connection that goes unpaid. What Comcast or any other network encountering a free rider should do is simply enforce a symmetric rate on their peering connection. If Comcast sends 1 Gbps of traffic to Level 3 Communications, then they should enforce a 1 Gbps settlement free receive rate from Level 3. This might result in Level 3 failing to deliver the performance they promised to Netflix. If Level 3 can’t deliver an adequate service level to Netflix because they refuse to pay for the extra traffic going to Comcast, then Netflix is well within its rights to terminate the new agreement with Level 3 and return to honest CDN providers like Akamai. From a Net Neutrality standpoint, Comcast would not be discriminating against any source or application type in particular and they would only be enforcing existing contractual agreements with Level 3.

> What Comcast or any other network encountering a free rider should do is simply enforce a symmetric rate on their peering connection.

Thats an asinine idea, ISP are very heavy on inbound since people download content all the time (streaming movies, audios, etc). It doesn’t make sense to force a transit provider to provide symmetric rate when they are content providers. Comcast would expect to be unbalanced since they do not server content. Comcast users are the ones pulling netflix data, Comcast’s customers already paid for the bandwidth. If Comcast believe the peering is unbalanced, they need to better model their pricing, it seems to me they are trying double-dip on bandwidth charges.

# 30 November 2010 at 12:23 PM

Jack Moves said:

Comcast’s current (prior?) relationship with Level 3 was that of a TRANSIT CUSTOMER. They pay Level 3 money, and Level 3 re-advertises their routes to its peers and customers. In other words, no “theft” involved. This is easily verified on a variety of public resources.

You’d be well-served to read up on fundamentals of Internet routing prior to posting in this area.

Wes, no offense, but I would have thought you’d understood all of this based on how knowledgeable you are in general.

Anyone sending or receiving *transit* traffic from the Internet coming via Level3, Level3 is providing you with a transit service. You pay for transit service on a leased basis and it’s for upstream and downstream.

Now if you have a peering connection with Level 3 (which only covers traffic from Level 3 network to you and back) and Level 3 sends more traffic to you than you send them, you can charge them money. Now you might ask what if an individual set up a cheap server in a Level 3 data center and downloaded more stuff than they uploaded from other servers on the Level 3 network, and if that entitled them to payments. The answer is probably no since Level 3 doesn’t have to peer with you.

The bottom line is that peering agreements are only formed when *both* parties agree to the terms. A peering agreement where only one party is happy is not an agreement but theft. This is true of all contractual agreements between two parties.

# 30 November 2010 at 12:25 PM

Mike Hunt said:

Where are you getting this market research, Equinix co-founder Bill “Dr. Peering” Norton?

Comcast may indeed be a transit customer of Level 3, but that’s independent of the peering agreement dispute. These are entirely separate issues.

Comcast paying Level 3 for transit is for Level 3 carrying Comcast packets to other non Level 3 networks on the Internet. The peering agreement/dispute only covers the traffic directly between the Level 3 and Comcast network. Just because Comcast is a Level 3 transit customer doesn’t mean Level 3 gets to violate their peering agreement and refuse to pay for out-of-compliance peering.

You’d be well served if you stopped conflating transit and peering agreements. They’re different and they can coexist.

# 30 November 2010 at 12:32 PM

Mike Hunt said:

There is no technical evidence that there was any “peering” relationship in addition to the transit relationship. What are your sources?

Only, Level3 never said that, you’re now inventing facts to support your position (or relying on inside information you’ve not credited)…

# 30 November 2010 at 1:17 PM

Noname said:

As requested multiple times thus far, please provide your source, including specific references and documents – not “read this agreement or the other”, but specific content within those agreements that you are basing your arguments upon.

Failure to do so will make this article appear to be a piece intended to malign Level 3, without offering any substance to back up your assertions.

Level 3 has a settlement free peering agreement with Comcast. This is independent of the transit agreement where Comcast pays Level 3 for Internet bound traffic.
Akamai had a paid peering and/or co-location agreement with Comcast where it leased rack space and bandwidth from Comcast.

You’ve missed my point entirely. Of course broadband networks receive way more traffic. That’s the whole point, but they also spend hundreds of times more money building those networks. If you want to connect to that broadband network directly and bypass the Internet transit system and the fees associated with transit, it’s only going to happen if you reach a peering agreement. And since the cost of server bandwidth is nothing compared to last mile bandwidth, the only way the two parties can provide equal value is a fee-based settlement. That peering fee is actually much cheaper than coming over the Internet. This is how the Internet has always worked.

Now if you want free peering, be prepared to offer something of equal value to the broadband provider such as a long distance high capacity circuit. If you can’t do that, the ISP doesn’t need to peer with you and you can still get to that ISP over the “open” Internet via transit.

“The network that spent all the money up front in capital expenditures provides value to the network that spent little money building the network and it needs to recoup its costs by charging the network that didn’t build the most expensive part of the network.”

The above statement would be true for Comcast and Level 3 IF AND ONLY IF Comcast is NOT already being paid by its end-user subscribers for the bandwidth.

The infrastructure and capital expenditures that were spent to build up the Comcast network is already being paid for by the end-user subscribers. The Comcast subscribers are already paying for the right to receiving the Internet content, including the Netflix content.

If anything, nobody should be paying anybody anything for the peering or content exchange arrangement. Level 3 is getting paid by the content providers like Netflix for the right to use the Level 3 network. Comcast is being paid by end-user subscribers for the right to use the Comcast network. Why should Comcast or Level3 even pay each other anything for anything? Their infrastructure is already paid for by their respective end-users.

This is where the FCC needs to step in. FCC needs to tell Comcast: hey you are already being paid by your own subscribers. No double-dipping allowed. If you don’t think the extra bandwidth from Netflix can be supported by current subscriber fees then just increase the fees or reduce the bandwidth caps. And while we’re at it, we should talk about those local franchise monopolies that you have in place. . . .

# 30 November 2010 at 2:30 PM

Dave said:

So, no technical proof, just a cnet article? Way to cite sources George!

# 30 November 2010 at 2:55 PM

Dave said:

Also in case anyone was unclear, not only is Bennett a bumbling idiot (see: his past NANOG posts), but a hypocrite and Comcast lobbyist to boot. I’d not exactly trust him as a neutral party or subject expert.

# 30 November 2010 at 2:57 PM

John said:

The big problem here is that comcast subscribers PAID comcast for access to the internet and internet content. Comcast is double dipping here – and this is a net neutrality issue – if I pay comcast for access to content then that is what I expect – access to content – not access_if_traffic_is_balanced.

# 30 November 2010 at 3:04 PM

Nick R Brown said:

@Dave, Where is your evidence that Richard Bennett is a lobbyist? Is it because Brett Glass called him one in a Twitter post? “Way to cite sources”…?

# 30 November 2010 at 3:16 PM

Ronnie said:

@John
Yes…. comcast subscribers paid comcast… netflix paid L3. If L3 wished to put out their content in the open net, that’d be fine. It would be a net neutrality issue if comcast disallowed L3 content. Comcast is not threatening that. L3 content would always be available in the open net (unless L3 pulls a dick move by closing their network and starts blaming that comcast had cut them off). Comcast is re-negotiating their peering agreement. How is that net neutrality? If comcast should not expect payment, should they be forced to have a “free” peering agreement with every tom, dick, and henry that comes along? Or, maybe you’re suggesting we just illegalize peering agreements?

# 30 November 2010 at 4:37 PM

Sanjath said:

As someone mentioned already, comcast charges us for having the internet service, it is not for them to decide what traffic they can let us in depending on whether or not they have a deal with the content provider. I understand that there may be costs involved, let them charge it to customer or bring in the tiered model where customer can opt into a lower priced model when the content provider is subsidizing. But it is too much of a mess, comcast should stick to being pure ISP and charge accordingly. All these ISPs keep complaining that they do not want to be a dumb pipe, unfortunately we do need dumb pipes and customers should pay for dumb pipes alone, no mucking up with content providers. W.r.t this specific dispute, I dont know why a CDN would get into peering agreement with an ISP. It just doesn’t make sense to me, how can a content provider expect to have equal traffic (in reverse direction) from content consumers?!

# 30 November 2010 at 4:57 PM

Sanjath said:

@Ronnie,
You are saying comcast is not blocking the content but trying to renegotiate the contract, what if L3 did not have an agreement? (starting fresh) would they have just blocked the content? Let’s forget for a minute that there is L3 in the middle, but netflix decides to bring the content till the comcast hubs, should the Netflix pay comcast to deliver it to their customers as Netflix cannot have a peer agreement with comcast? What I am not clear is what is the boundary of an ISP and backbone infrastructure. Up to what point should the content provider deliver his part?

# 30 November 2010 at 5:09 PM

Gonzo said:

One small problem here. L3 is a tier one provider, they (most likely) don’t pay a transit fee. They might have peering fees, but I would imagine that comcast would be losing out on losing peering and transit with a major tier one provider. It’s a game of chicken between the two companies, for sure. So L3 is pushing around their tier one status, which akamai doesn’t have, but comcast won’t be happy if their customers all call them the same day to complain about access.

In the end, it’s the comcast subscribers that are requesting the data, so I believe that the requester should be the one to pay for this, even if it requires a price hike on the consumers. Netflix might have to pay more to increase the size of the pipe for their customers as well, indirectly, but this should have been fleshed out before they changed their provider. And I really doubt that this data was every synchronous upstream/downstream between L3 and comcast, considering all their subscribers are heavily downstream weighted.

This isn’t the first pro-comcast article you have drafted either. I’m not sure if they are paying you or what.

> Level 3 will refuse to pay for the bandwidth they use from Comcast by free riding on their existing peering connection

That traffic is there because *Comcast’s* paying customers asked for it; it’s not *netflix’s* traffic, it is that of those customers.

Comcast runs an “eyeball network”, with asymmetric cablemodems at the edge, and not a lot inside it that the Internet at large wants to see; they’re *always* going to be asymmetric at their peering points as well.

I concur with whomever suggested that this story sounds like press-release journalism.

> If Comcast sends 1 Gbps of traffic to Level 3 Communications, then they should enforce a 1 Gbps settlement free receive rate from Level 3. This might result in Level 3 failing to deliver the performance they promised to Netflix.

I hope Comcast tries. If they do, *their own customers will have a right of action against them*, for failing to deliver to those customers what the customers asked for, the carrier then trying to parade itself as a common carrier.

That Comcast *itself* failed to make proper arrangements for traffic to get in and out of its network to satisfy its customers’ reasonable desires *is Comcast’s responsibility*.

They do have some peering with some other major carriers (Savvis it looks like), but I’m not seeing transit connections, so this is very dangerous to Comcast to get in this fight. I also don’t know the prices they are paying for these alternative peers, but I think it could get expensive.

# 30 November 2010 at 5:40 PM

Nick R Brown said:

@Dave,
So your evidence that Bennett is a lobbyist is because someone in the comments of that post said that he was a lobbyist because ITIF has a K Street address on their website?

So are all the folks that work at the movie theater on K Street near the bridge lobbyists too? Lobbyist by association, that’s some great evidence.

The tell here that this is not a real net neutrality dispute is that Netflix has said nothing in support of Level 3. If Netflix genuinely believed its streaming business was being threatened by Comcast, acting out of a desire to gain strategic or competitive advantage over Netflix (as opposed to a simple pricing dispute) Netflix would be beating the drum for FCC action alongside Level 3. The fact that it isn’t tells you that Netflix does not feel threatened here by Comcast.

This sentence from Level 3′s own press release is also a tip-off:

“On November 22, after being informed by Comcast that its demand for payment was ‘take it or leave it,’ Level 3 agreed to the terms, under protest, in order to ensure customers did not experience any disruptions.”

Translation: Netflix said it didn’t have our back because this is really just a pricing dispute with Comcast, so we had to cave.

[...] 30: Commentary on the web accelerates with some accepting the commercial disagreement argument, and others seeing in the move a chance to implement a double-sided revenue model for ISPs that [...]

# 30 November 2010 at 6:36 PM

WLJ said:

Comcast was much closer to balanced in/out with Level 3 than people realize. They’ve been in the process of trying to become a content originator for a while now. Fancast is one example. Also consider that their customers are serving content via BitTorrent and DynDNS techniques.

Netflix has changed all that, practically overnight. They decided not to build a CDN (unlike some other large originators, such as MSFT), and instead took an offer from L3 that they knew was lower than cost.

Btw, how the heck did you get through this entire article without mentioning Limelight? LLNW is still handling a large chunk of Netflix traffic, and in fact Netflix is probably going to go to a primarily 2-CDN relationship by the first of the year instead of splitting between all three. Limelight actually pays for peering, too.

# 30 November 2010 at 7:08 PM

B Waters said:

People buy comcast internet to access things on the internet. They would not buy it if not for the services available. Perhaps comcast should be paying to be allowed to serve the content that lets them sell their internet access. Otherwise, let the access go through the regular internet and clog up that channel more, and give customers a reason to go to another ISP. It is as much if not more of a favor to comcast to peer with them and allow them a cheaper way to give their customers access to the services they bought their connections to receive.

# 30 November 2010 at 7:22 PM

Mike Hunt said:

Probably because he’s funded by Comcast (fact) and regularly shows up at regulatory/think-tank-related discussions in DC (fact).

In any event, we’re 26 posts deep in this thread, and I don’t see any technical evidence that Comcast and Level3 had a PEERING relationship going prior. Only vague references to some cnet article.

I think George started this article under false pretenses, just to stir the pot.

# 30 November 2010 at 7:54 PM

ksj said:

Actually, you are the one conflating transit and peering (or rather helping Comcast to do so) by quoting Level3′s comments regarding their transit relationship with Cogent.

As you said, they are different.

You also neglected to mention that Comcast has plenty of traffic it could route through Level3 to help with the imbalance. They just chose to start charging for peering instead because their revenue source (Akamai) just dried up.

I think you’re wrong on this one.

# 30 November 2010 at 8:06 PM

Ben said:

What does having paying subscribers have to do with not needing to charge for a peering agreement’s capital cost??

Paying subscribers obviously DON’T pay for all of the costs associated with running an ISP, so the rest of the money needs to come from somewhere else. In business there is rarely a 1:1 relationship between capital expeditures and funding sources. This is a good thing because it keeps the prices for end users reasonable even for products with high capex requirements or maintenance costs.

And don’t point to profit either. Both of these companies want to make a profit. Making a profit is why companies exist. You’re just being obtuse when you rant about a company trying to make more profit by trying to get a better deal for itself from another company as if it were a bad thing. OF COURSE they are. Both sides are. That’s the point of their existence. That’s why Level 3 cut this shady deal to get Netflix. That’s why Comcast isn’t willing to eat it.

Do you have a current copy of the peering agreement between Level(3) and Comcast? I have a version from 2003 that would allow Level(3) to send the Netflix traffic directly to Comcast. I would love to see the new contract. Did the Comcast PR folks even read the agreement? Did they share a copy with you or did you take them at their word? I think the agreement is covered by an NDA and neither party can share the details with you – did someone violate the NDA?

In any event, Level(3) doesn’t need to terminate the Netflix directly through Comcast peering to make money on the deal. The reality is that Akamai’s business model is too dated to be competitive as their loss to Level(3) reveals.

Are you still trying to claim there is no peering agreement between Comcast and Level 3? Level 3 *admitted* that they’ve agreed to change from a symmetric traffic free peering agreement to an asymmetric fee-based settlement peering term that Comcast demanded even though they’re protesting it. That’s a settled fact.

# 30 November 2010 at 8:30 PM

Matt W said:

The infrastructure and capital expenditures that were spent to build up the Comcast network is already being paid for by the end-user subscribers.

WRONG. Comcast makes money from a variety of sources. They used to get money from Akamai because Akamai was pushing a large amount of data into their–very expensive compared to Akamai’s– network. L3 then takes the contract from Akamai, Comcast loses a revenue stream, and seeks to recoup losses from a legitimate business practice: private peering agreements.

Take a second to actually read and understand the article at hand. This is not a net neutrality issue. Comcast is not throttling Netflix traffic, it is attempting to (ostensibly) throttle all traffic from a single peer. Ever try to download data from a network very distantly connected to yours? IE- sites hosted in Asia. It’s a lot slower than downloading from an American based CDN. It is a violation of net neutrality? No! It’s just a lot more expensive to get that data here, so it comes slower.

# 30 November 2010 at 9:20 PM

Formergreybeard said:

CDNs are a different sort of peering. I’m reasonably confident that Level 3 would be willing to provide Comcast with dozens of content CDN servers peppered thoughout their network so that virtually of the traffic would be just going over their cable “edge” and not through central peering points.

Comcast is by definition an edge content distributor, with massive download (to end users) and very little server upload. Level 3 could provide servers inside their facilities. This is not a normal internet peering relationship.

The fact of the matter is that Level 3 cannot easily become a CDN without either Level 3 or Akamai. Why? The answer is Patents. Level 3 (though Sandpiper/Digital Island acquistion) and Akamai basically between them own the important intellectual property that enables modern CDNs. CDNs allow the internet to scale to millions of users watching TV content by distributing the bulk of the content activity to the edges freeing up peering relationships for genuine traffic that isn’t the same video or similar content over and over again.

Equating Akamai or Level 3 as stealing bandwidth is just not representing what they do fairly. They do something valuable for the internet, and deserve the right to do. They do something valuable for Comcast and it’s users as well.

Would we be happier with millions of copies of Netflix content going all the way ‘cross the internet without CDNs moving content to the edges? No we wouldn’t, because all the ISPs would have to invest much more to make peering points oh so much bigger.

# 30 November 2010 at 10:41 PM

Samir said:

Comcast propaganda by people who clearly don’t understand the internet.

# 30 November 2010 at 10:56 PM

Mike Hunt said:

Where in the Level3 press releases is that stated?

(Hint, it’s not, as it’s incorrect.)

# 30 November 2010 at 11:04 PM

Abhishek Sethi said:

What Netflix really need to do is to re-write their client so that it sends a packet back to netflix server, every time it receives one. Level 3 or netflix can throw away this dummy packet. Now Comcast will send equal amount of traffic to L3 as it receives from L3, thus making it symmetric peering.
The cost of filtering this dummy traffic would be much cheaper for Level 3 than $$ comcast is asking for.

# 30 November 2010 at 11:14 PM

Garrett said:

George,

I like the explanation about the difference between peering and transit contracts among ISP’s. Thank you for spending the time to do this it helped a lot! I can also see the points about violation of contracts concerning peering agreements. It does seem to indicate that L3 may be trying to violate the contracts its in.

But I have to admit… It looks like the relationships are twisted at the top. If Comcast is PULLING more traffic than L3 is… why isn’t Comcast paying L3 for the additional content that it’s customers are requesting?

The internet is a request based system, not a push based one… I don’t understand why anyone would ever pay to serve up data to someone else…

# 1 December 2010 at 12:31 AM

ksj said:

BTW, peering is being used in both a business and routing context here. You “peer” with another providers router using BGP, the protocol, which is done regardless of whether it’s a transit or “peering” business relationship.

# 1 December 2010 at 12:33 AM

ksj said:

Matt W said: “No! It’s just a lot more expensive to get that data here, so it comes slower.”

I agree that the trans-oceanic cost is higher, but it’s generally slower because of latency, not bandwidth, or cost.

# 1 December 2010 at 2:22 AM

Dave R. said:

Comcast might reasonably be entitled to charge L3 if this were transit data. It isn’t. It’s data being requested by Comcast subscribers, who have already paid Comcast for exactly this.

This is a clear cut case of Comcast leveraging it’s quasi-monopoly to extract rents. They are demanding to be paid twice.

Would any ISP be able to extract payments from L3 if they had serious competitors? No, they wouldn’t. And there’s your answer.

“But I have to admit… It looks like the relationships are twisted at the top. If Comcast is PULLING more traffic than L3 is… why isn’t Comcast paying L3 for the additional content that it’s customers are requesting?”

You’re thinking transit in the sense that a business has to pay both directions (upload and download) when they use an ISP to reach the rest of the Internet. Comcast pays Level 3 for traffic that it sends and receives from the Internet through Level 3 and this is independent of the peering agreement and the peering ratios. This is what you’re thinking of.

This peering dispute is nothing like that. The traffic imbalance is purely from “on-net” traffic coming directly from Level 3 (largely their new Netflix) customer to Comcast. Level 3 wants to deliver that traffic to Comcast so that it can charge Netflix a CDN fee. To do that, they were already offloading 2:1 more traffic to Comcast and Comcast was forgiving that.

Then Level 3 demanded 30 new peering ports from Comcast at no extra charge to handle the additional Netflix traffic. Comcast gave them 6 new ports (we’re talking 10 gigabits per port) for free and told them than if they want another 20, they’d have to pay for it. Level 3 agreed to pay for the 20 ports but they’re angry because they had assumed they could just violate their free peering agreement at a 5:1 ratio when the agreement was for 1:1 ratio.

Now ask yourself this. Can you bring a few gigabit switches into an Internet exchange and demand that the network give you 30 free 10-Gbps ports? Does that not seem like a fantasy.

# 1 December 2010 at 3:28 AM

Mike Hunt said:

There is no evidence that Comcast had any settlement-free peering relationship with Level3 prior. Their only relationship was that of a customer. Nowhere in any public resources (route-views, etc), or even in Comcast or Level3′s published works over the past few days, is it being claimed differently.

George is overlooking this point as the truth is far more boring than the fantasy he’s constructed, and he doesn’t seem to understand how peering works.

[...] commentators, such as the reliably telco-centric George Ou, have taken Comcast’s side: Level 3 outbid Akamai on Netflix by reselling stolen bandwidth. Others, such as Stacey Higganbotham of GigaOm, who favors competition and startups, have taken [...]

Whether, or not, Level 3 was abusing any agreement it had with Comcast, it’s still business malpractice for it to have done the deal with Netflix without first nailing down what its real costs would be to deliver what it promised. It’s even worse that Level 3 then tried to drag Netflix into the middle of its fight with Comcast by trying to make it into a net neutrality issue.

If I were Akamai I’d be rubbing my hands together with delight at the thought of getting the Netflix business back.

# 1 December 2010 at 4:21 PM

Dan Thies said:

Mike, if there was no peering agreement, how was Level3 co-located with Comcast? How does that happen without a contract? Answer: it doesn’t. Unless Level3′s army of Ninjas snuck into a data center and installed switches. In which case, Level3 has no business demanding anything.

The scary thing is that there seems to be very little regulation here. Obstenance could cause some major issues with paying customers that are the small fish individually but are the profitable side of the ledger.

# 2 December 2010 at 12:47 AM

Mike Hunt said:

Richard, you work for a “think tank” with a paper trail of funding by Comcast. How does that make Dave a liar?

# 2 December 2010 at 1:15 AM

Mike Hunt said:

Dan Thies, by buying service from Level 3. That’s how. Comcast was a customer not a peer.

Think Tanks are not lobbying companies, they’re non-profits who are forbidden by law from lobbying. Unlike lobbying firms, Think Tanks do not do work for hire; we have sponsors, but not clients. Among our sponsors are a number of foundations and government agencies as well as actual capitalist firms. The companies have opposing interests in the net neutrality debate, so any position we take on this issues may be good for some and bad for others.

So it’s naive and frankly idiotic to try and paint Think Tanks with the lobbyist brush, as anyone who is even vaguely in touch with our political process knows. Using this ploy is like wearing a button that says: “Kick me, I’m clueless and nasty.” It’s also a sign that the debate is probably over your head.

People make arguments, and the arguments are good or bad depending on their content, not on who you imagine pays the person making the arguments. If the arguments are bad, refute them; it should be easy enough. And if it’s all over your head, just call names and stamp your feet. Nobody’s fooled either way.

Yesterday, Comcast (CMCSA) posted another entry on their blog entitled, “10 Facts About Peering, Comcast and Level 3.” Amongst some of the points that Comcast wants us to believe is that this entire debate with Level 3 (LVLT) is “not about online video.” Comcast says that it is simply an, “old-fashioned commercial peering dispute.” If that’s the case, why is Comcast imposing this new fee to Level 3 only a few short weeks after Level 3 announced a new contract with Netflix (NFLX)? We all know that the growth of traffic on the net is coming from video and for Comcast to say that this is “not about online video,” is laughable. Companies like Comcast are struggling with trying to combat over-the-top services like Netflix. That’s exactly what this debate is about: video.

Another one of Comcast’s points is the idea that Level 3 is forcing the burden and cost onto Comcast’s customers, which again, is absurd for them to suggest. Comcast says, “This is all about Level 3 gaining an unfair advantage over its competitors by gaining enormous additional capacity at no cost to itself, instead shifting the financial costs to Comcast’s high speed data customers.”

Level 3 has spent billions building out a network. The idea that there is no cost to Level 3 to support all of the additional video traffic they have to deliver is just wrong. Did Comcast listen to Level 3′s last earnings call when Level 3 said they expected to spend $14M in capex, in the fourth quarter, just to support Netflix? Of course there is a cost to Level 3. In addition, how can Comcast say that Level 3 is shifting the costs to Comcast’s customers? The customer is the one who is demanding this content. The customer is the one who is paying Comcast for the ability to get the content. Next thing you know Comcast is going to raise their rates again, and then blame Level 3 for it. Does Comcast really think we are this stupid?

Comcast also says that, “our customers can and do watch video from any online video provider, including Netflix and dozens of others, on our high-speed Internet service.” Only if Level 3 agrees to Comcast fees they can. If they don’t, it’s going to be hard for Comcast customers to get Netflix content unless all of the requests from Comcast’s network gets delivered by Limelight Networks or Comcast caches Netflix’s entire library inside the Comcast network, which I don’t see them doing any time soon. Comcast also makes a point to say that, “our agreement with Level 3 is no different than our agreements with its competitors.” Really? What CDN, that owns their own network, does Comcast have this kind of agreement with? Akamai and Limelight don’t count.

And finally, Comcast says that, “we charge a flat fee for our high-speed Internet service and do not charge any additional price to consumers to watch these online video services.” So when Comcast says only four months ago that they are raising rates to help offset the costs associated with their Xfinity streaming service amongst other digital products, that’s their idea of calling the service free? I don’t think so.

Comcast is trying to treat the industry and consumers like we’re all a bunch of idiots and don’t know what type of content is growing or what we’re consuming. Not a smart move on Comcast’s part.

[...] 30: Commentary on the web accelerates with some accepting the commercial disagreement argument, and others seeing in the move a chance to implement a double-sided revenue model for ISPs that [...]

[...] extortion, is this simply an obscure commercial dispute between two Internet giants, or is Level 3 trying to pull a fast one by playing the net neutrality card at a moment when Comcast is especially vulnerable to criticism [...]

[...] 30: Commentary on the web accelerates with some accepting the commercial disagreement argument, and others seeing in the move a chance to implement a double-sided revenue model for ISPs that [...]

# 9 December 2010 at 12:52 PM

RichC said:

Why is it that so many commenters on this blog think that the laws of economics and supply and demand are irrelevant because we are talking about “the Internet”?

It matters very little whether, prior to this dispute, Comcast was a peer or transit customer of L3. What does matter is the following:

Prior to this dispute, Comcast received a total amount of revenue (most from flat-rate subscriptions for residential Internet access service from eyeballs, but some from transit payments by CDNs such as Akamai) that apparently was adequate to balance out its total costs.

As a result of the deal L3 made with Netflix: (a) Comcast’s revenue from its eyeballs and total traffic to/from its eyeballs would (to a first approximation) remain unchanged; (b) its revenue and traffic from Akamai would decrease; and (c) its traffic from L3 would increase greatly, but its revenue would not. Further, Comcast faced additional costs to serve this increased L3 traffic (e.g. 30 extra ports).

Now in the world of economics, when such a disruption to a prior equilibrium occurs, one generally sees a reaction. In this case, four types of reaction seem possible: (1) Comcast could do nothing — and simply accept lower total revenues and higher costs; (2) Comcast could block the augmented Netflix traffic from L3; (3) Comcast could raise the rates it charges to its eyeballs; or (4) Comcast could request payments from L3.

If you think Comcast should have just sucked it up and chosen #1, you aren’t very familar with business (and this is a business). If you think Comcast should have selected #2, then you aren’t very much a fan of the “no blocking” principle of the FCC’s Internet Policy Statement or network neutrality. While Door #3 is a possible solution, there are many reasons why Comcast may have preferred not to go down that road (e.g., customers who don’t pull a lot of traffic would be encouraged to leave, usage-sensitive billing is expensive and customers don’t like it, etc.) Instead, we know that Comcast choose Door #4 — and what’s even more important, it appears that L3 quickly agreed with this choice. When something like this happens, it is quite likely that economic supply and demand are telling you that this decision was the most rational market choice.

To be sure, L3 would have preferred Comcast choose Door #1 (and it appears to be crying out to Mama FCC to use the force of government regulation to demand that Comcast accept this); but because L3 is a business, too, it recognized that its interests would not be served by Door #2 — because its Netflix packets would never get delivered; or by Door #3, because usage-sensitive eyeball pricing might suppress the demand for its client’s (Netflix’) video services.

This isn’t about personalities and politics. It’s just business and economics — and the Internet is not immune.

Comcast could have told level3 that what you see is what you get, we’ll keep the original peering contract and charge for any additional BW you may require, which from what I’ve read throughout the internet is how its played out so far.

Now, Level3 could say we’ll just stick to our original peering agreement and we will not pay for additional BW. It will be just a matter of time before Comcast customers start getting annoyed that they are having latency/QOS issues while trying to stream Netflix content and give Comcast the boot.

Now I am wondering about the transit agreement that Comcast has with Level3. Comcast pays Level3 for this service, can any overflow from the peering agreement get funneled to the transit agreement. I honestly don’t know but I suppose its possible and if thats the case Comcast could be paying Level3 for the influx of traffic into the comcast network

Comcast could just give Level3 the boot and go with another transit provider and cancel the peering agreement all together. But guess what? Comcast customers are going to still watch Netflix. And Comcast customers are still going to be annoyed by the latency/QOS issues. Comcast will be paying a huge price to its new transit provider because of the bandwidth issue and still having to build up its network to handle traffic.

What Comcast should do is advertise this working partnership with Level3 to provide an enriching experience for Netflix Subscribers and explain the benefits of becoming a Comcast Customer, emphasizing the Netflix Brand.

“Now I am wondering about the transit agreement that Comcast has with Level3. Comcast pays Level3 for this service, can any overflow from the peering agreement get funneled to the transit agreement. I honestly don’t know but I suppose its possible and if thats the case Comcast could be paying Level3 for the influx of traffic into the comcast network”

No, you must not mix transit traffic with peering traffic. They’re separated by the source of traffic coming from Level 3. If it comes from the CDN, it’s peering. Now Level 3 might try to push the traffic from an IP range that’s considered transit, but that would be considered metering fraud as far as their agreement is concerned. Furthermore, if Level 3 tried to bill an additional 100 Gbps of their own CDN traffic as transit to Comcast, and even if their agreement allowed that, Comcast would have even less incentive to improve the performance. At that point, Netflix would likely move back to their original CDN provider Akamai to regain the performance they once had.

If you wish to learn more about this, please see my detailed video on the topic.

# 14 December 2010 at 7:49 PM

Ed said:

I agree with your reply, but….

Comcast has done little to update its transit links. In my mind Comcast has intentionally congested its transit ports to force their position for higher cost to a direct connection from providers such as Level3. A smart business move, maybe, as long as the likes of Level3 or others fall for this.

Another thing….
Comcast’s own Xfinity brand is in direct competition with Netflix. Xfinity is a CDN customer of Akamai which in turn is COLO’d with Comcast. I find it more than a coincidence that this peering dispute comes shortly after Level3 announced they were selected by Netflix to be the primary CDN. I would venture a guess (only a guess) that Akamai was provided the COLO space in exchange for hosting XFINITY. Since Akamai lost there money maker (Netflix), the need to be providing free CDN service to Comcast may not be as lucrative as before. Understand this is all speculative, but I’ve seen arrangements like this before.

And again I say…..
What Comcast should do is advertise this working partnership with Level3 to provide an enriching experience for Netflix Subscribers and explain the benefits of becoming a Comcast Customer, emphasizing the Netflix Brand.

They ultimately want to build their customer base, especially since growth has been stagnant. Netflix is here to stay, Xfinity or not. Why not use this to build revenue and build their network at the same time. Or better yet, work a deal to have Level3 provide its CDN service in exchange for the required circuits that Level3 needs.

Rushabh Sheth nailed this issue on the head! Read again what he wrote (included here again for convenience). In the middle of all the back and forth banter on this page, his comment is the only one that makes sound logical sense.

Think about it, internet service is analogous to wireless service. It should be we, the consumers, who should have to pay according to our bandwidth consumption, not the up-line providers.

Comcast needs to be reminded that they have large quantities of captive customers (i.e., no other broadband ISP to choose from!!) who are charged the same rate, regardless of whether they casually browse the internet or download reams of movies every night. Why doesn’t Comcast charge its customer according to their usage!? Becasue Comcast is just like Level 3…they only care about “fair” when it affects their bottom line!! When bashing Level 3, I believe George referred to that as being “hypocritical” and somehow even abstracted, perverted, and conveniently morphed that whole concept into being the same thing as “theft”. Clearly George bought too much Comcast stock and is now just pissed because he missed the golden rocket launched by Level 3. Complain about your broker, not about Level 3. Sorry George, your just dead wrong on this whole topic!

——————
Rushabh Sheth said:
George,

I take exception to the following statement:

“The network that spent all the money up front in capital expenditures provides value to the network that spent little money building the network and it needs to recoup its costs by charging the network that didn’t build the most expensive part of the network.”

The above statement would be true for Comcast and Level 3 IF AND ONLY IF Comcast is NOT already being paid by its end-user subscribers for the bandwidth.

The infrastructure and capital expenditures that were spent to build up the Comcast network is already being paid for by the end-user subscribers. The Comcast subscribers are already paying for the right to receiving the Internet content, including the Netflix content.

If anything, nobody should be paying anybody anything for the peering or content exchange arrangement. Level 3 is getting paid by the content providers like Netflix for the right to use the Level 3 network. Comcast is being paid by end-user subscribers for the right to use the Comcast network. Why should Comcast or Level3 even pay each other anything for anything? Their infrastructure is already paid for by their respective end-users.

This is where the FCC needs to step in. FCC needs to tell Comcast: hey you are already being paid by your own subscribers. No double-dipping allowed. If you don’t think the extra bandwidth from Netflix can be supported by current subscriber fees then just increase the fees or reduce the bandwidth caps. And while we’re at it, we should talk about those local franchise monopolies that you have in place. . . .

# 29 June 2011 at 5:31 PM

Twitter Feed

About Us

Digital Society is a digital think tank that believes culture and commerce are inseparable, that the digital economy flourishes when people are free and rights are secure, and that free markets free people.

Digital Society is an independent 501(c)3 non-profit organization, funded by donations from Jon Henke and from Arts+Labs. We advocate for a pro-culture, pro-commerce digital society through research, analysis and debate on emerging technology issues.

Reply Comments

Transparency and interactivity are trademarks of the Internet era, and we aim to foster them here at Digital Society. It is inevitable that some people will disagree with the technology policy positions we take. We want to have that constructive debate.

The Reply Comments feature gives our critics a chance to respond to our viewpoints and the Digital Society audience convenient access to competing arguments. Any time we directly challenge the views of an individual or a group on this site, the party in question may substantively respond in a guest post.