Verizon, bandwidth provider blame each other for slow Netflix streaming

Verizon, Time Warner, other ISPs won't use Netflix's own CDN.

Verizon and Cogent Communications, an Internet service provider that peers with Verizon, are blaming each other for poor Netflix streaming performance at customers' homes.

As Peter Bright explained in an exhaustive primer on how the Internet works, ISPs pass traffic between one another in an arrangement known as peering. A few days ago, Cogent CEO Dave Schaeffer told GigaOm that Verizon is "allowing the peer connections to degrade." Instead of adding more ports to accommodate increased traffic, Verizon is letting the ports fill up, he claimed. This is allegedly causing bad Netflix performance for Verizon's customers.

Verizon says Cogent's claims are incorrect, or at least incomplete. Verizon VP David Young wrote a blog post yesterday saying that Cogent is not living up to its end of the bargain and should have to pay Verizon. While the two companies send and receive traffic to each other for no charge, "Cogent is not compliant with one of the basic and long-standing requirements for most settlement-free peering arrangements: that traffic between the providers be roughly in balance. When the traffic loads are not symmetric, the provider with the heavier load typically pays the other for transit… This isn’t a story about Netflix, or about Verizon 'letting' anybody’s traffic deteriorate. This is a fairly boring story about a bandwidth provider that is unhappy that they are out of balance and will have to make alternative arrangements for capacity enhancements, just like any other interconnecting ISP."

To fix this disparity, Cogent (a "repeat offender" in Young's words) should take advantage of one of several services Verizon has for sale, he wrote. "Verizon offers a number of readily available solutions for interconnecting providers who send significantly more traffic than they receive from Verizon’s networks," Young wrote. "Solutions such as cloud, hosting, Partner Ports and others are designed specifically to provide a cost-effective means of delivering very large volumes of out-of-balance traffic. Other large streaming video providers (and/or network service providers carrying such one-way traffic) are already taking advantage of these solutions and seeing immediate benefits. These solutions are available today to Cogent, Netflix and any other content or network service provider with similar traffic profiles."

We've asked Cogent for a response to Verizon's claims but had not heard back at the time of publication.

UPDATE: In a phone interview, Cogent CEO Schaeffer provided some additional details to Ars. He noted that the amount of traffic swapped between Cogent and Verizon is unequal because of the traffic requested by Verizon customers who try to watch Netflix streaming video. "The very nature of [Verizon's] customer base makes it impossible" to make the amounts of traffic exchanged between Cogent and Verizon equal, he said.

Cogent is Netflix's primary Internet service provider, and thus plays a central role in bringing Netflix's content to end users. However, ISPs like Verizon that sell to homes are the ones that carry that traffic across the proverbial last mile. Thus, the connections between Cogent and ISPs that serve consumers are crucial. "We're not sending anything to Verizon that their customers don't ask for," he said.

Adding a port (each port provides another 10Gbps of bandwidth) costs about $10,000 each for Cogent and Verizon. That's a small amount of money for Verizon, Schaeffer said. He said he believes that Verizon's decision to limit Netflix bandwidth is due to Verizon wanting customers to use its own streaming video services, such as Redbox Instant.

Schaeffer said that other ISPs who offer content competitive to Netflix have also been slow to add ports to handle Netflix traffic. However, Schaeffer said Verizon has been the worst offender.

Should ISPs bear the burden—or should Netflix?

Not mentioned in Young's blog post is that Verizon could improve Netflix performance for its customers by taking advantage of Netflix's "Open Connect" content delivery network. "ISPs can directly connect their networks to Open Connect for free," Netflix says. "ISPs can do this either by free peering with us at common Internet exchanges, or can save even more transit costs by putting our free storage appliances in or near their network."

We asked Verizon today why it doesn't use Netflix's Open Connect and whether Open Connect could negate the Verizon/Cogent dispute's impact on Netflix performance. Verizon did not answer directly, instead pointing us to the portion of its blog that we quoted above about the "solutions" Verizon offers to Cogent, Netflix, and other content or network providers.

Why might Verizon object to taking Netflix equipment into its data centers? While Netflix offers Open Connect equipment for free to ISPs, it does cost money to operate the equipment. There is also the issue that Verizon offers streaming video that competes against Netflix.

The giant amounts of traffic Netflix streams over the Internet must be dealt with, whether by Netflix, ISPs, or both. In North America, "[a]udio and video streaming account for 65% of all downstream traffic from 9pm-12am and half of that is Netflix traffic," a report on Internet data usage by network equipment vendor Sandvine said last November.

While Netflix says Open Connect already has connections with "Frontier, British Telecom, TDC, Clearwire, GVT, Telus, Bell Canada, Virgin, Cablevision, Google Fiber, Telmex, and more," that only accounts for a minority of Internet traffic. Netflix "hasn't come to any agreement with the biggest US companies in the industry—Comcast Corp., Time Warner Cable Inc., AT&T Inc. or Verizon Communications Inc., which between them controlled 62% of the nation's broadband marketplace at the end of 2012, according to SNL Kagan data," the Wall Street Journal wrote yesterday.

As we wrote six months ago, Time Warner was apparently negotiating with Netflix to become part of Open Connect but objected to the technical requirements Netflix imposes upon partners.

While Netflix saves money on transit fees through its Open Connect arrangements, ISPs like Time Warner and Verizon don't seem to think the program offers them any advantages, even if it would help deliver a better video experience to their customers.

Promoted Comments

So Verizon wants additional money from Netflix to provide a service that their customer base is already paying for. How bout this Verizon, if you want to charge Netflix then fine, but since you're already being paid for carrying traffic at that point my connection should be free. No? Then quit whining.

RTFA.

Quote:

"Cogent is not compliant with one of the basic and long-standing requirements for most settlement-free peering arrangements: that traffic between the providers be roughly in balance. When the traffic loads are not symmetric, the provider with the heavier load typically pays the other for transit"

This is exactly how peering has worked for about as long as we have been peering networks. The way the Internet works near the core is nothing like buying your cable connection at the edge.

The problem is that Cogent doesn't actually have a load that it wants Verizon to carry. Actually Verizon (through its customers) is asking Cogent to transit a lot of traffic from service providers to Verizon that Cogent doesn't actually care about and wouldn't mind not having to carry at all.

In fact, since Verizon could easily bypass Cogent entirely by using the Netflix CDN, it is really clear that Verizon is choosing to demand that Cogent transit a heavy load that there is really no need for Cogent to transit, and then demanding that Cogent pay for the privilege of carrying Verizon's data.

Heard about this in NPR this morning. They brought up the fact that Verizon owns 65% of RedBox Instant, a direct competitor to Netflix. It sounds to me that there could potentially be some anti-competitive measures taken by Verizon here.

I've been proclaiming it for years, ISPs should not be allowed to sell services other than bandwidth. Allowing them to offer services over their own connection means that they will attempt to kill all competing services. It removes competition and results in a monopoly or duopoly of ISP/media conglomerates.

Granted, this would mean splitting up some corporate giants, like time-warner, verizon, AT&T, etc. But it is the only way that ISPs won't abuse their monopolies/duopolies. Customers are trapped because if the service is poor, it isn't like it is possible for a competitor to come out of nowhere and string utilities through existing cities. There simply isn't enough room on the utility poles and it wouldn't be in society's best interest anyway.

I hope that ISPs continue to abuse their market position to such an extent that something is done about it. And that something is to split break up ISPs, with the network division becoming a different company than that which provides services over the network. It is the only way we will see competition.

Verizon as an very old Incumbent Local Exchange Carrier (ILEC) almost certainly does not have a presence in a "common Internet exchange". that Netflix talks about for peering. Those types of facilities are often were several small and mid tier ISP have their facilities and peer local companies and upstream providers (i.e. Dallas DataMart) . Verizon almost certainly has it's own data centers and paid a significant amount of money to lay fiber to various upstream providers and have the ability to peer and interconnect locally using a large ATM Sonet Ring.

Verizon bought UUnet, and sells services in these facilities under their wholesale bandwidth side. Internal politics may necessitate the residential/non-wholesale side to "buy" bandwidth from them but I don't know.

The other problem is the "Free" hardware has a cost to the ISP. Netflix isn't the first CDN to have hosted machines that is a localized proxy. CDN's have been hosting machines at ISPs since the late 90's. What is different is Netflix expects the ISP to bear the costs of hosting the equipment. Most CDN's pay for power and a discounted rate for rack space. Some pay more if they put a Satellite dish on the roof. At the very least Netflix should be willing to pay for power to the equipment (which is quite a bit more than at home since it's online battery backed + generators).

I would say the benefits outweigh the cost here, but I work for a much smaller municipal ISP that already peers with Netflix. When looking at price per mbps, having local offloaded traffic pays for itself in reducing internal network congestion.

This could be a great time for CDNs to start offering proxy services to end users so the end users can circumvent ISP's purposefully bad routing.

As an end customer to an ISP, you don't get to control the routing. VPN service would only help if the ISP was traffic shaping, filtering, or otherwise screwing with your traffic based on traffic type or source/destination. Routing around a bottleneck is just not possible for an end user, short of changing ISPs. And most of us don't have that luxury.

I knew as soon as I saw the first story about this that it would just end up being a peering dispute. As others have said peering is usually a situation where it is mutually beneficial and traffic both ways is comparable. When it gets unbalanced it becomes transit and one side usually pays the other. Traditionally a CDN would be the one paying for transit which in this situation would be Cogent paying Verizon. I know some people see it as Verizon double dipping but Cogent is getting paid too. Netflix is paying them lots of money to deliver this content as a CDN and traditionally the CDN ends up paying some of that to the large providers that they peer with or colocate with etc.

This could be a great time for CDNs to start offering proxy services to end users so the end users can circumvent ISP's purposefully bad routing.

As an end customer to an ISP, you don't get to control the routing. VPN service would only help if the ISP was traffic shaping, filtering, or otherwise screwing with your traffic based on traffic type or source/destination. Routing around a bottleneck is just not possible for an end user, short of changing ISPs. And most of us don't have that luxury.

Well, if the route out of the ISP is via links that aren't as congested it could actually help. You can't control the routing, but you can abuse it if you can figure out what it is.

Ahh the Verizon Bean Counters just clued in to the fact they are missing out on a substantial amount of revenue by not double dipping for the Netflix data they are peering. They already get paid on the client side now they've discovered and want more from the Host connection side too. Its as much about impairing /impeeding NETFLIX as it is about maximizing revenue from each byte sent or received but its the same ol shit different day.

Hey everyone. Just want to point out that I've added an update to the story:

UPDATE: In a phone interview, Cogent CEO Schaeffer provided some additional details to Ars. He noted that the amount of traffic swapped between Cogent and Verizon is unequal because of the traffic requested by Verizon customers who try to watch Netflix streaming video. "The very nature of [Verizon's] customer base makes it impossible" to make the amounts of traffic exchanged between Cogent and Verizon equal, he said.

Cogent is Netflix's primary Internet service provider, and thus plays a central role in bringing Netflix's content to end users. However, ISPs like Verizon that sell to homes are the ones that carry that traffic across the proverbial last mile. Thus, the connections between Cogent and ISPs that serve consumers are crucial. "We're not sending anything to Verizon that their customers don't ask for," he said.

Adding a port (each port provides another 10Gbps of bandwidth) costs about $10,000 each for Cogent and Verizon. That's a small amount of money for Verizon, Schaeffer said. He said he believes that Verizon's decision to limit Netflix bandwidth is due to Verizon wanting customers to use its own streaming video services, such as Redbox Instant.

Schaeffer said that other ISPs who offer content competitive to Netflix have also been slow to add ports to handle Netflix traffic. However, Schaeffer said Verizon has been the worst offender.

The mood affiliation in these threads is always interesting. Everyone likes Netflix as a product, so of course they can't do wrong.

The bottom line is, if Netflix is consuming so much bandwidth, then they need to pay for it. Left unsaid in a lot of this is that I believe that Cogent is getting some revenue (indirectly through a CDN) from Netflix. Now, Netflix will happily allow Verizon to give them free bandwidth, but why the hell should Verizon take that deal?

The issues are pretty simple, and we've been down this road before with Netflix. If Cogent's peering is asymmetric, they need to pay to balance it out, or else be happy with the performance they're getting. This stuff ain't free, and one way or another, the money for this is gonna come from our pockets, as consumers, either through higher ISP fees, or in higher Netflix fees.

This is exactly why vertical integration is so dangerous: When you start owning the entire content chain, it's easy to abuse the power inherent in that relationship to try to force a monopoly. Even if that's not actually happening here, it's easy for people to think that Verizon is using abusive tactics.

Honestly, I'm very surprised that Verizon is not already co-locating NetFlix boxes; that seems like a no-brainer when NetFlix already accounts for a huge portion of Internet traffic.

The bottom line is, if Netflix is consuming so much bandwidth, then they need to pay for it. Left unsaid in a lot of this is that I believe that Cogent is getting some revenue (indirectly through a CDN) from Netflix. Now, Netflix will happily allow Verizon to give them free bandwidth, but why the hell should Verizon take that deal?

NetFlix isn't consuming the bandwidth, the customers are.

If it happens that 50 or 60% of Verizon's customers are all getting data from the same content provider, then it's incumbent on Verizon to open a wider path to that provider. That's called servicing your customers.

Hey everyone. Just want to point out that I've added an update to the story:

UPDATE: In a phone interview, Cogent CEO Schaeffer provided some additional details to Ars. He noted that the amount of traffic swapped between Cogent and Verizon is unequal because of the traffic requested by Verizon customers who try to watch Netflix streaming video. "The very nature of [Verizon's] customer base makes it impossible" to make the amounts of traffic exchanged between Cogent and Verizon equal, he said.

Cogent is Netflix's primary Internet service provider, and thus plays a central role in bringing Netflix's content to end users. However, ISPs like Verizon that sell to homes are the ones that carry that traffic across the proverbial last mile. Thus, the connections between Cogent and ISPs that serve consumers are crucial. "We're not sending anything to Verizon that their customers don't ask for," he said.

Adding a port (each port provides another 10Gbps of bandwidth) costs about $10,000 each for Cogent and Verizon. That's a small amount of money for Verizon, Schaeffer said. He said he believes that Verizon's decision to limit Netflix bandwidth is due to Verizon wanting customers to use its own streaming video services, such as Redbox Instant.

Schaeffer said that other ISPs who offer content competitive to Netflix have also been slow to add ports to handle Netflix traffic. However, Schaeffer said Verizon has been the worst offender.

How do we fight this? There are only so many ISP's. I can go to Verizon or Comcast but in the end both are going to want to limit access to Netflix. The only TV I watch is Netflix.

Is anyone surprised by this? Verizon has a long history of such customer-hostile behavior. Don't believe it? Feel free to search for information about Verizon disabling Bluetooth file transfer capabilities on their cell phones so their customers would have to use their extra-cost media transfer services.

Bottom line, this kind of anti-competitive stuff is going to continue to happen until something changes to separate ISPs from content owners/providers. Any ISP who also provides content services is always going to find a way to favor those services over third parties, in order to capture that revenue. It's basic modern business.

And unfortunately whatever happens to change this situation is going to have to be technical, in the area of making it less expensive to provide "last mile" broadband Internet service to large numbers of homes. It won't be government regulations or anti-trust actions that break up the ISP/MVPD/studio conglomerates. Politicians and bureaucrats are far too reliant upon the campaign contributions, cushy lobbyist jobs, etc. to allow that to happen.

This is why ISPs need to be regulated like electric companies. They should be forced to be dumb pipes that sell nothing other than internet access.

This. This.This. THIS!

To all the major broadband providers, esp. those who run it over coax:

I don't CARE about your damn cable/VoIP/streaming/whatever services. All I am paying you for is a dumb pipe. That's all you should be. You, like the electric company, should have nothing to do with how I choose to use your service - you should only provide the service as I pay for it, and fix any problems that arise with regard to providing said service.

How do we turn this into a real movement? I need the right wording for petitions and letters to Congress, along with advertising to get people to join the cause. Unless there already is something like this that I can support.

How do we fight this? There are only so many ISP's. I can go to Verizon or Comcast but in the end both are going to want to limit access to Netflix. The only TV I watch is Netflix.

This is really unfair.

The problem is that this is a losing proposition for the carriers. Cord cutters will use more Internet than people who subscribe to regular TV services, and this is forcing carriers to work to keep up. In my case, my Internet bill is roughly 1/3 of what my Internet+TV bill used to be, so I'm asking my carrier to work harder on my behalf for less money.

Barring some new, disruptive influence, I think the long-term "solution" is going to be entirely in the carriers' behavior: They will bundle more services as standard, hiking the prices as they go. In the end, you'll pay something like $200 a month for your "one size fits all" package, and the only way to get pure data service will be something that's tuned exactly wrong for video, telephony, and audio streaming.

I can see two ways this could change:1. Add more dedicated Internet carriers into the mix, companies that don't own media companies or content providers. Actual competition would force the big carriers to act a little more proactively.

2. Go back to the old Ma Bell system, where the telephone company was only allowed to provide connectivity: disallow broadband providers from also owning content providers. This would mean Verizon couldn't own a NetFlix competitor, and it would mean Time Warner could not own both a cable company and a TV network. In short, you can create the content or you can deliver the content to the user, but you can't do both.

The mood affiliation in these threads is always interesting. Everyone likes Netflix as a product, so of course they can't do wrong.

The bottom line is, if Netflix is consuming so much bandwidth, then they need to pay for it. Left unsaid in a lot of this is that I believe that Cogent is getting some revenue (indirectly through a CDN) from Netflix. Now, Netflix will happily allow Verizon to give them free bandwidth, but why the hell should Verizon take that deal?

The issues are pretty simple, and we've been down this road before with Netflix. If Cogent's peering is asymmetric, they need to pay to balance it out, or else be happy with the performance they're getting. This stuff ain't free, and one way or another, the money for this is gonna come from our pockets, as consumers, either through higher ISP fees, or in higher Netflix fees.

Netflix isn't consuming bandwidth. Netflix's subscribers are consuming bandwidth. They've already paid. Twice, even, every month - once to netflix for access to netflix, and again to the ISP for the ability to consume bandwidth.

Its great that they also work with ISPs to keep speed and reliability up, but Verizon isn't giving Netflix any bandwidth, they're providing it to their customers who have paid for it. (I know I'm beating a dead horse now as far as this comment thread is concerned - but apparently someone isn't reading any of them.)

Pretty clear conflict of interest here - I also think service providers should not be allowed to be content providers.

Netflix should just have their video player spew spam back to the Netflix servers. That would balance out the send/receive load on the ISP network and return the peering relationship back to balance. :-D

Actually, a lot of ISPs that have a peering connection that requires a ratio, use traffic generators to make fake traffic. Which is another reason why ratios are a horrible way to base peering. With so many different routes on the Internet, there are many ways to game the system. Keep it simple, if it benefits you, do it.

This could be a great time for CDNs to start offering proxy services to end users so the end users can circumvent ISP's purposefully bad routing.

As an end customer to an ISP, you don't get to control the routing. VPN service would only help if the ISP was traffic shaping, filtering, or otherwise screwing with your traffic based on traffic type or source/destination. Routing around a bottleneck is just not possible for an end user, short of changing ISPs. And most of us don't have that luxury.

Well, if the route out of the ISP is via links that aren't as congested it could actually help. You can't control the routing, but you can abuse it if you can figure out what it is.

Exactly, and I don't put it past Verizon to traffic shape in some way to make it worse and calling it 'network management'. Such as forcing all netflix data through a specific connection and not letting it get routed around the congested connection which they would call 'network management' because they don't want that netflix data filling up other connections, and just so happens make a service they don't like worse for Verizon's customers.

It's a bit more complicated than just money. ...The other problem is the "Free" hardware has a cost to the ISP. Netflix isn't the first CDN to have hosted machines that is a localized proxy. CDN's have been hosting machines at ISPs since the late 90's. What is different is Netflix expects the ISP to bear the costs of hosting the equipment. Most CDN's pay for power and a discounted rate for rack space. Some pay more if they put a Satellite dish on the roof. At the very least Netflix should be willing to pay for power to the equipment (which is quite a bit more than at home since it's online battery backed + generators).

If it was power, backed by batteries, generators et al., and even if disputed costs include rack space at the ISP, then it would be a non-issue. Powering the free equipment is negligible compared to the costs for reaching and maintaining the reach to the ISP.

I can say that most of them had five figuring hosting bills per month. That's for many data center sites nationally and it represented a discount compared to a company buying Us of rack at list.

The hosting fees are marginal and any CDN would be willing to pay for pure hosting fees if so required, unless, the ISP is secretly billing for infrastructure fees which seems to be (at least) the case here.

If Verizon and Netflix are meeting at an independent peering location then Cogent should certainly not be entertaining the thought of having to peer equal amounts of traffic. Because Cogent happens to be providing Netflix traffic and is a transit provider at the same time, Verizon is nefariously attempting to confuse people by arguing symmetric levels of traffic exchange required for peers, while any ISP simply has to pay for upstream or else transit traffic to serve their customers.

The crux is to understand that upstream traffic is different from the Netflix traffic being carried to the ISP by Cogent. If Verizon wanted free peering over Cogent's existing Netflix peering ports back to Netflix, and only Netflix, then their argument would be sound, but that's not the case here. Verizon is effectively asking Netflix to finance Verizon's upstream traffic, while Verizon's subscribers have already financed Verizon's downstream and upstream traffic.

Verizon should be happy Cogent is bringing the high levels of Netflix traffic close to the ISP (free for Verizon nonetheless), but Verizon (or Comcast for that matter), wants to have its cake and eat it too.

Hey everyone. Just want to point out that I've added an update to the story:

UPDATE: In a phone interview, Cogent CEO Schaeffer provided some additional details to Ars. He noted that the amount of traffic swapped between Cogent and Verizon is unequal because of the traffic requested by Verizon customers who try to watch Netflix streaming video. "The very nature of [Verizon's] customer base makes it impossible" to make the amounts of traffic exchanged between Cogent and Verizon equal, he said.

Cogent is Netflix's primary Internet service provider, and thus plays a central role in bringing Netflix's content to end users. However, ISPs like Verizon that sell to homes are the ones that carry that traffic across the proverbial last mile. Thus, the connections between Cogent and ISPs that serve consumers are crucial. "We're not sending anything to Verizon that their customers don't ask for," he said.

Adding a port (each port provides another 10Gbps of bandwidth) costs about $10,000 each for Cogent and Verizon. That's a small amount of money for Verizon, Schaeffer said. He said he believes that Verizon's decision to limit Netflix bandwidth is due to Verizon wanting customers to use its own streaming video services, such as Redbox Instant.

Schaeffer said that other ISPs who offer content competitive to Netflix have also been slow to add ports to handle Netflix traffic. However, Schaeffer said Verizon has been the worst offender.

For those that are not in America. Why can't users subscribe to Cogent as their ISP? (ie use the same ISP as Netflix)

For those that are not in America. Why can't users subscribe to Cogent as their ISP? (ie use the same ISP as Netflix)

You're suggesting that because line-sharing has not rooted in the US that the rest of the world defaults to line-sharing, which is not entirely the case.

In any case, you need some level of separation and associated competition. Think of movies where the content producers (Paramount and Warner Bros being Google et al.) have been legally separated from the content distributors (Lions Gate and Tristar being transit providers) and the content providers (Verizon and Comcast being ISPs).

Competition and diversity is not just some ideological principal. It's what Darwinism teaches us is the best for the ecosystem as a whole.

It's a bit more complicated than just money. ...The other problem is the "Free" hardware has a cost to the ISP. Netflix isn't the first CDN to have hosted machines that is a localized proxy. CDN's have been hosting machines at ISPs since the late 90's. What is different is Netflix expects the ISP to bear the costs of hosting the equipment. Most CDN's pay for power and a discounted rate for rack space. Some pay more if they put a Satellite dish on the roof. At the very least Netflix should be willing to pay for power to the equipment (which is quite a bit more than at home since it's online battery backed + generators).

If it was power, backed by batteries, generators et al., and even if disputed costs include rack space at the ISP, then it would be a non-issue. Powering the free equipment is negligible compared to the costs for reaching and maintaining the reach to the ISP.

I can say that most of them had five figuring hosting bills per month. That's for many data center sites nationally and it represented a discount compared to a company buying Us of rack at list.

The hosting fees are marginal and any CDN would be willing to pay for pure hosting fees if so required, unless, the ISP is secretly billing for infrastructure fees which seems to be (at least) the case here.

If Verizon and Netflix are meeting at an independent peering location then Cogent should certainly not be entertaining the thought of having to peer equal amounts of traffic. Because Cogent happens to be providing Netflix traffic and is a transit provider at the same time, Verizon is nefariously attempting to confuse people by arguing symmetric levels of traffic exchange required for peers, while any ISP simply has to pay for upstream or else transit traffic to serve their customers.

The crux is to understand that upstream traffic is different from the Netflix traffic being carried to the ISP by Cogent. If Verizon wanted free peering over Cogent's existing Netflix peering ports back to Netflix, and only Netflix, then their argument would be sound, but that's not the case here. Verizon is effectively asking Netflix to finance Verizon's upstream traffic, while Verizon's subscribers have already financed Verizon's downstream and upstream traffic.

Verizon should be happy Cogent is bringing the high levels of Netflix traffic close to the ISP (free for Verizon nonetheless), but Verizon (or Comcast for that matter), wants to have its cake and eat it too.

ISPs pay the same for bandwidth, no matter the direction. The main difference between upstream and downstream is that you can typically host hundreds to thousands of customers on 10Gb of downstream, but it only takes a few heavy users to saturate the upstream, effectively doubling their bandwidth bill.

You also run the issue that only a select few services consume most of your downstream, like youtube/netflix/etc. You can optimize your heavy downstream by getting good deals for specific peering relations to get these services. But if a few customers start uploading to random people all over the Internet, then a large portion of that bandwidth will be going over the default link, which is more expensive.

The only thing that makes uploading more expensive is the usage patterns.

The problem with Verizon is that they are both an ISP and a Tier1. Normally, an ISP will do anything to get peering, but in this case, Verizon is a Tier1 and doesn't want to peer unless it really has to.

It's a conflict of interest. The Interest of the ISP is to best serve the customers, the interest of a Tier1 is to squeeze as much money out of ISPs as possible.

When the two merge, you get the worst of both worlds, an ISP that tries to squeeze as much money out of both "peers" and customers, while trying to give customers as little as possible.

If Verizon can't control themselves, then they should be forced to split.

Hey everyone. Just want to point out that I've added an update to the story:

UPDATE: In a phone interview, Cogent CEO Schaeffer provided some additional details to Ars. He noted that the amount of traffic swapped between Cogent and Verizon is unequal because of the traffic requested by Verizon customers who try to watch Netflix streaming video. "The very nature of [Verizon's] customer base makes it impossible" to make the amounts of traffic exchanged between Cogent and Verizon equal, he said.

Cogent is Netflix's primary Internet service provider, and thus plays a central role in bringing Netflix's content to end users. However, ISPs like Verizon that sell to homes are the ones that carry that traffic across the proverbial last mile. Thus, the connections between Cogent and ISPs that serve consumers are crucial. "We're not sending anything to Verizon that their customers don't ask for," he said.

Adding a port (each port provides another 10Gbps of bandwidth) costs about $10,000 each for Cogent and Verizon. That's a small amount of money for Verizon, Schaeffer said. He said he believes that Verizon's decision to limit Netflix bandwidth is due to Verizon wanting customers to use its own streaming video services, such as Redbox Instant.

Schaeffer said that other ISPs who offer content competitive to Netflix have also been slow to add ports to handle Netflix traffic. However, Schaeffer said Verizon has been the worst offender.

For those that are not in America. Why can't users subscribe to Cogent as their ISP? (ie use the same ISP as Netflix)

Ive seen the Sandvine reports and the other half of that streaming is YouTube. Google also offers to in-house their CDN caching servers to save on transit bandwidth. The ISP I work for has done this a long time ago. So my question for Verizon would be, do you have Google caching servers? And if the answer is yes then clearly this is a conflict since they have the option to resolve the Netflix issue in the same manner.

We are seeing old school cable mentality. They're hanging onto their video business as much as they can. Id rather see the cableco's start partnering with Netflix, Apple and Google etc to provide next gen video services that have the potential to transform the market and gain a competitive edge. but at least for now theyre going into protectionism mode just like the wireless providers did by banning Skype over HSPA in the early days.

Its only a matter of time before video takes the same path. Dumb pipe here we come!

Hey everyone. Just want to point out that I've added an update to the story:

UPDATE: In a phone interview, Cogent CEO Schaeffer provided some additional details to Ars. He noted that the amount of traffic swapped between Cogent and Verizon is unequal because of the traffic requested by Verizon customers who try to watch Netflix streaming video. "The very nature of [Verizon's] customer base makes it impossible" to make the amounts of traffic exchanged between Cogent and Verizon equal, he said.

Cogent is Netflix's primary Internet service provider, and thus plays a central role in bringing Netflix's content to end users. However, ISPs like Verizon that sell to homes are the ones that carry that traffic across the proverbial last mile. Thus, the connections between Cogent and ISPs that serve consumers are crucial. "We're not sending anything to Verizon that their customers don't ask for," he said.

Adding a port (each port provides another 10Gbps of bandwidth) costs about $10,000 each for Cogent and Verizon. That's a small amount of money for Verizon, Schaeffer said. He said he believes that Verizon's decision to limit Netflix bandwidth is due to Verizon wanting customers to use its own streaming video services, such as Redbox Instant.

Schaeffer said that other ISPs who offer content competitive to Netflix have also been slow to add ports to handle Netflix traffic. However, Schaeffer said Verizon has been the worst offender.

For those that are not in America. Why can't users subscribe to Cogent as their ISP? (ie use the same ISP as Netflix)

Cogent only sells to large businesses and in limited areas.

A better way to explain it is you need to peer with them at an exchange point with 10Gb ports and use several Gb of bandwidth. No single person could do that. You need to be large to do that.

i'll bet that verizon wants a direct conect to netflix... if effect a free conduit to netflix, ie all of the content to verizon is free. But if Cogent ever agreed to that, verizon would abuse that by rerouting all of their data though Cogent to verizons customers. (because verizon does not want to build out it's network, to provide for netflix, and perhaps is why they don't want the open connect boxes from netflix. in effect making verizon deal with the traffic not Cogent...)

To be fair, it's unusable with almost any ISP. Back when I lived at home and had Time Warner, I had the same problem with YT as I do now with FIOS.

Google's CDN just sucks, universally.

While I do periodically find a video that has "buffering issues", I almost never have issues with YouTube. They send data very fast; but my ISP does absolutely no throttling.

My last ISP used to be pretty good with Netflix and would only buffer only once or twice per night. My new ISP hasn't buffered Netflix once in over 2 months.

Switching from an incumbent ISP to a neutral ISP with no throttling has made the Internet totally different. It is no longer a place that pauses and stutters. It's most like having everything on a LAN.

I used to think my 30ms pings were nice, but now I get low teens with no jitter, even during peak.

Verizon is a cock-up of a tech organization, who also (apparently) regularly throttles/can't deliver on non-VZ services. They're a bunch of greedy, half-competent douchebags. I wish they'd fall off the earth, and several other more competent providers would spring up from the ashes (I say this as a FIOS user).

A good bar bet is always to offer a Verizon phone use that they should compare phones with you. Whichever phone person has the most amount of ISP names on their phone, pays the other person $100 bucks per ISP name found on the found.

Verizon will always have their name pasted or displayed on the phone more times than any other ISP. When I used to have Verizon, it did astound me that they had their name on my phone in 4 places. Then a change happened in their software and the home page was changed. Voila. I then got to see the Verizon name on the home page of my display screen as they had added it to every screen's display. And of course this was not editable. They just won't get the f out of your face. Money, money, money. They took so f'ing much of my money that I NEVER EVER will go back to them regardless of my options.

To expand on my last post, there is a whole other Internet out there once you get away from incumbents.

I grew up on 9600bps dial-up and have been used to being "patient" and overall just saying "well, that's the Internet for you" when ever something bad happened.

My last ISP, Charter, was actually quite good. I got my rated bandwidth with most of my services, buffering mostly happened during peak hours and I still had my rated bandwidth to other services so it must have been an issue with Netflix or YouTube,I had few connection issues to my node, my ping to Chicago was about 20ms, but could jitter up to 25ms during peak. And when connection issues to other services happened, it always seemed that the packetloss and high pings were far down the line of hops on another network.

To me, Charter was just fine and connection issues were just a "normal" part of the Internet as they always seemed to happen on other networks.

Well guess what. That is NOT normal. I just switched ISPs to one that has a net neutral policy. They also use Level 3 Comm, which also has a very net neutral stance and evangelize that bandwidth is cheaper and easier to manage than traffic shaping.

Now I get 10ms pings to Chicago with 1 ms of jitter, I no longer get those "issues on other networks", buffering is a thing of the past.

Nearly every issue that I used to think was "just part of the Internet"... They're gone!

Jitter, congestion, packet-loss, latency of almost any kind that affects your services. All of that stuff is anything but normal. Every time you are playing a game and get a lag spike, remind yourself, your ISP is providing sub-par services.

The Internet is no longer the Internet of yore. It is like a LAN, and if you're getting anything less, you're not getting what you're paying for.

The last time I was this shocked about what I have been missing out on, was when I purchased my Voodoo2 and got 3d-acceleration for the first time.

Hey everyone. Just want to point out that I've added an update to the story:

UPDATE: In a phone interview, Cogent CEO Schaeffer provided some additional details to Ars. He noted that the amount of traffic swapped between Cogent and Verizon is unequal because of the traffic requested by Verizon customers who try to watch Netflix streaming video. "The very nature of [Verizon's] customer base makes it impossible" to make the amounts of traffic exchanged between Cogent and Verizon equal, he said.

Cogent is Netflix's primary Internet service provider, and thus plays a central role in bringing Netflix's content to end users. However, ISPs like Verizon that sell to homes are the ones that carry that traffic across the proverbial last mile. Thus, the connections between Cogent and ISPs that serve consumers are crucial. "We're not sending anything to Verizon that their customers don't ask for," he said.

Adding a port (each port provides another 10Gbps of bandwidth) costs about $10,000 each for Cogent and Verizon. That's a small amount of money for Verizon, Schaeffer said. He said he believes that Verizon's decision to limit Netflix bandwidth is due to Verizon wanting customers to use its own streaming video services, such as Redbox Instant.

Schaeffer said that other ISPs who offer content competitive to Netflix have also been slow to add ports to handle Netflix traffic. However, Schaeffer said Verizon has been the worst offender.

For those that are not in America. Why can't users subscribe to Cogent as their ISP? (ie use the same ISP as Netflix)

Cogent only sells to large businesses and in limited areas.

A better way to explain it is you need to peer with them at an exchange point with 10Gb ports and use several Gb of bandwidth. No single person could do that. You need to be large to do that.

I was just talking to the Cogent CEO yesterday. He said they have two businesses. One is the one you just described. The other is being an ISP to large businesses in the same way Verizon is the ISP to home customers (that's what I was referring to in my comment). This more traditional ISP part of its business accounts for half of Cogent's revenue, but only 3 percent of its traffic.

I was just talking to the Cogent CEO yesterday. He said they have two businesses. One is the one you just described. The other is being an ISP to large businesses in the same way Verizon is the ISP to home customers (that's what I was referring to in my comment). This more traditional ISP part of its business accounts for half of Cogent's revenue, but only 3 percent of its traffic.

No, Jon, nooooooooooooo.

You've summoned him. Soon he will be here to lecture us on the "real" costs of running a last mile ISP and question why everyone isn't running their own ISP if it is so easy and profitable (which is actually a good question, mind you).

But yes, anecdotally many of us are aware that there are large profit margins in running fiber and providing last-mile connectivity for businesses, particularly large ones with modest but highly fungible bandwidth needs like hospitals, medical centers, CDNs, etc.

I was just talking to the Cogent CEO yesterday. He said they have two businesses. One is the one you just described. The other is being an ISP to large businesses in the same way Verizon is the ISP to home customers (that's what I was referring to in my comment). This more traditional ISP part of its business accounts for half of Cogent's revenue, but only 3 percent of its traffic.

No, Jon, nooooooooooooo.

You've summoned him. Soon he will be here to lecture us on the "real" costs of running a last mile ISP and question why everyone isn't running their own ISP if it is so easy and profitable (which is actually a good question, mind you).

But yes, anecdotally many of us are aware that there are large profit margins in running fiber and providing last-mile connectivity for businesses, particularly large ones with modest but highly fungible bandwidth needs like hospitals, medical centers, CDNs, etc.

Being a small ISP is anything but "large profit margins". Bandwidth approaches free as you get really large, but until then, you pay through the nose.

You can purchase 1.544Mb/s of dedicated bandwidth for $600/month, which is almost $400/mbit, or you can purchase 10Gb of bandwidth for $10k/month, which is $1/mbit.

Capital costs and demand are two huge barriers to making decent bank from being an ISP.

Of course there are other fees to the 10Gb of bandwidth, like leasing dark-fiber or burying your own and purchasing a 10Gb port at your upstream provider, but the bandwidth is dirt cheap.

I was just talking to the Cogent CEO yesterday. He said they have two businesses. One is the one you just described. The other is being an ISP to large businesses in the same way Verizon is the ISP to home customers (that's what I was referring to in my comment). This more traditional ISP part of its business accounts for half of Cogent's revenue, but only 3 percent of its traffic.