Netflix packets being dropped every day because Verizon wants more money

Verizon wants to be paid by consumers and Cogent, but Cogent refuses to pay.

The battle over who should pay to carry Netflix traffic is heating up again, and one of the main players blames Verizon's greed for the poor performance that many consumers see when trying to watch streaming video.

Cogent Communications CEO Dave Schaeffer made his case in an interview with Ars yesterday, saying Verizon is refusing to upgrade the infrastructure that carries Internet traffic from one network to another unless outrageous demands for payment are met.

The network connections between Cogent and Verizon, crucial for carrying streaming video and other content to Verizon's home Internet customers, "are full," Schaeffer said. "They are more than full. They are so full that today a significant amount of packets are being dropped between the networks."

First, some background. Cogent is an Internet bandwidth provider that sells transit to Netflix and other companies. When Netflix purchases transit from Cogent, Cogent is responsible for distributing the traffic to all corners of the Internet. But no single company controls the entire Internet. Thus, Cogent must exchange traffic with other network providers, including Verizon.

Further Reading

The connections between Cogent and Verizon take the form of peering. It is a point-to-point connection that doesn't necessarily guarantee passage of traffic to any networks beyond the two involved in the deal. Peering generally happens without any money changing hands, particularly if the two companies involved are of similar size and influence.

Theoretically, the peering arrangement between Cogent and Verizon benefits both. Cogent gets to distribute Netflix traffic, as Netflix pays it to do, and Verizon ensures that its customers, who pay Verizon to access the Internet and are demanding to watch Netflix movies, get the content they request.

Verizon wants to ditch the "settlement-free" peering model and get money from Cogent, arguing that it has to accept far more traffic from Cogent than vice versa because of high-bandwidth applications like Netflix.

Cogent has refused to pay. As negotiations stall, Netflix performance has dropped measurably for months on both Verizon and Comcast. Cogent claims this is because Verizon and others—but especially Verizon—are refusing to upgrade the connections between networks. Cogent points out that Verizon offers its own streaming video services, such as Redbox Instant, and thus has an incentive to harm Netflix traffic.

There are about 11 Cogent/Verizon peering connections in major cities around the country. When peering partners aren't fighting, they typically upgrade the connections (or "ports") when they're about 50 percent full, Cogent says. They can do this by adding ports, adding capacity to ports, or peering in new locations.

With Cogent and Verizon fighting, the upgrades are happening at a glacial pace, according to Schaeffer.

"Once a port hits about 85 percent throughput, you're going to begin to start to drop packets," he said. "Clearly when a port is at 120 or 130 percent [as the Cogent/Verizon ones are] the packet loss is material."

The congestion isn't only happening at peak times, he said. "These ports are so over-congested that they're running in this packet dropping state 22, 24 hours a day. Maybe at four in the morning on Tuesday or something there might be a little bit of headroom," he said.

Slower traffic, and not just on Netflix

Dropped packets can be re-sent, but users could see slower loading times or page load failures. This problem is particularly bad with streaming video because it requires so much bandwidth, with Netflix alone accounting for more than 30 percent of downstream Internet traffic at peak usage times. It does affect all traffic, however.

"If you've got VoIP and you're trying to run Skype, it may not work because the pipe was full, with your neighbor down the street trying to watch Netflix," Schaeffer said. "It's not only the video user. Every Internet user is suffering today in their ability to access all the applications, content, and other users across the Internet."

This isn't the kind of problem consumers can solve by purchasing a faster Internet connection, because they'd be paying only for a faster path to Verizon's network, not a faster path to the rest of the Internet.

In some cases, Verizon has actually purchased and installed the necessary equipment to upgrade ports, but not turned it on, according to Schaeffer. "They actually put it in, so they spent the money, but they just politically have not been willing to turn it on in order to ensure that Netflix will not work as well as Redbox," he said.

When Netflix can't get through to Cogent, it sends traffic through other providers, but Schaeffer claims that "everybody is full."

So what are they fighting over? Schaeffer said Verizon "wants a price that is about 10 times the market price for transit." Again, transit is a service that takes traffic across the entire Internet, whereas peering is just a connection between two networks.

"It's a less robust product at ten times the price," Schaeffer said.

Cogent sells transit for an average price of $1.31 per megabit, though large volume users like Netflix get discounts.

Verizon responds: We don't degrade Netflix to help Redbox

Verizon Senior VP of Public Policy Craig Silliman spoke to Ars today, saying that Cogent is unique in taking such an inflexible stance in negotiations.

There is a wide range of Internet interconnection agreements, he said. "We have settlement-free agreements. There are some ISPs to whom we pay money and there are others who pay money to us," Silliman said. "There is a whole range of commercial options, and they get worked out commercially and smoothly, except for this one ISP who seems to have problems with not just us but a lot of others."

Silliman added that "the whole premise of settlement-free peering is that you have a roughly equal exchange of traffic." He did not say whether any of Netflix's transit providers have agreed to pay Verizon, noting that specific commercial arrangements are confidential.

He also did not reveal how much money Verizon is asking for, saying, "we are open to negotiation for a commercially reasonable solution that works for both parties."

Further Reading

Silliman said he'd have to check with Verizon's engineers to respond to Cogent's allegations regarding ports being more than 100 percent full and Verizon allegedly installing equipment but refusing to turn it on.

He did, however, say, "it is categorically false that we are doing anything to adversely impact Netflix traffic to benefit Redbox."

A recent Wall Street Journal report said that "[e]xecutives at major broadband providers ... privately blame the traffic jam on Netflix's refusal to distribute its traffic more efficiently."

Silliman noted that Cogent has been at the center of public disputes over peering more often than any other company. "It's noteworthy that Cogent is always the one in the middle of these disputes, and they tend to use the media as negotiating leverage," he said.

Cogent digs in for a long standoff

Schaeffer didn't deny that Cogent has fought more public peering battles than anyone. One reason for that is "we only have one business, the Internet. Everybody else, the Internet is a sideline and they've got all these other businesses, but they're kind of internally conflicted. They're not willing to fight."

Level 3 has also been in Cogent's position, fighting Comcast over Netflix traffic in 2010. Level 3 has complained about the sway consumer ISPs hold because they face so little competition and control the only network path to consumers. The company has argued that peering arrangements should be calculated based on "bit miles," the distance traffic is carried and the number of bits carried, regardless of which direction the traffic flows.

Further Reading

Cogent said its battle with Verizon isn't unique. Comcast and Time Warner Cable are demanding payments similar to the ones Verizon wants, and Cogent's connections with those ISPs have hit the "red zone" of 85 percent throughput, Schaeffer alleged.

Level 3 and Comcast declined to comment when contacted by Ars. A Time Warner Cable spokesperson told Ars, "It’s a business discussion with Cogent, and we don’t discuss our business dealings in the press." Executives at major broadband providers, meanwhile, privately blame the traffic jam on Netflix's refusal to distribute its traffic more efficiently.

Schaeffer said Cogent isn't willing to pay any price for peering with Verizon.

"We believe firmly in bill and keep," he said. "We bill our customers and we connect them to the Internet, and it's our job to deliver to them the best quality possible. Verizon is billing its customers, and it's got a lot more of them and it bills a hell of a lot more, and they're not committed to delivering their customers what they promise the customers they're going to deliver."

Once you pay anything for peering, "it's a slippery slope in two respects," Schaeffer said. "One, everyone wants to get paid then. And then two, once you pay it's like blackmail, they've got you, there's nowhere else to go. They'll just keep raising the price in a market where prices [for transit] are falling."

Netflix attempts to peer directly with ISPs through its Open Connect peering and caching program. British Telecom, Cablevision, Google Fiber, RCN, and others have accepted the offer, but Verizon hasn't. Netflix declined to provide comment to Ars about disputes with Verizon, but Schaeffer said Verizon has demanded money in exchange for peering with Netflix.

"Netflix is free to do whatever they want," Schaeffer said. "I think if Verizon sold them a comparable product at an equal or lower price to Cogent, they should take it. But what Verizon is saying is, 'pay us 10 times as much to get a less robust product, but it's only to get to our customers, and that's the only way you can get to them,' and Netflix understands that's closer to blackmail than a market transaction."

Entrepreneurs trying to become the next Netflix will have a rough time if they and their transit providers have to pay for entry into consumers' homes, Schaeffer said.

"We're committed to driving down the price of Internet," Schaeffer asserted. "If you allow people to impose monopoly taxes on Internet service, it's going to end up resulting in higher prices and lower quality."

Schaeffer said he isn't opposed to senders paying for traffic, but not if it's in addition to consumers paying. "I didn't see Verizon roll out a product that says, 'well if Netflix pays to send content to you, you've got free broadband connectivity,'" he said.

It might get worse, but is there a shred of hope?

The Federal Communications Commission hasn't regulated the peering market, although the topic could come up during proceedings on the proposed Comcast/Time Warner Cable merger. FCC Chairman Tom Wheeler has supported the idea of "two-sided networks" in which ISPs like Verizon can charge both Netflix and their home customers. Wheeler wants to impose some type of net neutrality rules on Internet access, but that would apply only to the "last mile" of connectivity from consumer ISPs to homes, and not the interconnection agreements at issue here.

In Europe last July, the European Commission opened an antitrust probe into whether Orange, Deutsche Telekom, and Telefónica abused market positions in negotiations with content providers. The investigation, which led to a search of those ISPs' offices, was spurred by Cogent's complaints.

Schaeffer thinks the US government will eventually intervene. Individual consumers can't do much to fight, but as peering controversies get more attention and inspire customer outrage, state prosecutors could intervene, he believes.

"I think there will be more press in major mainstream publications. I think there will be more customer ire," he said. "I think service qualities will degrade, and I believe eventually some states' attorney generals will probably step in and sue on behalf of the consumers."

Promoted Comments

I don't get how they can do this, and then with a straight face, offer you higher speed packages. What's the point of getting a 50/5 connection if you're going limit the traffic that would benefit from those speeds?

Verizon internet customers need to call Verizon and ask what they're paying for if not for Verizon to serve them internet. And if they are paying Verizon to serve them internet, will they be getting a discount from Verizon if Netflix starts paying for Verizon to serve internet.

I'm a Verizon customer, and I've called several times in the last few months and they just don't care. They know I don't have a good alternative, because Time Warner is so bad in NYC. And Verizon has all the power here because it is ONLY Netflix that this is happening too. I get 75/25 and everything else including Amazon Prime Video is blazing fast. Netflix constantly has problems. So what am I going to do cut off the entire internet because Netflix performance is bad?

It's infuriating, and especially so because otherwise I love Fios as a total package.

498 Reader Comments

Verizon internet customers need to call Verizon and ask what they're paying for if not for Verizon to serve them internet. And if they are paying Verizon to serve them internet, will they be getting a discount from Verizon if Netflix starts paying for Verizon to serve internet.

I've been reading this series with growing frustration (but thank you Ars for providing such thorough coverage!). Is there any sort of consumer action possible? I happen to live in an area where Cox is an alternative, which at the very least has good Netflix streaming performance according to previous articles. Until recently I've been very happy with FIOS, since (except for this) they've provided me the advertised speed with more consistency than any other ISP I've used.

But this blatantly anti-competitive behavior makes me want to switch. We need some way to get lots of FIOS/Comcast customers who have viable other options to make a credible threat to switch ISPs. If consumer pressure can convince these ISPs to behave then I'll happily stay. Otherwise I will be switching ISPs in a month or two. Verizon should know why.

I don't get how they can do this, and then with a straight face, offer you higher speed packages. What's the point of getting a 50/5 connection if you're going limit the traffic that would benefit from those speeds?

I wonder what happens when every single Netflix feed starts with a banner explaining why their customers are getting poor quality. Better, provide the service number for Verizon along the way. Just as with SOPA/PIPA the power of the content providers should not be underestimated.

Verizon internet customers need to call Verizon and ask what they're paying for if not for Verizon to serve them internet. And if they are paying Verizon to serve them internet, will they be getting a discount from Verizon if Netflix starts paying for Verizon to serve internet.

I'm a Verizon customer, and I've called several times in the last few months and they just don't care. They know I don't have a good alternative, because Time Warner is so bad in NYC. And Verizon has all the power here because it is ONLY Netflix that this is happening too. I get 75/25 and everything else including Amazon Prime Video is blazing fast. Netflix constantly has problems. So what am I going to do cut off the entire internet because Netflix performance is bad?

It's infuriating, and especially so because otherwise I love Fios as a total package.

For years, I've had verizon fios service. The customer service is abhorrent but the product has been good compared to the other option in my area. For years, the speedtest.net tests have been stellar. The PSN connection test (via the PS3, wired connection directly to router) has been a fraction of the speedtest results. I've often wondered if Verizon games the speedtest results and has restricted PSN connections (directly via throttling or [more likely] indirectly by not upgrading ports as referenced in this article).

If only there were true competition and not having to choose the best of the worst of two choices.

I don't get how they can do this, and then with a straight face, offer you higher speed packages. What's the point of getting a 50/5 connection if you're going limit the traffic that would benefit from those speeds?

You are looking at this from ENTIRELY the wrong perspective. Flip it around. Imagine it's your responsibility to draw more revenue to Verizon.

Silliman added that "the whole premise of settlement-free peering is that you have a roughly equal exchange of traffic."

There is never going to be an equal exchange of traffic between residential ISPs and other networks. Ever. Ever ever. Period. ISPs have made asynchronous connections the only option and made all things that send data from your home connection to the internet against ToS for one reason: they want to keep the status quo. They want to be able to walk into negotiations and use this as a strongarm tactic, saying "Well, you send more traffic to us than we do to you, so you have to pay!" The dumb prats.. of course this is the way it is, they designed the system to work this way.

All of this will continue to be a problem until there is competition in the broadband market. I can't speak from experience because the Netflix experience on Comcast is miserable, thankfully not unwatchable, but I'm sure anyone with FiOS that can't watch Netflix or Youtube during peak hours would have already jumped ship if they had any other choices. It would be amazing what innovation we could get if we separated the duty of infrastructure and content delivery.

Thankfully, there's no influence of the big business of ISPs in government, and there's definitely not the head of their old lobbying effort chairing the only government entity out there designed to protect consumers from this sort of blatant exploitation, so I'm sure this will get resolved very quickly.

Why is Verizon's behavior not "false advertising"? They are an ISP selling x/y MBit connections but they don't deliver.

As a customer you reasonably expect that the speed you buy is not isolated to the "last mile" between your house and the closest DSLAM but to the internet in general (source throttling excepted but that is clearly not the case here).

There is a wide range of Internet interconnection agreements, he said. "We have settlement-free agreements. There are some ISPs to whom we pay money and there are others who pay money to us," Silliman said. "There is a whole range of commercial options, and they get worked out commercially and smoothly, except for this one ISP who seems to have problems with not just us but a lot of others."

Silliman added that "the whole premise of settlement-free peering is that you have a roughly equal exchange of traffic."

So what should happen when it isn't roughly equal? Should one ISP pay fees to the other, or should the ISP receiving more data just cover the cost?

Before this even turns into a mainstream news story, in the court of public opinion you've already lost. People like Netflix. People DO NOT like Verizon. And people mainly don't give a crap about who Cogent is.

Verizon internet customers need to call Verizon and ask what they're paying for if not for Verizon to serve them internet. And if they are paying Verizon to serve them internet, will they be getting a discount from Verizon if Netflix starts paying for Verizon to serve internet.

I'm a Verizon customer, and I've called several times in the last few months and they just don't care. They know I don't have a good alternative, because Time Warner is so bad in NYC. And Verizon has all the power here because it is ONLY Netflix that this is happening too. I get 75/25 and everything else including Amazon Prime Video is blazing fast. Netflix constantly has problems. So what am I going to do cut off the entire internet because Netflix performance is bad?

It's infuriating, and especially so because otherwise I love Fios as a total package.

Use a proxy to stream Netflix. That way you can get the Netflix traffic from another direction. It's a thought, YMMV.

A stopgap solution would be for Netflix to implement a controllable buffer option in their software. Allow me to set a 15/20/30 min buffer, so I can delay the start of playing, but be ensured of HD quality the whole time through.

This might help put pressure on Verizon\Comcast\et.al. because they lose out on any complaints Netflix subscribers might send to Netflix to just give in.

Verizon internet customers need to call Verizon and ask what they're paying for if not for Verizon to serve them internet. And if they are paying Verizon to serve them internet, will they be getting a discount from Verizon if Netflix starts paying for Verizon to serve internet.

I'm a Verizon customer, and I've called several times in the last few months and they just don't care. They know I don't have a good alternative, because Time Warner is so bad in NYC. And Verizon has all the power here because it is ONLY Netflix that this is happening too. I get 75/25 and everything else including Amazon Prime Video is blazing fast. Netflix constantly has problems. So what am I going to do cut off the entire internet because Netflix performance is bad?

It's infuriating, and especially so because otherwise I love Fios as a total package.

Heh, well, I don't have Netflix, but Amazon Prime video is complete crap where I'm at (LA) over the last couple of months. Basically unwatchable, or if I do get it, it's SD with very low bitrate. Horrible. I need to hope on a VPN and do a comparison. I'll bet you practically anything it'll work just fine on a VPN.

I'm beginning to agree with the likes of the ISPs – the FCC shouldn't be in control of network neutrality. It's obvious Wheeler isn't going to do anything beneficial for users, so I vote it should be sent down to a baser, more primal part of the government where neutrality will be much more difficult to unseat in the future. It should be closer to a basic human right.

If only I could send a bucket of dicks and a fork to Wheeler's office.

There is a wide range of Internet interconnection agreements, he said. "We have settlement-free agreements. There are some ISPs to whom we pay money and there are others who pay money to us," Silliman said. "There is a whole range of commercial options, and they get worked out commercially and smoothly, except for this one ISP who seems to have problems with not just us but a lot of others."

Silliman added that "the whole premise of settlement-free peering is that you have a roughly equal exchange of traffic."

So what should happen when it isn't roughly equal? Should one ISP pay fees to the other, or should the ISP receiving more data just cover the cost?

If anything, the ISP serving more data should be paid. It is not Cogents' fault Verizon has to manage Netflix traffic, it is Verizon's customers demanding to use the access they paid for. Verizon is handling the data it requested on its own.

It is like asking an employee to pick up a package at the post office and him asking the post office for money to carry it back. The post master will look up and say "WFT?". It does not matter if it is one package or a 100. We already paid the employee for his work and the post office already got paid to deliver the package(s).

G.R.E.E.D.^2

Edit: I should say I do not believe any side should be paid but rather that if Verizon insists money has to exchange hands it should not be into their coffers because their claims are ridiculous.

Exibit #1 on why Network Neutrality is needed more now than ever. Verizon is being greedy. They are already getting paid by their customers.

This is a peering issue, not a Net Neutrality issue. VZ is not blocking anything.

NN is not just about blocking -- it's also about purposely degrading a user's connection. By purposefully letting your network become slow for services outside of your network (Netflix) but keeping your own service (RedBox streaming) running just fine within your own network, is a NN issue.

Verizon wants to ditch the "settlement-free" peering model and get money from Cogent, arguing that it has to accept far more traffic from Cogent than vice versa because of high-bandwidth applications like Netflix.

Why should Verizon upgrade its network to support Netflix's business model?

residential broadband is not guaranteed bandwidth and never has been. it's a shared pool among many households. for better bandwidth you need to go with a business class circuit which costs more, a lot more.

You are right. Why should an ISP be forced to relay data all day when they can simply trash it?

Give Verizon what it wants. Alter the Netflix client to cache videos and then go peer to peer. That will generate a large amount of traffic in the other direction so that the Cogent load is more balanced.

The balanced argument is BS. End user ISPs will never have balanced traffic. But those end users are paying to receive their unbalanced traffic and they should get it.

Verizon wants to ditch the "settlement-free" peering model and get money from Cogent, arguing that it has to accept far more traffic from Cogent than vice versa because of high-bandwidth applications like Netflix.

Why should Verizon upgrade its network to support Netflix's business model?

residential broadband is not guaranteed bandwidth and never has been. it's a shared pool among many households. for better bandwidth you need to go with a business class circuit which costs more, a lot more.

They're not, they're upgrading their network to support their customers' desired usage scenarios. Verizon (and all the other ISPs) advertise heavily about rich media and fast internet and video instantly everywhere. Their stock in trade is offering a fat pipe to end users who want that pipe for rich media out on the internet.

The problem is, their real customers, the end-users, are a captive audience, so Verizon doesn't need to treat them like customers. They can treat them like resources to resell. If customer-facing ISPs were truly a competitive market, I would be ok with Verizon's attempts to get revenue coming and going. They could bill themselves as a premium ISP with better access to Netflix if they wanted. But they aren't, and they won't.

Verizon wants to ditch the "settlement-free" peering model and get money from Cogent, arguing that it has to accept far more traffic from Cogent than vice versa because of high-bandwidth applications like Netflix.

Why should Verizon upgrade its network to support Netflix's business model?

residential broadband is not guaranteed bandwidth and never has been. it's a shared pool among many households. for better bandwidth you need to go with a business class circuit which costs more, a lot more.

Because we paid Verizon millions of dollars in subsidies and subscription fees to upgrade its infrastructure? Just a thought...

When you get a tire low on air pressure, you put air in that tire, not sue the manufacturer for miniscule leaks in the tire. When your customers want Netflix, it's your job to fix the congestion issue to Netflix, especially since those customers are paying you, AND the government is paying you for exactly that reason.