Further Reading

When Netflix agreed to pay Comcast for a direct connection to the ISP's network, video performance improved immediately. Verizon subscribers aren't so lucky. Although Netflix and Verizon confirmed on April 28 that they had struck a paid peering deal, performance continued to drop in May and could remain poor for months while the companies upgrade infrastructure.

"Verizon FiOS is down two slots and now ranks behind DSL providers Frontier and Windstream," Netflix wrote today after releasing its monthly speed index.

In the US, Netflix performance on Verizon FiOS dropped from 1.99Mbps in April to 1.90Mbps in May, and performance on Verizon DSL dropped from 1.08Mbps to 1.05Mbps. This is the average performance of all Netflix streams on each ISP's network. The drops are small, but they show that the paid peering deal didn't make any immediate impact.

Netflix has continued to blame Verizon for poor performance, and Verizon last week sent a cease and desist letter demanding that Netflix stop bad-mouthing the ISP. Verizon says this dispute isn't preventing network upgrades, but they are proceeding slowly nonetheless. A Wall Street Journal article from last week says:

The connections are just getting set up. Data from network research firm Renesys show the two networks already set up a test connection to carry video in the Dallas area, but major networks typically need links in dozens of cities to deliver traffic effectively.

Verizon said it plans to fulfill the terms of its agreement with Netflix over the next few months.

"We are working on the first 13 cities, and we do plan to have everything done in 2014," said Verizon spokesman Bob Elek. "All of this kerfuffle that is going on isn't affecting that."

A Verizon spokesperson confirmed to Ars today that the Journal's "reporting is correct," but they company didn't provide any further details in response to our questions. Netflix declined to say why it's taking longer to set up connections with Verizon than it did with Comcast.

Verizon VP David Young told CNET, "We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year." When finished, Netflix's connection to the Verizon network will supply "adequate capacity to satisfy the needs of their subscribers."

The cease and desist letter Verizon sent to Netflix was spurred by Netflix sending error messages to customers that blame ISPs for poor performance in cases when a customer's stream quality declined. "We are testing this across the US wherever there is significant and persistent network congestion," Netflix said today. "This test is scheduled to end on June 16. We will evaluate rolling it out more broadly."

Netflix performance on Comcast also fell a bit in May, but just from 2.77Mbps to 2.72Mbps. Comcast previously moved from 1.51Mbps in January 2014 to 1.68Mbps in February, 2.5Mbps in March, and 2.77Mbps in April. A Netflix spokesperson told Ars that the links between Netflix and Comcast are "almost" complete.

Netflix was previously sending traffic over congested links between the ISPs and transit providers such as Cogent and Level 3. Those companies have also argued with ISPs over whether they should have to pay the Internet providers for direct network access.

Netflix performance on AT&T was just 1.7Mbps in May, worse than Verizon FiOS. AT&T said in February that it's negotiating to get payments from Netflix in exchange for a direct network connection, but no deal has been struck yet.

Promoted Comments

You know it's bad when I need to torrent Orange is the New Black to watch it in a decent resolution. Streaming it was just ugly. Yes, I know it's technically illegal, but do have a valid Netflix account. I don't feel bad about it.

If the money Verizon is blackmailing from Netflix isn't going to system upgrades and the money Verizon conned out of the state of New Jersey for implementing FiOS everywhere isn't going to network upgrades, where is all that money going? Does the CEO of Verizon have a giant Scrooge McDuck money bin on some Caribbean island?

A bit ironic that Verizon refers to them as "their (Netflix's) subscribers". One could make a compelling case that they are also Verizon subscribers who are paying Verizon for ""adequate capacity to satisfy [their] needs".

This is why the US is the laughingstock of the world in terms of broadband.

Yeah... much of the world laughs at us because both parties to a mobile phone cal pay.I wonder if there is any other country where both parties pay...

In civilized countries, only the person making the call pays.No double dipping.

Considering how mobile traffic is handled here, asking both the receivers of data (Cable Co. customers) and the providers of data (NetFlix or whoever) to both pay isn't really that odd... Falls in line with mobile model of business.

When did this become okay. Bag phones.Anyone remember 'let me call you back' so billing could be shared for long distance calls over land lines. Customers haven't always both been charged. I wonder if people that have only ever had cell service know this...

This is why the US is the laughingstock of the world in terms of broadband.

Yeah... much of the world laughs at us because both parties to a mobile phone cal pay.I wonder if there is any other country where both parties pay...

In civilized countries, only the person making the call pays.No double dipping.

Considering how mobile traffic is handled here, asking both the receivers of data (Cable Co. customers) and the providers of data (NetFlix or whoever) to both pay isn't really that odd... Falls in line with mobile model of business.

When did this become okay. Bag phones.Anyone remember 'let me call you back' so billing could be shared for long distance calls over land lines. Customers haven't always both been charged. I wonder if people that have only ever had cell service know this...

It's been a while but yeah, I remember how long distance used to be. I remember when MCI was hated by tons of people too.

Netflix and Comcast were already working on setting up direct peering relationships when Comcast at the last minute demanded extra money before turning them on. That is why Comcast's performance improved so quickly; all they had to do was enable the new peerings.

Verizon on the other hand, demanded money before building out any infrastructure. They have the paid agreement in place now and are just now working on building out the new peering infrastructure. This stuff can take time. Comcast just had a head start.

Maybe I can only give you fuel for 3/4s of your train journey, too. It's insane for Verizon to do business this way, and it wouldn't be able to if it weren't stealing* from people in order to pay off politicos.

*And in this case, it is theft, because Verizon have not lived up to many of their legally-contracted obligations elsewhere.

from what i understand EC2 is only for authentication. for playback they rent data center space around the USA to host servers close to the users. san diego is where they are peering with comcast

Could be, although why use a self-described "elastic" compute service for just signing in/authenticating users and now the actual heavy lifting of streaming the content? Renting data centers is all fine and good -- until you have unexpected demand hit you, and you can't find a provider with affordable and available nandwith fast enough.

The authenticating code is a relatively small amount of data. Easy to transfer the authentication code to a new cloud server and add it to the pool.

The movie data itself is *huge*. Adding more cloud servers that don't have the data on them doesn't do you any good for scalability, because they all have to copy the data from the already-loaded servers that have the data.

True, but you're not talking about copying the entire Netflix library with huge amounts of data. You're talking about a very targeted approach of taking what is -- or what you believe will be -- the most requested content for the streaming service. So if you know that House of Cards is coming in February, and you're already using Amazon's cloud servers, then it doesn't seem that hard to order up more capacity ahead of the launch. It certainly seems easier to me that going out and renting another entirely new data center.

If the money Verizon is blackmailing from Netflix isn't going to system upgrades and the money Verizon conned out of the state of New Jersey for implementing FiOS everywhere isn't going to network upgrades, where is all that money going? Does the CEO of Verizon have a giant Scrooge McDuck money bin on some Caribbean island?

Moreover, what about all the money that subscribers are paying to Verizon? What is all that being used for if not to ensure that Verizon's infrastructure is capable of delivering the internet services requested by their customers at the advertised speeds?

I have Verizon and can (unfortunately) confirm that Netflix is slow as well. It seems to pause to load a lot more than it used to.Verizon and other ISPs in general need to stop crying like babies and realize they charge us for access and we're using it like it was intended to - they advertise fast speeds and when we use it - whether it's Netflix or League of Legends - it should work as advertised.Lastly, internet access is way to expensive here in the states. I don't want cable tv, just a fast, inexpensive internet.

If the money Verizon is blackmailing from Netflix isn't going to system upgrades and the money Verizon conned out of the state of New Jersey for implementing FiOS everywhere isn't going to network upgrades, where is all that money going? Does the CEO of Verizon have a giant Scrooge McDuck money bin on some Caribbean island?

Moreover, what about all the money that subscribers are paying to Verizon? What is all that being used for if not to ensure that Verizon's infrastructure is capable of delivering the internet services requested by their customers at the advertised speeds?

I cannot upvote this enough. I was going to post literally that. Virtually literally, anyway. Are you some other me?

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

Newer content is typically under higher demand, and so the servers/routes hosting that content is probably under a heavier load. When you're streaming older stuff, you're probably pulling it from a different resource that is under less load. I noticed the same thing on Time Warner Cable with the Breaking Bad finale coming up and the few hours after House of Cards went live.

it doesn't make any sense. each netflix server holds like 5TB of data so it's not like it's only house of cards and a few other shows on it. and being closer to the user it would perform better than a far away server even if it's not hit as hard

Old mech HD models of Openconnect CDN servers had 100TB of storage and the newer SSD versions have 16 960GB SSDs. And Netflix uses something similar to JBOD, no RAID, so they get nearly 100% effective usage.

Their newest SSD OpenConnect appliance has a peak 165watt usage and has a quad port 10gb nic.

If the money Verizon is blackmailing from Netflix isn't going to system upgrades and the money Verizon conned out of the state of New Jersey for implementing FiOS everywhere isn't going to network upgrades, where is all that money going? Does the CEO of Verizon have a giant Scrooge McDuck money bin on some Caribbean island?

Moreover, what about all the money that subscribers are paying to Verizon? What is all that being used for if not to ensure that Verizon's infrastructure is capable of delivering the internet services requested by their customers at the advertised speeds?

The rest of the world doesn't have problems with their broadband infrastructure like this for one simple reason: regulation. We're being trounced by countries large and small around the world in terms of broadband speed and price, and it's not because our system works and theirs doesn't.

Verizon VP David Young told CNET, "We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year."

Comcast can. Is it just me, or does Verizon seem particularly slow or reluctant to upgrade their own network?

I'm surprised that nobody here has mentioned Verizon's ownership share in Redbox Instant, which is trying to compete directly with Netflix. Verizon has a direct financial interest in Netflix failing.

Well...didn't a lot of people criticize Comcast when their network capabitilies went up TOO quickly after payment? I hate the ISPs too, but it seems like this statistic is being used to condemn them both ways.

Well...didn't a lot of people criticize Comcast when their network capabitilies went up TOO quickly after payment? I hate the ISPs too, but it seems like this statistic is being used to condemn them both ways.

No. People said that was proof that Comcast was deliberately screwing with Netflix and all the payments were actually 'protection money' to keep Comcast from doing it again. Being the guy running a protection racket is enough to warrant complain.

People are pissed here because Verizon is running a protection racket too, but instead of stopping when they got paid, they kept going. That also warrants complaint.

Could be, although why use a self-described "elastic" compute service for just signing in/authenticating users and not the actual heavy lifting of streaming the content? Renting data centers is all fine and good -- until you have unexpected demand hit you, and you can't find a provider with affordable and available nandwith fast enough.

What a ridiculous lie, Verizon says they need new facilities? What kind of idiots build their infrastructure with no room for scalability? While it is true that this sort of thing does take a bit of time but saying it will be done this year is not encouraging at all. If you can't even commit to a shorter time frame it means they are really dragging their feet.

I hope netflix holds back any sort of payment until things actually start improving as it makes little sense to start already paying for a service that is not performing any better yet.

You don't actually build buildings in that short a timeframe, they would be talking about installing electronics in existing structures. Or turning more of them on. Or whatever.

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

That's likely a matter of quality. The older episodes are in a lesser quality format, so you don't see any issues, whereas HoC is shiny and new and pushing the limits of your connection.

A bit ironic that Verizon refers to them as "their (Netflix's) subscribers". One could make a compelling case that they are also Verizon subscribers who are paying Verizon for ""adequate capacity to satisfy [their] needs".

It's a flat out fact. I pay Netflix for their library. I pay my ISP for the bandwidth to their library. If their bandwidth isn't up to the task, I'm on the phone to my ISP, not Netflix. What Verizon is suggesting is like getting mad at Ford or Honda because there are potholes in the roads. That doesn't make even the tiniest bit of sense.

A bit ironic that Verizon refers to them as "their (Netflix's) subscribers". One could make a compelling case that they are also Verizon subscribers who are paying Verizon for ""adequate capacity to satisfy [their] needs".

It's a flat out fact. I pay Netflix for their library. I pay my ISP for the bandwidth to their library. If their bandwidth isn't up to the task, I'm on the phone to my ISP, not Netflix. What Verizon is suggesting is like getting mad at Ford or Honda because there are potholes in the roads. That doesn't make even the tiniest bit of sense.

But Ford is selling cars, which is putting more load on the roads, so Ford should help pay the cost of road maintenance. Ford should be punished for increasing demand, because that increases capital investments for the road owner.

So one of two things. They were full of shit. They didn't have the proof to back up that claim. Or they didn't want to spend the money on a trial knowing full well Verizon would release fleet of lawyers at them.

In any case this makes me sad.

That's consistent with this article. Netflix said they were sunset-ing the whole think in a week. Not caving immediately == not caving.

I didn't think it was possible, but there is a worse company than Comcast. Comcast is like the Mafia in the movies. Pay your protection fee and you're alright. Verizon takes your protection money, but still trashes your shop.

A bit ironic that Verizon refers to them as "their (Netflix's) subscribers". One could make a compelling case that they are also Verizon subscribers who are paying Verizon for ""adequate capacity to satisfy [their] needs".

It's a flat out fact. I pay Netflix for their library. I pay my ISP for the bandwidth to their library. If their bandwidth isn't up to the task, I'm on the phone to my ISP, not Netflix. What Verizon is suggesting is like getting mad at Ford or Honda because there are potholes in the roads. That doesn't make even the tiniest bit of sense.

But Ford is selling cars, which is putting more load on the roads, so Ford should help pay the cost of road maintenance. Ford should be punished for increasing demand, because that increases capital investments for the road owner.

The analogy here is imperfect. I'll try and improve it a bit. I pay Verizon to gain (unlimited) access to roads the build and maintain. They tell me that I will be able to go a certain speed on these roads. Ford makes a car that everyone now wants (I know it's a stretch, but bear with). Many more people buy this car and pay to access Verizon's roads. The roads are now congested and I can know longer go at the speed they said I could. Verizon then tries to get Ford to pay to improve the roads. After all, it's their cars that are causing the congestion.

If Verizon can't keep up with the demand on their networks, they need to stop signing up new people and give discount to current member's whose performance has been degraded by them overselling their capacity.

You know it's bad when I need to torrent Orange is the New Black to watch it in a decent resolution. Streaming it was just ugly. Yes, I know it's technically illegal, but do have a valid Netflix account. I don't feel bad about it.

A bit ironic that Verizon refers to them as "their (Netflix's) subscribers". One could make a compelling case that they are also Verizon subscribers who are paying Verizon for ""adequate capacity to satisfy [their] needs".

It's a flat out fact. I pay Netflix for their library. I pay my ISP for the bandwidth to their library. If their bandwidth isn't up to the task, I'm on the phone to my ISP, not Netflix. What Verizon is suggesting is like getting mad at Ford or Honda because there are potholes in the roads. That doesn't make even the tiniest bit of sense.

But Ford is selling cars, which is putting more load on the roads, so Ford should help pay the cost of road maintenance. Ford should be punished for increasing demand, because that increases capital investments for the road owner.

The analogy here is imperfect. I'll try and improve it a bit. I pay Verizon to gain (unlimited) access to roads the build and maintain. They tell me that I will be able to go a certain speed on these roads. Ford makes a car that everyone now wants (I know it's a stretch, but bear with). Many more people buy this car and pay to access Verizon's roads. The roads are now congested and I can know longer go at the speed they said I could. Verizon then tries to get Ford to pay to improve the roads. After all, it's their cars that are causing the congestion.

If Verizon can't keep up with the demand on their networks, they need to stop signing up new people and give discount to current member's whose performance has been degraded by them overselling their capacity.

I do like your improvements. The only thing missing is somehow adding that the congestion is less general and more between Verizon and Netflix.

You could turn Netflix into "Walmart" and claim that lots of people want to go to Walmart, which is driving up demand for road time. And Verizon is both the road owner and the car dealership that only leases cars.

Let's just take Verizon at their word that they are building and testing all of these changes to their system after the direct payment took place. If that's true, then Comcast is lying. The fact that their connection changed immediately after the payment was made, shows that something is amiss on either Verizon or Comcast's side.

Personally, I think they're both lying jerks and that together they prove that ISPs are purposefully not upgrading their infrastructure to keep up with demand.

No its possible comcast setup their peering points with larger switches and had available ports allowing them to plug in netflix quicker. Verizon has probably not been upgrading peering points and has no ports available requiring swapping out switches/routers to add ports which takes longer.

Let's just take Verizon at their word that they are building and testing all of these changes to their system after the direct payment took place. If that's true, then Comcast is lying. The fact that their connection changed immediately after the payment was made, shows that something is amiss on either Verizon or Comcast's side.

Personally, I think they're both lying jerks and that together they prove that ISPs are purposefully not upgrading their infrastructure to keep up with demand.

I think what we have here is a difference of approachComcast builds all the upgrades then refuses to turn them on out of spiteVerizon doesn't build any upgrades and sends all the money to build money thrones for its board meetings.

yeah and its only made worse by the fact that their pr statement makes it seem like they need custom designed hardware "We have to do the equipment engineering. Build it and test it." if they didnt have the team and labs they needed to order and test this off the shelf hardware in under a week they wouldnt be an isp at all.

its probably a case of the highest level ceo ordering: "put the acquisition work order at the bottom of the work orders pile and when it comes to the top put another weeks worth of orders on top a few more times out of spite ...we will tell the public the magic tech guru takes time and must invent to solve... what are they gonna do find another isp? *lights cigar with $100 bill* "

Verizon does have a point. This does not have anything to do with network congestion. Here's what the message should say. "We're sorry your video is not loading at this time. This due to the fact that one of your ISP's chief lobbyists is now running the organization charged with the regulation of said ISP. Because of this, Verizon is allowed to squeeze you for a connection, and us for the pleasure of sending you the data you request over that connection. We have paid the danegeld but Verizon has not taken the steps to provide you with the service you are paying for.

Raise your hand if you're tired of following these corrupt, monopolistic, dictatorial, corporate-greed shenanigans and are deciding to give the finger to them all and just move to torrents.

It's not like our lawmakers and/or regulators are going to rule in favor of the consumer/taxpayer anyway. Besides some bullshit 90s-era comment system on the FCC site, we really have no other way to influence this than simply refusing to play the game.

"We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year."

Gee, then maybe you shouldn't have sold capacity to YOUR subscribers that you didn't have. You are going to compensate your subscribers for not providing the service they pay you for, right? Right?

What a ridiculous lie, Verizon says they need new facilities? What kind of idiots build their infrastructure with no room for scalability? While it is true that this sort of thing does take a bit of time but saying it will be done this year is not encouraging at all. If you can't even commit to a shorter time frame it means they are really dragging their feet.

I hope netflix holds back any sort of payment until things actually start improving as it makes little sense to start already paying for a service that is not performing any better yet.

You don't actually build buildings in that short a timeframe, they would be talking about installing electronics in existing structures. Or turning more of them on. Or whatever.

My point stands regardless, they knew there was congestion and that they are mandated to deal with it. Holding back on upgrades only hurts Verizon's own customers yet they did it.

They also knew this would happen since they gutted net neutrality in court and knew Netflix would be forced to cave in and pay. Shoddy planning, management and execution is Verizon's own problem they have years of experience to know better. They are not deserving of your pity, want to pity someone? The paying customers are worlds more deserving of it after being held hostage for the negotiations since they got now increased costs and poorer service than before this fiasco started.

I didn't think it was possible, but there is a worse company than Comcast. Comcast is like the Mafia in the movies. Pay your protection fee and you're alright. Verizon takes your protection money, but still trashes your shop.

I do know movie mafia and that does sound like Comcast (they came through following the payoff)Taking money and still trashing things sounds like real life mafiaAs I would imagine mafia to be, not knowing their methods,,,