Further Reading

When Netflix agreed to pay Comcast for a direct connection to the ISP's network, video performance improved immediately. Verizon subscribers aren't so lucky. Although Netflix and Verizon confirmed on April 28 that they had struck a paid peering deal, performance continued to drop in May and could remain poor for months while the companies upgrade infrastructure.

"Verizon FiOS is down two slots and now ranks behind DSL providers Frontier and Windstream," Netflix wrote today after releasing its monthly speed index.

In the US, Netflix performance on Verizon FiOS dropped from 1.99Mbps in April to 1.90Mbps in May, and performance on Verizon DSL dropped from 1.08Mbps to 1.05Mbps. This is the average performance of all Netflix streams on each ISP's network. The drops are small, but they show that the paid peering deal didn't make any immediate impact.

Netflix has continued to blame Verizon for poor performance, and Verizon last week sent a cease and desist letter demanding that Netflix stop bad-mouthing the ISP. Verizon says this dispute isn't preventing network upgrades, but they are proceeding slowly nonetheless. A Wall Street Journal article from last week says:

The connections are just getting set up. Data from network research firm Renesys show the two networks already set up a test connection to carry video in the Dallas area, but major networks typically need links in dozens of cities to deliver traffic effectively.

Verizon said it plans to fulfill the terms of its agreement with Netflix over the next few months.

"We are working on the first 13 cities, and we do plan to have everything done in 2014," said Verizon spokesman Bob Elek. "All of this kerfuffle that is going on isn't affecting that."

A Verizon spokesperson confirmed to Ars today that the Journal's "reporting is correct," but they company didn't provide any further details in response to our questions. Netflix declined to say why it's taking longer to set up connections with Verizon than it did with Comcast.

Verizon VP David Young told CNET, "We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year." When finished, Netflix's connection to the Verizon network will supply "adequate capacity to satisfy the needs of their subscribers."

The cease and desist letter Verizon sent to Netflix was spurred by Netflix sending error messages to customers that blame ISPs for poor performance in cases when a customer's stream quality declined. "We are testing this across the US wherever there is significant and persistent network congestion," Netflix said today. "This test is scheduled to end on June 16. We will evaluate rolling it out more broadly."

Netflix performance on Comcast also fell a bit in May, but just from 2.77Mbps to 2.72Mbps. Comcast previously moved from 1.51Mbps in January 2014 to 1.68Mbps in February, 2.5Mbps in March, and 2.77Mbps in April. A Netflix spokesperson told Ars that the links between Netflix and Comcast are "almost" complete.

Netflix was previously sending traffic over congested links between the ISPs and transit providers such as Cogent and Level 3. Those companies have also argued with ISPs over whether they should have to pay the Internet providers for direct network access.

Netflix performance on AT&T was just 1.7Mbps in May, worse than Verizon FiOS. AT&T said in February that it's negotiating to get payments from Netflix in exchange for a direct network connection, but no deal has been struck yet.

Promoted Comments

You know it's bad when I need to torrent Orange is the New Black to watch it in a decent resolution. Streaming it was just ugly. Yes, I know it's technically illegal, but do have a valid Netflix account. I don't feel bad about it.

If the money Verizon is blackmailing from Netflix isn't going to system upgrades and the money Verizon conned out of the state of New Jersey for implementing FiOS everywhere isn't going to network upgrades, where is all that money going? Does the CEO of Verizon have a giant Scrooge McDuck money bin on some Caribbean island?

A bit ironic that Verizon refers to them as "their (Netflix's) subscribers". One could make a compelling case that they are also Verizon subscribers who are paying Verizon for ""adequate capacity to satisfy [their] needs".

120 Reader Comments

If the money Verizon is blackmailing from Netflix isn't going to system upgrades and the money Verizon conned out of the state of New Jersey for implementing FiOS everywhere isn't going to network upgrades, where is all that money going? Does the CEO of Verizon have a giant Scrooge McDuck money bin on some Caribbean island?

What if the Netflix payment to Verizon was a super chess move by Netflix. Netflix knew that Verizon could not abide by the terms of the contract for faster speeds? Verizon breaches the contract and Netflix gets their money back plus damages and Verizon is exposed, via discovery, on how they can provide such super crappy service.

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

i'm on TWC, can stream everything fine but a lot of stuff isn't in HD. at least not real HD

Interesting. I've been keeping a record of which shows/movies have performance issues and which ones don't, and so far I can't see HD as being much of a factor. My experience seems almost totally based on the popularity of the content (though it's obviously hard to judge that since I don't have raw numbers and am doing some guess work). But I'll keep an eye on it.

netflix has their servers in peering centers with the most popular content. if you stream something that's not popular it will probably have to go a lot farther away and will play like crap. i don't think they have their entire library at every peering location

Verizon VP David Young told CNET, "We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year." When finished, Netflix's connection to the Verizon network will supply "adequate capacity to satisfy the needs of their subscribers."

Why wasn't their network upgraded to begin with? Why did they wait until Netflix paid them to start doing something that would benefit everyone? This is precisely the reason why the US's broadband speed are so low in comparison to other nations.

What if the Netflix payment to Verizon was a super chess move by Netflix. Netflix knew that Verizon could not abide by the terms of the contract for faster speeds? Verizon breaches the contract and Netflix gets their money back plus damages and Verizon is exposed, via discovery, on how they can provide such super crappy service.

so why is netflix pulling the messages after being threatened by verizon?

I'm hoping the FCC enforces truth in advertising with the ISPs. No more hiding behind 50Mb/s to the speed test server and 2Mb/s to everything else. If the average website responds at 2Mb/s make them advertise 2Mb/s.

So if I'm paying for 50Mb/s I should be able to get 50Mb/s. Maybe not 100% of the time, but most of the time. Before dropping FIOS they forced me to upgrade from 25Mb to 50Mb by dropping my 25Mb offering. Of course that was $10 more a month. Nothing except the speed test server got faster after that upgrade. It was a complete waste of money.

I'm hoping the FCC enforces truth in advertising with the ISPs. No more hiding behind 50Mb/s to the speed test server and 2Mb/s to everything else. If the average website responds at 2Mb/s make them advertise 2Mb/s.

So if I'm paying for 50Mb/s I should be able to get 50Mb/s. Maybe not 100% of the time, but most of the time. Before dropping FIOS they forced me to upgrade from 25Mb to 50Mb by dropping my 25Mb offering. Of course that was $10 more a month. Nothing except the speed test server got faster after that upgrade. It was a complete waste of money.

What if the Netflix payment to Verizon was a super chess move by Netflix. Netflix knew that Verizon could not abide by the terms of the contract for faster speeds? Verizon breaches the contract and Netflix gets their money back plus damages and Verizon is exposed, via discovery, on how they can provide such super crappy service.

so why is netflix pulling the messages after being threatened by verizon?

Mission Accomplished and damage is already done to Verizon. No need to beat dead horse. Maybe not but its seems like Netflix is taking baseballs bats to ISPs knees.

If they're doing anything like our ISP did a few years back for a major upgrade they'd need to order all the equipment from China, where it could be sitting on a ship for six weeks before it even heads out, then once it gets here have to distribute it to where it needs to go. It took months on end for our ISP to finally get the new equipment installed.

You know it's bad when I need to torrent Orange is the New Black to watch it in a decent resolution. Streaming it was just ugly. Yes, I know it's technically illegal, but do have a valid Netflix account. I don't feel bad about it.

I do the same thing for Fringe, Caprica, and most of the other shows that I would normally watch on Netflix. I'm not ashamed of it, though I'd greatly prefer to use a single box rather than tie up storage on my pc.

So is Netflix slut-shaming Verizon to get them to speed up the process, or is Verizon failing to comply with the letter of the agreement? Given Verizon's propensity to breach contracts of late, I wouldn't be surprised if they had effectively breached by failing to implement caching in a timely manner.

While I think Netflix has somewhat harmed the larger neutrality debate in service to itself (and maintaining customers, which is totally something I would expect an intelligent company to do), this slut-shaming of ISPs is really really really important. Highly visible, objective(ish) graphs help put public pressure on the ISPs and the FCC, and any level of state government that intercede to make ISP operation more difficult in their area of influence.

Fun fact: Comcast and Verizon are not at the top of the graph, despite paying. Remember kids, premium != value.

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

i'm on TWC, can stream everything fine but a lot of stuff isn't in HD. at least not real HD

Interesting. I've been keeping a record of which shows/movies have performance issues and which ones don't, and so far I can't see HD as being much of a factor. My experience seems almost totally based on the popularity of the content (though it's obviously hard to judge that since I don't have raw numbers and am doing some guess work). But I'll keep an eye on it.

netflix has their servers in peering centers with the most popular content. if you stream something that's not popular it will probably have to go a lot farther away and will play like crap. i don't think they have their entire library at every peering location

Right, that's why I thought I'd be having the opposite problem -- the popular stuff would be solid, and the older/obscure stuff would be shaky. But alas, that's not the case.

Let's just take Verizon at their word that they are building and testing all of these changes to their system after the direct payment took place. If that's true, then Comcast is lying. The fact that their connection changed immediately after the payment was made, shows that something is amiss on either Verizon or Comcast's side.

Personally, I think they're both lying jerks and that together they prove that ISPs are purposefully not upgrading their infrastructure to keep up with demand.

John Oliver has a pretty compelling chart to back up this assertion. That being said, getting Netflix performance up quickly implies that at some level the operators WANT to provide good customer experience, and that's just not true in monopoly conditions, especially where Verizon is involved.

Verizon VP David Young told CNET, "We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year."

Comcast can. Is it just me, or does Verizon seem particularly slow or reluctant to upgrade their own network?

See, I wanted to say that too, but this shows two problems. First off, apparently Verizon had done no footwork to upgrade their network if this is their scapegoat. And second off, Comcast did have the groundwork, did have it tested, but just decided not to upgrade their network until it was forced to. That's just how it appears to me, but if half of this is true, ISP's clearly need to get their shit together and/or someone needs to create a Kickstarter to crowdfund for a new ISP.

If the money Verizon is blackmailing from Netflix isn't going to system upgrades and the money Verizon conned out of the state of New Jersey for implementing FiOS everywhere isn't going to network upgrades, where is all that money going? Does the CEO of Verizon have a giant Scrooge McDuck money bin on some Caribbean island?

Nope, it's going on stealing more money from Congress to stop Netflix.

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

Newer content is typically under higher demand, and so the servers/routes hosting that content is probably under a heavier load. When you're streaming older stuff, you're probably pulling it from a different resource that is under less load. I noticed the same thing on Time Warner Cable with the Breaking Bad finale coming up and the few hours after House of Cards went live.

What this says about Verizon is that they do not have any concern for their customers. Their hardware was so overloaded and they did nothing (were going to do nothing) to address it. Customers are paying Verizon for a connection to the internet. If the content customers are downloading is overloading Verizon's network, Verizon has a responsibility to upgrade their network, yet this shows Verizon had no intention of ever doing this. They only doing now cause they being paid to specifically do it. Look at small ISPs who do not get paid by Netflix and yet have way faster Netflix rating. They are spending the money for their customers, Verizon is not.

AT&T and Verizon are scum companies. I am amazed anybody does business with either of these companies at all. I have to assume the only reason anyone does business with them is because they hold a monopoly in the area. If there is no monopoly and customer is doing business with them, I have no sympathy for that person's broadband issues. Why are you paying these companies for service?

wireless = switch from verizon to t-mobilelandline = switch from at&t/verizon to small ISP who rates well on netflix scale

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

Newer content is typically under higher demand, and so the servers/routes hosting that content is probably under a heavier load. When you're streaming older stuff, you're probably pulling it from a different resource that is under less load. I noticed the same thing on Time Warner Cable with the Breaking Bad finale coming up and the few hours after House of Cards went live.

it doesn't make any sense. each netflix server holds like 5TB of data so it's not like it's only house of cards and a few other shows on it. and being closer to the user it would perform better than a far away server even if it's not hit as hard

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

Newer content is typically under higher demand, and so the servers/routes hosting that content is probably under a heavier load. When you're streaming older stuff, you're probably pulling it from a different resource that is under less load. I noticed the same thing on Time Warner Cable with the Breaking Bad finale coming up and the few hours after House of Cards went live.

Right, but to @anon_coward's above comment, I was under the impression that Netflix devotes more bandwith infrastructure to the content that has the highest demand. In fact, I believe they use Amazon's EC2 for hosting in large part because they can increase the amount of data center capacity when needed.

If Netflix *didn't* have such a strategy of devoting more resources for its best and most popular content, then it would be like a Blockbuster store having just one copy of the latest Marvel superhero flick instead of several dozen DVDs on the shelf.

When Netflix agreed to pay Comcast for a direct connection to the ISP's network, video performance improved immediately. Verizon subscribers aren't so lucky. Although Netflix and Verizon confirmed on April 28 that they had struck a paid peering deal, performance continued to drop in May and could remain poor for months while the companies upgrade infrastructure.

To me, this means that Verizon really didn't have the capacity in place to handle Netflix's volume, and it will take some time to build up the infrastructure. And it also means that Comcast had the capacity all along, and was artificially throttling Netflix until they paid the ransom.

Netflix and Comcast were already working on setting up direct peering relationships when Comcast at the last minute demanded extra money before turning them on. That is why Comcast's performance improved so quickly; all they had to do was enable the new peerings.

Verizon on the other hand, demanded money before building out any infrastructure. They have the paid agreement in place now and are just now working on building out the new peering infrastructure. This stuff can take time. Comcast just had a head start.

The article makes it sound like Verizon is now routing the connections through the building they're installing and setting up servers in; instead of you know, just putting them over the regular internet while they set up the datacenter.

Verizon VP David Young told CNET, "We can't just snap our fingers and the network is upgraded. We need new facilities. We have to do the equipment engineering. Build it and test it. We are doing all of that right now. And it should be completed during this year."

Comcast can. Is it just me, or does Verizon seem particularly slow or reluctant to upgrade their own network?

Well my first thought when reading this was "wait, how is this different than the rest of the connections they already have set up?" So are they doing this from scratch? Setting up connections for the first time ever that it involves extended engineering, deployment and testing? seems to me that they are just drawing this out for the sake of retaliation.

Its funny, its not like they have a competing service with a similar name that runs substantially faster on their system. Why would they want to slow down Netflix if they have nothing to gain?...

I have a feeling they will be true to their word. I mean they did roll out broadband to all of NJ on schedu- Well.. They did roll out service to all of NYC per their contra- ah crap.

from what i understand EC2 is only for authentication. for playback they rent data center space around the USA to host servers close to the users. san diego is where they are peering with comcast

Could be, although why use a self-described "elastic" compute service for just signing in/authenticating users and not the actual heavy lifting of streaming the content? Renting data centers is all fine and good -- until you have unexpected demand hit you, and you can't find a provider with affordable and available nandwith fast enough.

Let's just take Verizon at their word that they are building and testing all of these changes to their system after the direct payment took place. If that's true, then Comcast is lying. The fact that their connection changed immediately after the payment was made, shows that something is amiss on either Verizon or Comcast's side.

Personally, I think they're both lying jerks and that together they prove that ISPs are purposefully not upgrading their infrastructure to keep up with demand.

I used to work on phone systems for a living in the north east, so I had to deal with Verizon all the time. The big V has serious internal organization issues. I can't tell you how many times I'd get out to a job site for a cut-over to new T1s or PRIs and they weren't ready, or worse, had no note of the scheduled cut-over...... that they had sent us an e-mailed reminder about.

Verizon: "Ugh, it's not our fault we're so terrible! This stuff takes time and a lot of effort! BTW, don't ever say that we're terrible."

Verizon: "Oh and hey Netflix, wanna have more bandwidth for your service? Why not pay us for something we don't even have in place so that we can actually put the equipment in service later on while still providing the same (or worse) service until then?"

from what i understand EC2 is only for authentication. for playback they rent data center space around the USA to host servers close to the users. san diego is where they are peering with comcast

Could be, although why use a self-described "elastic" compute service for just signing in/authenticating users and now the actual heavy lifting of streaming the content? Renting data centers is all fine and good -- until you have unexpected demand hit you, and you can't find a provider with affordable and available nandwith fast enough.

The authenticating code is a relatively small amount of data. Easy to transfer the authentication code to a new cloud server and add it to the pool.

The movie data itself is *huge*. Adding more cloud servers that don't have the data on them doesn't do you any good for scalability, because they all have to copy the data from the already-loaded servers that have the data.

As a Verizon subscriber, I noticed an odd pattern with my Netflix streaming service and was wondering if anyone had experiences similar to the following:

I had virtually no issues with Netflix streaming (through my PCs, tablets, and consoles) until House of Cards season 2 debuted earlier this year. After that, whenever I watched an episode, I had serious performance issues, from blurry video to episodes cutting out or not loading at all. Same thing for other popular Netflix items or new releases -- performance issues are pretty much a guarantee.

However, I hardly ever have an issue with older or more obscure stuff. Not to wear a tin foil hat, but....anyone else experience this? And if so, do you think it's possible that Verizon is selectively altering the streaming service based on what content is being streamed (i.e., choking House of Cards because Verizon knows that will affect the most Netflix subscribers)?

Newer content is typically under higher demand, and so the servers/routes hosting that content is probably under a heavier load. When you're streaming older stuff, you're probably pulling it from a different resource that is under less load. I noticed the same thing on Time Warner Cable with the Breaking Bad finale coming up and the few hours after House of Cards went live.

it doesn't make any sense. each netflix server holds like 5TB of data so it's not like it's only house of cards and a few other shows on it. and being closer to the user it would perform better than a far away server even if it's not hit as hard

Being closer to the user may or may not result in better performance. If your issue is at the interconnect, it's entirely possible that being farther from the user could lead to better performance, if that distant interconnect happens to be less congested (after which the data transits on Verizon's network). This is generally why you hear so many stories of "it works great through my VPN."

So one of two things. They were full of shit. They didn't have the proof to back up that claim. Or they didn't want to spend the money on a trial knowing full well Verizon would release fleet of lawyers at them.