Slashdot videos: Now with more Slashdot!

View

Discuss

Share

We've improved Slashdot's video section; now you can view our video interviews, product close-ups and site visits with all the usual Slashdot options to comment, share, etc. No more walled garden! It's a work in progress -- we hope you'll check it out (Learn more about the recent updates).

Barryke writes: Verizon has blamed Netflix for the streaming slowdowns their customers have been seeing. It seems the Verizon blog post defending this accusation has backfired in a spectacular way: The chief has clearly admitted that Verizon has capacity to spare, and is deliberately constraining throughput from network providers. Level3, a major ISP that interconnects with Verizon's networks, responded by showing a diagram that visualizes the underpowered interconnect problem and explaining why Verizon's own post indicates how it restricts data flow. Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that's the case, we'll buy one for them. Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it. Heck, we'll even install it." I'm curious to see Verizon's response to this straightforward accusation of throttling paying users (which tech-savvy readers were quick to confirm).

Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct.
If Verizon doesn't want network neutrality, then punish their customers.

Find locations where you will hurt Verizon customers, and cut the cables. Do so publicly. Precondition repair on upgrades of Verizon's network as you direct. If Verizon doesn't want network neutrality, then punish their customers.

I wish it was that simple, but I'm on board with the general idea. I wouldn't publicly cut the cables. That's too extreme. I would, however, like to see Level 3 turn the tables and publicly (as noisily as possible) accuse Verizon of using up all their bandwidth and that if Verizon

If you are paying for 75/25 or 50/25 and they are throttling it at the borders of their network, then you aren't getting the bandwidth you are paying for... downgrade your service. That $10, $20 or more per month they aren't getting from you because of their throttling practices should get their attention.

Verizonâ(TM)s Accidental Mea CulpaMark Taylor / 18 hours agoDavid Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflixâ(TM)s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizonâ(TM)s own paying broadband consumers.

His explanation for Netflixâ(TM)s on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know itâ(TM)s bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post, I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But hereâ(TM)s the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizonâ(TM)s depiction of their own network. We engineer it that way. We have to maintain adequate headroom because thatâ(TM)s what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3â(TM)s network interconnects with Verizonâ(TM)s in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So letâ(TM)s look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the con

It looks like the Level 3 post has been pulled. It goes to their 404 page which has a link to recent posts which lists the very post linked in the article.....and the recent post link ALSO takes you to a 404.

Strange, the link works fine for me. Your ISP isn't Verizon by chance is it?

No, if he's getting a 404, there's a connection to the server. Would be hilarious if Verizon had something to do with it, but we can't pin that one on them.

It was a joke, but if we are going to take it seriously it is certainly not out of the realm of possibility for an ISP to redirect a specific URL to a different URL. Just get the 404 page from the site and redirect it there as it passes through the provider's network gear. Similar process to the one used by internet providers in countries that have mandatory blacklists for "pirate" sites.

A better way of thinking of it is that Verizon should be representing the interests of its customers, because Verizon is the gateway between the customers, and the rest of the internet. It's not doing that job -- it's trying to play both sides against each other. This is what middlemen do, of course, and they're entitled to do it, but as long as they have a monopoly (which they do), then there should be limits, oversight, and accountability.

Nice sentiment, but, unfortunately, a public corporation's responsibility is to its shareholders and their interests - which is simply $$$. (and probably executives and cushy bonuses, etc...)

And a government's responsibility is to take action against a company which is committing wholesale fraud against its customers by selling them Internet Service which promises bandwidth speeds which they are then purposefully not providing in order to shake down their customers and companies trying to provide services to those customers more money.

A government's responsibility is to ensure that companies that are given government licenses and franchise agreements which restrict competition in certain geographic areas are providing the service that the people of that area want and need at a fair price.

A government's responsibility is to ensure that companies which get too big, hold too much market share and are too horizontally or vertically integrated are broken up so that there can be real competition and a real free market.

Except that Verizon Enterprise (formerly Verizon Business) is also a Tier 1 backbone provider. Different part of the company, but the behavior does appear to be a conflict of interest, of exploiting the Verizon's ISP (last-mile) business actions (failing to resolve congestion to L3) to make a competitor (L3) to Verizon Enterprise (formerly UUnet, AS 701 / 702 / 703) less desirable to Level 3 customers, namely Netflix.

Arguably, Verizon is abusing its ISP customers as pawns in making a competitor to one its Enterprise IP business less desirable, in a very anti-competitive fashion. [ftc.gov]

The problem is still the lack of competition in the market. If everyone had the choice between 4-5 ISPs, considering the popularity of Netflix, consumer ISPs would be paying Level 3 truckloads of money to ensure Netflix works flawlessly...and the roles may even be reversed (where Level 3 tries to gouge Verizon, since they'd know Verizon would have no choice or lose a ton of customers).

But since there isn't any competition, Verizon takes their own customers hostages...

The problem is still the lack of competition in the market. If everyone had the choice between 4-5 ISPs, considering the popularity of Netflix, consumer ISPs would be paying Level 3 truckloads of money to ensure Netflix works flawlessly...and the roles may even be reversed (where Level 3 tries to gouge Verizon, since they'd know Verizon would have no choice or lose a ton of customers).

I've lived in Europe and I got to see first hand what very strong competition means.

Every ISP peers to the max with every other ISP it can, and with the backbone providers. Nobody charges for peering either way, everybody wants to open the pipes as much as possible.

At one place I lived at I had a choice of the biggest 3 providers in the country and 2 small ones. All of them offered bandwidth in the range of 100 Mbps, both up and down, to/from anywhere inside their network (which for the big ones meant pretty much the entire country) and varying levels of outside bandwidth (but 10-30 Mbps was usual). This was pretty much the standard on cable or copper connections in the cities. Outside it went down but you'd still typically get 30-50 Mbps. Fiber was only available in the cities – but it meant 1000 Mbps down (yes, 1 Gbps).

Lowest basic monthly subscription started from around 10$. It was 25$ for the fancy fiber stuff. I wish I was making this up.

Was there throttling, blocking, or shafting customers with lower-than-advertised bandwidth? You betcha, and plenty of it. Did anybody call for government regulation? Nope. They bitched about it to the ISP, and if the ISP didn't fix it (or couldn't) they switched to another one. Or they decided they don't care that much and stayed on. Whatever. Even with the most crap of the crappiest ISP's you still got something like 10 Mbps so, yeah, some people didn't care.

In case you're honestly asking: city-owned roads are much the same as they were 50 years ago, same for sewer pipes. We can do things like that in a publicly owned manner due to the long cycles, and even then it's often not optimal. Note, too, that most roads that you travel on were not built by a government but were built by a developer and simply maintained by the government.

Trying to get government to run networks would work well up front, but in 5 years it would be outdated and there would be no money

Prove it. At this point with stagnation or even reduction of service from the Internet providers it isn't at all clear that private companies are doing anything other than gauging customers with the exclusive franchises or licenses they are getting from communities in order to be the only one running wires.

All evidence is pointing to it being better for communities to treat wired communications along public ways as a public utility.

Much is made about the private capital that is used to invest in installing all these wires, but it is the capital of customers which is paying back those original investments. I would say the customers who are actually paying for this should be the ones that decide how they want their communications network managed.

Trying to get government to run networks would work well up front, but in 5 years it would be outdated and there would be no money to upgrade it.

What is the difference between that and what we have now? It isn't like Comcast is working around the clock to beef up their last-mile connections.

I feel the same way about internet service that I feel about Healthcare: I am all for private for-profit alternatives, but there needs to be a solid free or low-cost (non-profit) public option. Everybody should have a minimum standard with the option to go with a private entity if they want something better. Something needs to shake up the Internet monopolies and

That long cycle worked fine for the telephone lines that serve your house. They served us well for a long time. The biggest flaw was that instead of managing the infrastructure themselves, they gave it to private companies to manage. Then, when they started abusing the monopoly, the government had to turn around and start requiring them to allow CLECs to use the lines. The phone companies, predictably, hated this, and did the absolute least that they could do to comply with the regulations, often refusing to fix problems with lines while blaming it on the CLEC (and vice versa).

All those problems would have been avoided if the government had simply maintained exclusive control over the lines and leased them out to third parties. That's how next-generation fiber networks in cities should be set up. The entire premise of letting a few companies maintain exclusive control over critical infrastructure is fundamentally flawed and can only lead to more of the same bulls**t we've had for the last two decades.

The only scheme that works is the public utility scheme, where the government owns the wires and private companies provide the service. We know this model works because it has worked with our interstate highway system and private shipping companies for decades. Is it perfect? No. The government historically hasn't charged those shippers enough money in gas taxes to cover infrastructure maintenance costs, resulting in some roads falling into disrepair. But that's mainly a problem caused by lack of a single management body that manages all of the roads in a region. Dozens of city governments working together isn't a great way to get things done except on an "It burns! It burns!" emergency basis. The solution to that problem, of course, is for all the cities in a metropolitan area to get together to form a non-profit corporation, and make that corporation responsible for the management and leasing of lines a la TVA.

city-owned roads are much the same as they were 50 years ago, same for sewer pipes

What shithole do you live in where they haven't upgraded any roads in 50 years? Is your city shrinking in size because theres no way any city, with normal growth, has the same traffic it did 50 years ago. No roads have ever been made wider? No new roads have been added?

Your argument is bunk, its just your ignorance of how your city has dealt with the need for additional capacity either due stupidity or willfully ignoring the obvious upgrades that have been made.

I'm just assuming you're saying that and not judging from personal experience, because last I checked, the State of Michigan monitors road quality state-wide [state.mi.us] and funds repairs accordingly.

Too big to fail, too arrogant to concede, too greedy to care. This news is all the more reason to regulate.

But, but, but... regulation is the antithesis of the Capitaist way that our republican Democracy has weaned its children on since it was formed!!I do tend to agree though - regulation of ISPs is probably the only way to deal with this.Capitalist theory says that if an incumbent merchant/provider is too inefficient to provide a good service or if another potential merchant/provider thinks they can do a better job for a lower price, then that new provider will step in and provide said service. The threat of that is what keeps the incumbent lean and competitive, and the result is a competitive environment that is generally good for the consumer and rival providers seek to offer better deals to entice custom away from their competitors.However, that theory assumes that there is a very low or non-existent barrier to entry into that competitive marketplace. Given the initial infrastructure setup costs and, in many cases, exclusivity contracts between providers and the municipal areas which would present the profits to drive services out into more marginal areas, the barriers to entry into the Tier 1 ISP market are prohibitive, to the point where you need to be a corporate entity the size of Google to be able to reasonably make the capital investment required.As such, the local markets for each ISP more closely resemble non-competitive monopolies with the illusion of choice being provided by third party suppliers who typically have to by access to the resources from the incumbent monopoly - they get wholesale prices, and the consumer sees some small price reductions if the third parties can make enough money to operate by charging the consumer slightly less than the discount they got from the incumbent. But fundamentally, everything is still controlled by that original monopolistic provider, so services suck, progress is stifled because there is no incentive for change, innovation is discouraged, and the level of capacity/reliability is never going to be any more than "just barely enough so that we can maximise our profit margins".

Impressive way to flip that around. Any restrictions on a person's behavior is equivalent to slavery? Seriously?

Think about what you're saying, that anyone should be able to anything they want? Anything? or should there be some rules governing behavior? Unless you're in favor of wild west anarchy, you're in favor of *some* type of regulation on society and the blessed free market capitalism.

Individuals, and individual rights, are like single atoms. They only exist in the abstract sense. The real world is entirely dominated by groups and collective actions.

You're a confused anarchist. The problem with non-coercive government is that all government is coercive. Government is primarily a set of restrictions on the use of force, or alternately the monopoly on that use of force. Getting rid of a government, or disarming it, merely allows anyone with a larger arsenal to set up their own government -- anarchy is an unstable system. We all have a right to violence, because it cannot be taken from us except in extreme situations. Remember, the Code of Hammurabi was instituted, "...so that the strong might not harm the weak." Coercive government is a necessary evil, and it will remain necessary so long as men are capable of harming their fellows, for that is its justification and primary purpose.

Rights are not inherent, except in some abstract sense. In the real world, your rights are what the men with guns say they are. You may feel fortunate that the world has had a long, bloody time to work out semi-cooperative frameworks to restrain our darker impulses. Individual rights are an important conceptual counterbalance to the overwhelming powers of the collective, but they are no justification for anarchy, economic or otherwise. The "free market" is an ideal, even a good one, but in most cases removing government interference makes markets less free, more subject to collusion and fraud. In some cases, where the service is required to be universal, or when the barriers to entry would be insurmountable, it makes sense for the government to assume these functions directly. Govenment can also be thought of as the natural monopoly of natural monopolies, in that sense.

Slavery is a word that has a specific meaning; your definition is specious. You just fundamentally don't like being told what to do. To some degree this idealism is admirable. For the true individualist, I can recommend (from long personal experience) the Alaskan wilderness; you can get land for free still up there, provided you build upon it. Whatever romantic images your mind conjures upon thinking of Alaska are all true; I can't stand the weather, personally, but it's as close to a pure state of nature as you will ever find. If you'd like to enjoy the benefits of society, however, you have to play by the rules. "Slavery" isn't an option -- it's mandatory.

Regulation for the public benefit = good. Examples: Public Utilities, Healthcare, Agriculture, Air Quality/Environmental Protection.Regulation for the sake of Regulation = bad. Examples: 70,000 + pages of IRS Regulations and 30,000 pages of tax code written by special interests and bureaucrats.

Was obvious people were going to figure out everything Verizon was saying is BS, and that they'd continue to get bad press about this.
You'd think the PR droids spouting this stuff would talk to their tech people and listen. But they probably said "look, just give us a pretty graphic right?" "But, techs will see through your spin" "Leave that to us" "But it'll make us look even worse" "You don't get paid to deal with this"
All too predictable, and the same techs are probably still being yelled at.

Just connect to a VPN first and then use Netflix. You'll be able to clearly see how much Verizon is throttling. I've been using this as a workaround for a while now. I'm not sure why more people don't think of pointing this out when Verizon's tech support people claim there is no throttling.

I wonder if Netflix themselves could provide such a service, maybe even run it through Comcast? Now there would be a fun bit of PR... "yes, your netflix connection runs better if we route from your Verizon DSL through Comcast, imagine how much better it would be if you just switched to Comcast?"

It's not artificial because of the details of the technical implementation, it's artificial because it's a scarcity that would not be expensive or difficult to resolve. Drought is geographic scarcity that cannot be readily resolved; an undersized water treatment plant is systematic scarcity that can be resolved but would be expensive and slow; a faucet that's rusted half-closed is artificial scarcity.

What you're doing is using a VPN connections which has an different inbound interconnect than the one which the majority of Netflix traffic comes in on. Verizon is 100% correct. There's no throttling going on because the connection that is being "throttled" as you put it is actually at 100% utilization and congested causing dropped packets and bad performance.

Verizon is most likely telling the truth in a highly deceptive way. Technically there is no throttling. There are not fucking with any packet that comes through that connection. They are not actively doing anything to inhibit the flow from Netflix.
They've simply chosen to architect a bottle neck which only impacts one provider. The end result is the same but they can truthfully say they're not throttling.

Which this article seems to implies it takes Verizon a year to send a technician to 7 cities to connect up a few cables between routers. (And / or maybe install a couple of cards). Maybe Verizon should stop having their techs travel by horseback, they might get it done faster.

End Of the Year is a schedule determined carefully by verizon so that Verizon can figure out how to offer their own competing service "Without the problems of netflix!"....or possibly because Verizon has to force netflix into a complex solution to hide the fact that it's a simple problem.

After some Google searches, it looks like a dump truck would be about 27 cubic yards [yahoo.com]. A penny is about 0.44 cubic cm [yahoo.com]. This gives us about 46,900,000 pennies in a dump truck (rounding down to the nearest hundred thousand) or $469,000 worth.

We all know most top tier network providers are running over multiple bands of fiber just sitting there idle. What Verizon is saying is Level 3 has not worked out an agreement with Verizon to upgrade capacity. The physical part is the easy part; it's just about upgrading port usage. Now, if Level3 is paying for X bandwidth and they're not getting X bandwidth because Verizon hasn't upgraded their equipment, I'm sure Level3's lawyers would be all over that.

Level 3 doesn't pay Comcast for bandwidth. Why should they? Comcast customers have already paid Comcast for the links to their house and they're the ones pulling data from Level 3. Level 3's customers pay Level 3 to deliver to the edge of their network. As the Level 3 post points out, the cost for Verizon to add more bandwidth between the Level 3 network and the Verizon network is minimal.

Part of the issue is that Verizon is a last-mile network, and does not sell symmetric bandwidth to its subscribers. So, the typical agreement between providers - where they each send about the same amount of traffic to each other and upgrade the interconnects to handle that traffic - will not work between Verizon and Level 3. Verizon (and the vast majority of other last-mile providers, including Comcast) will NEVER have a balanced interconnection with Level 3, because the home subscribers can all download far faster than they can upload.

Really, it's Verizon's customers who are causing all this bandwidth usage, so it should be Verizon ensuring that their interconnects can handle the requested bandwidth. If anything, Verizon (and Comcast) should be paying Level 3 for additional download capacity... but we all know that is never going to happen.

Costs should be driven by the party responsible for the traffic being on the network. In the case of neflix traffic, that's _me_, the end recipient. And I've already ponied up to the cable company to cover their cost to transfer the bits to me. The cable co just wants to double dip.

Bingo. Well, almost; it's a little more nuanced than that. Costs should be driven by the party responsible for the traffic being on *your* network. For Verizon, that's Verizon's customer; for Level3, that's Netflix. And they both already pay their providers. Where Verizon and Level3 peer, it's a matter of recognizing that the imbalance of traffic across that link is caused by Verizon's customers requesting more traffic than they (can) return. Thus, Verizon caused the imbalance and should therefore pay for it. If Verizon primarily sold symmetrical access and allowed their users to run servers, there would likely be a balance, and if there was not, they'd have a leg to stand on here, but they don't sell symmetrical access to the end user and they don't have a leg to stand on in this debate; what they do have is a monopoly on Verizon customers, which they're attempting to abuse right now, which should warrant an anti-trust suit, if anything. No additional regulation needed.

Ahh... I thought of this as soon as I clicked the button... If Netflix *really* wants to drive the point home, they can simply start peering or buying transit (more likely) from the providers Verizon has non-congested links with and stop routing to Verizon through L3 and Cogent. When Verizon refuses to upgrade *those* links, Netflix will be able to say "Either Verizon is refusing to upgrade their links, as we've been saying, or they only use providers which, as they claimed of Level3 and Cogent, can't handle the throughput their customers are requesting. In either case, this should be a wake-up call for Verizon customers to stop giving them money for a service they aren't, for whatever reason, delivering." And they shouldn't stop there; after that statement, they should re-enable routing to Verizon over all available links and watch the congestion continue; regardless of Verizon's response (which will likely be something along the lines of "Netflix performance continues to be slow because they have disabled routing to our network over multiple providers"), Netflix can stand up and say "We are currently routing to the Verizon network through every provider Verizon also uses, and make our routing decisions based on performance metrics, including packet loss and ping time to each user, to ensure that our users get the best possible experience we can provide. Unfortunately, as every link Verizon maintains appears to be congested, packet loss and ping times are high in all cases; the only solutions that exist are for Verizon to upgrade their links or peer with us, or for Verizon customers to find an alternate provider."

I don't think Netflix *wants* to fight dirty, or they would have done this already (and with Comcast, as well).

"it's Verizon's customers who are causing all this bandwidth usage"Let's pause to re-read that: Verizon's customers. Ah yes, those people who pay Verizon $x each month for y mbp/s of bandwidth. Those foolish people who actually expect Verizon to deliver on what is being paid for.

And then Verizon passive aggressively acts like the problem is not of their own making.

Yeah, that would make sense. Verison's blog just had a small hidden reference to that. Its not immideatly obvious, probably because thats a difficult thing to explain to people not versed in peering agreements.

Verizon to level3: "Our traffic from netflix moved over to Level3 last night... very strange, anyways we need to increase our capacity..."Level3 to Verizon: "Ok, that will be $X"Verizon to level3: "um... That's 300% higher than any other provider out there..."Level3 to Verizon: "suck it... your monies are belong to us"

Except that this fictional exchange you've created, in which Level 3 is extorting Verizon for more, is easily refuted by using either blog post. For instance, from Verizon:

Netflix did not make arrangements to deliver this massive amount of traffic through connections that can handle it.

[...] Netflix is responsible for either using connections that can carry the volume of traffic it is sending, or working out arrangements with its suppliers so they can handle the volumes. As we’ve made clear before, we regularly negotiate reasonable commercial arrangements with transit providers or content providers to ensure a level of capacity that accommodates their volume of traffic.

Which is a nice way of saying, "Level 3 is refusing to negotiate rates for more capacity with us, so we've refused to give them more." Level 3's blog post also affirms that the issue is Verizon's refusal to act:

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused.

Even without the blog posts, it should be obvious your notion makes little business sense. Level 3 is in no business position to play hardball like you've suggested. If they sacrificed on performance as a ploy to double-dip (i.e. get both Netflix and a lower-tier ISP* like Verizon to pay), Netflix would simply take its traffic to a different Tier 1 ISP that doesn't play those sorts of games, since the double-dipping would be hurting their bottom line. Or, at the very least, they'd be calling out their own ISP, rather than calling out the customer's ISP.

On the other hand, as a lower-tier ISP, Verizon has a monopoly on its own end users: if you want to reach them, you MUST go through them. If Verizon tries to double-dip by getting money out of both the higher-tier ISP and its end users, the end users won't understand what's going on, and in many cases they lack any viable alternatives anyway. Meanwhile, the higher-tier ISP can't switch out for a different peer, since Verizon is the only way to get to those end users.

Besides which, it's not like Netflix's switch from Akamai to Level 3 took Verizon by surprise, as you suggest, since it happened way back in 2010 and has been working fine for most of that time. If there was a problem resulting from the switch, it would have come up before now. Which is to say, this isn't a "Wow! Level 3's traffic is suddenly skyrocketing and we can't keep up!" situation. Rather, it's almost certainly a, "Hey, that Comcast company had a good idea to try getting money out of both sides...let's see if we can do it too!" situation, given the timing of it all.

* A quick aside: I'm well aware that Verizon also maintains a Tier 1 network, but Tier 1 networks rarely connect directly to end users. That's what lower-tier networks do. Moreover, the defining characteristic of a Tier 1 network is that it enjoys free peering with other Tier 1 networks. As such, the Verizon network being discussed here is clearly not their Tier 1 network, but rather a lower-tier one they control (e.g. a Tier 2 or 3 network) that has direct access to their end customers.

So no this isn't a case of exclusive peering, Level 3 being such a large provider just happens to be the best connection between Verizon and Netflix.

Secondly, that whole thing of 'Level3 to Verizon: "Ok, that will be $X"' has no bearing on a peering agreement, the statement would have been more like "The link between us is congested, want to upgrade the link?" each side upgrades their switch (if neccesary) and they connect the cable / fiber (given that they are in the same location we are talking about a multimode fiber patch at the high end).

Level3 also offered to pay for the necessary upgrades to Verizon hardware: "... these cards are very cheap, a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more

Verizon's response was "Ok, but these cards tend to wear out pretty quickly so we'll need you to pay that amount each month. 5,000 streams may sound like a lot, but they don't last very long. A person watches a few movies a week, maybe a couple of youtube videos per day, that's like 20 streams in one week, and that's only one customer. Before you know it, you've used up all 5,000 of those streams and the card needs to be replaced."

"Oh yeah, and if it's coming from Netflix then we're using twice as many streams. We use one stream from Netflix to us, then another stream from us to our customers. Maybe you should really pay us that amount every week."

Maybe they can't afford the small piece of cable between our two ports. If that's the case, we'll provide it.

Does that sound serious to you? I'm sure the part about them being willing to provide the cable is serious. The part about maybe Verizon not being able to afford the cable... probably not. Verizon are trying to get other people to pay for the service their customers are already paying them to provide. They have to justify that somehow. L3 seems to be pointing

I have FIOS... Yes my Netflix performance is piss poor, but so are the connections to other services that just happen to use the same transit providers as Netflix.

Particularly the VPS providers that I was using (I just switched due to the latency). I have 2 VPS providers, 1 in Reston, 1 in the UK. The one in Reston is just down the street from Verizons datacenter (used to be UUNET), but the provider to the VPS company I use was Cogent, heavy latency right at the peering point.

Of course, Verizon likes to blame Netflix for picking crappy transit providers, but had it been Company XYZ instead of L3 and Cogent, Verizon would have done the exact same thing to XYZ and let the peers saturate.

I did manage to switch to a different VPS that does not use Cogent or L3, and I have consistent low transit times, which I use as a VPN endpoint. Seems to do the trick (I have been doing this long before any people started publicizing using VPN's to get around Verizon and Comcasts shenanigans, mostly to keep Verizons prying eyes from monetizing my internet behavior, not to keep gov spying eyes out. If VZ wants to pay me [no, not give me a discount on already overpriced service, but give me cold hard cash] for my browsing and internet habits, then I will more than be happy to let them snoop)

What happened was a bunch of salesmen and marketers at Verizon asked how they could explain the network throttling.
They obviously didn't understand the presentation so they assumed no one else would either.

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix’s customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon’s

David Young, Vice President, Verizon Regulatory Affairs recently published a blog post suggesting that Netflix themselves are responsible for the streaming slowdowns Netflix's customers have been seeing. But his attempt at deception has backfired. He has clearly admitted that Verizon is deliberately constraining capacity from network providers like Level 3 who were chosen by Netflix to deliver video content requested by Verizon's own paying broadband consumers.

His explanation for Netflix's on-screen congestion messages contains a nice little diagram. The diagram shows a lovely uncongested Verizon network, conveniently color-coded in green. It shows a network that has lots of unused capacity at the most busy time of the day. Think about that for a moment: Lots of unused capacity. So point number one is that Verizon has freely admitted that is has the ability to deliver lots of Netflix streams to broadband customers requesting them, at no extra cost. But, for some reason, Verizon has decided that it prefers not to deliver these streams, even though its subscribers have paid it to do so.

The diagram then shows this one little bar, suggestively color-coded in red so you know it's bad. And that is meant to be Level 3 and several other network operators. That bar actually represents a very large global network, and it should be shown in green, since, as we will discuss in a moment, our network has plenty of available capacity as well. In my last blog post , I gave details about how much fiber and how much equipment we deployed to build that network and how many cities around the globe it connects. If the Verizon diagram was to scale, our little red bar is probably bigger than their green network.

But here's the thing. The utilization of all of those thousands of links across the Level 3 network is much the same as Verizon's depiction of their own network. We engineer it that way. We have to maintain adequate headroom because that's what we sell to customers. They buy high quality uncongested bandwidth. And in fact, Verizon admits as much because they conveniently show one direction across our network with a peak utilization of 34%; almost exactly what I explained in my last blog post. I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network â" IN BOTH DIRECTIONS.

So why does Verizon show this red bar? And why do they blame Level 3 and the other network operators contracted by Netflix?

Well, as I explained in my last blog post, the bit that is congested is the place where the Level 3 and Verizon networks interconnect. Level 3's network interconnects with Verizon's in ten cities; three in Europe and seven in the United States. The aggregate utilization of those interconnections in Europe on July 8, 2014 was 18% (a region where Verizon does NOT sell broadband to its customers). The utilization of those interconnections in the United States (where Verizon sells broadband to its customers and sees Level 3 and online video providers such as Netflix as competitors to its own CDN and pay TV businesses) was about 100%. And to be more specific, as Mr. Young pointed out, that was 100% utilization in the direction of flow from the Level 3 network to the Verizon network.

So let's look at what that means in one of those locations. The one Verizon picked in its diagram: Los Angeles. All of the Verizon FiOS customers in Southern California likely get some of their content through this interconnection location. It is in a single building. And boils down to a router Level 3 owns, a router Verizon owns and four 10Gbps Ethernet ports on each router. A small cable runs between each of those ports to connect them together. This diagram is far simpler than the Verizon diagram and shows exactly where the congestion exists.

Level 3 Communications is (or at least was) a really great company to work with. When the company I worked for was a huge customer of theirs, they did anything and everything to satisfy us. The claim of them volunteering to install 10GE cards really does sound like something they'd just do to make a large customer happy.

You're paying for a service, and nowhere does it say that they will discriminate against a particular service, such as Netflix.

It's obstructive business, against your customer's best interests, for no particular reason. It will also violate any given "net neutrality" laws that are / may come into effect.

Those laws are the answers. The reason for their existence is this sort of unnecessary posturing. And governments make companies do a lot of things against a company's best interests - all the time. It would be in the company's best interest to not pay tax, screw over its customers, not ship goods that have been paid for, be monopolistic, collude with others to enforce market prices, etc. The laws are brought in to stop that shit in the PEOPLE'S best interest, not the company's.

Not saying it's anywhere near perfect, but your post seems to want to back a corporation screwing over its customers and then (falsely) blaming its competitors and random third-party companies for that.

"I want it and my government friends have guns..." Is this the best we can do?

The reason Verizon can stay in business despite having "very limited interest in what their customers want" is because of municipal and state granted monopolies, federal grants and subsidies, and the reason they even exist at all is because of a government approved corporate charter. Why is "government friends with guns" an acceptable argument for them getting their way, but not an acceptable argument against it?

Sure. The content streaming from Netflix has been requested by Verizion customers. They've paid for access to the internet, which includes Netflix. They are the ones being throttled. Basically Verizon is trying to double dip here - get money from regular customers plus shaking down more from content providers. If Verizon really cannot handle the flood of Netflix content, shouldn't they raise the cost to the consumers to build out the Verizon network?

Sure, the governments should break up the ISP monopolies that restrict access to only one or two ISPs. They should also try to reduce any barriers to entry in the market for new competitors to spring up. This would increase competition and force verizon to better their service to retain their customers.

The issue with this is that its good for consumers but bad for investors, and we all know who our esteemed congress men actually represent.

Perhaps a rule where cable or satellite TV providers are prohibited from operating centralized peering points. If Verizon had to buy their bandwidth from upstream providers, they wouldn't be able to choke L3. And L3 would have to bid against Verizon's upstream providers to get Netflix's business.

Essentially, less economic centralization in the network infrastructure would provide for more opportunities for competitive bidding all along the chain. Everyone would end up with more customer-focused incentive

The traffic isn't uniform in both directions. When it is, its an easy agreement. When its not, someone usually pays. Level3 offered to pay for the hardware to make additional connections.

But thats kind of like me telling verision that I'll pay for the router and won't charge them for using their network and expecting them to provide me with free service. There isn't enough of an incentive to allow them to agree to my free internet for Bill proposal. Same thing for most networks with unequal traffic pattern

How about because customers are paying them for Internet service, and going to Netflix is where they want to go? People don't want to pay for bad service do they? The real problem here is Verizon is a competitor to Netflix, and Verizon is not only being allowed to be anti-competitive but also hoping to get paid for it.

to make their customers happy. that is it. The customers are purchasing something, then finding out that they cant get what they thought they paid for. When customers find out that they can get some things at the speeds that they paid for , but not others, then it should be their ISPs job to provide that service. If there was true competition then this would happen quickly. Verizon doesnt really compete, so they can play these bullshit games.

The government should threaten them because they are not providing the service their customers have paid for and they advertised. The Internet works because tier 1's all connect to each other and upgrade ports as they get saturated (good ISP's are proactive, ok ones last minute and crappy ones only after the link is saturated). While this is related to net neutrality it's primarily a false advertising issue, if they change their wording to correct that it's a DPUC (or similar) issue as they are no longer

+1. Politicians at the state level have been paid off by cable companies and ISPs to squash competition from local municipalities. The cities who got in before the legislation are loving their services. It is absolutely insane what money can do in the political process.

How about to make their fucking customers happy. I pay Verizon (because my only other choice is Comcast, and I hate them more). I request a service, I expect my provider to give me access to this service. Netflix pays L3, L3 is their service provider. Service providers peer, that is the way the internet has always worked.

Netflix pays level 3 to get their bits from their servers all the way to the edge. Customers pay to get bits from the edge all the way to their house. Level 3 and Verizon make an agreement for the parts where those 2 networks touch each other.

Verizon is saying: "The direction of bits matter. Because our customers are paying to receive bits and not to send bits, YOU owe US more money. If our customers paid to SEND that much bit instead of receiving them, then we'd owe you money....like if it changes anything