from the too-many-experts,-not-enough-data dept

The mysterious performance problems customers on large ISPs (specifically Verizon, AT&T and Comcast) have been having with Netflix streaming has just gotten slightly clearer. For months people have posted traceroutes as undeniable proof that large ISPs like Comcast, AT&T and Verizon had intentionally been throttling subscriber Netflix streams. However, as we've been noting, simple route analysis doesn't tell the whole story, and it's not always possible to determine where video transit breakdowns are occurring without direct access to ISP network data (data they hide completely).

Adding a sliver of insight into the problems, Netflix and Comcast announced over the weekend that they had struck a new interconnection deal that involves Comcast providing Netflix with direct access to the Comcast network. The announcement itself says little, other than to insist that Netflix "receives no preferential network treatment under the multi-year agreement." A Wall Street Journal report had a little more insider detail, noting that the two sides bypassed core network operator Cogent in an arrangement that involves Netflix paying Comcast for direct access:

"(The connection to Cogent) was starting to become overwhelmed with Netflix traffic, congesting traffic and leading to slower Netflix streams for Comcast Internet users, people familiar with the matter said. At the same time, Comcast presented Netflix with more attractive deal terms than the operator had been offering, the people said. The deal spans several years. Netflix was aiming for a long-term deal to make sure its projected traffic growth wouldn't put it at a disadvantage, one of the people said. The connection is a so-called "paid peering" deal, which connects Netflix's network to Comcast's directly. Netflix was previously using several middlemen to access Comcast's network."

Why the sudden interest on Comcast's part in getting a deal done? They're likely not too excited about the fact that they've been receiving bad press for poor Netflix streaming performance at the same time they're trying to sell regulators on the purported technical benefits of their $42 billion acquisition of Time Warner Cable.

Despite a lot of Internet hand-wringing about the deal and counter-kvetching about said hand-wringing (all based on no actual data, mind you) -- the agreement on its surface appears to be good for everybody involved except Cogent. That's thanks to the fact that in this case, Comcast appears to have been willing to make apparent concessions on price and deal length in order to keep up appearances for the merger. Netflix pays the same money they were paying Cogent to Comcast (or less), Comcast makes revenue they weren't making before, and customers get better streaming performance at no added costs (since Netflix is probably paying about the same, for now).

"The fact that Netflix agreed to pay Comcast suggests that Cogent will likely lose its fight with Verizon as well. And as Cogent's chief executive Dave Schaeffer told Ars, "once you pay it's like blackmail, they've got you, there's nowhere else to go. They'll just keep raising the price in a market where prices [for transit] are falling." Indeed, in the long run, this development threatens the survival of independent backbone companies like Cogent. If it becomes industry practice for backbone providers to pay residential ISPs, companies like Cogent will become mere resellers of access to the networks of large broadband companies. Or they may be cut out of the loop altogether, as large customers such as Netflix cut deals directly with broadband providers such as Comcast."

I'm not sure Lee helps his argument by bringing net neutrality into the discussion though, as this is again more about market power than neutrality (for the moment). It's also worth noting that Cogent has a reputation of being difficult in peering disputes, and when a breakdown happens, more often than not Cogent is at the heart of it for not following through on truly balanced peering. Cogent was busy telling media outlets last week that it was Comcast and Verizon that were to blame for recent slowdowns. To hear Cogent tell it, Verizon and Comcast have been refusing to upgrade their interconnection links, largely because the ISPs want to move from an era of settlement-free interconnection, to one where the ISPs play a larger role in transit and get paid:

"This isn't the kind of problem consumers can solve by purchasing a faster Internet connection, because they'd be paying only for a faster path to Verizon's network, not a faster path to the rest of the Internet. In some cases, Verizon has actually purchased and installed the necessary equipment to upgrade ports, but not turned it on, according to (Cogeco CEO) Schaeffer. "They actually put it in, so they spent the money, but they just politically have not been willing to turn it on in order to ensure that Netflix will not work as well as Redbox," he said. When Netflix can't get through to Cogent, it sends traffic through other providers, but Schaeffer claims that "everybody is full." So what are they fighting over? Schaeffer said Verizon "wants a price that is about 10 times the market price for transit." Again, transit is a service that takes traffic across the entire Internet, whereas peering is just a connection between two networks."

To intentionally belabor a point again though, nobody can confirm anything either side says because the details aren't released. Meanwhile, while the Comcast deal might not be bad, the worry among consumer advocates is that in a world where ISPs get bigger and more powerful and transit options are reduced and replaced under absolutely no transparency, the overall industry could get less healthy. With Comcast controlling the last mile conduit, an ever-increasing slice of the core networks, and the media content itself, you start to see the monopsony concerns Comcast's expansion could raise among antitrust regulators.

Remember of course this deal only impacts Netflix on Comcast, there's continued ongoing problems for Netflix and YouTube traffic reaching the networks of a number of larger ISPs (Time Warner Cable, AT&T and Verizon), with no indication of when they'll get resolved. While I don't think the FCC's Wheeler is going to do much of anything meaningful on broadband competition, net neutrality, or prices, I do think he'll work to find some way to bring a dash more transparency to these disputes if everybody involved can't start getting along. Mainstream services grinding to a halt with nobody willing to admit fault and everything kept in the complete dark is simply bad for business. For everyone.

If this is all tl;dr for you, all you need to know is that if you're on Comcast and your Netflix streaming stinks you should either see improvements right away, or sometime over the next few months as the changes are implemented.

Netflix paid for a better service. It's pretty obvious what is going on is double dipping. The degradation in service was planned, not an accident. The fact that money fixes it tells all. It sets a damn poor precedent that will now be sought for all intensive bandwidth applications.

This was not a good move by Netflix, even if they see the majors as having them by the balls.

Here's a quick definition of net neutrality along with a handy example, for all those ignorant journalists out there:

A neutral network is one where filtering is performed at the protocol layer and not at the application layer. So filtering 'internet video' is not an example of neutrality violation, but filtering 'netflix' would be, if YouTube was left alone.

Destruction

Netflix's new deal with Comcast may not destroy the Internet...but it may destroy Netflix.

They haven't disclosed the terms of this agreement, but Netflix doesn't have very much margin at all to play with. They had only $112 million in net income for all of 2013. How long is that going to last if they've got the major ISPs lining up with their hands sticking out, right behind the movie studios?

Either they'll have to jack prices (and watch subscribers jump ship), or hemorrhage money. Either way, they're in trouble. "House of Cards," indeed.

(I don't actually have a dog in that fight...I will never be a Netflix customer, because they sent me unsolicited commercial E-mail. I do not do business with spammers.)

Netflix also talking to Verizon

Re: Destruction

Wow, if getting spam is your criteria that must eliminate half the internet and most national businesses.

Not to mention that you may at some point in the last decade have agreed to have your email sold for marketing purposes, meaning you solicited the spam you're getting. Or maybe you're just the purest of web surfers?

Re: Re: Destruction

Agreed. Give 'em an inch they'll take a mile and all that shtick.

Seriously though, that it's public knowledge that Netflix is willing to pay off/bribe/pay a toll in order to keep their customer base satisfied, it won't be long [I'd say a month or so] before all ISPs big and small are lining up to get their pound of flesh from the company's net income.

And it's not just the US ISPs we have to worry about. I can see a future worldwide ISP dogpile on Netflix thanks to this announcement.

Re: Re:

"Guess what: there's more of Internet than Netflix and Comcast."

Guess what, if Comcast can get Netflix to pay up for higher speeds then they can get others to pay up to. That's the danger we've been warning about, putting up that added tax on the Internet that harms tech start up companies most of all.

Re: Macro

While there may be more to this story, it sure seems as if Netflix exchanged one connection point for another, of course the cost followed the connection. The question that I have not seen answered yet is "Is there a cost difference?".

Re: Re: Re:

Bingo. And as an AC pointed out: clearly double-dipping is also at stake.

Now that Comcast has tasted sweet, sweet cash, it's just a matter of time before it goes after Google, Sony (Crackle), Disney (ESPN), and any other content pusher, regardless of "degradation" of connection.

Re: Destruction

It really depends on the terms of the deal (and their deal with Cogent). Let's assume Netflix is paying Comcast the same price / Mbps as they're paying Cogent. Let's also assume Netflix can easily scale up and down the bandwidth they're purchasing from Cogent and Comcast. Because each byte sent directly through Comcast to a Comcast subscriber is a byte that doesn't need to transit through Cogent, that allows Netflix to offset any additional costs from Comcast by a proportional decrease in costs from Cogent. Ditto if Verizon and other ISPs start hopping on. All that's happening is that

These assumptions may obviously not be true. Netflix may be paying Comcast a higher per unit price than Cogent. And I'm sure there are plenty of fixed costs and overhead involved with dealing with more ISPs, rather than just one. But it's theoretically possible at least for these deals to be structured in a way that has a negligible impact on Netflix's bottom line.

After some research, I've come to the conclusion that my Netflix was being throttled. I don't have any solid evidence, but after a few weeks of my video working fine until exactly 6:00 pm then slowing down to a crawl until around 10:30, what other conclusion could you come up with? Especially after you connect to a VPN and suddenly the entire problem goes away...

Speaking of VPN's... and again I'm no expert. But this entire situation has led me to do quite a bit of research and I've come to the conclusion that regardless of your internet usage, a VPN is a good idea... especially if you use one that takes privacy and security seriously, preferably one outside the U.S.

cogent

To be fair, Cogent's network has sucked for a long time. I've used them in previous jobs for high bandwidth connectivity, and they've always had issues and been terrible to deal with.While I agree this may not be a good precedent, Cogent is as much at fault here as anyone.

Re: Re:

I don't know about cheap...DWDM equipment to support 100Gbps connections can be quite costly in both modules (Cisco CPAK) and equipment to support it. It's one of the reasons people are still prefering SFP+ 10Gbps connections and bonding, but it probably wouldn't scale well for very large interconnections like Cogent and Comcast. Comcast uses Ciena equipment and does have a deal with them, so it might be a bit cheaper but not by that much.

Re: Re: Destruction

He's not alone in how he determines who he will do business with. When I go to the grocery store that is one of my prime considerations. Has someone spammed me, either with email (which I rarely use) or a commercial that has somehow slipped through the process of blocking.

I'm amazed you encourage their behavior by not considering these items.

Re: Re: Re: Destruction

Re: Re: Re:

Thus the use of my qualifier "relatively". It's a lot cheaper to resolve a chokepoint issue (something that doesn't necessarily require deploying more equipment, by the way) than expanding total carrying capacity.

In 1914-15, when mass production of automobiles was getting underway, the Ford Motor Company decided it needed some drill presses which would drill eight or ten holes at a go, in a fixed pattern. When this was put to the machine tool makers, they insisted that you could not possibly make a drill press like that, and that it was immoral besides. Ford had to design and build its own drill presses before the machine tool makers relented, and agreed to make copies of them. Blindly accepting the machine tool makers' advice would have been a cop-out, and the first generation of Ford managers were remarkably tough-minded.

This was a recurrent event in the emergent mass-production industries. They tended to bring out the bottlenecks in tools designed for custom production, and, every so often, heated exchanges took place.

The same thing applies to Telco/ISP switches. Naturally, Cisco is not very keen on the idea of a switch built on the Redundant Array of Inexpensive Devices principle, but a certain sort of customer will simply have to insist, even to the point of building its own. The designer will simply have to find out what kinds of mass-produced components are inexpensively available, and devise some means of hooking them together to form a big system. For example, I find that there are some Ethernet switches intended for hooking offices full of diskless desktop computers together, so that they can boot from a common server, and be centrally managed, and that these switches are very powerful and quite inexpensive, say 50 Gbits for a thousand dollars. There are vastly more such offices than there are Telco switchrooms, so a greater economy of scale exists (*). With modifications of firmware, it ought to be possible to find ways to hook up arrays of such machines to make a central switching office.

(*) parallel example: In railroading, there is something called a Genset locomotive, developed by the Union Pacific Railroad, which is powered by three 18-wheeler truck engines. There are about a hundred trucks for every locomotive, and a truck engine is much more of an off-the-shelf item than a locomotive engine.

Falling Away From Logistics

I think NetFlix hoisted the white flag of surrender some time ago, when they turned away from their disk-renting business. The First Sale Doctrine is such a great charter that to abandon it for an internet-streaming business model, dependent on studio licenses, was a sign of doom. An internet video streaming service is inevitably caught between the studios and the ISP's, and with mergers, the two sets of entities are increasingly likely to have interlocking ownership.

NetFlix chose not to find a means of shortening its physical logistic lines, something like Redbox. More generally, whenever someone has a business model which essentially involves sitting behind a website and trading stuff, that is a danger sign. The winners will be the ones who are doing something meaningful on the ground.

Re: Re: Re:

It's technically not paying for higher speeds though. Yes it results in a more efficient, faster transmission but that is because it the traffic doesn't have to travel through a smaller middle man like Cogent. It's similar to the move that Netflix made earlier by getting access through Level3. You have to understand how network traffic normally works on the Internet. Normally the request comes from the user through their ISP thru an upstream provider through a backbone provider or two before then going through an upstream provider for the datacenter to get to the server which routes the traffic in response in the reverse order. When Netflix got a connection directly through the backbone provider Level3 they cut one of the upstream providers (the one between their datacenter and the backbone) our of the loop. It made content delivery faster and more efficient for them. Now they are going to be cutting the other upstream provider out of the loop as well when the response comes from a Comcast user. It's not about Comcast increasing the bandwidth for Netflix traffic as much as it is shortening the route the traffic has to travel.

Re: Re: Re: Destruction

I never signed up to receive E-mails from Netflix. They sent unsolicited commercial E-mail. Granted, this was quite a few years ago, but I have a long memory. (I didn't stop boycotting Amazon until long after even the FSF gave up on it, for instance...)

But Netflix's problem can be neatly summed up as: "Once you pay the Danegeld, you never get rid of the Dane."

Re: Re: Re: Re:

"It's technically not paying for higher speeds though."

Yes, it is paying for higher speeds. The consumer is being charged to have traffic delivered to them and the content provider is being charged to deliver the content. and since there is little competition in the U.S. consumers pay more for worse service.

Re: Re: Re:

If comcast thinks its too expensive then what the government needs to do is to remove exclusive rights of ways and enable competitors who don't think its too expensive the opportunity to enter the market and compete. Government exclusivity exists for good reason, because if it didn't new competitors would enter the market and offer consumers services worth paying for. Otherwise there is no point in having exclusive rights of ways.

Re:

Re: Re: Re: Macro

"You don't get to know that because the details are confidential."

Oh I don't know. Maybe not today, but in a year, maybe two, or sooner with quarterlies, a careful reading of financial statements might indicate changes. A good analyst might sus out just what actually happened. Or come really close.

Costs to Netflix unknown

I don't know how you or anybody else can write that Netflix will pay Comcast less than Netflix paid Cogent. Or the same amount. Nobody knows. According to the quote you offer, Comcast only was willing to deal with Netflix for less than the terms Comcast was offering before. And even this is "according to people who know something" -- anonymous cowards, that is.

Nobody knows what Netflix has agreed to pay, now and in future, who is not in on the details of the contract. And that means nobody outside Netflix and Comcast.

Some other blogger offered that this must be a lower amount, because "why would Netflix pay Comcast more than they were paying Cogent? They wouldn't." But maybe Netflix might pay more so that customers could watch Netflix streaming and not cancel the service, and watching on Comcast was becoming, according to a lot of people, unwatchable.

So, yeah, nobody knows -- but there is a good reason why Netflix would pay Comcast more than they paid Cogent. And there is every reason why we should expect that Comcast would seek just as high a fee as they possibly could get.

Service quality maybe a big problem.

While this may eliminate the throttling issues, I'm fearful that the quality of Netflix's programming (Netflix original movies and shows) may suffer tremendously. Due take note, that after Comcast bought out the Syfy channel, the quality of their programming went on rapid downward spiral. It went from great shows like Stargate, to utter garbage movies like sharknado being repeatedly shown in marathons.