from the sunlight-is-the-best-disinfectant dept

If you've followed the ongoing feud between Netflix and the nation's biggest ISPs, you'll recall that streaming performance on the nation's four biggest ISPs (AT&T, Verizon, Comcast and Time Warner Cable) mysteriously started to go to hell earlier this year, and was only resolved once Netflix acquiesced to ISP demands to bypass transit partners and pay carriers for direct interconnection. Though the FCC has refused to include interconnection in their consideration of new net neutrality rules, we've noted how it's really the edges of these networks where the biggest neutrality battles are now being waged.

While Netflix's deals did resolve the congestion issues for users on these ISPs this past fall, the debate over this practice has since broken down into two distinct camps. One is Netflix, its transit partners, and consumer advocacy groups, who claim that the nation's biggest ISPs have intentionally let key transit points degrade, intentionally creating significant congestion and forcing Netflix to pay them directly if they want their video services to work. On the other side is the mega-ISPs, their PR folks and a smattering of industry consultants, who argue that Netflix is to blame by choosing Cogent as a cheaper transit link to these carriers.

The problem is that because so much of this data is confidential and ISPs aggressively prevent any meaningful access to raw network data (in part because congestion has always been used as a policy bogeyman to justify anti-competitive behavior and usage caps), clearly illustrating who is at blame here can be (quite intentionally) difficult.

However, as we noted late last month, a new study from M-Labs (an organization ISPs have historically tried to hamstring) clarified at least a few key points in the debate. One, the report put to bed (or should have) the mega-ISP claim that Netflix was intentionally choosing congested partners like Cogent to save a buck, noting that other mid-sized ISPs not pushing for direct interconnection deals (like Cablevision) were seeing no congestion over Cogent connections to Netflix. It also made something very clear: natural congestion wasn't responsible for the degrading of user Netflix traffic: it was a conscious ISP business decision for users to have degraded experiences.

Now to be clear there are no saints here, as Netflix, transit partners and last mile ISPs are all running face-first at the video cash trough. Also to be clear, M-Lab went out of its way to avoid naming ISPs specifically in the report, in part to avoid liability (remember Verizon threatened to sue Netflix for even suggesting the problem could be on Verizon's end), but also because the data proving culpability isn't yet complete.

Now if you've followed the adventures of AT&T, Comcast and Verizon over the years, it's very easy to come to some conclusions based on their long, proud history of anti-competitive behavior. Since most of the data is obscured, intentionally degrading transit connection points is a brilliant way to grab the extra cash from content companies through the kind of troll tolls they've long talked about. The FCC has requested confidential data from all parties and is investigating Netflix's claims, but it's not yet possible to completely prove shenanigans without stepping foot in the very rooms that provide interconnection and examining both the hardware and the data.

If the M-Lab study pushes the needle in any direction, it's indisputably toward the ISPs being at fault. Still, it has been entertaining to see some folks on the ISP side of the argument continue to declare Netflix the villain. Though that M-Lab study pretty clearly illustrates that Netflix's choice of Cogent wasn't the core problem (again because Cablevision saw no Cogent congestion), Frost and Sullivan analyst Dan Rayburn magically comes to the complete opposite conclusion. When Cogent this week acknowledged that the company had utilized some traffic management to try and improve Netflix streaming issues earlier this year, that was enough evidence for Rayburn to again declare Netflix the bad guy:

"This morning, Cogent admitted that in February and March of this year the company put in place a procedure that favored traffic on their network, putting a QoS structure in place, based on the type of content being delivered. Without telling anyone, Cogent created at least two priority levels (a ‘fast lane’ and ‘slow lane’), and possibly more, and implemented them at scale in February of this year. What Cogent did is considered a form of network management and was done without them disclosing it, even though it was the direct cause of many of the earlier published congestion charts and all the current debates.

In short, everything is Cogent and Netflix's fault. Again, nobody disagrees that there's too much secrecy in these agreements. But to hear Cogent tell Ars Technica their side of the story, the QOS (quality of service) implemented was only necessary because of intentional ISP failures to upgrade network hardware on their end of the equation:

"The system was put in place only because Internet service providers refused to upgrade connections to Cogent to meet new capacity needs, Cogent Chief Legal Officer Bob Beury told Ars. "The problem was, after years of upgrading connections as necessary to accommodate the flow of traffic as necessary at these peering points, Comcast, Time Warner Cable, Verizon, and AT&T refused to do so and let their customers be hurt,” he said.

It's worth noting that Cogent has been in more than a few major disputes over the years when it comes to settlement-free peering, and hasn't made a whole lot of friends in the process. It's also worth noting Cogent does appear to have been caught in a lie, its website proudly proclaiming it doesn't engage in prioritization of any kind. Since then, both Rayburn and a number of Comcast employees have been making the rounds, proclaiming this is, again, proof positive that Cogent is the bad guy in this entire affair. Except it's not.

The problem is that the idea that ISPs are neglecting basic upgrades to create a problem isn't just the opinion of Cogent, it's also the suspicion of nearly every consumer group in existence, smaller independent ISPs (like Sonic.net) as well as Cogent competitors like Level3. Level3 VP Mark Taylor made the compelling case back in July that Verizon was to blame for the Netflix streaming slowdown on Verizon's network:

"We could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on (Verizon's) routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they've run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it."

Another, earlier Level3 blog post claimed that all of the biggest ISPs were engaged in this behavior. Netflix too has numerous times now claimed that while the big ISPs insist these are all just run of the mill transit debates, what's happening with this latest debate is very, very different. Netflix has repeatedly insisted that ISPs are using their mono/duopoly control over the last mile to effectively charge for functional access to their customers, imposing arbitrary tolls to offload network operation costs to other companies.

Now it's possible that Netflix, Cogent and Level3 are all lying (something Comcast engineers have been telling telecom reporters), but you can't ignore historical context. If you've been paying attention, using monopoly power to impose arbitrary and unnecessary new tolls on content companies is precisely what the biggest ISPs have spent the last decade promising they would do. AT&T, Verizon, Comcast and Time Warner Cable have spent the last decade engaging in some of the most anti-competitive behavior in the history of any technology market, whether that's writing laws prohibiting cities and towns from deploying their own broadband, to using astroturf and propaganda to push anti-consumer policies. The discussion of their current behavior can't simply exist in a context-free vacuum.

This week, Rayburn's contrarian, vacuum-sealed analysis of the issue resulted in an interesting Twitter exchange between himself and Netflix's David Tempkin. While Rayburn is busy claiming he has confidential but unpublishable knowledge from the industry that somehow proves Netflix is the villain in this equation, Temkin, in turn, accuses Rayburn of engaging in "biased conjecture," arguing that only the FCC currently has all of the data they need to validate Netflix's claims that ISPs are up to no good:

And there's the rub. As somebody who has written about and studied Verizon, AT&T, Comcast and Time Warner Cable behavior for fifteen years, my gut tells me that yes -- they probably are intentionally degrading transit connection points, knowing that the resulting cacophony of often dull engineering analysis would act as cover for their bad behavior. Hiding anti-competitive shenanigans under intentionally over-complicated technical analysis is status quo for incumbent ISPs (again, see the usage cap debate or the wireless carrier blocking of disruptive tech for just two examples). But even if Level3 and and M-Lab's data confirms part of my suspicions, I can't claim to fully prove it without direct access to transit connection points and a close look at ISP hardware and traffic data -- no matter how many compelling charts I post. Neither can Dan Rayburn, Susan Crawford, or anybody else. Not quite yet.

That means there are two things we need here if we care at all about network neutrality and the future of Internet video. One is significantly more transparency across the board, from Netflix, transit partners, and last-mile ISPs, something Rayburn is spot on about. To that end, Jon Brodkin at Ars Technica filed a FOIA request with the FCC earlier this year in an attempt to to get a better look at the data behind this debate, and ran into a brick wall thanks to all of the companies involved in the fight:

"Verizon, Netflix, and Comcast filed requests for confidential treatment of the agreements in their entirety," the FCC's response to Ars said. "In support of its request for confidential treatment, Comcast asserts that, if its agreement were disclosed, competitors would gain valuable insight into the parties' business practices, internal business operations, technical processes and procedures, and information regarding highly confidential pricing and sensitive internal business matters to which competitors otherwise would not have access."

In other words, Comcast wants to quietly accuse everybody of lying, but neither they (or Netflix) want to release the data that proves it. Hiding ISP data from the public has also long been a favorite FCC hobby. Take for example the FCC's $300 million broadband coverage map, which took ISP claims of speeds and locations as gospel, but also completely omits any data on pricing at ISP behest. That lack of data pricing has allowed ISPs for years to claim that the broadband market is more competitive than it actually is. Without transparent access to hard data, most telecom policy discussions wind up being little more than verbose games of patty cake.

The other thing desperately needed is for the FCC to wake up and acknowledge that the debate over net neutrality has very intentionally been immeasurably more complicated, shifting from concerns about throttling or simple service blocking, to issues like interconnection and usage caps (and all the dangerous business models under consideration therein). If the FCC's new rules (hybrid or otherwise) don't incorporate both of these concerns (and so far Wheeler has stated they won't), they're going to fall well short of protecting consumers from the mega-ISPs with a noncompetitive stranglehold over the last mile.

from the ouch dept

As you know, despite Netflix having already agreed to pay Verizon's shake down fee to avoid interconnection congestion for its streaming, Netflix and Verizon are fighting a very public fight about who's to "blame" for the network congestion. Netflix took a public shot at Verizon by publicly highlighting Verizon's congested network to users, leading Verizon to send a cease-and-desist letter, claiming it was misleading. As we noted, there's no doubt that it's actually Verizon at fault, because it's not delivering exactly what it sold customers (and we wonder why the FTC is still not getting involved yet). If Verizon's network is getting congested, that's on Verizon to fix, since it sold its users a promise that they could reach anywhere on the internet, including Netflix.

Verizon tried to spin the story back in its favor last week, with a blog post about "the congestion myth," in which it claimed that the real problem was how Netflix chose to route its traffic to Verizon. It presented the following nifty chart, claiming that there was no congestion at all on Verizon's network, and saying that it was all about how Netflix was choosing to deliver its traffic to Verizon's network:

As you can see in the "red" arrow, it's showing that there's 100% utilization at the interconnection link. And Verizon claims this is all on Netflix. Here's Verizon's explanation:

One might wonder why Netflix and its transit providers were the only ones that ran into congestion issues. What it boils down to is this: these other transit and content providers took steps to ensure that there was adequate capacity for their traffic to enter our network. In some cases, these are settlement-free peering arrangements, where the relative traffic flows between an IP network provider and us remain roughly equal, and both parties invest in sufficient facilities to match these roughly equal flows. That is the traditional basis for such deals. In other cases there may be traffic imbalances, but the networks or content providers have entered into paid arrangements with us to ensure connections and capacity to meet their needs for their out-of-balance traffic.

Some reporters took this at face value, but it never made any sense at all. The chart above pretty clearly shows that the congestion point is actually Verizon's border router. And if it just made a basic upgrade to accept the traffic that it has promised to consumers, there would be no problem at all.

Level3 has now jumped into this debate as well, with even more data showing that Verizon is the real culprit here. Level3 is carrying a bunch of that Netflix traffic, and notes that it has more than enough bandwidth to carry it. It says the only problem is Verizon refusing to take 5 minutes to upgrade its system:

Verizon has confirmed that everything between that router in their network and their subscribers is uncongested – in fact has plenty of capacity sitting there waiting to be used. Above, I confirmed exactly the same thing for the Level 3 network. So in fact, we could fix this congestion in about five minutes simply by connecting up more 10Gbps ports on those routers. Simple. Something we’ve been asking Verizon to do for many, many months, and something other providers regularly do in similar circumstances. But Verizon has refused. So Verizon, not Level 3 or Netflix, causes the congestion. Why is that? Maybe they can’t afford a new port card because they’ve run out – even though these cards are very cheap, just a few thousand dollars for each 10 Gbps card which could support 5,000 streams or more. If that’s the case, we’ll buy one for them. Maybe they can’t afford the small piece of cable between our two ports. If that’s the case, we’ll provide it. Heck, we’ll even install it.

Level3 has provided a (not quite as nicely designed) image to zoom in on the border router situation, showing that it has plenty of capacity ready -- all it needs is for Verizon to let it connect more ports:

Again, this is what plenty of people have been saying since the beginning of this interconnection fight. Verizon, Comcast and AT&T have deliberately made the decision not to make rather basic and inexpensive upgrades to their interconnection points that would solve the congestion problems with Netflix. In doing so, they are the ones creating the bottleneck and congestion -- and effectively using it to shake down Netflix, getting them to pay extra for the bandwidth that the broadband providers' customers have already paid for.

Looking at this, it once again becomes clear that it's Verizon, AT&T and Comcast that have deliberately caused this congestion, using their positions as dominant players with monopoly control over the last mile to force Netflix to pay them extra. As Level3 notes, it takes two parties to take the "trivial" steps to remove the congestion, and it's Verizon that's the party who isn't cooperating:

All of the networks have ample capacity and congestion only occurs in a small number of locations, locations where networks interconnect with some last mile ISPs like Verizon. The cost of removing that congestion is absolutely trivial. It takes two parties to remove congestion at an interconnect point. I can confirm that Level 3 is not the party refusing to add that capacity. In fact, Level 3 has asked Verizon for a long time to add interconnection capacity and to deliver the traffic its customers are requesting from our customers, but Verizon refuses.

As we've discussed, Verizon, Comcast and AT&T know exactly what they're doing here. People hadn't been so concerned with interconnection disputes in the past, because they didn't think the big broadband players would be so crass and so anti-consumer to purposely let interconnection points clog up. But, those three companies have such control over the market at this point that they are able to do that and can effectively shake down internet companies to get them to double pay for the bandwidth that subscribers are already paying for. Today it's Netflix, but soon it's likely to be lots of other companies as well. That's why, in our own comments to the FCC, we noted that the interconnection fights need to be a part of the open internet discussion.

from the but-still-won't-call-it-net-neutrality dept

Throughout the most recent version of the net neutrality fight, Tom Wheeler and the FCC has worked hard to keep interconnection issues a separate debate, even though interconnection is likely where the real fight has moved. In short, if you haven't been following this closely, net neutrality has historically been about discrimination on the last mile -- from your broadband access provider to your home -- but over the past couple of years, the big broadband companies (mainly Comcast) have realized they can get the same basic result (getting big internet companies like Netflix to double pay for your bandwidth) by clogging up the transit, and getting Netflix to pay up to interconnect directly. However, in the minds of most people, it's the same thing. Even while the congestion is happening on the network, the end result for everyone is effectively the same: Netflix has to pay to get a quality stream to you, your connection sucks if they don't pay, and Comcast collects all the money. So, when John Oliver did his piece on net neutrality, he actually illustrated it with the interconnection battle. And that's because it's really the same thing.

Well, it's really the same thing to just about everyone... except the FCC. The FCC's request for comments explicitly tries to avoid delving into the interconnection fights, but thanks to things like John Oliver's coverage, many of the comments the FCC has been receiving have been about those issues anyway. Apparently realizing that he can't avoid the issue, FCC boss Tom Wheeler has announced that he's now "gathering information" on these interconnection fights and has specifically asked Comcast, Verizon and Netflix to hand over the details of their arrangement. Wheeler even quotes one of the comments that the FCC has received during the NPRM comments that talks about the interconnection battle, and notes that there are many more like that.

In reading the emails I receive, I thought this one from George pretty well sums up public concern:

Netflix versus Verizon: Is Verizon abusing Net Neutrality and causing Netflix picture
quality to be degraded by “throttling” transmission speeds? Who is at fault here? The
consumer is the one suffering! What can you do?

We don’t know the answers and we are not suggesting that any company is at fault. But George has gone
to the heart of the matter: what is going on and what can the FCC do on behalf of consumers? Consumers
pay their ISP and they pay content providers like Hulu, Netflix or Amazon. Then when they don’t get
good service they wonder what is going on. I have experienced these problems myself and know how
exasperating it can be.

Consumers must get what they pay for. As the consumer’s representative we need to know what is going
on. I have therefore directed the Commission staff to obtain the information we need to understand
precisely what is happening in order to understand whether consumers are being harmed.
Recently, at my direction, Commission staff has begun requesting information from ISPs and content
providers. We have received the agreements between Comcast and Netflix and Verizon and Netflix. We
are currently in the process of asking for others.

To be clear, what we are doing right now is collecting information, not regulating. We are looking under
the hood. Consumers want transparency. They want answers. And so do I.

That sounds good, but we'll see what actually comes of it. The fact that Wheeler has tried hard to separate interconnection from net neutrality hasn't been particularly encouraging. The personable "I've experienced these problems myself" is nice, but it means little if the FCC doesn't actually realize what's going on here. Also, the quote at the end about transparency also sounds good, but we'll have to see if the FCC actually lives up to it and shares the details or keeps the whole process secret.

AT&T launched a new billing program called Sponsored Data Monday at its developer conference at CES, which shifts mobile data costs from the consumer to the content provider. The idea is to create a two-sided charging model for mobile data, letting app developers and content providers foot the bill for their customers’ data use.

AT&T and other carriers have been hinting at such a subsidized mobile internet for some time, but this is the first time that it’s actually put those ideas into practice. Under the program a content or service provider would pay AT&T to exempt their app, websites or even specific bits of content from consumers’ mobile data plans. Anytime someone consumed such exempted content on the mobile network, AT&T customers wouldn’t see it deducted from their data buckets. Instead, AT&T would subtract that data from a kind of universal data pool bought by the content provider.

By having developers and providers pay the "freight" for data, AT&T will once again be derailing net neutrality. A system like this will obviously favor deeper-pocketed entities, raising the barrier to entry for everyone else. Subscribers, nearly all of whom now have data caps, will be much more likely to use services and apps that don't cut into their monthly allotment. AT&T will also effectively collect twice on the data, once with the monthly service charge to subscribers (that isn't reduced if customers don't hit their caps) and once from any developers/providers who buy in.

AT&T tried to spin this positively, saying that purchasing a data allotment would work as advertising for apps, services or other content providers. "Free" means more subscribers should take advantage of the offerings, resulting in more traffic and business. Its reps were careful to mention that those "sponsoring" data would not receive preferential treatment in terms of having competitors throttled (as it has done in the past), but that obviously won't be necessary when data-hungry customers are looking for the best deal, data-wise.

AT&T even suggested this new model could be useful for BYOD businesses, allowing charges for data consumed on work-related apps to be covered by that business, rather than the subscriber. For subscribers, this may look like a great deal, but for AT&T, it's a new revenue stream.

But it's what's tacitly admitted by this program -- something AT&T avoids addressing -- that's the most interesting. Giving providers and developers the option to pay freight on data exposes these data caps for what they are: an arbitrary limit that exists only as new source of revenue.

Data-heavy apps and services will be the ones most likely to take advantage of AT&T's sponsored data program. Customers will naturally gravitate towards anything that doesn't eat into their data caps, especially if it's something like a streaming audio or video app/site. Any developer that buys in will see an uptick in use and the more data-heavy the offering is, the more likely it is that customers will take advantage of the "free" data.

Because of this, data usage by customers will go up, with the most data-heavy apps and services seeing the biggest increase in usage. According to the company line, caps are in place to prevent data hogs from creating network congestion and degrading service. But this program effectively encourages users to consume more data and use more data-heavy services. As long as AT&T is still making money (via data "sponsors"), then all previous handwringing about network stability no longer matters.

AT&T is likely facing a decline in income for data overages as consumers become more adept at staying under their data caps. The trend should remain unchanged, meaning fewer overages as time goes on. This fixes the income problem but does nothing to address the (false) concern AT&T deployed to justify its implementation of data caps. AT&T wants everyone to use more data, which will increase the amount it can collect from providers and developers. All in all, it's more evidence that equating data caps with network capacity is nothing more than a lousy spin job attempting to justify the replacement of unlimited data with multiple revenue streams.

from the of-course-not dept

For years, the key rationale given by broadband providers for implementing data caps was that it was the only way they could deal with "congestion." Of course, for years, independent researchers showed that this was bogus, and there was no data crunch coming. If you actually caught a technologist from a broadband provider, rather than a business person or lobbyist, they'd quietly admit that there was no congestion problem, and that basic upgrades and network maintenance could easily deal with the growth in usage. But, of course, that took away the broadband providers' chief reason for crying about how they "need" data caps. The reality, of course, is that data caps are all about increasing revenue for broadband providers -- in a market that is already quite profitable. But if they can hide behind the claims that they need to do this to deal with congestion, they can justify it to regulators and (they hope) the public.

Of course, enough people have been calling this explanation out as completely bogus that it appears that even the broadband companies' own lobbyists may finally be dropping this line of reasoning. Former FCC boss Michael Powell, who is now the cable industry's chief lobbyist (president of the National Cable and Telecommunications Association -- NCTA), has finally admitted caps aren't about congestion:

Michael Powell told a Minority Media and Telecommunications Association audience that cable's interest in usage-based pricing was not principally about network congestion, but instead about pricing fairness...Asked by MMTC president David Honig to weigh in on data caps, Powell said that while a lot of people had tried to label the cable industry's interest in the issue as about congestion management. "That's wrong," he said. "Our principal purpose is how to fairly monetize a high fixed cost."

Of course, as Broadband Reports notes, Powell is jumping from one myth (congestion) to another (fairness) that is just as ridiculous. If it was true, we'd see at least some prices going down. But we don't.

Except the argument that usaged pricing is about fairness has been just as repeatedly debunked. If usage caps were about "fairness," carriers would offer the nation's grandmothers a $5-$15 a month tier that accurately reflected her twice weekly, several megabyte browsing of the Weather Channel website. Instead, what we most often see are low caps and high overages layered on top of already high existing flat rate pricing, raising rates for all users. Does raising rates on a product that already sees 90% profit margins sound like "fairness" to you?

Data caps are about one thing only: increasing profits for the broadband providers, who already have massive control over the market with limited competition. It's nice to see them give up on one myth (even if we still see pundits repeating it without criticism), but it would be nice if we could get past the others as well.

from the not-nice dept

While the broadband providers often talk up the need to break network neutrality in order to avoid "congestion" problems, most people have recognized that's just a smokescreen. The congestion issues are not an issue at all. Broadband costs have been going down, consistently, and most network engineers admit that with basic upgrades (nothing out of the ordinary), there's no bandwidth crunch to worry about. The real reason why broadband providers are interested in breaking network neutrality is because many of them want to get into the content business -- and they don't want to compete on even ground.