The Cliff and the Slope

The proof is in: Detailed report shows how U.S. Internet access monopolies punish rivals and catch innocent bystanders in the crossfire—legally.

Devan Dewey, the Chief Technology Officer of midsize investment consultancy NEPC, has an orderly office and a highly organized mind. So naturally, when some at-home employees near Boston complained they could barely work because their connections to the company data center had slowed to a crawl, Dewey and his team determined to find out why.

His team’s research led him to suspect something astonishing and dark: that NEPC, and probably many other businesses and consumers, were caught in the crossfire of an ongoing battle between “eyeball networks” run by Internet access providers, such as Comcast and Verizon; and “transit networks” used by competing video services, such as Netflix. He came to wonder whether, in their attempts to charge Netflix for access to their subscribers, Comcast and some other networks were recklessly affecting Internet connectivity for businesses like NEPC. Could that possibly be true?

The answer is yes. What started out as suspicion is now fully documented, in a study that has just been released by a nonprofit research consortium called M-Lab. M-Lab’s data suggests the logical conclusion that Verizon and Comcast, as well as Time Warner Cable, CenturyLink, and AT&T, are intentionally squeezing data coming from some incoming networks — in particular, networks associated with Netflix, which competes with these companies in video entertainment. Customers of these eyeball networks are getting degraded service that cannot be explained by anything other than business decisions. And these eyeball networks are acting with an apparent disregard for users not affiliated with Netflix, affecting all kinds of traffic and all kinds of users. By tacitly allowing network traffic jams — affecting only the highways of fiber that Netflix was using to send its bits — everyone else using those routes was getting stuck. NEPC employees working from home, for instance, could barely operate.

The revelations of clear service bottlenecks offer rare first-hand evidence of the power of large Internet access monopolies to force companies that require access to their networks into costly service arrangements, or else suffer degraded connectivity. This is the kind of game-playing feared by the millions of Americans who have pushed for rules preventing what faux news comedian John Oliver calls “cable industry fuckery.”

Here’s what happened to NEPC: Beginning in November 2013, employees who were working remotely began having trouble accessing NEPC’s servers. Employees who access their NEPC files by way of a remote desktop platform were waiting for minutes at a time for their files to open, and the quality of their phone calls — which are also routed over NEPC’s network — was wildly inconsistent. By January, the drip of employee complaints had become a torrent. Remote access had become untenable. Calls were dropping right and left. Files were freezing and not opening for minutes at a time. Productivity suffered and NEPC employees were, understandably, enormously frustrated — which led to a deluge of complaints to Dewey’s support team. Employees who had been used to having state-of-the-art access to their work materials from home or on the road started working in the middle of the night on the off chance that connectivity would be better. One employee said she would have to start working from the Boston office; she could no longer work from home, even though she’d made the move to full-time remote work so as to be available to her four children in a house far away.

Dewey and his staff were frantic. It was a complete mystery to them why NEPC employees were having trouble accessing their files and making phone calls; they couldn’t replicate the problems their employees were experiencing. Dewey’s team spent six weeks investigating every detail of NEPC’s network services.

NEPC has a private fiber network connecting its seven offices (and connecting those offices to the Internet), but NEPC’s data center connects to the Internet through Cogent, a company that sells both Internet access and inter-city transit services; according to Dewey, Cogent’s $600/month 100 Mbps service has historically been trouble-free. NEPC employees working remotely have VPN software that allows them to access the firm’s files. Dewey’s staff checked the software: no problems. Dewey and his team help employees set up their home Internet access — and require employees to subscribe to at least 10 Mbps download and 5 Mbps upload services — and there were no problems with either the Comcast or Verizon FiOS Internet access services the employees were using. Everything checked out, but nothing worked.

Dewey was baffled: Who or what was choking NEPC?

Although many people use the Internet in America, the physical lines that carry data from one place to another are invisible — and therefore mysterious to most of us. Let’s assume you’re one of the twenty-one million Comcast high-speed data subscribers. Comcast owns its own lines that transport your data from your living room to a handful of points in the U.S. for handoff to other networks. Seven key interconnection points — in New York City, Chicago, Seattle, San Francisco, Los Angeles, Dallas, and Miami — collectively handle about ninety percent of all handoffs among networks in the country. In these cities, hundreds of disparate networks meet in buildings, co-locating their facilities so that data packets can be easily handed off from one network to another. They are extremely large hubs of Internet inter-networking activity; three New York City points, for example, complete handoffs for most of the east coast of the United States.

Why handoffs? Comcast, in this story of NEPC’s travails, is wearing the hat of an “eyeball” network; it serves end-user individuals and businesses that are using browsers to request data from computers in other places. These end users often ask for data from networks and data centers that are not provided or hosted by Comcast. (Like Google. Or YouTube. Or Netflix.) And that means that a handoff between networks is needed.

Indeed, the point of the Internet Protocol was to allow disparate networks to interconnect easily; having a standard size unit of data (known as a “packet”) and addressing system allows traffic to travel around the world without asking permission from individual systems, making it possible for end-users to request packets from computers that are not physically controlled by their Internet access providers — their eyeball networks.

That original inter-networking idea came from an era in which inter-city “transit” (sometimes called “backbone”) networks were competing for customers. Policymakers assumed that it would always be in the interest of physical network suppliers to cooperate with one another at interconnection points. The market would ensure that these forces remained in balance, protecting both consumers and data packets from exploitation.

Things have changed. For starters, there has been tremendous consolidation in the last-mile wired marketplace. Just three providers, Comcast, Verizon, and Time Warner, account for almost half of the wired last-mile access marketplace in America. End-users, encouraged by access providers interested in replicating the pay TV model online, now ask for far more data than they generate. And the architecture of U.S. Internet access encourages this passive downloading behavior: the cable modem Internet access network, on which most Americans rely, substantially favors downloads over uploads. The three major last-mile providers also have nationwide networks of their own, which means that they do not have to rely as much on transit networks to carry traffic throughout the U.S. You can think of Comcast’s eyeball network as a giant castle surrounded by a giant moat. Inside the castle are all of Comcast’s territories in thirty-nine states. Outside the moat is the rest of the Internet, including all the transit networks carrying traffic requested by Comcast subscribers. Finally, the entire high-speed Internet access marketplace has been deregulated, meaning no level of government exercises oversight over the castle — or the terms on which the castle allows the moat surrounding it to be crossed.

These changes have put vast power in the hands of Comcast, Verizon, and Time Warner Cable nationwide, and in the hands of AT&T and CenturyLink in many U.S. markets. These companies are gigantic “terminating monopolies,” because the only way for traffic destined for Comcast’s subscribers coming from data centers or networks not controlled by Comcast — traffic coming from outside Comcast’s castle moat — to reach those subscribers is to go over Comcast’s lines.

In the past, if two networks transferred so much data between themselves that they were about to exceed the capacity of their connection, they would have gotten in touch to solve the problem. As M-Lab notes in its report, “[T]he traffic that flows through these interconnections is the lifeblood of the Internet — nearly all of the value of the Internet comes from the exchange of traffic, even when the ISPs involved are fierce competitors.” The engineers would have worked out a solution to open the access network’s door to the outside world more broadly. And they would split the minor costs of doing this upgrade — a $300 piece of fiber, a $10,000 souped-up router. A January 2013 OECD report found that 99.5% of Internet interconnection agreements at Internet Exchange Points happen without any formal contracts; engineers easily make deals to share the very low cost of trading traffic between networks in the same building.

But that was the past. Today, we have communications giants who see no need to adhere to traditional Internet niceties. Comcast, Verizon, and Time Warner Cable are now powerful enough that they can demand that they be paid for connecting with other networks. Their power comes from their huge numbers of subscribers; other networks need Comcast, Time Warner, and Verizon more than these eyeball networks need them. If the eyeball networks aren’t paid, they will refuse to upgrade the doors between their eyeballs and the network seeking to connect. If that upgrade doesn’t happen but the eyeballs keep asking for more and more data — because, say, they want to watch movies online from Netflix — packets get dropped. And if packets get dropped, hourglasses spin and screens freeze.

In other words, it’s possible to mess up the user online experience by doing nothing — not upgrading facilities when traffic patterns call for it. If the congestion just happens to affect the business of a competitor, it’s in the interest of an eyeball network to do so.

Now, thanks to M-Lab, we have definitive data about what is going on between end-users and sites on the other side of the (metaphorical) bridge that runs between the user’s eyeball network and an outside network: we can measure how many packets can be sent between two networks successfully each second (“throughput”), how quickly packets can travel — how many milliseconds it takes them to make the trip — across an interconnection point (“latency”), and what gets lost over an interconnection point that hasn’t been upgraded to higher capacity, as packets that aren’t acknowledged get re-sent (“retransmission”). All of these measurements are captured for a representative sample of the U.S. population by M-Lab at very frequent intervals over a long period of time — beginning in 2009 and continuing to this day — which allows us to compare what is going on at (for example) 7pm each day in different locations. And guess what — the painstaking research reveals patterns that look exactly like the destructive jamming-by-doing-nothing gambit described above. It turns out that the evening hours between 7pm and 11pm local time, peak hours of Internet usage according to the FCC, are particularly painful times for packets flowing between the eyeball networks and other places on the Internet.

The M-Lab data reveals two striking patterns of interconnection problems; The Cliff and The Slope. The Cliff afflicted Cogent in its relationships across the country with Comcast, Verizon, AT&T, CenturyLink, and Time Warner Cable from the spring of 2013 until early 2014. The Slope afflicted another non-eyeball network, Level 3, in its relationship with Comcast, Time Warner Cable, and Verizon over this same time period in several markets, and its effects appear to be continuing.

First, The Cliff. In the spring of 2013, Netflix began moving more of its business to Cogent. Cogent’s traffic to Comcast’s, Time Warner Cable’s, and Verizon’s key interconnection points across the U.S. went way up — because Comcast, Time Warner Cable, and Verizon customers were asking for Netflix videos. In a competitive market, the eyeball networks would have had every incentive to upgrade their connections to Cogent to ensure that their subscribers continued to have a good online experience. But the opposite happened.

M-Lab’s data conclusively shows that Cogent’s connections with the large eyeball networks, in every part of the country where M-Lab gathered data, subsequently fell off a cliff. Conditions were bad for Cogent’s packets in Dallas, Los Angeles, and Seattle: in Dallas and Los Angeles, Cogent connections with Comcast, Verizon, Time Warner Cable, AT&T, and CenturyLink were congested. In Seattle, Cogent had problems with Comcast and CenturyLink. Life for Cogent packets was particularly rough in New York City. Before the Netflix-Cogent deal, peak rates (when throughput matters most) for traffic traveling from Cogent to Comcast and Verizon customers averaged 20–25 Mbps. But in January 2014 traffic traveling from Cogent to Comcast and Verizon subscribers dropped to speeds of less than 0.5 Mbps during peak use hours — the minimum rate needed for web browsing and email, according to the FCC. Users of Comcast, Verizon, and Time Warner Cable also saw significant slowdowns in the form of packet retransmission rates and increased round trip times for traffic coming from Cogent.

The data shows that for most of each day between May 2013 and March 2014 major New York City interconnection points between Cogent and the major eyeball networks were running at full capacity — meaning there was insufficient headroom for the packets being requested by subscribers. Packets were being dropped.

Shockingly, Comcast, Time Warner Cable, and Verizon failed for about nine months to provide their customers with throughputs above 4 Mbps (the FCC’s minimum for a “broadband” connection) when connecting to Cogent traffic. Within a few days after Netflix and Comcast agreed to a deal, traffic carried by Cogent was flowing normally to Comcast subscribers — for whatever reason — again.

When Comcast and Verizon customers complained during this period about malfunctioning online applications — which were not always Netflix — they were unable to figure out what the problem was.

Post on a Comcast user forum, dated February 25, 2014:

My needs are simple — I work in a local university hospital, and sometimes need to connect from home overnight or on weekends for urgent patient cases. So when I’m not using the connection as a home internet connection, I primarily connect to a VPN with a Citrix server, which hosts some proprietary software that displays certain patient data and relevant video. Video is vital to what I do, so I require reasonable speed. At certain times of the day I’ve managed to get 15mbit/s down, and video runs at a decent speed. At peak times, however, I rarely see speeds upward of 700kbit/s down from the VPN, and the video is so slow as to be unusable, I might as well hop in my car and drive to work. . . .

I have tried our local IT contacts, but they have been of limited assistance (of the “unplug and reboot your computer” variety).Thanks!

Paul Davis, a network engineer for American Fiber, Inc., said in a Comcast user forum in February 2014,

Now for The Slope. In Atlanta and Chicago, transit network Level 3's connections with eyeball networks Comcast, Time Warner Cable, and Verizon have suffered from congestion issues.

Roundtrip time — latency — is up, particularly between Comcast and Time Warner Cable and Level 3. Throughput — the number of packets allowed through the door between the two networks — is down, particularly between Verizon and Level 3. The uniformity of these issues in different geographic areas between the same pairs of companies signals that this is not a technical problem. This is a business dispute that is harming American consumers.

Across many of these connections, the effect on consumers continues. The practical dimensions of these effects vary. If a delayed or dropped data packet is part of an email, then there is probably a short delay in the delivery of that message. If the data packet is carrying information for a performance-sensitive application like a voice phone call or a streaming video, the user may be experiencing much more trouble — garbled calls and degraded video. Applications (like Netflix, Sony, Apple, Google, Amazon, and others) likely are having more trouble working and online streaming services are likely affected (Major League Baseball, for example). Problems are particularly bad at peak usage times; the doorway between the two networks is simply too narrow for the amount of data that needed to be handled. Level 3 is the largest transit provider in the world, but data coming from Level 3 is not being treated well.

Why would Comcast, Verizon, and Time Warner Cable make their users’ experience accessing the online world worse? The obvious answer: money. In the Cogent “cliff” context, Cogent had signed a deal with Netflix, and Comcast and Verizon wanted to make sure that they were getting a slice of Netflix’s consumer revenue. Also, Comcast sells its own transit network services, and would be happy to sell those services to customers — so Comcast is effectively competing with Cogent, Netflix’s transit carrier, and Level 3 — as well as Netflix.

In the Level 3 “slope” context, it is the naked principle of the thing: eyeball networks want payment before they will upgrade their interconnection points to the transit networks.

And the payment will be whatever the eyeball networks ask. “Paying for upgrades,” in the eyeball networks’ view, means paying an invented cost for traffic volume plus whatever equipment costs that Comcast and the others say they need to invest to handle the traffic. Such costs may be minimal—a $300 piece of fiber or other cheap interconnection equipment upgrades. But as last-mile monopolies, the eyeball networks can invent fees and then assess them, and force everyone else in the system to pass those costs on to subscribers.

In this case, there seemed to be no upgrades, despite a clear need for them. Thus, as traffic predictably grew on Cogent’s network from increasing numbers of Netflix subscribers streaming videos, but the width of Cogent’s doorway into Comcast’s and Verizon’s network didn’t change, packets started dropping. And Devan Dewey started getting inexplicable complaints.

Before M-Lab’s data, of course, affected customers and businesses had no way of knowing why the Internet access they had paid for was not being delivered. Dewey scrambled for weeks, trying to figure out why his employees could no longer do their work. Finally, after hiring an Internet “health” monitoring firm, Dewey’s team discovered a chokepoint: the interconnection point in New York City. Packets traveling from NEPC’s offices to NEPC’s workers via Cogent were being lost at the moment Cogent handed those packets to Comcast and Verizon in New York City for delivery over the eyeball networks to the worker. Just two percent of packet loss can make a perceptible difference in the quality of a phone call; at the height of NEPC’s problems as many as 20% of their packets weren’t making it through, which is why the firm’s remote work ground to a halt.

Cogent confirmed there was an issue: “We have a problem in New York,” Cogent told him. Dewey ended up buying a second Internet connection from NEPC to the outside world in addition to his Cogent line, and spent dozens more hours getting that circuit up and running. All told, Dewey estimates that his firm spent about two hundred employee-hours trying to diagnose and fix the problems the firm’s employees were encountering — not counting the time wasted by frustrated NEPC employees who were unable to work without interruption or delay.

In part, the reason why it took Dewey so long is that the answer is counter-intuitive. NEPC had nothing to do with Netflix, so why was it affected by Comcast’s and Verizon’s apparently anti-competitive activity? The answer requires a bit of explanation.

Cogent sells two flavors of services to businesses: Internet access — the connection that NEPC bought for its operations center — and Internet transit, or connections between networks.

Cogent has assembled its network by leasing segments of fiber across the country and it aggressively charges less in order to gain more customers; it connects to 1400 office buildings and would like to have more data centers on its list. NEPC’s problem was that it had bought the first flavor of service from Cogent: NEPC is one of many professional services firms (legal, financial, advertising, consulting) located in multi-tenant office buildings that use Cogent for Internet access.

But Cogent was being systematically disfavored by Verizon, Comcast, and the other providers because of its sale of the second flavor of service — transit — to bandwidth-intensive Netflix. Both flavors of services look the same from the perspective of a fiber network; they’re just packets flowing over Cogent’s leased or owned facilities. When the Cogent transit network seeks to interconnect with the Comcast or Verizon eyeball network to hand off packets, things can evidently go wrong. NEPC’s communications cataclysm was a byproduct of the battle between eyeball and transit networks. In order to make life miserable for Netflix and force that company to share its revenue with the eyeball networks, Comcast and the others had simultaneously made life miserable for many other companies.

Comcast’s response: this is business as usual.

“The M-Lab Interconnection Study only confirms what network engineers have always known: If a network does not obtain sufficient capacity to deliver its content to another network that content runs the risk of being transmitted over congested links which could degrade performance.

That is why Comcast reached out [to] transit providers, CDNs, and content providers to negotiate business agreements that would provide them with the capacity they needed, as we explained in our sworn declaration before the FCC.

As our declaration shows, Netflix hailed our agreement as great for consumers, and affordable for them.”

[The Comcast statement ignores Netflix’s more recent clarification to the FCC on why that agreement occurred; “It is extortion when Comcast fails to provide its own customers the broadband speed they’ve paid for unless Netflix also pays a ransom.”]

This interconnection problem is not confined to the world of telecommuting. Andrew Boegly is the Chief Information Officer of the Colonial School District in Plymouth Meeting, Pennsylvania — a Philadelphia suburb. He oversees a state-of-the-art fiber optic network, a 10 Gbps backbone, that serves the district’s 4,633 students and 718 teachers and staff. Since about May of this year, however, Colonial’s Internet access connection to the outside world has been intermittently experiencing severe packet loss, causing enormous practical problems. Colonial’s payroll data is due to the processing company by 8pm every other week in order for employees to be paid two days later, and Colonial is required to send its payroll information online. But recurring connection issues have meant racing against the clock to re-submit the payroll data multiple times until it goes through. “A lot of times it came down to the last minute with us calling the end [payroll] provider who actually did it manually for us because we couldn’t do it,” he says.

Boegly does not have a large tech staff at his beck and call, but he and his team put in countless hours trying to trace the problem. “It was a lot of wasted time, a lot of impact for us in terms of employee work,” says Boegly. They determined that they were suffering packet loss — a network along the way was ignoring and not forwarding some of the data they were sending — a sure sign of network congestion. Because Boegly uses a Citrix application for payroll data, and “Citrix doesn’t like packet loss,” according to Boegly, the process was failing. After getting in touch with Cogent, with whom Colonial’s aggregating larger school district, the Montgomery County Intermediate Unit or MCIU, contracts for Internet access service, Boegly learned what was happening and why.

After Colonial’s payroll data leaves its network it is transmitted via fiber to the MCIU. From there, the MCIU forwards it on to Cogent, which transports it to an interconnection point with Verizon for delivery to its final destination. And guess what happens next? Just after the handoff from Cogent, Verizon loses many of the packets, often causing the data transfer to fail. “Any time you have packet loss over a Citrix connection, it’s going to dump you,” says Boegly. “And that’s exactly what’s happening in the middle of the process. It has to then be started all over again and then begin the transfer process to get it to connect.”

When Boegly complained, he got a classic runaround: Cogent blamed Verizon for not upgrading its ports, and told Boegly the problem had to do with Netflix. After working with Verizon, going up the ladder from the online tech support team to senior level Verizon customer service, Boegly was told by Verizon that the problem was Cogent’s fault. Meanwhile, Colonial’s problems persist.

Boegly says he feels “helpless.” “You know, I contacted the FCC. I contacted the [state] utilities commission, just for help because you couldn’t make any ground with Verizon, Cogent, or anyone,” he says. “It’s just not a priority for anyone.” Colonial pays quite a lot of money for its connection, and Boegly has spent an enormous number of hours trying to sort out this problem. There has been no progress. As a stopgap, Boegly purchased a wireless card to use when he has connectivity issues. The school district with state-of-the-art fiber will now submit its payroll using the equivalent of a cell phone connection. “It’s the giants that are out there that are not really paying attention to these little incidents, which may be small for them, but they’re large for us,” says Boegly.

M-Lab’s thorough research substantially narrows the likelihood that the problems faced by consumers as a result of these congestion issues are caused by anything other than business decisions about interconnection.

Consider this: The lab’s newly-published study shows that NYC-interconnected traffic from eyeball subscribers of Verizon, Comcast, and Time Warner Cable had no trouble reaching M-Lab nodes hosted on Internap, a different content network — a network that isn’t Cogent or Level 3. Performance between Internap and users was exponentially healthier than that between Cogent and those same users: data traveling from Internap to Comcast users zipped along at download speeds of 12 or more Mbps and experienced far lower retransmission and latency rates.

This means that what went wrong with the traffic coming from Cogent happened at the interconnection point. There does not appear to be any congestion inside the eyeball networks themselves.

One possibility, of course, would be that the problem is with Cogent’s network. But that is not the case. We know this because the M-Lab data also show that Cablevision users in NYC did not experience problems with traffic coming from Cogent between June 2013 and March 2014. Cablevision maintained parity and high throughput for both Cogent and Internap throughout this period.

Why would Cablevision, which is not available in Comcast or Time Warner Cable territory, act differently? Because, unlike Comcast, it has to compete. More than half of Cablevision’s territory overlaps with that of Verizon FiOS; only 7% of Comcast’s and 11% of Time Warner Cable’s does. And so Cablevision (but not Comcast or TWC) makes every effort to ensure that Netflix works well for its customers, including by allowing Netflix to bring its content inside its network—via Netflix’s “OpenConnect” content delivery network—at no cost. Cablevision, unlike Comcast or Time Warner, is at risk of losing its customers to Verizon’s fiber services.

In a competitive market customers can switch operators on the basis of poor performance. Here, however, entire connecting networks are being discriminated against in a way that will be difficult for a normal human being to detect. And if the customer does figure it out, in most parts of America they will have little ability to switch. The large cable operators — who never compete with one another — control 80% of lines capable of 25 Mbps or greater Internet data downloads. Verizon FiOS, which could compete effectively with cable, will stop rolling out services once it reaches 20 million households. And even where it does have fiber optic services in place, Verizon is playing the same interconnection games. So there is nowhere for customers to go.

Furthermore, this is a national problem. M-Lab data shows that customers of Comcast, Verizon, AT&T, CenturyLink, and Time Warner Cable, in different markets and in different ways during the period between June 2013 and February 2014, have experienced persistent and extreme degradation of performance when requesting content residing outside the castle moats of the eyeball networks. In some cases, these issues are continuing today. The pattern is clearly consistent with a systematic and simultaneous refusal on the part of these major eyeball networks to upgrade connections between their networks and Cogent and Level 3. Even if it ruins the business of some of their customers.

Verizon’s response to the M-Lab report (from David Young, the company’s Vice President for Federal Regulatory Affairs), toes the party line:

“It is a very interesting report which brings new data to light that clearly documents how the changes Netflix made to its routing of video traffic suddenly caused congestion on links between the transit networks it was using (Cogent, Level 3 and XO) and the end-user ISPs who were not directly connected to Netflix’s Open Connect CDN (everyone except Cablevision and Cox). It confirms [that] Netflix [is] ultimately responsible for the dramatic, simultaneous decline in Netflix performance for all non-OpenConnect ISPs.”

And, from AT&T:

“Commercially negotiated peering relationships have existed since the inception of the commercial Internet. The goal of commercial peering arrangements is to allocate infrastructure costs between the two networks as close to equal as possible. Consequently most commercial peering arrangements have a component that takes into account balance of traffic. When one side of a commercial peering arrangement sends significantly more traffic than it receives, the allocation of infrastructure costs described above gets skewed. If the sending party refuses to take steps to bring balance back into the arrangement, congestion can result. Since the entire Internet operates on a best efforts basis, severe congestion can only be addressed by a new commercial arrangement involving steps to either reduce the traffic flows, route the traffic in a more efficient way, invest to augment capacity, or some combination of all three. This is how the Internet has always worked— and it works very well.”

While one cannot get inside the corporate minds of the Internet access giants, one does not need a conjurer to see the obvious motive for this: the eyeball networks think they should be paid for access to their subscribers.

In the past, requests for upgrades were routinely granted. Now, suddenly, upgrades are impossible without painful negotiations over fees that have no perceptible relationship to the cost of making the upgrade — and Comcast and the other eyeball networks are making no promises about restraining themselves in the future.

Most of the Americans who are online rely on the eyeball network services of Comcast, TWC, AT&T and Verizon. More still use regional networks that also rely on these providers. If you add in wireless access, almost two hundred million subscribers have been affected by the tussles between the eyeball and content networks in 2013–2014 — not counting the corporate customers of Cogent, Level 3 (and XO, another transit network mentioned in the M-Lab report) who, like NEPC and Colonial, experienced problems. Every one of those customers likely expects that their monthly payments give them unfettered access to everything on the Internet — outside the castle moat — on a non-discriminatory basis. But that is not happening, and the tussles continue. More hospitals are using telemedicine services these days; more schools are providing distance education. All of these consumer uses and services are hostage — potentially — to the power of the eyeball networks to exact tribute.

But in the meantime, customers are at the mercy of the Internet access giants. On Sunday, February 23rd, 2014, Comcast and Netflix issued a joint press release with the title “Comcast and Netflix Team Up to Provide Customers with Excellent User Experience.” Netflix now contends that behind the sunny headline is a tale of what they consider a form of extortion: pay or lose customers. They paid, and now Comcast customers will pay twice for Internet access: the price they pay to Comcast, and the price of the Comcast interconnection fees reflected in their Netflix bill.

According to Matt Wood, policy director for Free Press, the M-Lab report makes clear just how bad the situation is: “Cable companies always say that somebody has to pay for all of this traffic. Well, somebody does: cable customers pay hand over fist for high-speed Internet access, and they should have the right to use the connections they pay for however they want.

We see even more clearly now how Comcast and other Internet access providers cause artificial congestion, and make their own paying customers suffer the consequences, to generate new revenue streams for their bottleneck businesses.”

And what of NEPC, the company whose telecommuting employees stopped being able to function? While the flow of bits has returned, there is no comfort. “We’re still at risk,” Dewey says. Because Comcast and Verizon (and, it turns out, Time Warner Cable, AT&T, and CenturyLink, in their respective markets) have the market power to shape traffic flow across the connections into their networks from the outside world, they can make life miserable for packets trying to reach subscribers at any moment, for any reason. Think Chris Christie and the George Washington Bridge: these eyeball networks have the power to move the traffic cones at the chokepoints into their networks for any purpose, and the cars trying to cross the bridge (the equivalent of individual users’ data sessions) don’t know why they are having such difficulty.

Dewey, for his part, doesn’t care whose fault it is. He just wants someone to protect consumers and businesses. Dewey, the Colonial School District, and millions of consumers are not impressed that the eyeball networks, if they wish, can shrug off M-Lab’s research with three words.