Share this story

Accidental fiber cuts caused by construction workers took out telecommunications service for more than 750,000 customers in the New York City area yesterday.

The fiber cuts hit the network of Level 3, an Internet backbone provider, and lasted for hours before being fixed. Problems hit several states: customer reports on DownDetector indicate that outages primarily affected Time Warner Cable (TWC) in New York and Cox Communications in large parts of Connecticut and Rhode Island and small parts of Massachusetts. Level 3's network serves both TWC and Cox.

The most specific outage numbers came from New York. The New York Department of Public Service (NYDPS) issued a statement saying that "more than 750,000 customers in the New York City area were unable to complete telephone calls." Most or all of those customers are apparently Time Warner Cable users. Internet and TV service was also affected.

Level 3 confirmed the outage, telling CNN and other media outlets, "Our network is experiencing service disruptions affecting some of our customers with operations in the Northeastern United States due to a fiber cut caused by third-party construction. Our technicians are on site and working to restore service." Time Warner Cable said the outage was caused by "multiple fiber cuts at one of our network providers."

The NYDPS statement noted that Level 3 provides service to both TWC and Verizon in New York. But Verizon's network did not suffer any problems related to the Level 3 trouble yesterday, a Verizon spokesperson told Ars.

Time Warner Cable acknowledged the outage on Twitter at 3:22pm ET yesterday and said at 9:48pm that service had been restored to NYC customers.

NYC customers, we are aware of issues concerning all services. Techs are investigating. Services will be restored as soon as possible. ^MR

Cox's customer service team on Twitter "was responding to questions about outages across Connecticut and Rhode Island all afternoon," Fox61 in Hartford noted yesterday. Cox said that service was restored late last night.

I am sure the expensive is whats stopping them, but these companies should really dedicate an effort to having fallback lines in case of things like this; or at least minimize the outage size, similar to how the power grid is set up.If one cut line could affect nearly a million people, it's a major concern for communications.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

but that would cost MONEY. And we cant take any profit away from our investors, now can we? That would hurt competition! -Cable industry.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

but that would cost MONEY. And we cant take any profit away from our investors, now can we? That would hurt competition! -Cable industry.

I am sure the expensive is whats stopping them, but these companies should really dedicate an effort to having fallback lines in case of things like this; or at least minimize the outage size, similar to how the power grid is set up.If one cut line could affect nearly a million people, it's a major concern for communications.

They took something that was designed to route around problems in the network. And created single links for most parts of the network.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

but that would cost MONEY. And we cant take any profit away from our investors, now can we? That would hurt competition! -Cable industry.

This was a backbone company's line that got cut. Not an ISP.

Same difference. Level 3 is a publicly traded company. Nothing I said is exclusive to cable ISPs, it goes equally for publicly traded backbone providers who dont bother laying redundant lines, who have investors that want higher profits. Investment in infrastructure doesnt make bigger profits appear this quarter.

Not that I have anything against profits, but many of these publicly traded companies make tons of short sighted decisions to increase the profits of the next quarter. Infrastructure is a dirty word to them (TWC, comcast, ece) . Level 3 is no different. If they were, this kind of outage wouldn't happen.

Given the increasing level of convergence on Fiber from previously independent and potentially redundant channels, Cu telephone/DSL and coax cable/broadband, having a backup networking path via LTE tethering has become a more important asset for a technologist, but doesn't obviously solve this type of problem if cell towers in the area are served by the same backbone.

It used to be that stores could still sell goods during a power outage; can anyone envision successfully completing a purchase at Target if the power or network is down for a day? How about emergency medical care at a hospital if the only radiologist is remote and there is no backup channel to send scans for review?

With the spate of fiber cuts in California and deprecation of Cu lines by Verizon, it would be helpful if new disruptive networking efforts by Google and others could provide a viable, redundant alternative path for data, otherwise we may find basic infrastructure outages like this having a greater impact on everyday life.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

If you cut a busy enough pipe, all of that traffic will then try to go along the other routes, and it will all clog up. They could build out enough capacity for even huge outages like this, but that would be extremely costly for an event that will probably only happened every few years.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

Originally yes it was essentially a peer to peer network where each node has multiple redundant connections to other nodes and each node could relay traffic so traffic could be relayed around down links.

How many redundant connections does your end node (home) have to the rest of the internet?

A pure peer to peer system is highly survivable (which was the intent of the ARPANET) but it is also very costly. Today we have a few backbone providers interchanges where those backbone providers connect and then a hierarchy of smaller networks.

I am sure the expensive is whats stopping them, but these companies should really dedicate an effort to having fallback lines in case of things like this; or at least minimize the outage size, similar to how the power grid is set up.If one cut line could affect nearly a million people, it's a major concern for communications.

Ah, but you've made a classic mistake: you're framing the problem in terms of best engineering practices. The more useful observation would by that Level 3 has insisted upon a form of infrastructure deployment that makes it especially vulnerable to terrorist attacks, risking the lives of everyone served by Level 3 in the wake of a terrorist attack with no means of conveying or coordinating information. Assuming wireless isn't running off the same backbone, watch 4G LTE crash like a motherfucker after six minutes of everyone in NYC trying to call everyone else.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

but that would cost MONEY. And we cant take any profit away from our investors, now can we? That would hurt competition! -Cable industry.

This was a backbone company's line that got cut. Not an ISP.

Same difference. Level 3 is a publicly traded company. Nothing I said is exclusive to cable ISPs, it goes equally for publicly traded backbone providers who dont bother laying redundant lines, who have investors that want higher profits. Investment in infrastructure doesnt make bigger profits appear this quarter.

Not that I have anything against profits, but many of these publicly traded companies make tons of short sighted decisions to increase the profits of the next quarter. Infrastructure is a dirty word to them (TWC, comcast, ece) . Level 3 is no different. If they were, this kind of outage wouldn't happen.

Going through Level 3's finances for the last couple of years, their total revenue is less than half of Verizon's net profit, and their income pitiful, with lots spent on the network. Not done an extensive study, but best I can tell, they don't seem like they're deliberately neglecting their network. They just have less than 1/25th of Verizon's revenue. (A net income of $124 million for Q1 2016, which is... pocket money for the big cable providers).

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

Verizon apparently had other routes, likely their own backbone connections, which is why they didn't have an outage.

Not sure about TWC or Cox, but either they didn't have another backbone provider, or didn't have proper failover.

Wasn't the original goal of the internet a network that could still operate after, like, multiple nuclear strikes?

That was when it was a government project. Now that the capitalists have control of it you're lucky that it's up and working at all and the price hasn't gone up again today.

But, but, but... they told me government projects were evil and communist !?!? BTW, could we get some taxes and national infrastructure maintenance going? Future is looking a bit cracked and pothole-y.

Wasn't the original goal of the internet a network that could still operate after, like, multiple nuclear strikes?

That was when it was a government project. Now that the capitalists have control of it you're lucky that it's up and working at all and the price hasn't gone up again today.

But, but, but... they told me government projects were evil and communist !?!?

My dearest Wormwood, it's true, the government is evil communism. Now take your Rosary and do 10 Praise Reagans at the altar of money then go to sleep. There's a big day of exploitation of political illiterates ahead of us tomorrow.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

Verizon apparently had other routes, likely their own backbone connections, which is why they didn't have an outage.

Not sure about TWC or Cox, but either they didn't have another backbone provider, or didn't have proper failover.

That's more likely. TWC/Cox might only have a single provider. It's likely level 3 had redundant options but TWC chose not to purchase that option. I can ask my current carrier for geographically disperse routes but it's going to cost me to have them build out their network from the other direction, something like 5+ miles of new fiber. Not going to happen! Or find another carrier that brings in their fiber using a different route than the other providers. Also not likely because of my location. Lots of different things to consider.

There isn't enough information in the article to determine who dropped the ball in regards to redundancy. I'm leaning toward TWC/Cox.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

but that would cost MONEY. And we cant take any profit away from our investors, now can we? That would hurt competition! -Cable industry.

This was a backbone company's line that got cut. Not an ISP.

I believe the original facetious post was the ISP not having multiple backbone providers for redundancy, not the backbone provider themselves.

I can't speak for TWC and New York, but up here in Rhode Island, if you cut over to a non-Cox DNS server, you could get online. I don't think their transit was the issue, I think maybe they were hosting their east coast DNS in a Level 3 datacenter and maybe this cut made those DNS servers unreachable.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

If you cut a busy enough pipe, all of that traffic will then try to go along the other routes, and it will all clog up. They could build out enough capacity for even huge outages like this, but that would be extremely costly for an event that will probably only happened every few years.

Wasn't the original goal of the internet a network that could still operate after, like, multiple nuclear strikes?

That was when it was a government project. Now that the capitalists have control of it you're lucky that it's up and working at all and the price hasn't gone up again today.

Yes, you are totally correct with your snarky observations. The internet today is much slower, less reliable, and less secure than the internet of the mid 90s, or even the Darpa days in the 70s and 80s. No improvements at all, just increasing costs to the consumer!

Wasn't the original goal of the internet a network that could still operate after, like, multiple nuclear strikes?

That was when it was a government project. Now that the capitalists have control of it you're lucky that it's up and working at all and the price hasn't gone up again today.

Yes, you are totally correct with your snarky observations. The internet today is much slower, less reliable, and less secure than the internet of the mid 90s, or even the Darpa days in the 70s and 80s. No improvements at all, just increasing costs to the consumer!

Considering the vast majority of consumers are stuck with the same technological infrastructure that was running to their house when it was still darpanet (coaxial cable or twisted pair phone lines) along with the fact that the price continues to rise at all when the cost per speed delivered continues to decline and carriers try to figure out ways to stifle internet video solutions I wouldn't be quick to call out government run projects inferior to capitalist run ones.

Wasn't the Internet meant to have been built with redundancy in mind? Really surprised that one fiber cut can cause a multi-state outage. I would have thought there were other routes for the data to flow along - apparently not?

Okay unplug your cable modem and see if you can route around that. The Internet was designed to work, so that an outage in New York City doesn't cause an outage in California, and this one didn't. Imagine if the Internet was a token ring network, and the ring was cut. That's how most pre-Ethernet networks were setup. Rings, or spoke and hubs. A bad connection in one place affected the entire network.

You can setup redundant connections, and if there are more than way of routing information, the Internet would choose the best of the various paths. And, if one path was down, the Internet could switch to another path. However, there was always nodes that had a single connection to the rest of the network since the Arpanet days.

The difference is the amount of people, the speed, and amount of the data that's expected over this network. Now, a down connection doesn't affect a dozen college professors, but millions of people. Plus, fiber is extremely difficult to splice when cut. That's what took so long.

Level3 is dealing with two critical fiber cuts in the Northeast. One is in Auburn Mass the other is between Phili and Wilmington, this is causing a massive issue for services providers in NYC.

So their redundancy plans were basically undermined by two major routes being cut at about the same time. If only one had happened the outage probably wouldn't have even been noticeable.

WOW. I thought there was more to this story and there is. Hopefully there will be an update to this article.

Yeah, that makes much more sense. I was going to make the point that networks providers generally run line segments at levels of saturation that can exceed 50% depending on demand. So, a single fiber cut at the wrong time of day can over saturate a redundant line segment which can exacerbate an outage.

Wasn't the original goal of the internet a network that could still operate after, like, multiple nuclear strikes?

That was when it was a government project. Now that the capitalists have control of it you're lucky that it's up and working at all and the price hasn't gone up again today.

Yes, you are totally correct with your snarky observations. The internet today is much slower, less reliable, and less secure than the internet of the mid 90s, or even the Darpa days in the 70s and 80s. No improvements at all, just increasing costs to the consumer!

Considering the vast majority of consumers are stuck with the same technological infrastructure that was running to their house when it was still darpanet (coaxial cable or twisted pair phone lines) the fact that the price continues to rise at all when the cost per speed delivered continues to decline and carriers try to figure out ways to stifle internet video solutions I wouldn't be quick to call out government run projects inferior to capitalist run ones.

You are conflating raw ISP cost and service level with the mega packages that many of the bigger media companies have put together to keep profits rolling in. If you take out cable and "phone" packages and look at the cost it is ~$50/mo for 25mbps average in the USA. In 1995 when the private sector took over NSFNET a typical dial up 56k (.056mbps) connection cost $10/mo. I'd say that is a considerable improvement in cost/mbps.

I am sure the expensive is whats stopping them, but these companies should really dedicate an effort to having fallback lines in case of things like this; or at least minimize the outage size, similar to how the power grid is set up.If one cut line could affect nearly a million people, it's a major concern for communications.

Ah, but you've made a classic mistake: you're framing the problem in terms of best engineering practices. The more useful observation would by that Level 3 has insisted upon a form of infrastructure deployment that makes it especially vulnerable to terrorist attacks, risking the lives of everyone served by Level 3 in the wake of a terrorist attack with no means of conveying or coordinating information. Assuming wireless isn't running off the same backbone, watch 4G LTE crash like a motherfucker after six minutes of everyone in NYC trying to call everyone else.

I was tempted to make an observation about terrorism. But given the number of times things like this has happened because of construction crews, random passerbyer, someone with a utility truck and a vendetta, or wild animal(including sharks) compared to the number of terror related attacks on these lines, I figured it would be best not to speculate on the things that have happened 0 times.