Metro-North blames shutdown on 'human error'

Martin B. Cassidy

Published 11:35 pm, Friday, January 24, 2014

Photo: Mario Tama, Mario Tama/Getty Images

Image 1of/1

Caption

Close

Image 1 of 1

People work in Metro-North Railroad's Operations Control Center at Grand Central Terminal on January 31, 2013 in New York City. The terminal opened in 1913 and is the world's largest terminal covering 49 acres with 33 miles of track. Each day 700,000 people pass through the terminal where Metro-North Railroad operates 700 trains per day. less

People work in Metro-North Railroad's Operations Control Center at Grand Central Terminal on January 31, 2013 in New York City. The terminal opened in 1913 and is the world's largest terminal covering 49 acres ... more

Photo: Mario Tama, Mario Tama/Getty Images

Metro-North blames shutdown on 'human error'

1 / 1

Back to Gallery

In the Operations Control Center high above Grand Central's elegant grand concourse Thursday evening, Metro-North rail traffic controllers were monitoring a routine rush-hour commute on banks of computer screens as thousands of passengers made their way home to the northern suburbs.

It was a welcome change of pace for a center that had just dealt with a stranded train the night before in Westport, where the metal arms that reach up from the tops of the rail cars to the overhead catenary lines had pulled those wires down.

The extreme cold makes the miles of catenary taut and brittle, and prone to snapping as the train pantographs pass along them.

Despite the persistent deep cold spell, Thursday evening's commute was running smoothly as of 7:30 p.m. Thursday on all three of Metro-North's main lines.

Then, at 7:45 p.m., the control center was thrown into chaos.

The computer screens went black, leaving the rail traffic controllers blind, in essence. They had no way to be sure that the more than 50 trains spread across hundreds of miles of track were moving, stopping and switching tracks.

They put out an immediate radio call to all train engineers to stop their trains at the next station and stay there until further notice. Trains that were near junctions that were no longer controllable without the signal system, were forced to stop where they were until maintenance workers could be dispatched to guide them onto the right tracks and to the nearest station platforms.

As the traffic controllers scrambled to pin down the locations of all the trains, and other control center staff members were trying to get the power back on, several stories below them on the Grand Concourse, commuters were beginning to pile up, waiting. All across the Metro-North system, others sat patiently stuck on trains in between stations, stopped at stations wondering whether to give up and find a taxi or waited in the station for something to happen.

Given all of the beleaguered railroad's recent problems caused by aging infrastructure and bad weather, one might have expected the total shutdown to be related to that. But in fact, it was one simple little error that brought the whole railroad down.

An employee working to repair a battery backup power system "inadvertently unhooked the wrong wire. It was just human error, a stupid mistake," Metro-North spokeswoman Marjorie Anders said Friday morning.

How could such a little mistake with such massive consequences happen? The railroad said Friday it was still working to understand that.

Railroads in the U.S. have been operating on electrically controlled systems for more than a century now, long before computers came on the scene beginning in the early 1970s, according to Don Thomas, a railroad operations consultant in Marietta, Ga.

"If you go back 40 years or more, railroads would operate using what was called a `twist and punch' board," he said. "It was still electrified, but in the control center, you would twist a knob to indicate whichever way you wanted the rail switch to go, then you would punch a little button underneath it to send the signal out to the switch."

But even in those days, a power failure in the operations center would grind the railroad to a halt. It might be a bit safer today though.

Steve Ditmeyer, an adjunct professor for the railway management program at Michigan State University, said, "The computer systems are designed so that if there's a power failure, the systems will revert to a fail-safe mode. The electro-mechanical devices automatically stop the trains and make sure that there are no green signals displayed."

It's unclear whether the fail-safe mode kicked in properly Thursday night, but there were no accidents reported, just inconvenience.

All of that computing power relies on electricity, however. And when a repair crew doing maintenance work on power units fails to check whether a backup unit is in place to ensure continuous operation and cuts the electricity, that computing power is useless.

The project should have been analyzed before it began, and it should certainly not have been done at the height of rush hour.

"The best time you would normally do that is in the middle of the night between Saturday night and Sunday morning, when you'd have the least impact on as few people as possible," Prendergast said.

The electricians who conducted the repair will be subject to discipline pending the outcome of the railroad's investigation of the outage, said Aaron Donovan, a spokesman for Metro-North Railroad.

Discipline is key

Because the recent failures seen on Metro-North are all dissimilar in nature, there won't be an easy fix, Michigan State's Ditmeyer said.

"When one looks at these incidents, you find that there are different people is different situations making mistakes," he said, rattling off the series of incidents that began with the Bridgeport collision of May 17. "So it's hard to find a single thread that ties everything together."

A likely solution, he said, could involve a systemwide analysis of all of the managers and employees, and making sure that all of them are aware of their duties and responsibilities.

"People have to be reminded to pay attention to what they're doing," he said. "Railroads require a great degree of discipline -- not in the punishment sense, but rather like what you see in the military. You have to obey the rules, because if you don't, bad things happen."

"It should never have taken place during commuting hours, and you should schedule these things for when there is little to no chance of any impact," Redeker said. "From what I know, this was not absolutely critical and should have waited for overnight hours."

Variation on a theme

Redeker said the lack of backup power echoes an incident in late September when the New Haven Line's capacity was cut drastically for 13 days after Con Edison took a secondary power line out of service during a $50 million substation replacement project.

In that incident, Metro-North had studied the cost of establishing a secondary power source as a hedge against the event of a power failure on the line and deemed the risk too low to spend millions to put a backup feeder cable in place.

"Obviously, there were some things that weren't checked before they tried to install the backup system," Redeker said.

U.S. Sen. Richard Blumenthal, D-Conn., said he was outraged by the lack of oversight that would allow non-emergency maintenance to take place on a critical system during busy travel times.

"User error seems to be Metro-North's story these days," Blumenthal said. "Routine maintenance during peak user hours, particularly under the most severely adverse weather conditions, is absolutely unacceptable and inexplicable."

The Federal Railroad Administration started a 60-day assessment of Metro-North's safety procedures and protocols after the derailment in the Bronx, where four people died and dozens were injured, to review the railroad's compliance with federal safety regulations. On Friday, FRA spokesman Kevin Thompson said that investigating service disruptions is not part of the administration's statutory safety mandate. But given the onslaught of problems Metro-North has been having, situations such as Thursday's may fall under their purview in a way.

"The central focus of Operation Deep Dive is to identify and mitigate systemic safety and cultural issues that impact safety performance on the Metro-North Railroad and to the extent that last night's disruptions may have been caused by systemic safety culture, Operation Deep Dive may provide significant value."