Computer Failure Puzzles Metro

Metro officials are still trying to understand why their $20 million central computer system crashed before dawn yesterday, stalling the start of daily service for the first time in Metro's 23-year history and delaying thousands of morning passengers.

"I apologize profusely," said Bea E. Hicks, Metro's chief operating officer for rail service, who was roused from her sleep at 3:30 a.m. when the computer failed. She called the timing of the meltdown when Metro was gearing up for service a "freakish, never-happened-before incident."

A graphics generating device for Metro's finicky central computer system froze about 3:20 a.m., sending Metro managers racing to fix it before the scheduled start of daily service at 5:30 a.m. But they couldn't restore it until 5:46 a.m., which meant normal passenger service didn't begin rolling until about 6:15 a.m., Hicks said.

For passengers who take the earliest trains, that meant 45-minute delays. And once trains were running on normal schedules, three to six minutes apart, they were packed, particularly on the Blue and Red lines.

"I was mad. I was so mad," said Joann Brown, who lives in Forestville and took a Blue Line train to Metro Center, where she waited 20 minutes for a Red Line train to Farragut North. Brown has to be at work at the Capital Hilton at 6:30 a.m.; yesterday, she was a half-hour late. "I sat there for 20 minutes, waiting for a train. And the longer you wait, the more people keep coming onto the platform. People were pushing to get onto the train when it finally came. Overall, Metro is pretty good. But today it was frustrating."

Metro's three-year-old, $20 million central computer system has a troubled history. Last fall, the system crashed several times during rush hour, including one episode similar to yesterday's when computer-generated views of sections of the system were blacked out for nearly two hours. In the 15 months after the system was installed by McLean-based BDM International, it crashed 50 times, Metro managers said.

Yesterday, Metro's main computer that controls signals, power and other key elements continued to run properly and was not affected by the crash. But the system that provides real-time graphics in Metro's downtown Washington control center fizzled, blacking out computer-generated views of the 96-mile system that normally are projected on large screens in the control center.

Controllers rely on those computer images to make sure that trains are spaced far enough apart. The images allow them to see at a glance whether a train is disabled or delayed. Using that visual information, downtown controllers can reroute trains or change their speed by remote control. Without those graphics, the downtown controllers are blind to the location of trains, and their safe movement is strictly in the hands of the operators in the trains.

As computer technicians tried to restore the graphics system, Metro faced two challenges: It had to clear work trains that were spread across the system and position passenger trains at points along every line for the start of service.

To do this without guidance from the central control center, Metro workers fanned out across the 96 miles of track, set switches by hand and used radios to report the position of each train to the next worker down the line.

"They had to eyeball it, people looking at their segment, relaying that to the next person down the line, who relays it down the line," Johnson said. "It's like if you're at home in your bed and you hear a noise in your house. If you had a central computer system that shows all the rooms in your house, you could check it out by looking at the computer from the comfort of your bed. But if you don't have that, you have to get up and check each room."

Once the graphics system was back online at 5:46 a.m., Metro started putting trains into position that normally would be in place by 4 a.m.

Yesterday's meltdown came one day after Metro directors agreed to extend service on weekends and close the system at 1 a.m., instead of midnight, beginning in November.

Hicks said the public should not worry that extending hours on the weekends will further strain the subway's operations.

"Staying open late on Friday and Saturday nights is a completely different set of circumstances," she said. "We open late on Saturday and Sunday, 8 a.m. instead of the 5:30 a.m. opening on weekdays. That gives us more time for maintenance when the system shuts down on Fridays and Saturdays. It's a different set of circumstances altogether."

At 5:15 p.m. yesterday, a train stalled near the L'Enfant Plaza station on the Green and Yellow lines. Passengers inside waited for 45 minutes before the train got moving. Other trains went around the stopped one on another track.

Johnson said the train had problems with its brakes. The delays had nothing to do with the morning's problems, she said.

Staff writer Emily Wax contributed to this report.

CAPTION: Morning commuters file off a train at McPherson Square. The early-morning crash of a Metro computer yesterday delayed the rush and packed trains.

Lyndsey Layton has been running coverage of local politics in D.C. and state politics in Virginia since 2016. Prior to that, she was a reporter and covered a wide range of local and national beats, including national education, food safety and chemical policy, Congress, transportation and the U.S. invasion of Iraq.