Washington, D.C. - Fail Gracefully, Recover Quickly

by Rodrigo (Roddy) MoscosoTue, November 11, 2014

Since the issuance of Presidential Policy Directive 8 in 2011, which established the objective of strengthening the security and resilience of the United States across five core mission areas, the nation’s city leaders have been developing and implementing plans to improve their ability to protect and respond to a variety of manmade and natural disasters. Unfortunately, since 2011, homeland security and emergency management personnel have had many examples of critical infrastructure failures, including: the earthquake and tsunami on 11 March 2011 that resulted in Japan’s Fukushima Daiichi nuclear power plant meltdown; the North American derecho in the summer of 2012; and Hurricane Sandy, which made landfall in the northeastern United States in October 2012. The increased focus on preparedness and resilience planning coupled with an emphasis on continuous learning from critical infrastructure failures elsewhere is enabling emergency managers to improve their ability to plan for, and respond to, major incidents.

Hazardentification & Lessons Learned From Other Cities
“First, we want to ensure that we fail gracefully,” said Christopher Geldart, director of the District of Columbia’s Homeland Security and Emergency Management Agency (HSEMA), in a telephone interview on 19 September 2014. “You can never prevent a failure from happening, but the key to prevention and protection is to take steps to ensure that you fail gracefully and also do the work necessary to shorten the recovery time as much as possible,” he added. Recently, D.C. HSEMA took the Federal National Preparedness Framework and created a family of plans that focuses on the five core mission areas, the first of which is Prevention and Protection.

As part of its planning effort, HSEMAentified 18 hazards that are unique to Washington, D.C., and reviewed all previous plans and efforts to improve resilience in these areas. This included looking at ways to harden physical infrastructure and to improve supply chains that aid and shorten the recovery process. Based on the implementation of these plans, HSEMA officials expect only partial power outages with faster recovery times rather than a citywide power failure during a major storm.

Maintaining electrical power continues to be a core focus for the city, as it does for most emergency managers, service agencies, and lifesaving organizations. Hurricane Sandy demonstrated that even well planned and implemented resilience efforts could fail dramatically with significant effects on the public. In New York City, two major hospitals – Langone Medical Center and Bellevue Hospital Center – had to evacuate after their backup generator systems failed. Although the generators were located on upper floors, the pumps and diesel tanks supplying fuel to the generators were located in “flood-protected areas” of both hospitals’ basements. Unfortunately, these protections failed and the generators had no fuel to operate.

At Fukushima Daiichi, 12 of 13 backup generators failed following the tsunami. Although the earthquake itself did not significantly damage the plant, the two large tsunami waves submerged the emergency generators, seawater pumps, and batteries located in the basement of the plant, rendering them inoperative. In short, the lack of power to run the water-cooling system and monitor the reactor itself was directly responsible for the nuclear reactor meltdown.

Location-Specific Considerations
The topography of Washington, D.C., makes it a challenge to provide water to higher elevations while, at the same time, removing the water from low-lying areas. For Geldart, this was the biggest surprise encountered when developing the city’s new preparedness plans. “Water is a huge issue, and it is a much more complex system than I originally imagined. As soon as you get backpressure in the system, people must boil water in their homes, and schools and businesses close, including government buildings – and you don’t want to tell the White House that they need to boil water,” he said.

Power, then becomes the critical element to water delivery as well as to water treatment. “We have two [power] feeders into our Blue Plains water treatment plant but, if we do lose all power there, we have the choice of either dumping raw sewage into the Potomac [River] or backflow the entire city,” Geldart added. This scenario actually occurred in May 2006, when 17 million gallons of raw sewage spilled into the Potomac River. Geldart noted that Washington, D.C., now has portable generators staged for responding to such situations.

During the development of the new preparedness plans, HSEMA created separate task forces to focus on key areas, including the ability to quickly deploy emergency power generation across the city. HSEMA is now working on a 10-year plan of investing in water infrastructure enhancements that will better harden the system and help it to fail more “gracefully.”

Neighboring Infrastructures & Private Sector Cooperation
On the other side of the Potomac River in Virginia, the issue of power resilience also is at the forefront of preparedness planning. Dominion Power, Virginia’s largest power utility, announced this year that it had embarked on a legislatively approved plan to bury approximately 4,000 miles of electrical distribution lines underground in areas where above-ground lines are most susceptible to storms. During the 2012 “derecho” storm, more than one million people lost power in Virginia. Although burying power lines will not guarantee power to residents during a major storm, Dominion anticipates that restoration time will be significantly shorter when outages do occur.

However, funding remains an issue for the plan. In 2005, a Virginia State Corporation Commission estimated the cost of burying all 58,000 miles of Dominion’s power lines at $83.3 billion. The current plan calls for moving 350 miles of power lines each year for the next 10-12 years at a cost of $175 million annually. Funding for the plan will come from customer rate adjustments or riders that still await approval by the State Corporation Commission.

Another important element in disaster preparedness and service restoration, particularly electrical services, involves the active cooperation with the private sector. “When Craig Fugate was director of Florida’s Division of Emergency Management, he called it the ‘Waffle House index’,” noted Waffle House spokesman, Pat Warner, vice president of culture, in a telephone interview on 24 September 2014. “Since Waffle House is a 24 by 7 by 365 operation, the status of our open locations was a good measure of power availability after a storm, which we have been sharing with emergency management agencies across the southeast for many years,” he added. The Waffle House Corporation has worked to restore power to their locations after major storms as quickly as possible, and has implemented a rapid response effort, including the delivery of generators and fuel to the hardest hit locations. “For many people, [Waffle House] is the first hot meal they have in the aftermath of a hurricane,” said Warner.

The information between Waffle House and other businesses and Florida emergency management agencies increased over time, including regular status updates from them and other businesses. When Fugate was appointed as Administrator of the Federal Emergency Management Agency (FEMA) in 2009, information exchange with the private sector increased, resulting in the establishment of the National Business Emergency Operations Center (NBEOC), which “serves as FEMA’s clearinghouse for two-way information sharing between public and private sector stakeholders in preparing for, responding to, and recovering from disasters.”

For the Waffle House, which has its own emergency operations center in Atlanta, Georgia, the focus now is to use social media to “push” information to customers about which locations are open in order to manage the flow of people. The company also has developed a new “Waffle House on Wheels,” which can go to the hardest hit or underserved locations following a major storm.

Emergency managers always must be ready for any challenge presented to them, particularly those that former Defense Secretary Donald Rumsfeld called, “the unknown, unknowns.” However, mitigating strategies, including the development of thoughtful and comprehensive preparedness and resilience plans, can help to reduce the severity and impact of even an “unknown” event. Coupled with a strategy of collaboration with the private sector, a more holistic view can be achieved by emergency managers, which is the first, and most important, step in addressing the problem at hand, and which also serves as the basis for future preparedness planning.

Rodrigo (Roddy) Moscoso currently serves as executive director of the Capital Wireless Information Net (CapWIN) Program at the University of Maryland, which provides software and mission-critical data access services to first responders in and across dozens of jurisdictions, disciplines, and levels of government. Formerly with IBM Business Consulting Services, he has more than 20 years of experience supporting large-scale implementation projects for information technology, and extensive experience in several related fields such as change management, business process reengineering, human resources, and communications.