[ISN] Surviving Rita: A tale of two data centers

http://www.computerworld.com/securitytopics/security/recovery/story/0,10801,104940,00.html
By Todd R. Weiss
SEPTEMBER 24, 2005
COMPUTERWORLD
For the second hurricane in a row, New Orleans-based Web hosting
vendor and Internet domain name registrar DirectNIC managed to stay up
and running overnight as Hurricane Rita pounded the Gulf Coast before
coming ashore early this morning.
Sigmund Solares, CEO of Intercosmos Media Group Inc., which owns
DirectNIC, rode out Hurricane Rita in New Orleans last night and was
busy assessing damage this morning.
Solares said his company, located on the 10th floor of a downtown New
Orleans office building, was still dealing with broken windows and
other damage left from Katrina, which plowed through the area Aug. 29.
During that storm - and despite massive flooding in much of New
Orleans after -- DirectNIC's operations continued uninterrupted
because it had an emergency generator and an adequate fuel supply, he
said.
DirectNIC has approximately 800 hosted servers.
"We definitely got hurt by Katrina and we still have a ways to go [to
get completely cleaned up] but we remained open the whole time,"
Solares said.
Even as Rita's strong winds brushed by last night, ripping tin pieces
from neighboring structures and hurtling them at DirectNIC's building,
Solares said he and his four team members were much less worried than
when Katrina struck. "It's still windy outside," he said this morning.
"It's still gusty. [But] right now, we are feeling much better than at
any time during Katrina."
Solares said he got a good night's sleep inside an internal office in
the building, where he was able to get away from the howling winds
outside. The building continued to have electricity until 7:59 a.m. -
more than four hours after the storm struck near the Louisiana-Texas
border. The power went out then, and battery backups kicked on
instantaneously, followed by a backup emergency generator, he said.
The outage only lasted 11 minutes, however, and the company's servers
remained up and running.
Looking outside, Solares said damage to neighboring buildings doesn't
look as bad as when Katrina hit. Floodwaters didn't reach his building
during the last storm, nor did they do so this time. "With Katrina,
when the levees broke, I could see the flooding a half a block away,"
he said. "I'm looking where there was flooding [before] and I don't
see any water there right now."
In other parts of the city, particularly the Ninth Ward, floodwaters
cascaded over newly-repaired levee walls yesterday, causing more
problems in low-lying areas.
During Katrina, and again last night, DirectNIC's employees had to
build a makeshift levee of their own. Rainwater began entering through
the ceiling from broken windows on the floor above, Solares said,
forcing him and his staff to use mops, a wet-dry vacuum machine and
bundles of promotional T-shirts to create what he called the company's
own "levee system" to keep water away from servers and other
electronic gear. Those same methods came in handy again last night, he
said, since building repairs have not yet been made. Before Rita drew
near, Solares and his crew installed plywood over the open windows and
used sealant and tarps, hoping to keep out as much rain as possible.
The makeshift repairs helped, he said.
Earlier this week, as Rita swirled in the Gulf of Mexico as a
dangerous category 5 storm, Solares called a competing Web hosting
vendor in Houston, EV1Servers.net, and offered help in case Rita hit
Houston dead-on.
"For us, there's a lot we can do," Solares said. "It's not much effort
for us to help people get back up and running." DirectNIC has extra
space and capacity for servers and can make that space available to
others in an emergency, he said.
"We're lucky that we've always saved for a rainy day and that we're
able to help our employees and try to help other people, even though
it's a very tough time for us overall," Solares said. "It's also good
for morale for our employees to see us helping other people in need."
In Houston, Robert Marsh, the CEO of EV1Servers.net, said early today
that because Hurricane Rita tracked farther toward the east, away from
Houston, the city was spared the storm's worst effects (see "Update:
As Rita drew near, Web hosting vendor prepared" [1]). The highest wind
gusts were in the 50-mph-to-60-mph range, far lower than the 120 mph
winds expected at one point. By this morning, only an inch of rain had
fallen in the area where EV1's data centers are located, he said.
"In this instance, we were very fortunate not to have received the
worst," he said. Overnight, the company switched over to its emergency
generators after some nearby power transformers blew up. EV1 plans to
return to utility power sometime today after stable power flow is
maintained for at least five hours, he said.
About 25 EV1 employees stayed overnight inside the company's offices
and two data centers to maintain services for customers. The company
had also sent seven Web technicians to a hotel in Wichita, Kan. to
remotely handle customer trouble tickets had its main data centers
been knocked out by the storm.
"Our trouble ticket load is essentially back to normal at this time,"
Marsh said today. "If the storm continues to clear, we hope to bring
our people back from Wichita as early as Sunday night."
[1] http://www.computerworld.com/securitytopics/security/recovery/story/0,10801,104892,00.html
_________________________________________
InfoSec News v2.0 - Coming Soon!
http://www.infosecnews.org