Allergic to 'whiskers'

It's better to look ahead than back at disasters

It is important to learn from experience but it's better to plan ahead. That's what officials from the Colorado secretary of state's office discovered when environmental problems shut down its data center last year.

The state has responded with a five-year lease on a data backup and disaster recovery facility being outfitted by data center provider ViaWest Internet Services Inc. of Denver. Some of the hard-won lessons learned from the experience include:

It's hard to plan for a disaster during a disaster. 'More agencies need to be looking at a proactive approach to disaster recovery,' said ViaWest CEO Ray Dimoff.

Let everyone climb aboard the bandwagon'there's no need for each agency to invent its own wheel. 'Often, things end up getting built one-off,' said Brian Balay, CIO for the secretary of state's office. 'Come up with one thing nice, rather than 23 of them.'

Find a service provider that works well with others to ensure the process goes smoothly. Outsourcing a data center does not mean you've taken it off your plate. 'We will have to work shoulder-to-shoulder with them,' Balay said.

Colorado will open a new disaster recovery center in June, offering state-of-the-art facilities for data backup and operational recovery of IT systems at little cost to most state agencies.

'The build-out is expensive, but the secretary of state's office is paying the way,' said Secretary of State Gigi Dennis. 'The only thing the agencies will purchase is their servers.'

The secretary of state's office is assuming the cost of building out the Enterprise Facility for Operational Recovery, Readiness, Response and Transition Services. ViaWest Internet Services Inc. of Denver is providing the data center and infrastructure. The secretary's office will pay $2.1 million in initial costs this year and an estimated total of $4 million over the life of the five-year lease with ViaWest for the dedicated facility.

This is a bargain for agencies that will receive a free berth for their backup servers. But the secretary of state's office found from experience that even though it is footing the bill, it is cheaper to plan for a disaster than to react to it.

'This stems from an accident called 'zinc whiskers' that kept the secretary of state's data center offline for 30 days in 2005,' Dennis said. 'We decided we needed a good emergency response and disaster recovery plan in place.'

Zinc whiskers is not as cute as it sounds. The problem began in June 2004 when the zinc coating on the underside of floor tiles in the data center began crystallizing. Vibration from footsteps above loosened the small clear crystals, which air conditioning blew into computers, shorting them out.

'You don't see them unless you're looking for them,' said Brian Balay, CIO for the secretary of state. 'But we couldn't keep our data center running. We needed a place to go, and it's difficult to find a place to go in a crisis situation.'

Modern floor tiles do not have the troublesome backing, so zinc whiskers is unlikely to recur. But the state's General Assembly approved the disaster recovery facility in 2005, with the secretary of state as the coordinator for the program.

'The state's needs have a lot of similarities with the private sector,' said ViaWest COO Nancy Phillips. 'What the state wanted was very forward-looking.'

The new center includes room for growth as additional state agencies opt to use it, and a high level of redundancy with duplicate cooling equipment, multiple power sources and network links from multiple carriers, Phillips said. ViaWest will provide operational and maintenance services for the facility, including physical security. The company also will offer a suite of managed services, including firewalls, systems and database administration, monitoring and data protection.

The state settled on a dedicated recovery center only after considering shared co-location services.

'We came to the conclusion that we needed a physical site,' to allow for growth and use by multiple agencies, Balay said.

The location provides a reasonable compromise between convenience and geographical diversity, Dennis said.

'If you put it out 20 miles, it's still reachable,' she said. 'It's far enough away from the state capital, but close enough to make it convenient to get to the servers.'

The lease agreement requires that ViaWest be able to stand up a shadow facility for the state in its Salt Lake City data center if necessary. Many agencies have inquired about the new facility and five plan to use it.

'We are working on memorandums of understanding with them,' Dennis said.

The secretary of state's office will initially house about 20 servers in the new facility, Balay said. The office's primary responsibility is the business community, and its online presence focuses on providing services for businesses. More than 40 applications for electronic filing for licenses and permits are on the secretary's Web site, and some certificates of compliance can be printed from the site at no cost to the user.

Demand for disaster recovery services is expected to grow among other agencies as the state pushes to deliver more services electronically through the official Colorado.gov portal. Funding for the site was obtained in 2003, and the Statewide Internet Portal Authority was created in 2004 as an oversight body for the site, which is being developed as the single point of access for government information and online services.

The site is a work in progress, and the authority has plans to make 20 services from six state agencies as well as municipal and county governments available online this year.

About the Author

William Jackson is freelance writer and the author of the CyberEye blog.