How a company can control high-tech chaos

Is this one company that realizes the potential problem we are facing here. Sorry its rather long.

BUSINESS CONTINUITY

The term, in technology speak, means to keep things going
after computers blow up. In the event of a disaster,
one expert says that firms should focus on ensuring that
critical business systems are recovered quickly.

PAUL DeGROOT
Special to The Globe and Mail
Tuesday, May 25, 1999

Michael Smith is one of the undertakers of the high-technology world. People come to him to talk about systems that died, data that disappeared into the bit bucket, businesses that croaked. In short, some form of disaster struck and they weren't ready.

"Business continuity" is technology speak for "trying to keep things going after a disaster," and Mr. Smith says far too few companies have made the right preparations.

Ernst & Young conducts an annual information-security survey around the world, and of 310 Canadian companies in the 1998 survey, virtually all had some kind of unexpected business disruption that resulted in financial losses. More than 30 per cent of those disruptions cost the companies involved more than $100,000.

Yet, only half of the firms surveyed had business-continuity plans in place, and nearly a third of those companies had not actually tested their plans, Mr. Smith says.

The most common disasters involve computer failures of some kind. In the survey, 89 per cent of the companies involved suffered system downtime or failure. "Inadvertent errors," usually human error, hit 84 per cent of the companies surveyed, and viruses hit 66 per cent of them seriously enough to cause financial loss. Other common causes of disruptions include malicious acts by employees (32 per cent), natural disasters (29 per cent) and malicious acts by outsiders (23 per cent).

A business-continuity plan should not focus on specific disasters but on ensuring that critical business systems are recovered quickly when they are interrupted by any cause, Mr. Smith says.

Although companies may be prepared for a fire, for example, they may not be prepared for "some wacko in the building" who is threatening to blow the place up. In this case, nothing may be physically damaged, but all staff may be required to leave the building, interrupting the company's business.

Disasters can come from partners or markets as easily as from the most common problem, a power interruption. Mr. Smith was visiting a client in the hospital-supplies business when a major telephone-service failure, well beyond the company's control, took out its communications lines. Most of the company's business came from purchasing agents in hospitals, who used computers in their offices to send orders directly via phone lines. When the telephone system collapsed, so did the company's business and its reputation for reliable, just-in-time delivery.

Companies dependent on Asian markets, such as the forest industry, need to be aware that many Asian businesses are poorly prepared for any year 2000 computer disruptions, or Y2K, Mr. Smith observes. Even though North American companies may be prepared, if their customers aren't the result is not easily distinguishable from other business disasters.

Indeed, Mr. Smith considers the Y2K issue to be one of the best things to happen to business-continuity planning. "As companies start to address it, they realize that they can't fix everything and can't trust everyone else. This has really increased everyone's awareness of their dependence on technology."

Although many disasters, such as those caused by human error, can be reduced by refining business processes, companies cannot anticipate every possible disaster, Mr. Smith says. Instead, they should focus on defining their most critical processes and determining how long they can last without them. "If you can define levels of criticality, that becomes the marching order for the disaster-recovery planner," Mr. Smith says. "I approach it from a time point of view. If a system isn't up when you need it, if a disruption is too long, too costly or too disruptive, then you need to execute your business-recovery plan."

Time often determines cost. "If you need to be back up in 10 minutes, that's really expensive. You can completely mirror your operations facility, and that doubles your costs. If you can afford to wait a couple of weeks or months, the costs are lower, so you don't need to plan as much."

Some current technology practices make business recovery more difficult. Most companies back up their business data each evening, after front-line staff have gone home, which means the backup is as current as the most recently completed business day. If disaster strikes at 3 p.m. the next day, before backup begins, the company has lost all of that day's data.

Furthermore, companies rely increasingly on digital systems to record critical business transactions.

"The application designer wants to make it efficient, and wants to cut out the paper. That means that in a call centre, there may be absolutely no other record of an order than what they are keying into the system," Mr. Smith says.

Craig McLellan, backup solutions manager for StorageTek Canada Inc. of Mississauga, says too many companies focus on getting backups done quickly, and much of the focus is on the narrow "backup window" -- a decreasing slice of the business day when a company needs to ram gigabytes of data through vast libraries of tape backups. As businesses handle more of their data digitally, and as the Internet creates a 24-hour business environment for many companies, more data must be squeezed through in less time.

Mr. McLellan says new technologies give small firms access to some of the business-continuity tools that big companies already use.

Standard backup techniques involve backing up data on tape and then taking the tape to a different location, ensuring that a disaster that hits the company's main premises will not erase data stored on another site.

Large companies with critical business systems back up their data in real time. Data are written to one or more "mirrored" systems simultaneously, and if one system fails, another should have intact data.

BACKUP PLANS
Companies putting a business-continuity plan together should ensure that they cover the basics, which include:

Understanding all of the company's systems and processes so that dependencies can be mapped.

Defining the most critical systems and determining how long they can be down before the business-continuity plan kicks in.

Preparing a written business-continuity document that defines how to restart all crucial processes.

Listing key outside companies and contacts.

Testing the plan at least once a year.

And ensuring that critical partners have, and test, their own backup plans. -- Paul DeGroot