Don’t implement changes without testing

A few weeks ago, we were called into a customer's office to examine some problems they were experiencing with a critical system that we had developed for them years ago.

This system had worked like a charm since it was implemented, so we were all quite perplexed as to what could be causing the problems.

Initially, we suspected that something in the technical environment had changed; perhaps a new server had been introduced into the mix or a network connection had gone down. Instead, we discovered that someone had made some seemingly simple changes to one of the programs, which resulted in the malfunctions we were seeing.

Seasoned folks in the IT industry are often appalled by such behavior. Not so much because the programmer made a mistake -- all programmers make mistakes -- but because this particular programmer made changes to a critical operational system without first fully testing the changes.

In the old days of the mainframe, this would not have been possible. Software developers were clearly and purposefully prevented from tinkering with systems that were up and running and supporting business needs. Such systems, called "production" systems, or "prod" for short, were the sacred cow of data processing. Nobody made any changes until such changes had been fully vetted. There were separate areas of the mainframe fenced off for development and testing, and these areas generally were the playgrounds for the development staff.

Nowadays, however, the very nature of personal computers makes it difficult to prevent software developers from accessing production systems. Programmers must exercise a reasonable amount of discipline to avoid touching the prod systems.

Furthermore, additional resources, such as workstations, servers, and software licenses may be required to emulate the old-fashioned environment. This of course is tempered by the fact that the cost of a mainframe is usually measured in millions, while the cost of Windows or Unix servers are usually measured in thousands.

This ideology should be applied not just to software changes. Any changes to critical production systems, including hardware and operating system upgrades should be thoroughly tested before implementation.

Many organizations may deem this type of testing process to be too expensive, and it may well be. But before making that determination, you should figure out what the cost would be if that system went down. Such a risk-benefit calculation could be an eye-opening experience.

John Agsalud is president of ISDI Technologies Inc., a Honolulu-based IT consultancy. Call him at 944-8742 or e-mail jagsalud@isdi-hi.com