COUNTY CLARIFIES ERRORS IN MEDIA REPORTS ON COMPUTER ISSUES

The county’s old server – a Data General machine – suffered a serious and fatal hardware failure to its system drive on Nov. 2 of this year. During attempts to repair the Data General server, it was discovered that the hardware maintenance contract was not in effect and had lapsed long before County Manager David King, Assistant County Manager Mack Wilson or Information Systems Director Raymond Long were employees of the County. The new Hewlett-Packard server, which was purchased to be the replacement for the Data General server, was already partially running and had been in testing mode for a few weeks. A slow, methodical migration of records and images from the old server to the new was in its embryonic phase when the old server crashed. The severity of the crash required a faster migration of the records than had been anticipated.

Within a week of the Data General crash, the HP had been brought up to a basic and standard level of operation, even though it was exhibiting some systematic quirks. Input of data into the new system began the week of Nov. 12, but on Nov. 16, the new system also suffered a hardware failure and all the inputted data was lost. At this point, Hewlett-Packard technicians were brought in to assist in the accelerated conversion process. By Sunday, Nov. 18, the system was partially operable again with new, fault-tolerant features, such as mirroring. On Nov. 21, the Hewlett-Packard technicians identified other hardware issues, which were resolved by Friday, Nov. 23. Working through the Thanksgiving holiday weekend, county staff and the Hewlett-Packard team rebuilt the operating systems and reloaded applications and data.

With the system operational, staff discovered software problems that were corrupting data and turned their attention to resolving those issues. By Nov. 29, the offices of the County Assessor, County Treasurer and County Clerk were testing the system from their internal computers. By Friday, Nov. 30, the Clerk’s Office was able to allow outside agencies to come to the Courthouse to perform routine research.

On Monday, Dec. 3 – one month after the initial system crash – the county was moving quickly toward having fully functional operations for internal County departments.

Moving all data to a new server in such a short time frame was never the plan. The original migration was planned over several weeks. The accelerated schedule was made necessary by a major hardware failing on a system with no maintenance contract. The process was further complicated by two hardware failures on the new system and no existing disaster-recovery procedures in place. There were also several other issues complicating matters, such as back-up tapes that had historically been sporadically prepared, poorly labeled and indexed and stored in dusty boxes under stairwells. Procedural neglect going back over a period of several years greatly contributed to a protracted recovery period. Even with all that has been accomplished and the herculean number of hours put in by staff, the county is not through this challenge.

Staff is renegotiating hardware and software contracts to ensure better coverage and quicker response to future problems. In the meantime, Hewlett-Packard representatives will carefully monitor the county systems until stability of their hardware is no longer an issue. Implementation of standard fault-tolerance and backup procedures is underway at an unprecedented level in county history. Mirroring capabilities have been added to the system in conjunction with newly ordered special tape drives for recovering the entire operating system. As part of their contract, Hewlett-Packard will dial into the county system nightly and check for any impending problems.

These are steps that should have been part of the department’s standard procedures for computer systems. The fact that they are only now being implemented points to shoddy direction of the department for a long time prior to the current management’s tenure.

To summarize, the recent system failures are rooted in historical mismanagement of the department coupled with shoddy back-up procedures and a lack of maintenance contracts. The migration of the system from the Data General server to the new Hewlett-Packard equipment did not cause the problems, but upon completion of the migration and the implementation of standard back-up protocol, the likelihood of future failures of this magnitude is almost eliminated.

Upon completion of the conversion, Doña Ana County’s system will be one of the most advanced in the region. County management has complete faith in Raymond Long’s leadership of the MIS Department.