California Keeps Its Mainframes (Thank Goodness)

Sometimes it's difficult to admit making a mistake and reverse course. Yet California's Department of Motor Vehicles finally took the correct action and cancelled its second costly attempt to replace its IBM mainframe applications for vehicle registration.

According to public information, the applications date back to 1965 when they were first written to run on an RCA mainframe. In a few years they were moved to an IBM mainframe, and they are still running on one or more IBM mainframes today. Along the way the DMV has enhanced the applications, and they now have far more interfaces (and far more users) than they had in 1965. The applications handle about 30 million vehicle registrations.

The State of California first tried to replace these applications in 1987. The State hired Tandem Computers (now part of HP) and Ernst & Young to undertake the work. Seven years and $44+ million dollars later, California terminated the project. The new applications created to that point lacked basic functions and were 10 times slower than what they have.

California tried again, hiring EDS (ironically also now part of HP) to do the work in 2007. After spending $135 million (and counting), California cancelled the project. California has also recently cancelled its failed project to replace its payroll system after spending $254 million. Yes, that's at least $433 million wasted, and that's without adjusting for inflation and not counting significant opportunity costs.

It's important to recognize that these applications are not perfect. They are, however, better than any viable alternative, as has now been well proven twice. I would advise the State of California to take a new direction involving two important elements. The first element is that they should make an investment — a much smaller one — to improve the resiliency of these systems and applications to avoid outages. These applications are mission-critical, and, especially compared to the vast sums wasted, it's inexpensive to improve the resiliency of IBM zEnterprise-hosted application environments and access to them. For example, a couple years ago IBM introduced GDPS/Active-Active technology which helps businesses and governments achieve continuous service and move past traditional synchronous update distance limitations in many cases. The second element is to execute a program of sensible, progressive remediation of all of the deferred application maintenance, application improvements, and documentation that the State of California has undoubtedly failed to accomplish in its many years of budgetary problems and unwise spending on these two costly replacement projects.

Yes, these are three big IT disasters — no question about it. But I'm hopeful that California, on behalf of its taxpayers, can take the right steps starting today. And I'm also optimistic that at least some companies and governments can learn from this experience.

UPDATE: A correspondent writes me with this link to the project plan. The story is at least a bit more complex. The project involved multiple interdependent project deliverables. The plan was to move VSAM data into DB2, to move applications hosted in an Assembler-based "Real Time Control" environment to COBOL applications in CICS, and to do a lot of user interface technology rework using WebSphere Application Server (platform not clearly stated but apparently AIX on Power). Unfortunately it didn't fly.

If you're going to migrate these big applications and databases then the California DMV picked some good target environments — there was some thought that went into that decision. However, what seems to have happened is that the project was so big it collapsed under its own massive weight. It strikes me as an awfully big project to pull off in just a few short years. What probably has to happen now is what I alluded to above: some type of progressive rehabilitation, stretched over a longer period of time, with more tactical incrementalism guided by immediate business requirements. And hopefully the bigger, grander project that didn't fly still yielded some salvageable assets, such as better documentation of the existing code and data models. The destination may end up being the same, but the journey will be longer.

Observing from afar, another possible problem that might have cropped up is a common one: "What are we doing this for again?" Frequently I've been reminded of the fact that IT organizations sometimes get wrapped up in work that might be important to IT, but delivering observable, tangible business results also has to be part of the picture. If the users and business managers don't know why you're doing something or why you're spending their money, and if they don't see sufficient progress toward addressing their needs, they understandably get impatient. For example, it might matter a great deal to IT whether data are stored relationally or non-relationally. Both approaches have their merits, and non-relational databases are increasing in popularity nowadays (including at big name California-based Internet companies like Google and Facebook). Do users care? Should they? Not in those terms.

As it happens, IBM has a bit of software called CICS VSAM Transparency which, despite the name, doesn't actually require CICS but which does support moving VSAM data into DB2 databases without changing applications. It can be used selectively or broadly, as appropriate. That particular software was not on the DMV's materials list. Should it be, as part of a more gentle incrementalism? Maybe, but like any other project initiative it would depend crucially on user needs, and of course it would also depend on technical viability in the circumstances.

Anyway, my thanks to that correspondent who contacted me. And let's continue to cheer on the California DMV for making what was probably a tough decision. I think they do a pretty good job in general, especially considering the many years of state budget challenges that put pressure on every state agency.

I understand at least some of you are having trouble commenting on blog postings. We'll see if we can get our small army of technicians to look into that problem.