Stuff Happens! So buck up and deal with it

By Phil Bertolini, Deputy County Executive/CIO, Oakland County, MI

Phil Bertolini, Deputy County Executive/CIO, Oakland County, MI

Life is an ever changing landscape where each day is completely different than the next. We, in government technology, fool ourselves into believing that we can simply get in the groove to be one step ahead of whatever comes our way. This false sense of security can lull even a seasoned technology professional into believing that we have it nailed. In reality, the world of technology loves to throw curve balls and force us into the notion that stuff happens. Our days are not ever the same and we soon find that we end up moving from one problem to another. Advanced planning for these frequent occurrences is at the heart of what we all know as Disaster Recovery or DR.

Many in technology see Disaster Recovery as a necessary evil that takes time away from our regular work. Some would say that the odds of needing a DR plan are slim to none but experts in this world know all too well that it only takes one disaster to define an organization. Respond too slow and be chastised in the community. Respond with a limited time consuming recovery and start working on your resume. Let’s be clear, the ever changing world of technology has forced every Chief Information Officer (CIO) to concentrate resources, time and money towards ensuring that a well thought out DR plan is in place. The main problem government technology professionals face is that we don’t know what we don’t know. We find ourselves planning for the unknown hoping that we have covered all the bases.

This issue breaks down into very logical parts. First, in the case of a cyber security outage, we must have the proper controls in place to block, capture and recover. Many organizations get bogged down into buying the latest and greatest protection software knowing full well that government, in many cases, is far behind in providing what is needed to protect our services. Throwing solutions against the wall to see if they stick limits government’s ability to maintain a structured cyber security posture. In fact, we are fighting nation states and sixteen year olds from Russia all of which are totally unpredictable.

"A thorough Disaster Recovery Plan must be exercised frequently or it is only as good as the paper it is written on"

Second, we must ensure that we have quality backups that have been tested. Backup strategies can vary greatly with some doing total database copies and others doing partial copies based on activity. Whatever the philosophy, government technology professionals must validate their backups to determine if they were effectively completed. Backups for cloud technologies have essentially the same issues as backups on premise. There is nothing worse than when backups are taken for granted, an emergency takes place where the backups are required and the backups fail. Frequent recovery tests of all platforms, on premise and the cloud, are required even though they are very time consuming.

Third, DR plans are explicably linked to Business Continuity and Recovery Plans or BCORP. The ability to recover technical components of your operation is very important but if the business cannot operate as well, then all you have are some interesting flashing lights with no output. Technology alone cannot substitute for a fully functional business operation. The creation and development of BCORP plans can also be time consuming, costly and perceived to be a low priority. One cannot be successful without the other. A template specifically related to BCORP and government operations is located on our G2G Marketplace or www. g2gmarket.com.

Fourth, a thorough Disaster Recovery Plan must be exercised frequently or it is only as good as the paper it is written on. DR exercises must be effectively planned to ensure that when needed, the recovery process simply happens as a routine occurrence. In real estate, professionals will claim that the panacea for their industry is location, location, location. In the world of DR it is exercise, exercise, exercise. People can meet and develop the best plans ever to only have them fail in an emergency situation.

Fifth, DR can be an expensive undertaking from both the technical side and the human resource side. Government technology professionals have so many issues on their docket that consume every bit of their day. Preparing for an event that may never happen can fall very low on a priority list. Making a strong business case both within the technology organization and to the entire overarching organization is imperative to being prepared for the unthinkable, a complete failure that requires a complete recovery. Having the proper executive support, the proper funding and the proper priorities will at least keep everyone at the ready.

In our world it is so much easier to sell a fancy new web site or a cool new application than it is to sell the mundane operational projects like DR. A successful DR plan will never be front page news unless you have a catastrophic attack or failure, then not having one could be an organizational death sentence. My advice to everyone is to properly prioritize, fund, operationalize and exercise often, your Disaster Recovery Plan. Imagine if the cast of Gilligan’s Island had a tested quality DR plan. They would have been rescued much sooner. OK, then there would have been no show to watch. Maybe?