A Written Record of Personal Self-Improvement

Accepting What Is

“The database is gone.” I was in Charlotte, NC, helping out another site with a brand new upgrade of our primary application. Columbia, SC, where I was based, would upgrade the following day. We had received promises over and over again that this particular release had been fully tested. This tells you all you need to know about the previous history of the application. And when we ran this upgrade on five of the laptops, we discovered the upgrade had wiped out the database on one of them. Those were my words back to my manager, who was based in Atlanta, GA.

At that point, the failure rate was 20%. However, the sample size was small: 5. Though we had problems in the past with the application releases, we didn’t have enough issues to be able to halt the deployment. I realized this, as did he, and that’s why he gave me instructions to proceed with another batch of installs. I turned and informed the Charlotte folks, who groaned. They had seen the problems before, too. So we bore down and hit another 5. Everything was okay. Then we did another 5. Another database lost. Still, we’re talking 2 out of 15, which wasn’t enough to stop the installation. The application was getting upgraded properly and there was a workaround, to tell the application to re-pull all the data from the server, basically re-establishing each local database. Another 5, and a couple more failures. Now we’re 4 for 20, and we’re maintaining at 20% failure rate.

So we huddled in Charlotte. We didn’t like the failure rate. However, the app was installing correctly other than wiping out a database on 1 out of every 5 systems. To stop the installs meant having to take the 20 we had installed back to their base image. That was time consuming. Also, we knew we’d just be back in a few weeks later, working yet another Saturday. We didn’t want that. We knew we could fix the problem when we encountered it, but checking the database was time consuming. However, the sync process was something that most of our end users ran nightly anyway, so you could run it on a perfectly healthy, but unmodified database. And there we had our answer.

We accepted the situation as it was. We were going to get failures. To stop meant another weekend of work. We could kick off the installs and then just go back and run the sync when it finished. Toggling the system to run the sync was faster than interrogating each and every database. That was the answer. We didn’t like the situation and we couldn’t fix the application. However, we could overcome what was before us. So we started down that road. Then my manager got back on the phone. He indicated that because multiple sites were experiencing the same failure rate we were, they were thinking about doing a rollback. I explained the decision we had come to and asked that we continue forward and that our approach be sent out to everyone else. Realizing that he didn’t want to work another weekend, either, he did just that. The next day, when I was working on the Columbia upgrades, we simply did the install and ran the sync. Everything worked out fine.

It worked out because we accepted the fact that we didn’t have a good install and we weren’t likely to get one. Knowing that, we had to develop another approach to get the job done with minimal impact to our end users and minimal time away from our families. In all of our lives there are numerous situations that don’t work out the way we’d like. Either that, or we have to go into a situation where we know things aren’t the way we’d want them. Some of those can be avoided and probably should be. However, others can’t. The key is to accept the situation for what it is. Once you do that, you can see about determining what you can do. This leads back to that Venn diagram of what I control and what is important. The other options don’t work:

Getting angry just leads to hurt feelings but likely won’t change the situation in a positive way.

Ignoring the situation can often lead to it getting worse.

Pretending its something different is just lying to yourself.

Throwing your hands up and being despondent is just giving in to the drama.

Accept the situation. See what about it you can change. Do what you can. And don’t stress over what you can’t. The stress doesn’t make things better, it only makes you worse.