Pages

2010-09-21

Murphy's Law

Let me introduce you to a good old soul by the name of Edward Murphy. Who was Edward Murphy? In short he was responsible for the famous saying, "Anything that can go wrong, will go wrong".

And so the story goes.

After preparing with the rest of my team members about what should be done - how it should be done, and going over all the current open issues that needed to be taken care of while I would be away on vacation, I thought, "Great! Finally some quiet and relaxing times while I go on vacation". But our dear old soul Murphy? Nuhuh, he had other plans …

I received an email 6 hours after I had flown out of the country - saying that a the following VM's were deleted.

Deleted? Deleted? How? What? When? Why???????????????????????

So what happened? User Error. Plain and simple. Someone on the storage team wanted to delete the snapshots on an NFS Qtree but by mistake, instead of deleting the snapshots they deleted the volume. And in the flick of a button, Boom! 40 VM's were gone!