Friday, July 29, 2016

Happy sysadmin day!

Also on a day like today, I'd like to reflect on my background as a systems administrator, long ago. My first job was managing Apollo DomainOS/AEGIS, HPUX, and SunOS servers and workstations for a small geographics company (we made custom maps for businesses). I was fortunate to have a wonderful boss who respected me and took me under her wing. She was my first mentor.

Every sysadmin has a story when they accidentally mis-typed a command or clicked on the wrong control and took down an important system. And after I'd been with the company for about nine months, I did that; I hosed a system with an rm -rf * in the wrong directory. I thought I was in the temporary directory, but I was in a system directory instead. I immediately freaked out! I went to my boss and told her what happened. I'm sure she was disappointed, but instead of yelling at me, she explained that in IT this happens to other people, and now it has happened to me. I wasn't the first to make a mistake like this. There was a way out of it.

She had me take a step back from the console, calm down, then think out loud about what we could do. I couldn't "undelete" the files, but I realized we had other servers exactly like this one. As long as I didn't shut down the system, the server would continue to function. I just had to copy over the files from another system like it, and make a few edits that were local to the system.

Over the next hour, I focused on repairing the system, which stabilized things. By the end of the day, I'd brought the system back to normal, and I was confident enough to reboot the system for a test. And it worked!

That was a scary moment, but I made it through with my boss's support.

I have since moved on to other companies as a system administrator, and never repeated that accident. Whenever I or someone on my team made an oops, I repeated the same thing my first boss told me: this has happened to other people, they fixed it, you can fix it too. I've moved into management and leadership roles, but I will never forget my first boss. As I've managed system administrators in my own teams, I hope I've provided the same support.