Useful sysadmin skills I’ve taught myself recently

GMass is a lean operation, and not only am I the product manager, but I’m also the sysadmin, which means maintaining and keeping watch over a dozen servers. I’ve been managing servers in some capacity for over 20 years now, but here are a few skills I taught myself in the last month that have increased my productivity and made me feel more like a sys-ninja than a sys-admin.

1. Tailing a log file

GMass generates tons of logs.

There’s IIS log files from our web server, but also internal logs created by a plethora of worker EXEs that do everything from sending mail merge campaigns to syncing stats with the campaign reports that you see. I used to open up a log file in Notepad, and then a few minutes later, close it and re-open it, so I could see what progress has been made in those few minutes. No longer. Now I’ve switched to Notepad++ which has a handy “tail” feature. Every few seconds, it checks for changes to the open file and refreshes it. Now I can just stare at my screen and watch the log file scroll by. I feel like Chloe from 24.

2. View just certain lines of a log file

I often want to see just log file lines that match some particular string. For example, if I have an abuser on my hands, I’ll isolate the IP address of the abuser, and want to see the web server log file lines matching just that IP address. I used to use Notepad’s “Find” function and search for the IP and scroll through all the matching hits. What’s proven to be much easier is using Notepad++’s “Find all in current document” feature, which isolates just the lines that match a particular string. No more endless scrolling through gigs of logs.

3. Query your text log file like it’s a database table

This has been a huge boon for my sanity. Microsoft has a little-known tool called Microsoft Log Parser Studio which lets you write SQL SELECT queries against any log files, even ones that haven’t been generated by IIS.

4. Search for text inside images

I take lots of screenshots.

There’s an error on the server? Screenshot it, so I can research it later.

Buying something from a shady website? Screenshot it as evidence of my order and the refund policy.

Booking travel on Expedia? Screenshot it, because their email confirmations never seem to arrive.

However, after collecting a plethora of screenshots over the years, they’re pretty useless unless I can search them. At the end of 2018, Dropbox released a search function that finds text inside images. Since I already use Dropbox to store all my screenshots, this made searching my screenshots a breeze. I did have to upgrade my account from Plus to Professional, and went from $10/month to $20/month, but totally worth it.

5. Emergency notifications that wake me up when I’m asleep

I’ve used an iPhone as my primary mobile phone for the last five years, mostly because my wife insisted I make the switch because she has one and this way my texts appear the more desired blue instead of the less desirable green. Turns out this wasn’t the best decision as a sysadmin, because Apple’s ecosystem is so “closed”. I need my phone to wake me up when there’s a big problem, like if GMass is entirely down, or if Gmail has made a code change that has broken the GMass extension. I searched and searched and even posed this question on reddit, and the home-grown cheap solution I settled on was an Android-based solution. Only Android was capable of doing what I wanted because Android apps are able to read text messages and take actions based on them. iOS is “closed”, meaning a third party non-Apple app has no visibility into text messages. So, I use Android’s FireAlert 2 app. I tell it what string to look for in a text, and if found, it sounds an alarm that could wake a sleeping bear during hibernation. I no longer wake up every morning in a rush to check my phone to make sure the servers are up. I can rest confidently knowing that I (and my wife and baby) will be woken up with the screeching sound of a 1,000 decibel alarm if anything goes wrong.