Network tools for outage troubleshooting

“Hey, it’s Steve from the NOC. We’re going to need you to head to the satellite office, they’re down hard and there’s no one local to assist.”

Well, damn… So much for that [Dinner|Sleep|Event|Whatever]. Many of us in the IT industry are all too familiar with off hours work and on-call schedules. As a network administrator or engineer, you likely will experience a call schedule in some form or another throughout a portion of your career. The dreaded 2 AM wake up to troubleshoot an outage is a necessary evil to ensure services are restored, and having a set of tools to assist already determined can provide you some comfort as one less thing to worry about. Obviously ideally you will want to troubleshoot remotely first if possible, even if only providing troubleshooting steps and questions to a user located at the outage. If initial details indicate that you have to get moving to that location, bring the right tools so you aren’t dead in the water upon arrival.

Tools for traveling to an outage

Should you have to travel, here are a few tools to consider bringing:

1. A Toolkit
Ideally, you want a grab bag that has most of what you need ready to go. This will save you time and the headache of tracking down that one fiber cable that’s in the back storage room under those 12 boxes that you’ve been meaning to dig out a while ago. A backpack or shoulder sling bag (what are these hipster bags even called?) will work and allow you flexibility to add new items over time if needed. There are also commercial solutions available that come stocked with plenty of tools such as crimpers and cable testers. If the case they provide is enough to meet your needs, that’s also another viable option.

2. Cell Phone
Let’s be real, the IT industry is made up of people who learned how to Google things better than their friends until they got so good at Googling that they didn’t need to do it as much. With that said, any abnormal issue you run across may require you to look up a solution. For all you know, the circuit that’s failing could be because of a recent network software upgrade causing a known bug to show itself. Only Google will be able to provide you with some of the information. This isn’t to short change actual troubleshooting steps and processes to isolate where the issue is, as those are critical in troubleshooting triage as well. Having a cell phone is nice when you’re dealing with a location who’s internet has been knocked offline, leaving no means of internet access.

3. Laptop / Console Cable
This one shouldn’t need any introduction if you’re familiar with networking in the slightest. If you don’t have management vlan access at a remote location, you’ll be stuck with a console cable into at least one device, depending on your VTY restrictions possibly all of them. Toss the baby blue, a USB adapter if you don’t have a serial port, and your trusty laptop into your grab bag before you head out.

4. Network Testing Tools
This particular set of items will depend heavily on your network. If you know your infrastructure is fiber heavy, you may wish to have a fiber testing light, or an OTDR if your company has the bucks to spend. An OTDR will tell you information on loss of light/signal as well as show you different patches/breaks the fiber has. For the ethernet side of things, a simple end to end pin tester will allow troubleshooting potentially bad cables. If you can’t find the other end of those cables (we’ve all been there), a Toner or Fox and Hound device would help you identify where they are terminated. There are also higher end network testers for ethernet and fiber that can pull CDP or LLDP information from the switchport they are connected to. Reference the NetPi project showing how to build one with a Raspberry Pi 2. For wireless heavy networks, bring your favorite wifi scanning software or tool to identify AP signal strengths/outages. You only need a few tools per type of cabling you use, but they could save the day in a pinch.

5. Fiber / Fiber termination kit / Ethernet / Crimpers / RJ45s
Fiber and Ethernet come each with their own unique problems. For example, if you have fiber out to the user level, users will crush it. Adversely, if you use ethernet out to the user level, they will somehow also figure out how to destroy that over time. Essentially, you should always bring replacement layer 1 for those simple layer 1 issues. Fiber termination kits are small allowing easy termination of broken ends without rerunning cable. Crimpers and RJ45’s are required for repairing ethernet breaks and broken clips, so toss a pair into your grab bag. Also, toss a flashlight in that bag, you’ll thank me later.

6. Miscellaneous ItemsThese items are on a case by case basis as most likely will be a bit bigger than everything else carried, and depend heavily on technologies used in your network. If you know the location you are heading to uses any special media converters, modems, line cards, or network devices that are prone to failure… a hot spare will make you look like a hero instead of having to call in reinforcements or a purchase request. In dire situations, some metro areas do have tech-centric stores where you can find more specialty networking gear. Often if you go beyond the basics, your local Best Buy is out of the question.

7. Documentation
Last but not least, documentation. If you have a means to VPN and access documentation from your phone or laptop, that is perfect. If you’re heading to a building who has lost internet access, just keep in mind that you may want to print some details or at the very least write a few key IP’s and ISP phone numbers down so you aren’t stranded once you are on site. If you don’t have decent documentation, you shouldn’t be reading this, you should be building decent documentation. Don’t let your dreams be dreams! DO IT!