Getting out of jail free

FREE

Become An Insider

Sign up now and get free access to
hundreds of Insider articles, guides, reviews, interviews, blogs, and other premium content
from the best tech brands on the Internet: CIO, CSO, Computerworld, InfoWorld,
IT World and Network World Learn more.

Innocuous details can bring a network to its knees, so expect the worst and prepare for widespread outages before they happen

Email a friend

To

From

Thank you

Your message has been sent.

Sorry

There was an error emailing this page.

InfoWorld|Jan 9, 2012

It's 1:30 in the morning. By some miracle, you were able to get approval for a four-hour downtime window to complete a long list of overdue patching and network maintenance. Even better, you're done a half-hour early. Life is good!

As you're about to email the third shift to let them know they can get back in ahead of schedule, you remember it: that one setting you always knew was wrong and wanted to fix -- and, you thought, shouldn't cause any service disruption -- so you never got around to correcting it. Little do you know that "fix" is going to be your undoing.

It doesn't matter what it is. For me, it's been an incorrectly set spanning tree bridge priority or UPS software configured with an inadequate shutdown delay. Either way, half a second after hitting Enter or clicking Apply, your terminal freezes, pings go unanswered, and panic sets in: You've brought down the entire network, and you have no idea why. You thought you finished 30 minutes ahead of schedule, but now that half-hour may not be enough time to run around with a laptop and console cable to figure out what happened, much less fix it.

You can't avoid situations like this all the time. Bad things happen when you least expect it -- the old adage "If it ain't broke, don't fix it" applies to IT as much as it does to any other field. Nonetheless, you can build safegaurds into your network that will drastically reduce the time it takes to fix problems when they arise.