The Day the Network Died

Ever notice how gear doesn’t break when you don’t need it? It always breaks when you need it most.

Last year our IT department began the process of replacing switches and other gear that was beyond 7 years old. They wanted to replace it all, but these things are expensive. So, slowly, they are swapping out old tech for new.

Unfortunately, this past weekend one of the old pieces that had not been replaced yet went down. It was in a core part of the network, and all access was severed. There was nothing internal and nothing external. No email, no servers, no internet, no http://www.firstorlando.com, nothing.

Well, church has been happening a long time without internet, so we should have no worries, right? For the most part, but you just don’t realize how much you rely on something until it’s gone. The network went down saturday afternoon, right before we began prepping for the service. We store lyric documents and sermon notes on a serve, for easy access. We could not get them. Luckily, I had downloaded them to my Outlook, and was able to put them on a flash drive and we loaded them up.

We could not stream the services. not only could no one get to the website to launch the player, we couldn’t even send the signal out. we were able to record the video for later upload in our On Demand section. But, we literally have hundreds of people who watch us each week. We went to Facebook and twitter to spread the word, using our 3G enable phones we could still access them. we use Planning Center Online to plan the services, and follow it during the service as well. On Sunday one of our technical people brought in his own mobile hotspot and shared it on the mini network we have in the Worship Center, so we could still follow. not powerful enough for the campus, but it worked like a charm for what they needed.

The IT guys worked some long hours, and at one point we had access. Then about 15 minutes before service start it went down again. They were able to get internal servers up and running, but external connection was still out. On Monday we got some help, and Monday afternoon it came back on. We hope it stays working.

While it was down we could not access anything we housed in “the cloud”. My Evernote files we unreachable, except by 3G enabled devices. We use Google Docs and an online service to help track workflow in communications. We pulled the project list on our phones, and wrote them down. If we had to have a file, we would need to run to a local fast food joint and use their wifi. It’s tempting to snag a 3G modem or hotspot just in case this were to happen again. Most of our creativity this morning was used in figuring out how to work around the outage.

In the end it was right at about 48 hours of no network access. I don’t want to go through it again. The internet/network has become like a utility. It’s not quite as important as power and water, but we do rely on it. When things work right, the technology we employ is a huge help. When it doesn’t work, it can be a roadblock. This was 2 days of trouble out of thousands without issue, so we will still be using the network and internet to help us get things done.