A Tale of When Things Go Wrong

Share this

In a down economy, it can be tempting for manufacturers to opt for cheaper networking components. However, before you pull the trigger on the cheapest price you can find, be sure to calculate what a network outage would cost you first. Here’s why.

In a previous life, I was responsible for a System Integrator (SI) Program for my employer. During that time, I had numerous conversations with SIs, and one shared a story that still sticks with me today. He did a job for a Fortune 500 food and beverage company. The installation and start-up went fine, but a few months later he got a call from the customer complaining that the system had been experiencing several annoying shut downs.

Basically, the system would stop for no apparent reason. It would then reset, be OK for a few weeks and then happen all over again. The user tried to fix the issue himself but couldn’t figure out the root cause of the problem.

The SI ended up flying an engineer to the site for a week. They ran several tests – verification of logic, etc. – but found no incidents. Two weeks later, the SI received another call from the customer - this time more irate. Again, the SI sent an engineer out to start checking the wiring. The engineer noticed a “subpar” brand was used, but could not show through test data that the wiring was the root cause.

Later in the week, the engineer noticed some flickering LEDs on sensors. Nearby there was a maintenance man leaning against a conveyor line where cable was run, and he had a walkie-talkie connected to his belt. He has the volume turned up very high, and occasionally you could hear chatter. As it turned out, the walkie-talkie induced enough electromagnetic interference (EMI) noise on the subpar cable that it caused false reads, which prevented the machines in the system from operating properly.

The moral of the story – don’t cut corners in the design of your network infrastructure. Excessive down time and frequent maintenance work costs can quickly add up. In my opinion, it’s better to pay now than to pay later.

Do you have a similar story? Feel free to share your thoughts or stories in the comments section below.