The Case of the Crazy Conveyor

A new conveyor wreaks havoc in a newspaper printing plant, but what's to blame?

An EMI consultant investigates the source of a new conveyor's erratic behavior.

Remember the I Love Lucy episode in which the conveyor belt transporting chocolate candy speeds up faster and faster?

That was pretty much the scene at a major newspaper printing facility when I was called in to investigate whether electromagnetic interference (EMI) was the source of a problem with a new handling system. It would occasionally and randomly stop, while the printing press kept going. Talk about chaos.

The handling system was like a Ferris wheel turned out its side. It had a dozen bins to carry bundles of newspapers to waiting trucks at a loading dock. Individual embedded controllers had the task of ejecting labeled bundles at the appropriate location. A minicomputer located in a control room had supervisory responsibilities and the entire system was networked by Ethernet.

The system was a prototype, and it was still being debugged. As such, nobody was even sure it was EMI-related. My task was either to find an EMI cause or to eliminate EMI from further consideration. I focused on the big three EMI threats: electrostatic discharge (ESD), power disturbances, and RF interference.

My initial concern was ESD -- both human generated and machine generated. After all, this was a printing press, and this type of equipment can generate a lot of static discharge. But my client insisted ESD could not be the problem, as the press operated in a humid environment. Furthermore, nobody had ever complained of getting an ESD shock.

My client suspected RF, as the trucks had VHF radios to communicate with the control room. In addition, floor personnel had handheld VHF radios. Although RFI was not my highest priority, I started there mainly to appease the customer.

Day one was spent probing the system with a handheld radio, trying to force a failure. My investigation included getting up into the ceiling to be sure the Ethernet cable was not located next to a radio antenna. No problems occurred, so we ruled out RFI.

Day two was spent checking out the power. Before my visit, I had the client install a power disturbance analyzer at one of the stations. I then examined the power wiring all the way back to the service panel. No anomalies were noted, so I ruled out any power disturbances.

On day three, I hauled out my ESD gun. When I hit the first truck loading station's display panel at about 4 kV, the controller hung.

Bingo. Upon lowering the voltage, subsequent stations would hang at 1,500 V -- well below the threshold of what a human can sense and very easy to generate. Further examination revealed the display panels were not bonded to the main cabinets. They were mounted on small rubber bushings to minimize vibration. This meant any ESD current was forced into the embedded controller via an interconnection cable. Bad ground connections were the culprit.

As a quick experiment, I cleaned off paint and connected a piece of copper tape between the display panel and station cabinet. Now we could withstand about 15 kV. As a redesign, a commercial copper strap was installed between the panel and cabinet.

A simple fix for a multimillion-dollar machine.

But wait, there's more. When the first system hung, someone yelled from the control room that the main control computer had crashed. I decided to check it out. The display screen was frozen, but it had multiple messages, such as: "System 1 (which I had hit first) down, system 9 down, system 11 down, attempting to reconstruct, unable to do so, etc." and "System 2 (which I hit next) down, system 7 down, system 5 down, etc."

Apparently, the ESD hits caused other dominoes to fall.

So I dutifully recorded the messages from the frozen screen and noted in my final report that the system software appeared to be unstable in the presence of ESD.

About three weeks later, I got a rather heated call from the VP of the software company. He wanted to know who I thought I was to point fingers at his software. After letting him blow off some steam, I shared my recorded screen data. Suddenly contrite, he admitted the software had some problems.

I explained that our ESD repair was just a patch, and that the software still deserved attention before something else toppled the dominoes again. He agreed, and we parted friends.

There are several lessons to be shared here:

The client's main concern may not be the problem.

ESD problems can occur at very low levels, even below human feeling.

Software can be a factor (and also a fix).

Record everything. You may need it later to cover your behind.

Daryl Gerke (PE) is a partner at Kimmel Gerke Associates Ltd., an EMI/EMC consulting and training firm. He and his business partner Bill Kimmel (PE) have solved or prevented hundreds of EMI/EMC/ESD problems across a wide range of industries. They have also trained more than 10,000 students in their public and in-house EMC classes. Gerke has a BSEE from the University of Nebraska and resides in Mesa, Ariz. You can contact him at
www.emiguru.com or via his consulting blog at www.jumptoconsulting.com.

Good plan! It doesn't take much ESD, particularly when you are inside a system.

In this case, my client kept insisting it couldn't be ESD, since nobody ever felt a shock. But like you, we proved that ESD does, in fact, exist below the threshold of human feeling. Fortunately no damage - just upsets.

Glad to see you're still using the old (proven?) technology. I guess we've both reached the age where old is OK - just like fine wine :-)

I once lost a PC motherboard to ESD while installing a new hard drive--never felt the zap. Ever since then, I use an ESD wrist strap when working on a computer and I try to keep one hand on the chassis at all times (not always possible).

I opend the 9-year-old desktop computer just the other day for its semiannual cleaning. It still runs.

I totally agree, Adequate Grounding is the First Test. Not by ohm meter, Use a Megometer. Flow some real power and measure the resistance under load.

As a retired USAF Aircraft Electronic Tech I can relate to the pressure of "It can't be that" But eventually, trust your instints & training, it usually pays off.

Worst case for me: The F-111 had "Grounding Rings" where many cable grounds and part chassis grounds were gathered to be taken to Airframe ground (No Earth ground available in airborne items). The rings would build up corrosion internally until various systems would start getting flaky, and writeups (Pilot Complaints) were not very consistent. RADAR was a common early fail item due to the power flowing in that system.

Eventually it always seemed to come down to a grounding ring somewhere that had not been touched in over a decade. Crank them apart and capture the crud falling out for evidence. Clean with a wire wheel and then re-assemble. Conductive grease not allowed ! Problems in multiple systems would evaporate.

When a client insists on some inefficient or unnecessary procedure, there's a simple mantra to repeat to oneself to make the situation tolerable; "The meter is running, the meter is running.....". (As long as it's not a fixed-price contract.)

Good advice! Studies have shown that grounding is a major contributor to systems related EMI problems. Always a good place to start.

The reason for checking RF first was at the customer's insistence. Based on the symptoms, I felt that was the least likely cause. But as a consultant, it is important to keep the client happy.

Although we checked the power interface next, the power disturbance analyzer had been in place for a week and showed no perturbations. One parameter monitored was the voltage between the power neutral and safety ground. So in a sense, we did start with the power grounding.

A common power grounding problem is connecting the power neutral to the safety ground at more than one point. This often inadvertantly happens in sub-panels, which was part of my investigation. We chased the power wiring all the way back to the service panel, the point where the single return/safety ground bond should be made.

The reason for checking ESD last was again at the customer's insistence that ESD was not an issue (I suspected otherwise.) The poor ground connection between the panel and cabinet was not readily apparent. At first glance, it looked OK. Only after removing a display for inspection did it become apparent that the display was not properly grounded to the cabinet. Oh well...

Chasing problems in the field is often a process of elimination. Grounding is a good place to start as it is often relatively easy to check (or at least eliminate from further consideration.) Thanks for your comment!

Sometimes, it's the mundane things in electrical and electronic devices that make the difference. The strain relief on an air conditioner's line cord saved the day when the unit fell out of the window.