Worst Usability Issue?

by Tom Tullis
Posted on July 4, 2011

When I recently learned that I would be receiving the Lifetime
Achievement Award from the Usability
Professionals Association (UPA), it caused me to reflect on my 35+
years in this field. One of the things I started wondering about was the
worst usability issue I've personally encountered in my work. The one that
pretty quickly came to mind is something I worked on many years ago (early
1980's) when I was at Bell
Laboratories. Specifically, I was working in an area that was
designing systems and equipment for telephone repair personnel to use in
testing and maintaining the phone lines running from the central offices
to individual homes or offices. (This was back in the days when there was
such a thing as the Bell
System.)

One day I was asked to review a prototype of a new device being designed
for the field repair personnel to use in testing for possible high
voltage on something they needed to work on. Keep in mind that
telephone lines and eletrical lines are often run on the same poles, and
telephone technicians are often called out during or after a major storm
to repair downed phone lines where there might also be downed power lines.
So a portable device that can easily test for high voltage is very
important.

Here's a mockup of that original prototype:

Unfortunately, I don't have an actual photo of the original
prototype. (Note to self and others: Take photos of the things you work
on, whenever you can.) The probe is hand-held and vaguely resembles a
large flashlight with a pointed tip. The tip is designed to break through
any paint that might be on the surface to be tested. The user would
depress the trigger, touch the tip to surface to be tested, and then look
inside the base of the unit at the two green LEDs imbedded there. If the
LED on the "trigger" side was lit that means it is safe to proceed. If the
LED on the other side was lit, that means there is hazardous voltage
here that could kill you.

Hmmm... Does anyone think there just might be a usability
issue here? Let's see... A single green LED is being used to
indicate either "OK" or "Hazardous Voltage". And keep in mind that the
user can twist and rotate their hand and the probe in various ways to see
inside the bottom of the unit.

So I gave my feedback outlining in no uncertain terms my
concerns about the design. (The engineers did not want to use a red LED
for the hazardous condition because at the time they were not sure they
could find one bright enough.) The decision-makers weren't convinced yet.
They believed the users would always be able to tell which green LED was
which. They wanted proof that a problem existed. So I designed a quick
usability test.

For my test, I recruited 10 participants. They were
definitely a sample of convenience, consisting mostly of co-workers at
Bell Labs who were not involved in the project. The prototype we were
testing didn't actually work, so the engineers rigged it such that when
the tip was touched to a surface one of the LEDs would come on. They
designed it so that the "OK" LED came on about 80% of the time and the
"Hazardous" one about 20% of the time. (Once lit, the LED stayed on until
it was reset by touching a metal stud on the side of the probe.)

I set up five different stations, outdoors, for the participants to do
their tests on. Some even involved climbing on a step-ladder (to simulate
working on a telephone pole). Each participant would go to a station, do
the test, and announce to me what the result was ("OK" or "Hazardous
Voltage"). They would then hand the probe to me, I would record which LED
was lit, reset it, and hand it back to the participant. After going
through the five stations once, I had them go through again. So each
person did 10 tasks. The testing was done over 3 days, and the weather
ranged from clear to a light rain. A few of the sessions were done late in
the day, at sunset. Overall, there were 100 "participant-tasks".

Of these 100 participant-tasks, about 20 signalled the "Hazardous
Voltage" condition. The key finding was that in one of those
cases, the participant failed to recognize it as the "Hazardous Voltage"
condition. No one ever reported the "OK" condition as the "Hazardous"
condition. So technically the overall task success rate was 99%.
In most usability studies, you would probably be thrilled to get an
overall task success rate of 99%. But not in a case like this. That one
error would have been life-threatening in the real world.

Bottom line: The decision-makers were convinced and the design was
changed significantly. The two LEDs were moved from inside the base to the
top of the device, and the "hazardous" LED was changed to a flashing
red LED. The device, basically as it was redesigned, is still being
made today. Here's a photo:

You can also see a YouTube
video of one of these probes being used to detect hazardous
voltage behind a flat-panel TV.

So here's a case where just one participant making one
error was enough to convince people of the need to make a
significant change in the design. And I think this easily qualifies as the
worst usability issue I've encountered in my career.