Kill the Black Box – Living In A Widget-less World

I grew up on a farm and, in addition to growing crops, we processed and packaged crops for other farmers. Companies were always trying to sell us a new piece of machinery to increase the speed and/or efficiency of our packing operation. Every time someone showed up with a “grading machine” that had parts hidden in some encasement that my grandfather couldn’t access, he dismissed the salesman and the machinery out of hand. “I’m not buying something if I can’t see how it works. Period.”

He may have been a bit rigid in his disdain for the black box approach; however, at its core, my grandfather was on to something.

We were doing work for a contact lenses manufacturer and they wanted to conduct a strength and weakness analysis for their most popular brand. We (David) were pitted against a large, multi-national firm (Goliath) for this business. In the end, Goliath won because they offered a really whizbang black box solution – with very cool but complicated output and the all-important national norms.

Once the project was completed, I asked for a copy of the survey questionnaire. Turns out, near the end of the 35-minute phone survey, there was the big question fueling the black box. It had 27 contact lens attributes. 27 attributes? Who knew a contact lens could be so deep and multi-faceted to need a rating on 27 different items. But that’s not the best part. Not only did they want you to rate the brand of contact lens you are currently using, you also needed to rate two additional brands you may or may not have ever used.

So, every person interviewed had to rank 81 attributes – E-I-G-H-T-Y-O-N-E – each of them on a 10-point scale. Given that most contact lens wearers have only ever worn 1 (possibly 2 brands) and realizing that a 35-minute long interview is mind numbing, even on the most interesting of topics, one might question why you would put contact lens wearers through this exercise. You with me?

It seems that to create really cool but complicated output, the black box required all three major brands be rated. In addition, the more attributes included, the more reliable the predictor. So, the black box dictated this survey design. Hmmmm.

I saw the output. It was very cool and complicated looking.

But, I also couldn’t help but wonder if maybe in the pursuit of feeding the black box requirements, we were asking people to rate things they couldn’t, and asking them to do so in an environment where they surely must have lost interest long before the process ended. Did this thick and heavy report with the whizbang output in any way reflect reality? Or did this client end up with the equivalent of garbage in = garbage out.

Most proprietary research techniques are designed by statisticians. Though fine people, these statisticians are typically pretty far removed from the consumer and his/her decision-making environment. They know how the numbers work and what’s required to get “predictive” output. But the flaw in the process is that these statisticians assume the input they use is reflective of what people actually think and how they make decisions.

You and I don’t rate contact lenses on 27 attributes when deciding which brand to use. But the black box design assumes we do.

Not every black box is bad – but when black box techniques become the main driver for how the research is conducted, they can often lead the research astray – with no one the wiser.