Experimentation mental tools

Far be it from becoming a mad scientist, everyone in business needs to know the basis on which our investigations into business works.

I recently came across Charlie Munger’s 1995 speech, The Psychology of Human Misjudgment, which introduced me to an explanation of what I’d been doing all my business life – using the power of applying mental tools from a wide array of disciplines and applying these to the business questions I had.

Last week I introduced tools for understanding creative approaches. These mental tools or analogies, or models or heuristics are concepts you can use to help try to explain things to find the answers to the critical questions in your business.

This week lets talk about experimentation- whether that’s hypothesis testing or research we need some background models to help us create and interpret results.

Scientific Method— “Systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses.” This is the foundation of modern business where we seek explicit and measured relationships between cause(s) and effect(s).

Proxy — “A variable that is not in itself directly relevant, but that represents an unobservable or immeasurable variable. In order for a variable to be a good proxy, it must have a close correlation (acts in the same way, is acted on in the same way) with the thing you are interested in.”

Selection Bias— “The selection of individuals, groups or data for analysis in such a way that proper randomization is not achieved, thereby ensuring that the sample obtained is not representative of the population intended to be analyzed.” (related: sampling bias)

Selection bias comes in two flavours: (1) self-selection of interested individuals to participate in an activity or survey, or as a subject in an experimental study; (2) deliberate selection of biased samples or studies by researchers to positively support or disprove a particular hypothesis.

Response Bias — “A wide range of cognitive biases that influence the responses of participants away from an accurate or truthful response.” Its also the tendency of a person to answer questions on a survey untruthfully or misleadingly. For example, they may feel pressure to give answers that are socially acceptable. The respondent may not be aware that they aren’t answering the questions in the way the researcher intended: the format of the question or the nature of the previous questions may have an unwanted impact on how a person responds to a survey.People tend to want to portray themselves in the best light, and to be viewed by the surveyor in the best light so they select responses to fill fil this rather than offering an answer that iis a true reflection of their potential actions.

Observer Effect— “Changes that the act of observation will make on a phenomenon being observed.” Its also a form of reaction in which the researched subjects modify an aspect of their behaviour, in response to their knowing that they are being studied.

Survivorship Bias — “The logical error of concentrating on the people or things that ‘survived’ some process and inadvertently overlooking those that did not because they aren’t around to study anymore.”

Here’s a great example of how to avoid survivorship bias. US military statistician, Abraham Wald worked out the secret to placing armour on aircraft bombers in a way that saved countless lives.

Here’s the problem Wald was confronted with. The Airforce commanders wanted to place armour on their bombers, but clearly couldn’t put it everywhere because it was too heavy for the planes to fly. So where is it best to armour a plane to maximise its chance of survival?

Wald looked at all the planes that returned from missions and saw a pattern of bullet holes like this:

Distribution of bullet holes in aircraft that returned to base after missions. Sketch by Wald. In “Visual Revelations” by Howard Wainer. Lawrence Erlbaum and Associates, 1997.

So, where should you put the armour? Have a think before reading on.

The commanders saw it clearly. Put the armour where the most bullet holes are. That’s where the planes are getting shot the most.

And, of course, that would have been a complete disaster.

Wald showed that actually, you should put the armour where the bullet holes aren’t.

Why? Well the commanders had fallen for the classic fallacy of survivorship bias. They were only examining the aircraft that made it back to base. The survivors. The missing aircraft, with their locations of bullet holes that shot them down, were never seen by the commanders. And therefore not taken into account. Wald showed that it was odds-on that those missing aircraft had holes in very different places, on average, than the surviving aircraft.

In short, what Wald’s diagram showed was the places an aircraft could take hits and still get home. These were the places you didn’t have to put armour on. The exact opposite to what the top brass wanted to do.

The essence of survivorship bias is that you often don’t see the failures. In business, in life, and in war. And sometimes it’s the failures that have the most important lessons. Like the planes that didn’t make it back. Wald’s reasoning went on to save lives not only in World War Two, but also in Korea and Vietnam.

and one final word on selection bias impacting interpretation of data…