How to Tell if You’ve Made a Good Decision

Fifty years ago, after helping a bunch of nuclear engineers at General Electric figure out whether they should add a superheater to a new reactor, engineering professor Ronald Howard sat down to analyze what he’d just done. “Decision analysis,” he called it, and described it as “a logical procedure for the balancing of the factors that influence a decision” that factored in “uncertainties, values, and preferences.”

The takeaway is that you can’t tell the quality of the decision you are making by the outcome that will be produced. Or, to put it another way, you can’t tell by the outcome whether you made a good decision. It’s just a logical mistake to say, “I got the good outcome, I must have made a good decision.” And yet that’s what everybody thinks.

This way of thinking about decisions isn’t unique to Howard. The Decision Analysis 50th Anniversary celebration in San Francisco that brought me out to talk to him attracted a few hundred kindred spirits from industry (mostly the oil and gas industry) and academia, as well as the man considered the co-founder of the discipline, emeritus Harvard professor Howard Raiffa. All of them subscribe to the mantra that it’s how you make the decision that matters, not necessarily how it turns out.

So how do you make a good decision? Well first, Howard says, you make sure you’re framing it correctly. Otherwise you risk getting “the right answer to the wrong question.” After that it’s a matter of characterizing the alternatives, assessing the available information — which is expressed in terms of probabilities even if you don’t know what the exact odds are (“the whole idea of probability is to be able to describe by numbers your ignorance,” Howard says) — and expressing your preferences.

A lot of that is easier said than done. Psychologists and economists, with Daniel Kahneman and Amos Tversky initially leading the way, have collected tons of experimental evidence showing that people struggle with assessing probabilities and even expressing preferences in a consistent way. And psychological Gerd Gigerenzer and his academic allies have more recently identified some kinds of decisions for which simple rules of thumb deliver consistently better results than decision-analysis-style methods.

Getting consistently better results is one thing — that’s statistical information that should be incorporated into future decisions. But Howard is surely right that we put far too much weight on single-decision outcomes in business and everyday life.

He cites the example of the two-point conversion in American football. “The coach has decided to run for the two-point conversion, and it doesn’t work,” he says. “The announcer says, ‘Oh, that was a bad decision.’ It’s completely illogical!”

In a similar vein, UC Berkeley economist David Romer found in 2006 that professional football teams go for it on fourth down far less often than they would if they were really trying to maximize the number of points scored — and interpreted this as evidence that real-world businesses probably don’t maximize profits either. His findings have been widely publicized, but seem to have had little impact on NFL practice. Every Sunday the New York Times 4th-Down Bot, constructed along lines similar to Romer’s model, plaintively tweets again and again after punts and long field goal attempts that “I would have gone for it.”

Coaches don’t go for it presumably because they’ll be berated by fans, announcers, and owners for conversion or fourth-down attempts that fail, but not for punts or field-goal or extra-point attempts that, statistically speaking, leave points on the table. As a result they’re winning fewer games than they would if they followed a logical, points-maximizing approach. That’s what you call low-quality decision-making.