1. Identifying the attributes of consistently superior
forecasters, including their greater curiosity,
open-mindedness, and willingness to test the
idea that forecasting might be a skill that can be
cultivated and is worth cultivating;

2. Training people in techniques for avoiding common cognitive biases such as overconfidence and
overweighting evidence that reinforces their
preconceptions;

3. Creating stimulating work environments that
encourage the best performers to engage in collaborative teamwork and offer guidance on how
to avoid groupthink by practicing techniques
like precision questioning and constructive
confrontation;

Based on our experience, the biggest benefit of
prediction tournaments within organizations is
their power to accelerate learning cycles. Companies can accelerate learning by adhering to several
principles.

• The first principle involves careful record keeping.By keeping accurate records, it is harder to misre-member earlier forecasts, one’s own, and those ofothers. This is a critical counterweight to the self-serving tendency to say “I knew it all along,” aswell as the inclination to deny credit to rivals “whodidn’t have a clue.”• Second, by making it difficult for contestants tomisremember, tournaments force people to con-front their failures and the other side’s successes.Typically, one’s first response to failure is denial.Tournaments prompt people to become more re-flective, to engage in a pattern of thinking knownas preemptive self-criticism; they encourage par-ticipants to consider ways in which they mighthave been deeply wrong.

• Third, tournaments produce winners, which naturally awakens curiosity in others about how the
superior results were achieved. Teams are encouraged to experiment and improve their methods
all along.

• Fourth, the scoring in prediction tournaments is
clear to all involved up front. 9 This creates a sense
of fair competition among all.

Until recently, there was little published research
that training in probabilistic reasoning and cognitive debiasing could improve forecasting of
complex real-world events. 10 Academics felt that
eliminating cognitive illusions was nearly impossible for people to achieve on their own. 11 The IARPA
tournaments revealed, however, that customized
training of only a few hours can deliver benefits.
Specifically, training exercises involving behavioral
decision theory — from statistical reasoning to scenario planning and group dynamics — hold great
promise for improving managers’ decision-making
skills. At companies we have worked with, the
training typically involves individual and group
exercises to demonstrate cognitive biases, video
tutorials on topics such as scenario planning, and
customized business simulations.

3. Model the Experts in Your Midst

Another way to create a more intelligent enterprise is
to model the knowledge of expert employees so it
can be leveraged more effectively and objectively.
This can be done using a technique known in decision-making research as bootstrapping. 12 An early
example of bootstrapping research in decision psychology involved a study that explored what was on
the minds of agricultural experts who were judging
the quality of corn at a wholesale auction where
farmers brought their crops. 13 The researchers asked
the corn judges to rate 500 ears of corn to predict
their eventual prices in the marketplace. These expert judges considered a variety of factors, including
the length and circumference of each ear, the weight
of the kernels, the filling of the kernels at the tip, the
blistering, and the starchiness. The researchers then

FINDING THE SWEET SPOT

To create a more intelligent enterprise, executives need to leverage
the strengths of both humans and computers in order to produce
superior judgments. That will require a sophisticated understanding of
both human decision making (the “soft side”) and evolving technology-enabled capabilities (the “hard side”).