338-349. Note that these were highly trained professionals
making judgments central to their work. In addition, they
knew that their medical judgments were being examined
by researchers, so they probably tried as hard as they
could. Still, their carefully considered judgments were
remarkably inconsistent.

20. The average intra-expert correlation was . 76, which
equates to a 23% chance of getting a reversal in the ranking or scores of two cases from one time to the next. In
general, a Pearson product-moment correlation of r translates into a [. 5+arcsin (r)/π] probability of a rank reversal of
two cases the second time, assuming bivariate normal
distributions; see M. Kendall, “Rank Correlation Methods”
(London: Charles Griffen & Co., 1948).

21. A provocative brief for this structured numerical
approach in medicine can be found in J.A. Swets, R.M.

29. T. Gilovich, “Something Out of Nothing: The Misperception and Misinterpretation of Random Data,” chap. 2
in “How We Know What Isn’t So: The Fallibility of Human
Reason in Everyday Life” (New York: Free Press, 1991);
see also N.N. Taleb, “Fooled by Randomness: The Hidden
Role of Chance in Life and in the Markets” (New York:
Random House, 2004).

30. The best way to untangle the confounding effects is
through controlled experiments, and even then it may be
difficult. For a research example of how to do this, see
P.J.H. Schoemaker and J.C. Hershey, “Utility Measurement: Signal, Noise and Bias,” Organizational Behavior
and Human Decision Processes 52, no. 3 (August 1992):
397-424.

36. Prediction banks are a special case of the more
general notion of a setting up a mistake bank; see
J.M. Caddell, “The Mistake Bank: How to Succeed by
Forgiving Your Mistakes and Embracing Your Failures”
(Camp Hill, Pennsylvania: Caddell Insight Group, 2013).