clustering illusion

The clustering illusion is the intuition that random events which
occur in clusters are not really random events. The illusion is due to selective thinking based on a counterintuitive but false assumption
regarding statistical odds. For example, it
strikes most people as unexpected if heads comes up four times in a row during a series of
coin flips. However, in a series of 20 flips, there is a 50% chance of getting four heads
in a row (Gilovich). It may seem unexpected, but the chances are better than even that a
given neighborhood in California will have a statistically significant cluster of cancer
cases (Gawande).

What would be rare, unexpected, and unlikely due to chance would
be to flip a coin twenty times and have each result be the alternate of the previous flip.
In any series of such random flips, it is more unlikely than likely that short runs of 2,
4, 6, 8, etc., will yield what we know logically is predicted by chance. In the long run,
a coin flip will yield 50% heads and 50% tails (assuming a fair flip and a fair coin). But
in any short run, a wide variety of probabilities are expected, including some runs
that
seem highly improbable.

Finding a statistically unusual number of cancers in a given
neighborhood--such as six or seven times greater than the average--is not rare or
unexpected. Much depends on where you draw the boundaries of the neighborhood. Clusters of
cancers that are seven thousand times higher than expected, such as the incidence
of mesothelioma in Karian, Turkey, are very rare and unexpected. The incidence of
thyroid cancer in children near Chernobyl was one hundred times higher after the
disaster (Gawande).

Sometimes a subject in an ESP experiment
or a dowser might be correct at a higher than chance rate.
However, such results do not indicate that an event is not a chance event. In fact, such
results are predictable by the laws of chance. Rather than being signs of non-randomness,
they are actually signs of randomness. ESP researchers are especially prone to take
streaks of "hits" by their subjects as evidence that psychic power varies from
time to time. Their use of optional starting and stopping is
based on the presumption of psychic variation and an apparent ignorance of the
probabilities of random events. Combining the clustering illusion with confirmation bias is a formula for self-deception and delusion. For example, if
you are convinced that your husband's death at age 72 of pancreatic cancer
was due to his having worked in a mill when he was younger, you may start
looking for proof and run the danger of ignoring any piece of evidence that
contradicts your belief.

A classic study was done on the clustering
illusion regarding the belief in the "hot hand" in basketball (Gilovich,
Vallone, and Tversky). It is commonly believed by basketball players, coaches and fans
that players have "hot streaks" and "cold streaks." A detailed
analysis was done of the Philadelphia 76ers shooters during the 1980-81 season. It failed
to show that players hit or miss shots in clusters at anything other than what would be
expected by chance. They also analyzed free throws by the Boston Celtics over two seasons
and found that when a player made his first shot, he made the second shot 75% of the time
and when he missed the first shot he made the second shot 75% of the time. Basketball
players do shoot in streaks, but within the bounds of chance. It is an illusion that
players are 'hot' or 'cold'. When presented with this evidence, believers in the "hot
hand" are likely to reject it because they "know better" from experience.
And cancers do occur in clusters, but most clusters do not occur at odds
that are statistically alarming and indicative of a local environmental
cause.

In epidemiology, the clustering illusion is known as the Texas-sharpshooter fallacy (Gawande 1999: 37). Khaneman and Tversky called it
"belief in the Law of Small Numbers" because they identified the clustering
illusion with the fallacy of assuming that the pattern of a large population will be
replicated in all of its subsets. In logic, this fallacy is known as the fallacy of division,
assuming that the parts must be exactly like the whole.

further reading

Gawande, Atul. "The Cancer-Cluster Myth," The New
Yorker, February 8, 1999, pp. 34-37.