Monday, November 9, 2009

Genetic 'breakthroughs' in medicine are often nothing of the sort

During the second world war, the physicist Enrico Fermi asked General Leslie Groves of the US Army how many generals might be called "great" and why. Groves replied that any general who won five major battles in a row might be called great, and that about three in every hundred would qualify.

Fermi countered that if opposing forces are roughly equal, the odds are one in two that a general will win one battle, one in four that he will win two battles in a row, one in eight for three battles, one in 16 for four battles, and one in 32 for five battles in a row. "So you are right, General, about three in a hundred. Mathematical probability, not genius."1

There's an analogue of Fermi's "great general": the "great scientific discovery", or at least, as a case study, "the great genetic scientific discovery" as reported in the press. The discovery of genes for a certain behaviour, for schizophrenia, for happiness, always get good press coverage, usually based on publication in a respected scientific journal such as Science or Nature.

The research paper will include a statistic: the probability that the finding could have occurred by chance. The probability will have been sufficiently low that a reviewer for the journal was impressed and therefore recommended publication. Typically this probability or "P-value" will be less than 0.05, or 5%, which means the odds are less than one in 20 that the observed genetic correlation could have occurred by chance.