Syndication

August 31, 2005

Most scientific studies are wrong?

John Ioannidis, an epidemiologist at the University of Ioannina School of Medicine in Greece, says that small sample sizes, poor study design, researcher bias, and selective reporting and other problems combine to make most research findings false. But even large, well-designed studies are not always right, meaning that scientists and the public have to be wary of reported findings.

"We should accept that most research findings will be refuted. Some will be replicated and validated. The replication process is more important than the first discovery," Ioannidis says.

In the paper, Ioannidis does not show that any particular findings are false. Instead, he shows statistically how the many obstacles to getting research findings right combine to make most published research wrong.

Traditionally a study is said to be "statistically significant" if the odds are only 1 in 20 that the result could be pure chance. But in a complicated field where there are many potential hypotheses to sift through - such as whether a particular gene influences a particular disease - it is easy to reach false conclusions using this standard. If you test 20 false hypotheses, one of them is likely to show up as true, on average.

Meanwhile we are told that we must have absolute faith that when these same scientists claim to have proven Evolution to be a fact, it just has to be true.

Of course, the article did include this caveat...

Assuming that the new paper is itself correct...

Peer review is the normal process by which a scientific paper is determined to be true or false. Other scientists, presumably, will look to see if errors were made, or even attempt to duplicate the experiments that are claimed to prove the paper's conclusion. Sometimes this works by either confirming the conclusion, or exposing flaws. Sometimes it fails in that any real peer review is withheld until public opinion has so strongly accepted the conclusion that few scientists dare challenge it, as in the claim that HIV causes AIDS as well as the claim that life emerged through Evolution. Peer review is useless when there are outside pressures that would influence the peers toward accepting or rejecting a certain conclusion.

Here's an example...

Unusual marking have been found on the face of cliff X.

SCIENTIST A: These marks are the result of micro-organisms growing in a colony and expanding in specific patterns due to the chemical nature of the rock face.

SCIENTIST B: These marks are the result of erosion. The combination of wind patterns, sun angle and temperature produce the odd patterns.

SCIENTIST C: These marks are the result of some person at an unknown time in the past scratching them into the rock face with some sort of implement.

SCIENTIST A: RELIGIOUS FANATIC!!!

SCIENTIST B: FUNDAMENTALIST KOOK!!!

That is modern “science” in a nutshell.

Posted by Danny Carlton at August 31, 2005 06:34 AM

Trackback Pings

TrackBack URL for this entry:
http://www.jacklewis.net/cgi-bin/mt/jl-tb.cgi/1955