Addressing the Systemic Bias Problem

Jonathan Schooler is a professor of psychology at University of California, Santa Barbara.

Updated January 7, 2011, 1:27 PM

The frenzy of criticism that has greeted the recent publication of evidence for precognition highlights an important problem with its review process. However, this problem is not unique to this particular article.

An open repository that encourages researchers to log their methodology and predictions beforehand and report their results afterward would improve the review process.

As one of its reviewers I know that it received at least as rigorous an evaluation as other articles in high profile psychology journals. Rather the problem lies in the manner in which the peer review process restricts open access to scientific findings. This necessarily subjective vetting procedure produces systematic bias in that a sizable proportion of scientific studies by qualified researchers are unavailable for consideration.

Such a constraint on the open access to scientific findings has multiple implications for the present controversy. First, because peer review favors significant findings, we cannot know how many similar unsuccessful studies might exist. At the same time, because peer review has an important subjective component, we do not know how many successful studies with similar conclusions were rejected as a result of bias. As the present acrimony illustrates, there is a strong prejudice against this type of research, regardless of the rigor with which it is carried out.

One partial solution would be the development of an open repository of scientific findings that encourages researchers to rigorously log their methodology and predictions beforehand, and then report all of their results regardless of outcomes afterward. Such an open access repository of scientific findings might help to reduce the bias of the peer review process, and better enable us to discern spurious results from genuine discoveries.