Scientists know valid research - you can bet on it

Scientists have demonstrated the same principles applied to betting on sports events can be useful for determining the reliability of a scientific study. Image courtesy of Nathan Rupert.

An international team of scientists has shown how the ‘wisdom of the crowd’ extends to predicting how reproducible a piece of research is.

Researchers from Massey University, the Stockholm School of Economics, Harvard University and the University of Virginia have demonstrated the same principles applied to betting on sports events can be useful for determining the reliability of a scientific study.

Prediction markets allow investors to make predictions of future events by trading shares in the outcome of the event. The market price indicates what the crowd thinks the probability of the event is. Imagine, for example, shares that pay out $1 for the All Blacks winning a match. This price changes, however, as more people bet on the event occuring. If such shares are traded for 60c it means that the traders give the All Blacks a 60 per cent chance of a victory.

Reproducibility is golden in science because to truly know whether a result is valid, it needs to be reproduced in the same way, over and over again. However a recent study highlighted that more than half of the research published in three leading psychology journals did not hold up to re-testing – that is, the findings could not be reproduced.

The researchers allowed other scientists to estimate, or ‘bet’ on, the reproducibility of more than 40 experiments published in prominent psychology journals. They found prediction markets correctly predicted replicability in 71 per cent of the cases studied.

The next step in the research is to test whether or not prediction markets are accurate forecasters for the reproducibility of results in other fields, such as economics and cell biology.

One of the lead authors and Professor of Computational Biology at the New Zealand Institute for Advanced Study at Massey University, Thomas Pfeiffer, says prediction markets work because people will inherently try and correct incorrect odds.

“Participants can pick the most attractive investment opportunities,” he says. “If the price is wrong and I’m confident I have better information than anyone else, I have a strong incentive to correct the price so I can make more money. It’s all about who has the best information.”

The research was published today in The Proceedings of the National Academy of Sciences (PNAS). The full paper is available here.