Open Science Essentials: pre-registration

Open Science essentials in 2 minutes, part 1

The Problem

As a scholarly community we allowed ourselves to forget the distinction between exploratory vs confirmatory research, presenting exploratory results as confirmatory, presenting post-hoc rationales as predictions. As well as being dishonest, this makes for unreliable science.

Importantly, you can employ questionable research practices like this (“p-hacking“) without knowing you are doing it. Decide to stop an analysis because the results are significant? Measure 3 dependent variables and use the one that “works”? Exclude participants who don’t respond to your manipulation? All justified in exploratory research, but mean you are exploring a garden of forking paths in the space of possible analysis – when you arrive at a significant result, you won’t be sure you got there because of the data, or your choices.

Pre-registration is easy. There is no single, universally accepted, way to do it.

you could write your data collection and analysis plan down and post it on your blog.

you can use the Open Science Framework to timestamp and archive a pre-registration, so you can prove you made a prediction ahead of time.

you can visit AsPredicted.org which provides a form to complete, which will help you structure your pre-registration (making sure you include all relevant information).

“Registered Reports“: more and more journals are committing to published pre-registered studies. They review the method and analysis plan before data collection and agree to publish once the results are in (however they turn out).

You should do this

Why do this?

credibility – other researchers (and journals) will know you predicted the results before you got them.

you can still do exploratory analysis, it just makes it clear which is which.

forces you to think about the analysis before collecting the data (a great benefit).