moral psychology

On Rollercoasters, Data Collection, and RRs

Before Christmas some time I was talking to Aurélien about Registered Reports. I was trying to articulate my ambivalent feelings about them – which seem entirely disconnected from my not-at-all-ambivalent thoughts about them. By which I mean: I believe, even “know”, that RRs are An Excellent Idea, that in an ideal world I personally think all my studies would be registered reports, and that – even in a non-ideal world – they help decouple publication from p < .05, and are therefore one important way of getting out of this mess. And yet, I also find the idea extraordinarily intimidating. Why??

I/we didn’t get to the bottom of it, but I did discover one surprising (?) thing.

So. In the current/old system, I have an idea, I design a study to test it, I run the study, and then when I analyse the data… it’s super exciting. This is it! The test of my maybe-awesome-maybe-awful idea! If the answer is p < .05, I feel really happy. If the answer is p > .05, I feel bummed. (Sometimes I set up my studies so that both answers are nonetheless informative with regard to my research question, but not always.) Let’s also just say that this was such a perfect study that when p < .05, I can write it up and submit it for publication immediately.

With a Registered Report, I would have an idea, and design a study to test it. Then, I would write it up, and submit it for review to one of the many journals that now offer it. Let’s say after a couple of rounds of review, it receives In Principle Acceptance. I’m now ready to run the study, and analyse the data… and it’s not really exciting any more. If it’s a p < .05, I’m not especially happy. If it’s a p > .05, I’m not especially bummed. The answer is probably informative either way, and it is almost certainly going to get published either way.

This seems… good. The rational me recognises the value of a system which evaluates a study on the soundness of its theorizing and methodology, not on whether I happen to guess the right hypothesis or not.

The actual me, who was trained and had her emotional responses conditioned largely in the old system where p < .05 was a win and promised at least the potential for publication, and p > .05 was a fail and promised rethinking and uncertainty and wasted time, this actual me who as a result of this training feels the same heightened anticipation at the start of data analysis as at the top of a roller coaster, feels like… I will miss that feeling. It’s a dumb thing to miss. (What I’m “missing” is essentially the feeling of a gambler, and I can go to the casino if it gets too unbearable!) This local “cost” of Registered Reports that I just discovered – a p < .05 not feeling quite as awesome as it used to – is worth paying for the benefit of a p >.05 not being the undoing of a young career, and for the benefit of our literature not being littered with false positives.

To put this another way – “if you want the rainbow you’ve got to put up with the rain”, but with Registered Reports we can do away with both and it will actually be totally fine. It just feels weird to realise that one of the things I learned to value – viscerally, to get kicks out of, if not on like a True Value level – was just a perverse side effect of the publication system’s skewed incentives.

I think I can forgive myself – and others – for feeling ambivalent about the initiatives that are revealing these uncomfortable truths… even as it makes me all the more determined to not let my conditioning get in the way of reform.