Re-engineering Peer Review

Fixing peer review

Today Academic Karma comes out of beta with the launch of a new website and we thought it was the perfect opportunity to explain what we think is wrong with peer review, why this is bad for science and how Academic Karma aims to fix peer review.

Peer-review is – at its best – a cornerstone of science. Good peer review identifies potential weaknesses in scientific work, encourages authors to do further work to provide convincing evidence if necessary, and helps to ensure that details required for others to understand and replicate experiments are presented. Good peer-review should lead to greater reproducibility and fewer retractions.

While publication is heavily incentivized – and publication rates continue to grow dramatically (e.g. doi:10.1371/journal.pone.0068397 ) – there are meagre rewards for good peer-review. Editors recount stories of prolific scientists who barely peer-review at all. The reviewing academics do is largely on the basis a sense of duty or friendly coercion from editors. As a result peer review is rarely prioritised and leads to long delays in publishing.

As explained in our animation and faq, Academic Karma is a universal peer review platform. This means that as a reviewer, you could now do all your reviewing, for whatever journal, using Academic Karma. You would then have a complete copy of all your peer review in a single place, and you would have a public profile listing the number of times per calendar year you reviewed for each journal. Unfortunately there is currently one caveat – the journal has to accept your review via email. Academic Karma will customise the review form to match the format required by any journal, which in our experience makes it possible for most journals to accept the review. If the journal is not willing to process a review which contains all the relevant information simply because you sent it via email, then do you really want to review for that journal anyway?

But the point of this blog is not to sell you on why you should use Academic Karma to improve your experience of peer-review (although we think it will!) but how a universal peer review platform like Academic Karma can help fix peer-review.

The first way is that it can change the incentive structure around reviewing. Academic Karma does this using a reviewing currency (which we call karma). When you review you earn karma, and when your manuscript is reviewed you pay karma. By keeping a positive balance you are reviewing enough to support the rate at which you are publishing. As an extra incentive, if you have a positive balance you can see reviews for your paper as soon as they are completed by the reviewer (i.e. without waiting for all of the other reviewers and editors to finish). Karma can then be awarded either on the basis of timeliness and quality of a review. We are currently transferring karma on the basis of reviews completed within 10 days (50 karma) or 20 days (25 karma) to incentivise timeliness, but it would be possible to incorporate quality measures into this transfer.

The second way is that it keeps a publicly visible profile of an academic’s contribution as a reviewer. This provides extra visibility for reviewing activity, which we hope over time will be incorporated into the ways in which academics are evaluated.

The third way is that it decouples co-ordination of peer review from publishing. We sometimes mistakenly believe that we are peer reviewing ‘for’ a particular journal, when in fact we are peer reviewing for our peers, and for the scientific literature and science in general. However, peer review as a process has become something which is tightly tied to a journal. As a result, authors often have to re-start the peer review process with completely new reviewers when they submit a revised manuscript, even when many of the revisions have been done in response to reviewers reviewing ‘for’ the first journal. A universal peer review platform allows authors to keep the same reviewers even when resubmitting to a different journal. More broadly, however, it is worth noting that journals have not had to innovate much in terms of peer review – there has been substantially more innovation in other areas of publishing. By decoupling peer review co-ordination from publishing (while still allowing complete editorial control), it suddenly becomes possible to innovate purely in the area of peer review. This is exactly what Academic Karma is going to have to do if it is going to succeed in building a critical mass of enthusiastic peer-reviewers.

You can now sign in and register at Academic Karma using your ORCID credentials – so there is no need to remember yet another password/username. We hope this platform is a big step towards fixing peer review, but at the end of the day it is just a platform and we really need your support to turn it into a movement for reforming peer review.

Post navigation

2 thoughts on “Fixing peer review”

To me as reviewer in different journals in Mathematics Education Research (Didactics of Mathematics), academickarma is an interesting initiative, BUT: as far as I know, all (!!) the reviewing for the well kown Springer editor (a major editor in my field) is done by using an editorial manager, not by email to the editors. As far as I can see, academickarma has to take into account this way of reviewing in order to make it a success.
Rudolf Straesser
eml.: rudolf.straesser@uni-giessen.de

Yes, you are absolutely right. We are in discussions with various publishers about integrating Academic Karma into their existing systems, but of course the more people that want to use Academic Karma the easier it will be to convince publishers to do that. In our experience (more biomedical sciences) many journals can accomodate a review returned via email to the editorial office, although integrating into their existing systems would make the lives of editorial assistants easier.