Third-party review platforms

by Thomas Arildsen

I recently read this piece (http://chrissampson.me/2013/09/18/one-suggestion-for-a-new-model-of-academic-publishing/) by Chris Sampson. It is an interesting proposal and I thank Chris Sampson for writing it since it encouraged me to think so much about this possibility. I am mostly interested in the review aspect of it. Delegating the handling of peer review to third parties could for example be a good idea in terms of making the whole review process faster and more efficient – for example for the reasons argued by the team behind Libre: http://www.openscholar.org.uk/why-and-how-to-separate-scholarly-evaluation-from-academic-journals/.

I would like to weigh in with my thoughts on third-party peer review.

Funding

One important issue is that of funding such third-party “review companies”. What kind of companies are these and how do they make the money to fund their activities.

I suppose they could be non-profit organisations but could also be for-profit commercial companies. I find the former the most appealing. I guess there is nothing wrong in principle with companies making money from what they do if they do it well and provide a useful service to the scientific community, but somehow I think that such third-party review companies should exist for the sole purpose of providing competent, thorough, trust-worthy review of scientific papers and that a focus on profit might divert their attention from this goal. For example, unreasonably high profit margins are one of the reasons that large publishers such as Elsevier are currently being criticised.

However, some income is needed to run such a platform and where is that going to come from? I see several options:

Authors pay: this is probably not popular. It is my impression that traditional publishers are often criticised for their rather high publishing fees for (gold) open access publishing in their journals. An author fee for review could be seen as comparable to this, but I think its perception depends on how the review company is run. If it is a commercial company, an author fee might well be considered as unpopular as the mentioned OA publishing fees. If, on the other hand, the company is a non-profit able to argue that the fee simply covers the necessary (low) expenses to conduct the review process, it might be perceived as fair. Well, I am no game-theorist, sociologist, psychologist or anything relevant, so this is just my personal opinion.

Review company is paid by research funders: this could be a better option. In the above case of placing the fee on the authors, much of the funding for this will probably come from the researchers’ projects which are often funded by research councils etc. anyway – such as the public The Danish Council for Independent Research or The Danish Council for Strategic Research here in Denmark. It might be an idea for such research funders to simply fund the review companies directly instead, not placing the burden on individual researchers. In my opinion, this would only work fairly if the review company is a non-profit organisation. Otherwise, they would receive the same criticism for sponging off public research funding that many commercial publishers are currently subject to.

Review company is paid by publishers: this could be problematic. If a review company is paid for the review work by the publisher that eventually ends up publishing a particular article, the review company would be under suspicion for not being impartial. Maybe this could be less of a problem if publishers were to form a coalition that would collectively pay for all of the review being carried out. Anyway, this cost incurred on the publishers would just be carried over to publishing fees which would then be the authors’ problem.
By the way, could impartiality also be an issue if authors pay for reviews?

Review company is funded by advertising on its website: I guess one could imagine this scenario although I think it is unlikely. In my opinion, this would be distasteful at best.

Chris Sampson’s post also mentions that journals would bid on papers to publish, based on the reviews. I assume he means that journals would bid as in competing for who could publish at lowest cost? Since he also suggests that all papers should be published open access, the publishers cannot charge the readers for publishing, so I assume this cost must be charged from the authors. Therefore, the competitive element must lie in which journal is able to publish at lowest cost? If the author is to pay the fee, I suppose the author also gets to choose the journal. The author might not necessarily want to select the cheapest journal; maybe a slightly higher-cost bidding journal has a better reputation and the author prefers that one.

The last part leads me to another thought: the reviews should be openly accessible as well (an issue I have discussed before on this blog). This way, there is more information available to readers for judging papers by their actual merits rather than by the journal that publishes them. Maybe this would take some of the popularity contest element out of publishing and help make publishing fees more realistic?

Publishers’ interest

I am afraid a change to the proposed model would be lobbied hard against by exisiting publishers – for at least two reasons:

Review would no longer be part of the publishers’ services and so, they would have less to claim that authors need to pay for, making it more difficult for them to keep up their high profit margins.

Open reviews and thereby better assessment of papers based on content rather than publication venue could work towards more equal status of publishers. This is of course something the highest-ranking journals would want to work against too.

Like this:

Related

9 Comments to “Third-party review platforms”

Good post. I wonder though, couldn’t the review companies also act as publishers? It seems as if we are propping up old giants with preprints and external review. By doing this we are giving credibility to the idea that the name of the journal you publish in still matters, when it shouldn’t.

I guess they can. To me, the important part is really opening up the review process and also that the publications are openly accessible.
My main point here was to open up a discussion of Chris Sampson’s third party proposal. The “review companies” could in my opinion also act as publishers. Once you have said that, all existing publishers are also potential players provided that they are willing to open up their review process. But here I think newcomers to the game (such as The Winnower) may have an advantage: established publishers have their existing way of operating which they will be reluctant to change, while new players can start from a clean slate and get a head start if it turns out to catch on.
A few new publishers exist that have started an open review practice – I know of peerj.com, f1000research.com, and to a slight degree frontiersin.org – but we certainly need more.
BTW, have you at The Winnower considered if you could make use of third-party reviewer platforms for your publishing?

Thank you Thomas for this nice post that gives the opportunity to discuss the important issue of funding “third-party” peer review.

At open scholar, we have a clear take on the subject that I will try to summarize below.

Reviewing papers of colleagues should be part of a scholar’s standard work and as such it should be recognized and valued by the academic community just as it happens with other original scientific contributions. This can only be achieved when: a) the full text of the reviews is accessible to all, b) reviewers always sign their reports, and c) reviews can be cited as normal articles. This model provides necessary and sufficient motivation for reviewers to perform good quality reviews and advance their careers by achieving recognition and initiating new collaborations. Point 1: No money is needed for reviewers.

Reviewers should not be selected by journal editors. Why? First, journal editors care for their journal’s prestige, which is mostly indexed by the impact factor, and thus tend to favor articles that will be highly cited. Second, it is very difficult to find the most appropriate reviewer for each submitted article and that is why editors ask authors’ opinions for suggested reviewers anyway. Third, they do not have strong incentives to offer to reviewers (see point above) and are left with simply asking for a favor (do we really want the evaluation of the world’s scientific output to be a question of secret favors). So who should select reviewers? Our answer is authors themselves. Authors know the area of their article and the corresponding experts better than anyone. In addition, they have a strong motivation to attract good reviewers —including academic “enemies”— that will help them improve their work and make it have more impact in the community. Importantly, absolute transparency will filter possible bias in reviewer selection that the current system allows and even encourages. Point 2: No money is needed for “handling” the peer review process.

Most Universities have open access repositories with sufficient space to host vast amounts of articles and expert personnel (librarians) to curate the metadata. Authors can post their work on these repositories as soon as it comes out of the lab. Even if an author is not affiliated to a University there are disciplinary preprint servers and archives that could play this role. Peer review can be then be performed on these open access preprints using one of the available peer review services. Point 3. No money is needed for hosting papers.

Indeed, if we think about it, by taking advantage of existing free infrastructure and by openly crowdsourcing research evaluation to the scientific community itself we don’t need to spend any money for scientific quality control. The academic community already has the manpower and sufficient motivation to achieve this without intermediaries. Why haven’t we done so yet? First, because researchers are evaluated based on journal publications. Our hope and vision is that by opening up the review process, article evaluation will slowly gravitate towards the reviews record, rather than the publication record; an article that has been reviewed and approved by 20 known experts will be considered better that one all we know about is it has been published in a prestigious journal, meaning it has been reviewed (?) and approved by 2-3 anonymous experts (?).

The second reason is because the model we propose is free to all. There is no money involved. It is not a business opportunity for anyone. So, it has to be developed and supported by volunteers. There are people breaking their heads on how to transform the scientific evaluation problem into a business opportunity, but the truth is it should NOT be a business opportunity, but rather a service to science and society that, by the way, is already paying a lot of money for that.

Open Scholar was founded on those principals. Our founders and members are scholars who earn their lives by doing research and voluntarily join their efforts to develop a solution to what we think is science’s biggest problem at the moment. We are already 100 people from 16 different countries and always open to new members. Our journal-independent, author-guided, open peer review platform, LIBRE , will be released this November and it has been developed with minimum funding mostly for technical support and promotion expenses. Importantly, its maintenance and continued improvement will require a tiny fraction of what a single University library spends each year. In conclusion, let’s not worry about funding for once and simply channel our efforts into promoting the existing free solution for the benefit of scholars, science and the society at large.

Even if all “personnel” works for free (seems doable, we already do that now for the traditional publishers), I suppose there is still going to be some cost for running a website (maintaining software, hosting, user support)? Is that money going to come from donations, occasional grant money, and contributions from possible participating institutions such as universities?

This is exactly the idea. And of course we will also have costs for promotion as we would like to be able to organize conferences and smaller scale events to engage scientists and institutional repositories.

[…] peer review in Open Review of Scientific Literature and discussed the roles of such platforms in Third-party review platforms. I just wanted to mention the above document in Google Docs which seems to have been started by […]

Peer-review is the gold standard of science. But an increasing number of retractions has made academics and journalists alike start questioning the peer-review process. This blog gets underneath the skin of peer-review and takes a look at the issues the process is facing today.