Open post-publication peer review

Synopsis

Beyond open access, which is generally considered desirable, the essential drawbacks of the current system of scientific publishing are all connected to the particular way that peer review is used to evaluate papers. In particular, the current system suffers from a lack of quality and transparency of the peer review process, a lack of availability of evaluative information about papers to the public, and excessive costs incurred by a system, in which private publishers are the administrators of peer review. These problems can all be addressed by open post-publication peer review.

Open: Any scientist can instantly publish a peer review on any published paper. The scientist will submit the review to a public repository. Reviews can include written text, figures, and numerical quality ratings. The repository will link each paper to all its reviews, such that that readers are automatically presented with the evaluative meta-information. In addition, the repository allows anyone to rank papers according to a personal objective function computed on the basis of the public reviews and their numerical quality ratings. Peer review is open in both directions: (1) Any scientist can freely submit a review on any paper. (2) Anyone can freely access any review.

Post-publication: Reviews are submitted after publication, because the paper needs to be publicly accessible in order for any scientist to be able to review it. Post-publication reviews can add evaluative information to papers published in the current system (which have already been secretly reviewed before publication). For example, a highly controversial paper appearing in Science may motivate a number of supportive and critical post-publication reviews. The overall evaluation from these public reviews will affect the attention given to the paper by potential readers. The actual text of the reviews may help readers understand and judge the details of the paper.

Peer review: Like the current system of pre-publication evaluation, the new system relies on peer review. For all of its faults, peer review is the best mechanism available for evaluation of scientific papers.

In an open peer-review system, writing a review is the equivalent of getting up to comment on a talk presented at a conference. Because these reviews do not decide about publication, they are less affected by politics. Because they are communications to the community, their power depends on how compelling their arguments are to the community. This is in contrast to secret peer review, where uncompelling arguments can prevent publication because editors largely rely on reviewers’ judgments.

Signed or anonymous: The open peer reviews can be signed or anonymous. In analyzing the review information to rank papers, signed reviews can be given greater weight if there is evidence that they are more reliable.

Paper selection by arbitrary evaluation functions: The necessary selection of papers for reading can be based on the reviews and their associated numerical judgments. Any reader can define a paper selection function based on content and quality criteria and will automatically be informed about papers best conforming to his or her criteria. The evaluative function could for example, exclude anonymous reviews, exclude certain reviewers, weight evidence for central claims over potential impact of the results etc.

Webportals as entry points to literature: Webportals can define such evaluation functions for subcommunities – for scientists too busy (or too lazy) to define their own. Such webportals would provide a generalized access to the literature that transcends all journals. A webportal can be established cheaply by individuals or larger organizations that share a common set of criteria for paper prioritization.

Yes i support the post-publication peer-review system. i think a study should be conducted on the same. A journal editor can only tell how many comments they receive (as Letter to editor) post-publication and how they deal with them. i personally feel this is nothing wrong.

Thanks for mentioning Naboj on your blog. I fully endorse and support your views and efforts to create a viable post-publication scientific review process. The limitations of Naboj are mostly due to limited resources with which I have had to contend during its development, mostly in terms of time. I would be very glad to get some more developers working with me on it, and hopefully as the word spreads about it I can come in contact with people who would like to collaborate on what I think is a very important project for the future of scientific publishing.

[…] this article was published in PLOS ONE, which lets readers post comments on articles as a form of post-publication peer-review (PPPR). These comments aren’t just like comments on some random website or blog — they […]

EJOLTS (Educational Journal of Living Theories) has an open review process. http://ejolts.net/. A team of reviewers are appointed to work with the author to meet criteria for recommending publication so the review process is a generative and creative one as well as an educational one for both author and reviewers. The conversation is carried out in an open review space and others, as well as the reviewers, can contribute. We are continually trying to improve the review process so I would be interested in any thoughts you have.