¶ 1Leave a comment on paragraph 10This report is open for pre-publication review in advance of its release as part of the Academic Book of the Future initiative. It uses the CommentPress platform. To comment, please click on a paragraph to select it, then click the small “comment” icon to the right of the paragraph. For more detailed guidance on using CommentPress, please see the guidelines on the CommentPress website.

If you have any difficulties in viewing the site, or suggestions for improved accessibility, we’d be very grateful for the feedback. Please either comment on the appropriate section of the report or email d.butchard@stir.ac.uk.

1.Introduction

¶ 2Leave a comment on paragraph 21
Peer review constitutes one of the more paradoxical elements of academic research and dissemination: it is common for academics to complain about unhelpful feedback from their latest review, but the process is simultaneously seen as one of the bedrocks of assuring the quality of research. It does not take long to find anecdotal evidence of the value or pitfalls of peer review in trade publications such as the Times Higher Education or The Chronicle of Higher Education. Asked to share her own ‘horror stories’ in peer review for the Times Higher Education, Susan Bassnett comments that ‘it seems like a fine idea for work submitted to a journal, publisher or funding body to be assessed anonymously by independent experts’, but fears peer review ‘has grown into a monster’ as a result of an increasing volume of work requiring review, with detrimental effects for both authors and reviewers.[1] Such comments suggest an urgent need to reconsider review practices. However, it is rare to see a scholarly examination of the process, and this report sets out to address this by evaluating key aspects of academic discussion of peer review.

¶ 3Leave a comment on paragraph 34The following report considers the diverse range of practices that constitute peer review in both publishing and institutional structures, examining the history of peer review, and evaluating how innovative alternative models aim to resolve pressures on the current system. It does so with a particular focus on peer review in the Arts and Humanities (in connection with the AHRC Academic Book of the Future project[2]), while looking at wider disciplinary and publishing considerations. Peer review is an expansive topic, and our research has revealed a number of fruitful avenues for future evaluation which we have not been able to cover in detail here. These include the selection and crediting of reviewers, the role of peer review in creative practice, the advent of paid review platforms, and the use of metrics as an alternative means of quantifying research value and impact. In particular, our discussion of peer review for publications focuses on the example of scholarly journals, as that is largely where discussion in scholarly and other literature focuses. However, further investigation might consider equivalent issues in the field of monograph publishing.[3]

¶ 4Leave a comment on paragraph 40Given the parameters of our study, alongside an evolving environment for peer review, and our own wish to experiment with peer review modes, this report is offered for post-publication peer review. We encourage readers to submit comments and suggestions additional sources and references, and for new avenues of research. The current document is a beta version of our planned publication in the Academic Book of the Future’s “BOOC”, and we would particularly comments at this stage. Please see the top of this page for brief instructions on how to comment, and the guidelines offered by CommentPress for a more detailed explanation.

1.1 Structure and Methodology

¶ 5Leave a comment on paragraph 50This report provides an evaluation of the rationale, methodology and practicalities of peer review practices in the Arts and Humanities. The introductory section offers a brief overview of the historical development of peer review, establishing the context for perceptions of peer review as a ‘gatekeeper’ of research quality. Section 2, ‘Institutional Support for Peer Review’, examines the prevalence of peer review as a means of judging excellence and research quality within higher education institutions, demonstrated by peer review’s crucial role in decisions by large funding bodies, and recently reiterated by the Stern report’s advice that ‘panels should continue to assess outputs through peer review.’[4] Section 3, ‘Peer Review in Practice’, notes the variety of approaches and classifications, explaining key differences between review models, and illustrating varying levels of support for peer review among contemporary academics. Section 4 focuses on academic journal publishing as an area where traditional peer review models have come under particular pressure in recent years, with detailed examination of scholarly critiques. Finally, Section 5, ‘The Future of Peer Review’ introduces a number of models designed to address current concerns, noting the use of digital technology for streamlining or transforming review practices, and the complex issues involved in altering traditional models.

¶ 6Leave a comment on paragraph 60In addition to differentiating between anonymous, open, or post-publication review, this report identifies a number of key issues in the practical functioning of contemporary peer review models. We outline issues with timescales and workload (‘Efficiency’), cost of implementation (‘Overall Cost’), the level of reward or incentive offered to reviewers (‘Credit’), the extent to which authors respond directly to reviews (‘Respondability’) and the academic hierarchy of the reviewer in comparison with the author (‘Peerness’). The final section introduces examples of platforms which provide varying opportunity for revision of articles (‘Revisionality’), adjust the ease with which articles move from one journal to another (‘Mobility’), introduce automated selection processes (‘Automation’), and use dedicated software (‘Technical Mediation’). These are the categories used in our visualisations, which illustrate the different emphases of contemporary peer review models.

1.2 Why Evaluate Peer Review?

¶ 7Leave a comment on paragraph 70It is difficult to overstate peer review’s central role in higher education institutions and contemporary academic publishing. A recent British Academy report describes peer review as ‘a force making for enhancement,’[5] while the Research Information Network note the significance of peer review practices for a variety of research activities:

Peer review is both a set of mechanisms and a principle at the heart of the system for evaluating and assuring the quality of research before and after it is funded or published. […] Peer review is applied to a number of activities in the research process, particularly in the context of higher education.[6]

¶ 9Leave a comment on paragraph 90While peer review is commonly associated with the quality assurance of journal articles, conference papers, and monographs, the Research Information Network also highlight its use in grants, promotion, and tenure. Peer review is increasingly considered the standard for published scholarship across the globe; a study of the Faculty of Arts at Uppsala University found the number of faculty members publishing in venues using peer review increased ‘from around 20% or less in the years 2006–2010 to almost half the publications,’ suggesting ‘growing awareness regarding the importance of peer review.’[7] With peer review firmly established as a vital stage in career advancement, it is important to understand and evaluate its use in scholarly contexts.

¶ 10Leave a comment on paragraph 101
Despite the centrality of peer review in Higher Education institutions and publications, dedicated research into the development and use of peer review practices remains rare, particularly within the Arts and Humanities.[8] Shatz, in Peer Review: A Critical Enquiry, recognises a lack of investigation, particularly within the Arts and Humanities, declaring his book ‘the firstwide-ranging treatment of the subject by a scholar in the humanities.’[9] Biagioli expresses surprise at the absence of sustained research into scholarly peer review:

Given the remarkable epistemological and symbolic burden placed on peer review, it is surprising to find that so little research has analyzed it either empirically (in its actual daily practices) or philosophically (as one of the conditions of possibility of academic knowledge). While academics discuss it quite frequently, they do not frame it as an intellectual subject. Instead, they either confine it to private conversations or treat it as one of the practical aspects of the profession.[10]

¶ 12Leave a comment on paragraph 120
While this lack of critical reflection on the processes of peer review is acutely true in the Arts and Humanities, there has been a notable body of work examining peer review’s relationship to digital scholarship and the digital humanities as a discipline.[11] On the whole, this work tends to encourage a collaborative approach, summarised by Dougherty who advises ‘don’t venture into digital publishing alone.’[12]A similar emphasis on opening out the peer review process is evident in Gould’s discussion of anonymity in ‘The Scholar as E-Publisher’, with the assertion that ‘adding of more voices to the review discussion may be a source of distress for some, but it will result in a richer, more valuable conversation that will lead to greater success and progress.’[13]

1.3 History of Peer Review

¶ 14Leave a comment on paragraph 140
Scholars tend to discuss the origins of peer review in the context of developments in seventeenth-century publishing, though it has been placed earlier and later. Gould argues that peer review developed out of the ‘climate in Europe by the time of the Gutenberg press,’ with its emphasis on ‘learning, sharing, and collaboration on a scale unheard of just a few centuries earlier.’[14] In ‘The Scholar as E-Publisher’, Gould describes peer review as a well-established means of evaluating the quality and relevance of research:

Despite its frailties, peer review is still valued as a method of identifying research appropriate for publication and blocking work that might be considered inappropriate. This is the model that academia has relied upon, in one form or another, for more than 400 years.[15]

¶ 16Leave a comment on paragraph 162
Although peer review has a longstanding and important role in academia, Fitzpatrick finds that there has been ‘very little investigation of the historical development of peer review,’ particularly in the Arts and Humanities; she notes that ‘nearly all of the texts exploring the history of peer review focus on the natural and social sciences.’[16]Fitzpatrick places the advent of peer review in the eighteenth century, arguing that ‘authors date the advent of what we now refer to as editorial peer review – the assessment of manuscripts by more than one qualified reader, usually not including the editor of the journal or press – to the Royal Society of London’s 1752 creation of a ‘Committee on Papers’ to oversee the review and selection of texts for publication.’[17] Fitzpatrick notes that a number of scholars point to the existence of formalised peer review in journals of the Royal Society of Edinburgh as early as 1731.[18] However, Biagioli argues that the origins of peer review may lie in seventeenth century book publishing and that peer review of journals was implemented at a later stage.[19][20]

¶ 17Leave a comment on paragraph 170
Gould notes that ‘the exact beginnings of anonymous peer review are a bit more vague than those of peer review itself’, but suggests that early attempts at anonymous peer review emerged from the intent to generate more candid evaluations unaffected by personal feelings or institutional biases.[21]Peer review continued to evolve during the twentieth century and into the twenty-first century, and Fitzpatrick summarises its history – and historical purpose – as follows:

On the one hand, peer review has its deep origins in state censorship, as developed throughout the establishment and membership practices of state-supported academies; on the other, peer review was intended to augment the authority of a journal’s editor. Given those two disruptions in our contemporary notions about the purposes of peer review, it may well be less surprising to find that the mode of formalised review that we now value in the academy seems not to have become a universal part of the scientific method, and thus of the scholarly publishing process, until as late as the middle of the twentieth century. […] The history of peer review thus appears to have been both longer and shorter than we may realize.’[22]

¶ 19Leave a comment on paragraph 190
Pontille and Torny discuss peer review in terms of mid-twentieth-century changes to academic structures, situating the concept of anonymous review in June 1955, with the American Sociological Review’s ‘experiment in evaluation’ later popularised by other journals.[23] According to Pontille and Torny, the use of ‘single blind’, ‘double blind’, and ‘open review’ models were introduced in the 1980s and increased in use during the mid-1990s. Kennison finds a direct link between peer review practices and changes to the structure and funding of higher education institutions, arguing that ‘peer review became the norm during the rapid growth in higher education after World War II and the concurrent explosion in Cold War-driven research funding.’[24]

2. Institutional Support for Peer Review

¶ 20Leave a comment on paragraph 200Kennison describes peer review’s traditional role as ‘refereeing in the academy,’ manifested as ‘evaluation, assessment, and judgement.’[25] Certainly, the frequent use of peer review practices as a means of evaluating research quality indicates the trust placed in peer review by academic institutions and funding bodies. Amid increasing competition for jobs and funding in the Arts and Humanities, peer review continues to take a central role at several stages of institutional decision-making.

2.1 Peer Review for Funding Decisions

¶ 21Leave a comment on paragraph 210One of the key uses of peer review within institutional structures is for administrating grants and funding. In principle, peer review works to ensure proposals are judged fairly. The British Academy’s Peer Review: The Challenges for the Humanities and Social Sciences argues that grant proposals should be considered on a ‘case by case basis, taking pains to ensure that award decisions are not made by the application of formulae.’[26] However, the Research Information Network find that ‘competition for funds has increased,’ placing more pressure on traditional peer review practices:

It thus has become more important, and more difficult, to ensure that only the highest-quality research proposals are funded. The RCUK study of peer review in 2006 concluded that a success rate of between 20% and 50% represented “an acceptable balance between the benefits of competition and the cost/effort to support the system”. Since then, success rates have fallen further, in some cases to well under 20%; and such levels bring into question not only the balance between competition and cost, but the ability of the system to discriminate between the best and the very best.[27]

¶ 23Leave a comment on paragraph 230In addition to the difficulties of reduced success rates and risk that reviewers may not be equipped to determine the ‘very best’ proposals, the Research Information Network also recognises high financial costs of peer review in grant funding: ‘in 2006, Research Councils UK (RCUK) estimated that the cost of preparing and reviewing applications for funding from the UK Research Councils was approximately £196m a year. Again, the costs are particularly high where the rejection rate is high.’[28] Nevertheless, peer review remains a crucial factor in funding decision-making.

2.2 The Research Excellence Framework

¶ 24Leave a comment on paragraph 240Peer review plays a key role in professional advancement and evaluation. In the UK’s Research Excellence Framework (REF), review panels evaluate the quality and impact of colleagues’ work. Discussing the REF’s predecessor, the Research Assessment Exercise (RAE), Bence and Oppenheim detail how the transition from ‘quantitative’ to ‘qualitative’ assessment ‘meant a greater reliance on the system of peer review.’[29] The recent Stern report on the RAE confirms the ‘centrality and importance of peer review’, which remains a ‘gold standard’ for evaluation of research quality.[30] However, the Stern report also acknowledges that peer review is ‘not a perfect measure’ and notes ‘extra scope for the use of metrics’:

Applying the ‘gold standard’ of peer review does depend on panels having a very broad range of expertise and sufficient time to analyse each output in detail. At best, peer review is not a perfect ‘measure’, and with the time pressures on some REF panels, maintaining consistency and quality of review is very challenging. There is therefore a trade-off between considering a larger volume of outputs for each unit to provide more accurate benchmarking information, and the accuracy of an exercise based solely on peer review.[31]

¶ 26Leave a comment on paragraph 261The report concludes that traditional peer review should be supplemented by ‘the appropriate use of bibliometric data in helping panels in their peer review assessment,’ adding that ‘all panels should be provided with the comparable data required to inform their judgements.’[32]

2.3 Metrics and Institutional Review

¶ 27Leave a comment on paragraph 270As the Stern report’s recommendations indicate, metrics present an alternative way of evaluating research quality and impact. Pontille and Torny examine the role of bibliometric indications and their relation to peer review at institutional level, evaluating three different national organisations that produce journal ratings ‘as an alternative assessment tool’ in many disciplines including the Arts and Humanities. They then discuss the tensions that the process heralds through its dual function as both ‘political instrument’ and ‘scientific apparatus.’[33] Reale et al focus on Italy’s equivalent of the UK’s RAE to discuss the strengths and weakness of the grant funding and research assessment processes, including the role of bibliometric indicators, links between peer judgements and impact factor, and outcome credibility. The authors conclude that ‘bibliometric indicators should be used carefully at the individual level,’ and must be presented by specialists with the necessary expertise.[34] However, they find that ‘the combination of the peer review and bibliometric indicators can improve the overall assessment’ when used appropriately, with awareness of the limitations of bibliometric data.[35]

2.4 Peer Review and Professional Advancement

¶ 28Leave a comment on paragraph 281Peer review plays an important part in academic career progression within institutions. Fitzpatrick discusses peer review’s role in relation to promotion and tenure in the United States, where research outputs of assistant professors eligible for tenure are evaluated by colleagues and external examiners. Fitzpatrick warns that ‘faculty practices’ have ‘changed faster than have the ways that work gets evaluated,’ but concludes that tenure review offers an opportunity for ‘opening discussions about scholarly values, promoting innovations in both research and teaching, and supporting the new ways that scholars are connecting not just with one another but with the broader public as well.’[36] Dougherty (2013) explains how, when experimenting with new models of peer review for Writing History, he needed to create ‘a scholarly product that would be recognised by hiring and tenure committees.’[37] Writing with an Arts & Humanities focus, Wheeler notes that peer review plays an important role for career advancement in US departments, finding that ‘it is not uncommon for department chairs or deans to solicit copies of peer review when authors come up for tenure or promotion.’[38] As these examples indicate, peer review functions as a means of assuring and judging research quality within institutions, both for funding and career decisions.

2.5 Innovation and Risk

¶ 29Leave a comment on paragraph 290Given the importance of peer review for funding decisions, it is worth noting the Research Information Network’s concern that a need for consensus among reviewers may offer ‘particular risks for intellectually-innovative proposals, where the potential of the approach may be speculative, and where there may be marked differences between the evaluations of different reviewers.’[39] Risam likewise warns that consensus-based peer review models may discourage innovative research, finding that even in the digital sphere, projects on ‘canonical’ topics tend to thrive.[40] However, steps can be taken to safeguard funding for more speculative topics. The British Academy’s Peer Review: The Challenges for the Humanities and Social Sciences ‘commends’ the Arts and Humanities Research Council (AHRC) for supporting innovative research, by ‘seeking to ensure that the innovative proposals are dealt with through funds that explicitly recognise a degree of risk.’[41] The British Academy advises funding bodies to ‘ensure that considerations of applicability and relevance do not compromise judgments of quality,’ advising bodies to ‘set aside funds for risky, speculative projects,’ ensure ‘a healthy balance between strategic funding and responsive mode projects,’ and ‘encourage endowments within universities to support small grants for innovative, high risk research.’[42] More widely, the report recommends that ‘the process of selecting topics for strategic initiatives is also subject to peer review’ for humanities and social science research funders.[43]

2.6 Reliability and Bias

¶ 30Leave a comment on paragraph 301As indicated by the British Academy’s emphasis on ‘judgments of quality’, peer review’s role within institutions is often perceived as that of ‘a gatekeeper, the final arbiter of what is valued in academia.’[44] However, this ‘gatekeeping’ is not infallible. In a report on open access and peer review, Eve notes that ‘the gatekeeper model […] works on a series of unspoken ideological assumptions that are never wholly objective and apolitical, but rather based on a series of exclusions and marginalisations.’[45] In the specific context of institutional review, Marsh et al argue that peer review pertaining to grant funding in the Australian Research Council ‘lacked reliability,’[46] while Lamont highlights the issue of hierarchies and inequality of access, observing that ‘Peers monitor the flow of people and ideas through the various gates of the academic community. But because academia is not democratic, some peers are given more of a voice than others and serve as gatekeepers more than others.’[47] Bowman and Ulm raise the possibility of gender bias within institutions, noting ‘considerable gender disparity in a number of areas’ of the AHRC-funded ‘Discovery Projects,’ and suggest that ‘institutional factors’ may contribute to gender disparities in archaeology.[48] Discussing peer review and the RAE (now REF), Bence and Oppenheim argue that ‘the concept of quality of published output is very subjective.’[49] The authors detail historical accusations of bias in the RAE, quoting Doyle et al’s finding that an RAE panel had ‘fallen prey to just about all the different variants of the home team bias,’ favouring English universities, large and/or well-established institutions, and the home institutions of panellists.[50] The authors conclude that ‘it must be debatable just how far the subjective assessments in the RAE can provide a reliable “measure” of quality.’[51] Such critiques have since been addressed by changes to review structures, including the introduction of ‘external’ reviewers, but nevertheless indicate the difficulty of determining research ‘quality’ without unintentional recourse to subjective judgement and/or institutional bias.

3. Peer Review in Practice

¶ 31Leave a comment on paragraph 311The sheer range of practices that encompass the term ‘peer review’ provide a challenge for researchers seeking to analyse and evaluate its use. The Committee on Publication Ethics (COPE) include a brief definition of peer review in their Guidelines on Good Publication Practice, describing peer reviewers as ‘external experts chosen by editors to provide written opinions, with the aim of improving the study.’[52] However, Hammarfelt and de Rijcke argue that ‘to give a definition of peer review is impossible,’ since ‘a peer-review process could stretch from a highly structured double-blind procedure to a quick look by an editor […] there is no formal definition or control.’[53] The British Academy praise the ‘decentralised diversity’ of peer review processes, declare the variety of models in the Humanities and Social Sciences ‘a strength not a weakness’, and state that ‘there is no one model that all should follow.’[54] However, they do call for adherence to general ‘principles’ of ‘timeliness, transparency and verifiability.’[55] This viewpoint is echoed by the Research Information Network, with the advice that ‘there is considerable variety of practice [in peer review process], and it is a merit of the system that there is no single model of good practice.’[56] Here we outline practical features of peer review, before discussing positive and negative perceptions of traditional peer review models revealed by recent surveys of academics working in the Arts and Humanities.

3.1 Anonymous, Open and Post-publication Peer Review

¶ 32Leave a comment on paragraph 320Author and reviewer identification remain a key issue in discussions of peer review, and peer review practices are often classified according to levels of anonymity for author and/or reviewer. This prompts differentiation between traditional ‘blind’ review, where one party does not know the identity of the other, and ‘open’ models, where the author, and potentially readers, may know the identity of the reviewer(s). Taylor and Francis’s report, Peer Review in 2015: A Global View, identifies several models of contemporary peer review in scholarly publication, detailed in Table 1.

Table 1: Dominant models of peer review in scholarly publication

¶ 33Leave a comment on paragraph 330Source: adapted from Taylor and Francis’s Peer review in 2015: A Global View[57]—Double-blind and single-blind review models are often grouped together under the banner of ‘traditional’ or ‘anonymous’ peer review. In the double-blind model, both reviewer and author are anonymous, whereas single-blind reveals the author’s identity to the reviewer. In both cases, reviewers read submitted work prior to publication and return a report recommending whether it should be published (Figure 1). In contrast with ‘traditional’ blind review, ‘open’ peer review offers a more transparent model, revealing the identities of both author and reviewer. For ‘post-publication peer review’, reviews are invited subsequent to publication. This is often – though not always – preceded by an initial ‘traditional’ review phase, before opening work up to further commentary subsequent to publication (Figure 2).

Figure 1: The process of traditional peer review, regardless of knowledge of author and reviewer

Figure 2: The process of post-publication peer review

3.2 Popularity of Peer Review Models

¶ 36Leave a comment on paragraph 360Traditional ‘blind’ review models continue to dominate contemporary academic practice. Scholarly discussions often begin by acknowledging the value of conventional blind peer review as a means of establishing and encouraging rigorous research; Baverstock finds peer review has become ‘sanctified through long use,’[58] while Bourke-Waite recognises that ‘academics overwhelmingly think peer review is important, and surveys have consistently shown that they prefer papers to have been rigorously reviewed.’[59] In principle at least, peer review is associated with quality and rigour. A recent Taylor and Francis (T&F) survey found that ‘most respondents agreed that scholarly publishing is greatly helped by peer review of published articles (78% scoring this 7 or higher out of 10)’ and ‘68% of researchers believed they could have confidence in the academic rigour of published articles because of the peer review process.’[60] Such claims are further substantiated by findings in a survey by the Publishing Research Consortium (PRC) that ‘researchers continue to value the beneﬁts of peer review, with 74% agreeing that it improves the quality of the published paper.’[61] However, it is worth noting that in 2016 the PRC noted a 10% decrease in overall support for peer review since their 2008 report: ‘belief that peer review helps scholarly communication has fallen, though still high (85% → 75% A/SA).’[62] Addressing dissatisfaction with aspects of peer review, the 2015 T&F report found that respondents favoured ‘tweaks in the current system rather than a radically new way of accessing the quality of research outputs.’[63]

¶ 37Leave a comment on paragraph 370It is difficult to quantify levels of support for different peer review models within the Arts and Humanities. This is partly due to the limitations of surveys, which are often skewed toward science perspectives; only 3.7% of respondents to the PRC survey published in 2016 were from Arts and Humanities subjects.[64] There are also queries over methodology. For example, the 2016 PRC report reevaluates the earlier T&F survey, arguing that ‘results should be treated with caution’ since negative responses to single-blind review ‘may have been inﬂuenced by previously answering a series of questions that explicitly referenced problems […] associated with single blind review.’[65]

3.2.1 Traditional Anonymous Review

¶ 38Leave a comment on paragraph 382As noted by Bourke-Waite above, surveys tend to indicate preferences for pre-publication blind peer review. The T&F report finds ‘a strong preference for double blind review among all respondents, with a rating of 8 or above out of 10.’[66] This chimes with scholarly opinion; in ‘Rethinking Peer Review in the Age of Digital Humanities’, Risam describes double blind as the ‘gold standard for validating scholarly work.’[67] However, although the 2016 PRC report found ‘a clear preference for conventional pre-publication review (single or double blind)’, they detected ‘no clear-cut preference shown for single compared to double blinded review.’[68] This differs from the 2008 report’s evaluation of a preference for double-blind review as ‘a response to the potential for bias in single-blind review.’[69]

3.2.2 Open Review

¶ 39Leave a comment on paragraph 391Open review remains less popular than traditional models among surveyed academics, though it finds increasing favour in recent surveys. The 2008 PRC report found that ‘open peer review was an active discouragement for many reviewers, with 49% saying that disclosing their name to the author would make them less likely to review,’[70] while feedback on a Nature trial issue in 2006 indicated unpopularity amongst authors and researchers’ reluctance to offer comments.[71] Such attitudes may be changing, however. In the 2015 T&F report, open review divided opinion, with ‘approximately as many rating their level of comfort 6 or above as rating it 5 or below.’[72] The T&F report also noted disciplinary distinctions, where ‘notably, HSS [Humanities and Social Sciences] editors are less comfortable with open review.’[73] The 2016 PRC report broadly agrees, finding that ‘support for open review appears to have grown between 2007/2009 and 2015’, with ‘about 50–70% of researchers supportive of it.’[74] It is worth noting that the PRC report differentiates between ‘open’ and ‘open and published’, finding that support ‘falls to 35-55% if it includes publishing signed reviews alongside the paper.’[75] This suggests that reviewers have a preference for accountability to authors, but not to the wider scholarly community.

3.2.3 Post-Publication Review

¶ 40Leave a comment on paragraph 402The T&F report interprets ‘a mean rating below 5’ for post-publication review as an indication that ‘the majority of HSS researchers are uncomfortable with post-publication review.’[76] However, the report adds that post-publication review has value as an additional check to traditional review, ‘Some researchers feel that the ideal model is pre-publication double blind review supplemented by post-publication open review or commentary.’[77]The PRC’s 2016 report allows for this distinction, finding 50% support for a review model with work ‘assessed & rated post-publication, & reviewed prior to publication’, which dips to 17% support for work ‘NOT reviewed prior to publication.’[78] Notable examples of scholars advocating, or utilising, post-publication review in the humanities include Kathleen Fitzpatrick’s Planned Obsolescence, the History Working Papers Project, Postmedieval, Open Library of the Humanities,[79]MediaCommons, and Writing History in the Digital Age.[80] A 2012 issue of Shakespeare Quarterly entitled ‘Shakespeare and the New Media’ experimented with post-publication review; the issue was open for commenting for six weeks from June 18 to July 27 2012, and author feedback on the experiment was very positive.[81]

4. Criticisms of Peer Review: Journal Publication

¶ 41Leave a comment on paragraph 410A growing number of scholars argue that widespread support for peer review by academic institutions and individuals only occurs because no alternative model has been developed and adopted on a large scale. Macri and Khan write that ‘it has become popular to draw the parallel between Churchill’s opinion regarding democracy and the state of peer review itself (seriously, almost fatally flawed, but better than any alternative),’[82] while Shatz observes that ‘the most familiar cliché about peer review (akin to what Churchill said about democracy) is that “it is the worst form of evaluation – except for all the others.”’[83]In this section, we outline key criticisms of conventional peer review models, focusing on an area in which the usefulness of peer review has been discussed most rigorously: its use for evaluating work submitted for publication to academic journals.

¶ 42Leave a comment on paragraph 420The British Academy concede that ‘peer review has its critics who allege that it is costly, time-consuming, and biased against innovation,’[84] while the Research Information Network’s Peer Review: A Guide for Researchers outlines more detailed grounds for criticism: that peer review ‘brings delay; that it is not always effective in detecting misconduct and malpractice; that the selection of reviewers may introduce bias into the system; that the judgments made are subjective and inconsistent; that it tends toward conservatism and stifles innovation; that it disadvantages interdisciplinary research; and that it imposes increasing and unsupportable burdens on reviewers.’[85]The PRC note that opposition to peer review tends to centre upon ‘a lack of evidence that peer review actually works,’[86]while Sabaj Meruane et al. state that researchers ‘debate whether the process is valid or invalid, biased or unbiased, reliable or unreliable, predictably valid or invalid.’[87] There has also been discussion of peer review’s core purpose. In ‘Peer Review: The Current Landscape and Future Trends’, Jubb states that ‘lack of consensus as to its core purpose has led to a variety of views and concerns about its practice and whether it fulfills that purpose.’[88]

4.1 The ‘Black Box’ of Anonymous Reviewing

¶ 43Leave a comment on paragraph 431Sabaj Meruane et al invoke the metaphor of the ‘black box’ to discuss the inscrutable workings of peer review, concluding that ‘despite the massive amount of literature, this box is far from being open.’[89] Lamont agrees that ‘peer review is secretive. Only those present in the deliberative chambers know exactly what happens there.’[90] As shown in Table 5, anonymity is an essential feature of traditional peer review processes. However, commentators express concerns about this inscrutable process, particularly in single blind review where the reviewer knows the author’s identity but remains anonymous to the author. In ‘The Blind Shall See! The Question of Anonymity in Journal Peer Review’, Pontille and Torny ask ‘does the anonymity of the reviewers and/or authors guarantee or prevent an objective assessment?’[91] Warhol-Down argues that anonymity automatically signifies a hierarchical relationship and that this irresponsibly wields absolute power in contemporary scholarship. She writes that ‘what is at stake with academic anonymity is power: he or she who remains unnamed always has the upper hand,’[92] warning that ‘power in the hands of the anonymous referee is more absolute than power in the hands of evaluators who must sign their decisions.’[93]

¶ 44Leave a comment on paragraph 443Although anonymity in peer review frequently comes under fire, there are also significant concerns over alternative open models. This scepticism is usually focused on the issue of quality control; Fitzpatrick acknowledges that ‘open, post-publication review too often casts doubt on a project’s quality.’[94] However, traditional models are by no means a guarantee of fair judgement and high-quality feedback. The popular hashtag used on social media – #sixwordpeerreview – has been used to mock short, unhelpful feedback within the scholarly community.[95] Fitzpatrick cites evidence to suggest that ‘traditional closed, anonymous peer review processes and quality control aren’t quite as related as we often assume.’[96] Eve believes ‘luck plays an enormous part in the fairness of anonymous peer review and it is a system that has us at its mercy,’[97] while Odom states bleakly that ‘everyone knows anonymity breeds contempt.’[98] In ‘Is Peer Review Still the Content Industry’s Upper House?’, Baverstock states that peer review has ‘quasi-sacred status’, but argues that peer review is not necessarily the stamp of a ‘uniformly reliable gold standard.’[99]

¶ 45Leave a comment on paragraph 451One of the prevalent criticisms of peer review is the perception that it may not prevent biased judgements. This is acknowledged by many, including Shatz and Kennison, the latter concluding that ‘impartial assessment’ – is – or should be – a ‘goal of peer review.’[100] Murphy asks are ‘all authors unethical?’, noting author-nominated fake reviewers as a problem and explains how his journal, Lipids, has developed protocol to combat this.[101]For Tallaksen, traditional peer review models do not do enough to eliminate malpractice.[102] This is underpinned by recent high-profile cases: in August 2015, the publisher Springer retracted 64 articles from 10 different subscription journals ‘after editorial checks spotted fake email addresses, and subsequent internal investigations uncovered fabricated peer review reports’ (Springer website).[103] Although the British Academy report acknowledges that ‘peer review is sometimes criticised because examples of scholarly malpractice slip through the net,’ it does state that ‘it is not realistic to expect peer reviewers or editors to identify all cases of fraud or deception. Nor is it their responsibility.’[104] The Committee on Publication Ethics (COPE) guidelines for journal editors address the ethics of peer review by suggesting that ‘best practice’ should include ‘supporting initiatives designed to reduce research and publication misconduct’ and ‘encouraging reviewers to comment on the originality of submissions and to be alert to redundant publication and plagiarism.’[105]

4.2 Efficiency and Quality Control

¶ 46Leave a comment on paragraph 461The British Academy acknowledges a lack of training in peer review practice and recommends this should be funded and available at postgraduate level, since ‘peer review relies upon professional norms,’ and ‘it is important that these are reinforced by appropriate training.’[106] However, several critics attribute declining review quality to a surge in submissions to journals, rather than lack of professional training. The PRC indicates problems with the labour-intensive nature of traditional peer review processes, citing ‘evidence that peer review is too slow and that reviewers are overloaded,’[107] while some editors attribute the increase in the volume of submissions across all disciplines to the ‘drive to publishing in high Impact Factor journals.’[108] Bourke-Waite regards the traditional system as over-burdened, since ‘for the past two centuries the volume of peer-reviewed articles published globally has increased by 3.5% per year, with currently around 2.5 million articles, published in an estimated 28,000 peer-reviewed English-language journals, in 2014.’[109]

¶ 47Leave a comment on paragraph 470Bourke-Waite points out that the total pool of potential reviewers has not increased, and concludes that ‘at the rate research is growing, the current system is unsustainable in the long term.’[110] Bauerlein agrees, blaming a ‘productivity era’ in scholarly publishing for placing undue pressure on reviewers and editors.[111] Like Bourke-Waite, Bauerlein concludes that ‘the decline of peer review in the humanities will not end unless the amount of material published each year contracts sharply.’[112] The increasing burden of peer review, in addition to the increase in profitability of STM journals’ subscription models in particular, have led academics to protest at the free labour involved, and even to boycott some journals’ publishers. For example, a petition against Elsevier journals commenced in 2012 has gathered support from academics concerned by ‘exorbitantly high prices for subscriptions to individual journals’ and journals’ perceived efforts to ‘restrict the free exchange of information.’[113]

¶ 48Leave a comment on paragraph 480Critics have also noted the issue of timescales for publication. The British Academy include ‘timeliness’ as one of their 3 key criteria for ‘good practice’ in peer review.[114] However, many commentators discuss the often lengthy durations from submission to publication. Baverstock writes that traditional peer review ‘is often criticized for being unreliable and slow,’[115] while Tallaksen finds slow processing of articles highly problematic. Bjork and Solomon, in ‘The Publishing Delay in Peer Reviewed Journals’, find review times vary enormously between publications and disciplines and argue that the traditional model is inherently inefficient because ‘delays slow the dissemination of scholarship and can provide a significant burden on the academic careers of authors.’[116] T&F found that sixty days was considered a reasonable turnaround time for initial feedback, but many respondents had waited much longer, with researchers waiting between one and six months for an article to undergo traditional peer review.[117]

¶ 49Leave a comment on paragraph 491The timescales and workload of peer review have been linked to the cost of traditional peer review models. The Research Information Network suggest that ‘costs of the time spent by editors and reviewers of scholarly journal articles globally amount by now to £2-3bn, or over a quarter of the total cost of publishing and distributing journal articles; and that the UK contributes about 9% of those costs.’[118] Smart cites figures from the Publishing and the Ecology of European Research (PEER) project, where the average cost is $250 per submitted manuscript.[119] The British Academy acknowledge that ‘there is no practical way, particularly in the humanities and social sciences, in which the full economic cost (FEC) of peer review activity for the purpose of journal publication can be recovered’ and they state that ‘this reality needs to be recognised within the funding of research and universities.’[120] Tienari argues that contemporary academia functions like financial markets and has become fundamentally unsustainable, partly because practices such as traditional peer review squander ‘scarce intellectual and physical resources.’[121]

4.3 Peerness and Credit

¶ 50Leave a comment on paragraph 500The question of who constitutes a peer, and whether they are qualified to evaluate work, is a key factor in traditional models; usually it is assumed that reviewers will have equivalent, or greater, expertise than the authors whose work they evaluate. For example, postgraduate journals are set up to encourage research students to submit and review research undertaken by their peers. However, Biagioli writes that ‘While it would have been easy to define ‘peer’ around 1700 in terms of membership in a given academy, now the meaning of peer is highly situational, perhaps hopelessly so.’[122] Tsang argues that authors often have more specialist knowledge that the reviewer they are assigned, particularly in niche or innovative subjects,[123] while Fitzpatrick advocates new opportunities to open up scholarship and peer review practices in the digital sphere; she suggests a peer review process undertaken by reviewers both in and outwith academia. The editor of Postmedieval agrees, with advice to expand the definition of ‘peer’ to include ‘not just the specialized members of one’s narrow sub-fields but also members of the more broad intellectual community.’[124]

¶ 51Leave a comment on paragraph 514Wheeler outlines fundamental flaws in the traditional system, including complaints from scholars who regard peer review as ‘unrewarded activity’ both in financial and figurative terms.[125] She questions ‘what will become best practices, especially when the rewards for already over-worked and unrecognized editors are so low and the time cost for open [post-publication] review so high?’[126] In ‘Why do peer reviewers decline to review? A survey’, Tite and Scroter find that lack of formal recognition for peer reviewers is an influential factor when scholars decline to undertake peer review and recommend that ‘reviewing should be formally recognised by academic institutions’ and ‘journals should formally, and perhaps publicly, acknowledge the contribution of their reviewers.’[127] Some platforms, such as the Open Access journal Collabra, are taking active steps to address reward for reviewers (see Section 5.4.4: Payment).

4.4 Respondability: Engagement between Reviewers and Authors

¶ 52Leave a comment on paragraph 521Kennison suggests that peer review processes are traditionally based on exclusion and believes that new models must move towards inclusion, advocating a shift in the philosophy that underpins peer review and its attendant practices, to replace peer judgment with peer engagement as a primary value.[128] Smart writes that ‘all publishers have a strong role to play as facilitators of improved [peer review] systems,’[129] while Fitzpatrick asserts that peer review ‘demands to be transformed from a system of gatekeeping to a mode of manifesting the responses to and discussion of a multiplicity of ideas in circulation.’[130] Ashforth argues that a combination of traditional review processes and a ‘publish or perish’ culture in academia encourages scholarship to be risk-adverse and therefore bland, becoming the equivalent of ‘vanilla pudding’ with no real substance, flavour, or nutritional value.[131] He views this as a threat to future innovation, if ‘bold’ ideas are eclipsed and ‘study that offers little new thinking tends to have a smoother ride and better prospects for publication.’[132]

¶ 53Leave a comment on paragraph 531Critics have also questioned whether peer review is sufficiently relevant to scholarship today.[133] Fitzpatrick warns that ‘peer review threatens to become the bottleneck in which the entire issue of electronic scholarly publishing gets wedged’ and scholarship ‘runs the risk of becoming increasingly irrelevant to contemporary culture’s dominant ways of knowing.’[134] She argues that there is a need for scholars, reviewers, and administrations to ‘shift their perspectives and expectations as digital texts and objects become increasingly central to scholarly communication’ and claims that in the era of the digital network the traditional model of ‘externalising our judgement by deferring our authority to others and appealing to objective measures of value, in the long run, can only devalue all our work.’[135] Such criticisms have led a number of scholars to call for reform to current peer review practices. The following section considers alternative and proposed models for reforming peer review, emphasising key areas of concern and evaluating how review practices may evolve in the future.

5. The Future of Peer Review

¶ 54Leave a comment on paragraph 540Peer review is widely used in publication processes, grant-making decisions, and aspects of academic career enhancement, with challenges identified in all. But what of the future of peer review, particularly in relation to publication? At the time of writing, single blind and double blind peer review models continue to dominate publishing and institutional evaluation in the humanities. However, as this report has shown, there is ongoing scholarly discussion regarding the need to streamline or transform current peer review practices. Wheeler maintains that ‘the time has now come for editors and intellectuals to consider best practices and procedures for peer review. If we don’t assume agency, some agency will assume it for us.’[136] Sabaj Meruane et al. write that the study of peer review ‘requires not only further, but more comprehensive, approaches’[137] and Mole, in ‘The Future of Peer Review in the Humanities Is Wide Open’, finds that ‘nobody likes the current system, and most everybody agrees it should move online. But from there, opinions diverge.’[138]A number of publications have experimented with innovative review practices, and this concluding section considers the future of peer review by detailing the approaches of existing and proposed alternative models.

¶ 55Leave a comment on paragraph 550Considering how certain disciplines and publications have already modified their approach to reviewing offers an important means of determining possible alternatives for peer review in the Arts and Humanities. Table 3: Existing Alternative Peer Review Models outlines methods which depart from conventional single blind or double blind review. Drawn from STEM, law and social sciences, these examples indicate possible routes for Arts and Humanities peer review practices in the future. Although these examples vary in scale, they indicate the potential for success of alternative approaches that depart from traditional review models. Drawing on these and other models, a number of scholars have also suggested ways of modifying and improving peer review practices in the Arts and Humanities. These recommendations tend to build on traditional peer review structures, while maintaining rigorous evaluation and commentary by experts in the field as a core precept of peer review. The critics who propose these innovations suggest they would improve the process through greater transparency, streamlining the editorial process, or offering further incentives for reviewers. Table 4: Proposed Peer Review Models details these models.

¶ 56Leave a comment on paragraph 563The tables are accompanied by visualisations to demonstrate how various models differ from traditional blind processes of peer review (see Section 3.1: Anonymous, Open and Post-Publication Peer Review). These are mapped according to the terms defined in our methodology. ‘Efficiency’ indicates efforts to reduce timescale and workload, ‘Overall Cost’ refers to the cost of implementation for journals, while ‘Credit’ maps the level of reward or incentive offered to reviewers. ‘Peerness’ indicates the importance of academic hierarchy in reviewer selection, ‘Respondability’ shows the extent to which journals or platforms facilitate authors’ response to reviews, and ‘Revisionality’ the level of opportunity for revision of articles. ‘Mobility’ is the ease with which articles move from one journal to another, ‘Automation’ the use of automated selection, and ‘Technical Mediation’ indicates dedicated software used for the review process. The visualisations treat traditional double blind peer review as a baseline, indicated as a constant circle at the centre of the visualisation. Against this baseline, each visualisation plots how new models increase or decrease emphasis on particular features of peer review. For example, in Figure 3: Traditional Models of Peer Review, ‘open’ peer review’s central tenet of inviting comments from the widest possible audience is less concerned with traditional hierarchies, and tends to increase opportunities for revisions. Figures 4 and 5 detail alternative and proposed models respectively.Please note the visualisations are interactive – click on the images below to view the visualisation in full in a pop-up window. This allows you to highlight different areas of the visualisation and compare different methods of peer review directly.

Figure 3: Traditional models of peer review

Table 3: Existing Alternative Peer Review Models

Figure 4: Existing Alternative Peer Review Models

Table 4: Proposed Alternative Peer Review Models

Figure 5: Proposed Alternative Peer Review Models

5.1 Transparency

¶ 57Leave a comment on paragraph 570Several established alternatives to anonymous peer review address concerns about anonymity, bias and credit by making the review process more immediately transparent. Existing variants such as ‘Transparent Peer Review’, ‘Peer Review by Endorsement’ and ‘Collaborative Peer Review’ (see Table 3) and the proposed ‘Special Review Process’ and ‘Peer Reviewer as Collaborator’ (see Table 4) all advocate what we call ‘respondability’: direct communication between authors and reviewers at varying stages of the review process, giving authors the opportunity to respond to comments and criticisms. These practices are often facilitated by new media platforms, and aim to address bias and encourage helpful commentary by transforming the review process into a more responsive and directly constructive experience for authors.

¶ 58Leave a comment on paragraph 580New models designed to enhance the transparency of peer review processes tend to facilitate communication between authors, editors and reviewers. In the ‘Transparent Peer Review’ (TPR) model introduced at The EMBO Journal in 2009, referee reports, author responses, and editorial decision letters are published online as a largely unedited ‘transaction report’ alongside the paper. Confidential comments to editors are discouraged and authors can opt out at any stage.[139]Collaborative Peer Review similarly encourages reviewers to respond to one another in an interactive phase. In the case of Frontiers journals, reviewers respond to one another directly, and the names of reviewers and the handling Editor are acknowledged in published articles, although details of correspondence are not published.[140]

¶ 59Leave a comment on paragraph 590
Whereas the above practices deploy ‘open’ review models in advance of publication, others focus on post-publication review. ScienceOpen uses a transparent post-publication peer review process as standard, and offers a further option called Peer-Review by Endorsement (PRE), where authors may work with colleagues to improve their paper prior to submission and then publish it with their written endorsement. Reviewers must sign and publish the following statement: ‘I have read this article, given feedback to the authors and now feel that it is of appropriate quality to be included in the scientific literature and be part of the open scientific discourse.’[141]

Table 5: Transparency in Alternative Models

¶ 61Leave a comment on paragraph 610A number of commentators advocate increased dialogue between authors and decision-makers. Tsang proposes a ‘Special Review Process’ which ‘gives the author the privilege of responding to referees’ comments before the editorial decision is made,’ arguing this would help editors ‘make a more informed decision.’[142]Kairos, a journal in computers and writing, demonstrates how digital platforms can be used to facilitate a more flexible, multi-tier approach. The review process includes ‘several phases of open communication amongst editorial board members and between editors and authors’ via multi-media forums.[143] Other publications have found the flexibility allowed by multimedia review platforms similarly promising. In ‘Engaging Digital Scholarship: Thoughts on Evaluating Multimedia Scholarship’, the editors of Vectors, Anderson and McPherson, write that ‘At Vectors, we found that multiple models of peer review were required. Projects produced outside our collaborative production process (about a third of the pieces we published) underwent traditional peer review; in-house projects were reviewed at the proposal stage, during the development process, and near completion. An adequate review process often necessitated one aspect of review (or one reviewer) focused on content and another focused on design.’[144]However, models such as ‘collaborative’ review are reliant on individuals’ willingness to engage in discussion, and the prospect of a back-and-forth exchange between reviewers, authors and editors may also lead to a more protracted publication process. As Figures 4 and 5 (above) demonstrate, models with increased ‘respondability’ often also have longer timescales for publication.

Figure 6: Special Peer Review

5.2 Efficiency

¶ 63Leave a comment on paragraph 630In addition to transparency and quality control, excessive timescales and unfeasible workload for reviewers have become a key concern in traditional peer review models. The increasingly high volume of articles submitted for review places considerable pressure on journals and reviewers (Bauerlein), and may threaten the sustainability of traditional peer review models in the Arts and Humanities. This has prompted some to seek to increase efficiency, streamline the review process and decrease workload for reviewers.

¶ 64Leave a comment on paragraph 641Law journals in the United States reduce the workload of established academics by displacing the task of reviewing onto editorial teams of top-performing Law postgraduate students (Figure 7: Law Review). This system differs from the majority of other disciplines, which tend to emphasise scholarly experience as a requirement for peer reviewers. The use of postgraduate students has the advantage of reducing demands on academics in full-time employment, and provides an opportunity for individuals at an early stage in their academic career to develop significant professional expertise. Sciullo finds ‘several advantages’ to the Law review model, whose frequent publication and faster turnaround ‘does offer a corrective to peer review’s lack of timeliness.’[145] For Sciullo, ‘the dangers of student editors seem small in comparison to the benefit of more rapid diffusion of knowledge.’[146] Sciullo adds that ‘law reviews train students in ideas important to professional and academic success’ and ‘often publish innovative scholarship precisely because students are willing to challenge orthodoxies in their role as editors.’[147] However, he concedes that ‘Law reviews are, except for maybe the top ten or so journals, viewed as less rigorous than peer reviewed publications.’[148]

Figure 7: Law Review

¶ 66Leave a comment on paragraph 660Despite Sciullo’s enthusiasm, a shift to the Law model does not resolve concerns about the unpaid labour of reviewing, and has prompted other scholars to question whether postgraduates have sufficient time and expertise for this responsibility. In contrast to Sciullo, Mendenhall criticises the use of postgraduate students in Law review, finding that ‘for the most part, students […] are not experts in anything, and their acceptance decisions generally are based on surface-level first impressions.’[149]In ‘The Law Review Approach: What the Humanities Can Learn,’ Mendenhall advocates a ‘happy medium between law review and peer review,’[150] arguing that ‘peer reviewed journals and editors can draw from what makes the law review process efficient and thorough.’[151]

¶ 67Leave a comment on paragraph 670Competition is the key aspect of Law review processes praised by Mendenhall, where ‘the author competes for the best journal, just as the journals compete for the best articles or authors. Everyone is competing, and that speeds up the publication process.’[152] Mendenhall suggests that comparable competition in other disciplines would encourage efficiency:

If multiple, simultaneous submissions were allowed through electronic media such as ExpressO, and if authors and journals were able to bargain and negotiate over potential publications, competition would ensure that the peer review process moves more quickly than it does in current practice. […] it behooves humanities scholars to cultivate open competition among journals, editors, and ideas.[153]

¶ 69Leave a comment on paragraph 691In addition to competitiveness, Mendenhall identifies the following strengths in Law review: 1) a ‘quick and easy’ submission process, where ‘authors can submit to as many journals as they wish’ (53), 2) fast turnaround necessitated by the number of issues per year (54), 3) scrupulous copy-editing and citation checking by ‘multiple editors’ (54; original emphasis), and 4) a wider audience than Arts and Humanities journals, since Law Journals are ‘more interdisciplinary and have undertaken subjects previously reserved for scholars outside of the legal community.’[154]

¶ 70Leave a comment on paragraph 700In Law review, articles are ‘mobile’, meaning they are easily passed between journals. Several other models seek to reduce workload by transferring rejected articles to a subsequent prospective publication, including the reviews already undertaken. Addressing Mendenhall’s urge to ‘speed up’ the publication process, Cascading Peer Review endeavours to reduce the repetition of review work involved when a paper is rejected and then submitted for a new set of reviews by one or more further journals. It takes place in the instance that a paper has been submitted to a journal and rejected following peer review. The paper is then automatically sent, with accompanying reviews, for consideration in another journal from the same subject group. Baverstock outlines the merits of a ‘cascading’ system:

Technology is being harnessed to develop process efficiencies in peer reviewing. For example, BioMedCentral supports ‘cascade review’, which aims to reduce inefficiency and speed up publication by sharing review reports for rejected articles with other journals.[155]

Figure 8: Cascading Peer Review

¶ 73Leave a comment on paragraph 730A simpler approach is found in the ‘As-Is’ model suggested by Tsang and Frey, who argue that the popular ‘revise and resubmit’ option used by many journals simply stretches out timescales and increases workload, and should be streamlined to a simple ‘yes/no’ decision process with no opportunity to discuss or resubmit the article. Tsang and Frey advocate an ‘As-Is Review’ system, observing that ‘not all referee comments help to strengthen a manuscript’[156] and an ‘unduly long review process’ may result from ‘the practice of having multiple rounds of revision.’[157] Tsang and Frey recommend a double-blind system that encourages reviewers to give constructive feedback, but only permits one of two recommendations: ‘publish’ or ‘reject’. In summary, they state that ‘the as-is review process reestablishes the basic roles of authors, referees, and editors.’[158]

Figure 9: As-Is

Research output in the form of articles in prestigious journals is a key measure of performance for universities and individual academics. I suggest that in such a market journals are run like companies by CEO-editors who aim to create value for their shareholders. […] As a result, there is a thin line between smart, value-based management and unethical behavior.[159]

¶ 77Leave a comment on paragraph 772Tienari argues this could be countered by the adoption of a new system to combat the unsustainability of the current academic model, establishing ‘a rule whereby for every manuscript submitted the scholar in question would be required to review three manuscripts authored by others.’[160] Tienari argues that this ‘less is more approach’ would ‘restore equilibrium to the system’ and encourage researchers to ‘think extremely hard’ about where to submit work.[161]

Figure 10: Flexibly Anarchist

5.3 Communication and Credit

¶ 79Leave a comment on paragraph 790The need to provide incentives and/or professional recognition for reviewers can be addressed by dedicated platforms for reviewers to share comments and demonstrate prowess as reviewers. The ‘Publons’ platform allows academics to verify their contributions and compare their reviewing output with colleagues, while Belojevic et al introduce a ‘Peer Review Personas Prototype’ as a way for academics to archive and share review work. In these cases, the emphasis falls on demonstrating professional expertise by proving one’s capability as a reviewer. Other models offer the incentive of credit within the published article; the ‘Peer Reviewer as Collaborator’ model argues that co-crediting reviewers is an important way of ensuring their work is recognised and valued. Each of these examples indicates a tension between the precepts of traditional anonymous review practices, which value the anonymity of the reviewer as a hallmark of objectivity, and the need to demonstrate professional expertise in current institutional environments.

¶ 80Leave a comment on paragraph 801Addressing the question of what rewards or satisfaction reviewers gain for their work, Kumar argues that high-quality reviewing relies on ‘positive qualities of co-operation and collaboration.’[162] As an incentive for reviewers, Kumar proposes direct collaboration between reviewers and authors. This is a more radical version of the ‘Collaborative Peer Review’ model described above, since it suggests that in certain cases a reviewer could become a fully-acknowledged collaborating author on the manuscript. Kumar argues that ‘the reward of authorship outweighs any of the other rewards for peer reviewers suggested so far’[163] and believes it would benefit the author, the reviewer, and editors by enabling them to publish promising work that falls short of requirements for specific reasons.

¶ 81Leave a comment on paragraph 810
Noting that the ‘labor of review can seem ephemeral’, Belojevic et al propose a ‘Peer Review Personas Prototype’ to help reviewers ‘aggregate review content, search it, and present it as a core part of their scholarly labor.’[164] Allowing reviewers to track and demonstrate reviewer prowess, the prototype creates one or more ‘personas’ for reviewers to demonstrate their ‘diverse contributions.’[165]Inspired by social media (Twitter, Goodreads) and scholarly networks (PressFoward, Hybrid Pedagogy, HASTAC), the prototype facilitates collaborative practice, with the aim of encouraging reviewers to ‘view their manuscripts as opportunities for staging scholarly conversations, […] gathering on that stage to agree, disagree, reply, clarify, question and so on.’[166] Although the prototype does not include dialogue with authors, the article considers this as a possibility for the future: ‘Perhaps, too, authors could participate in the conversation as it emerges or after a certain milestone in review process.’[167]

5.4 Digital Platforms and Third Party Software

¶ 82Leave a comment on paragraph 821As early as 2002, Arms anticipated that peer review practices would change with the advent of digital platforms, suggesting ‘volunteer review’ would enable researchers to ‘build a reputation in the online world, outside the conventional system of peer review’ (n.pag). In 2013, Eve observed that ‘the born-digital medium of open access publications may lend itself to new modes that were impossible under the model of its print predecessor.’[168] This section details digital platforms and third party software designed to alter or re-emphasise aspects of the review process. Third parties now offer comprehensive manuscript and journal systems, whether managing the submission of articles (ExpressO), automated allocation of reviewers (NINES), dedicated software or web platforms to facilitate the entire process (Scholastica, ScholarOne), discussion forums for reviewers and authors, or commenting systems for open and post-publication review (PubPeer).

¶ 83Leave a comment on paragraph 830Although Arts and Humanities publications have been slow to adopt new systems in the past, such platforms may prove crucial to the future of peer review, offering new ways to reduce workload while maintaining the quality and consistency of reviews that is crucial for academic publications. It is also notable that such innovations are no longer restricted to large, established and well-funded journals and publications. Instead, ‘out-of-the-box’ solutions are increasingly available for independent organisations and individuals wishing to experiment with new forms of peer review; provisions such as Open Journal Systems and CommentPress are examples of a new generation of open source software available for small-scale projects and publications.

5.4.1 Technical Mediation

¶ 84Leave a comment on paragraph 840Mandernach et al evaluate the effectiveness of a ‘two–stage, hybrid evaluation process with open–public review, interactive discussion and final formalized private–peer review.’[169] Describing a peer review model developed by the Journal of Instructional Research (JIR), they find that ‘modern Internet–based interfaces allow collaboration for serious work’ and praise ‘the use of data sharing via technology to produce and check the quality of academic journal information.’[170] Mandernach et al find considerable advantage in ‘combining the strengths of traditional peer review with an integrative public review process’ and express surprise that few have tried to merge the advantages.[171] They admire the accountability for both authors and reviewers in open review processes, but suggest this could be strengthened by adding ‘a blind review process conducted by disciplinary experts’ to ‘allow the strengths of both models to be integrated and the drawbacks ameliorated.’[172] Surveying authors’ perceptions of the combined review processes, Mandernach et al found that authors were ‘highly satisfied’ with the quality of feedback from the public review process, and rated it as comparable to feedback from the blind review process.[173]

¶ 85Leave a comment on paragraph 850
Whereas Mandernach et al advocate combining traditional and new peer review models, Tallaksen calls for a move to entirely open review in the humanities, arguing that ‘putting all barriers aside, digital publication should become the norm in the humanities, and scholarly material should be disseminated through online networks.’[174] For Tallaksen, traditional peer review ‘promotes competitiveness and closed networks, rather than facilitating discussion, collaboration and sharing.’[175] Citing the examples of MediaCommons, OpenReview.net and the CASRAI/ORCID initiative – each of which use digital platforms for Open Review, Tallaksen advocates wholesale adoption of open review for Arts and Humanities journals. However, Tallaksen suggests ‘general unfamiliarity with digital technology’ is a barrier to innovation in the Arts and Humanities. For Tallaksen, ‘the real challenge seems to be of a social nature,’ with the need to seek solutions that are ‘accepted by each community.’[176]

5.4.2 Management and Processing

¶ 86Leave a comment on paragraph 861Several dedicated platforms endeavour to respond to a need for faster and more efficient ways to manage peer-reviewed journals. Editorial Manager is an ‘online manuscript submission and peer review system to manage submissions’ currently used by nearly 7000 scholarly publications, enabling them to manage different peer review processes depending on the article type and to match reviewers to articles through keywords via an integrated database.[177] Scholastica promises a similarly integrated system, stating that they are ‘a comprehensive academic journal management platform with all the tools you need to track submissions, automate administrative tasks, and collaborate with your editors, reviewers, and authors throughout every stage of peer review.’[178] Though the popularity of such platforms is increasing, humanities journals tend to be more reticent.

¶ 87Leave a comment on paragraph 870A notable open source example of online journal management is the Public Knowledge Project’s Open Journal Systems, offering an integrated platform for reviewing and editorial work, from submission to publication, as well as templates for online journal publication.[179] It is joined by an array of commercial options offering varying levels of control over editing and review processes. PubPeer is a scientific website that enables both named or anonymous post-publication review comments on manuscripts;[180] Blatt, editor of Plant Physiology, asserts that ‘the majority of posts on PubPeer are mounted anonymously.’[181] Aiming to cover the whole process of reviewing, ScholarOne describes itself as ‘providing scholarly publishers, societies, and associations with online, flexible workflow solutions since the mid-1990s.’[182] ScholarOne has become an important intermediary in the peer review process, offering a centralized workflow for large academic publishers including Sage, IEEE, and Taylor and Francis. Likewise ExpressO, owned by Be-Press at Berkley, is an online submission and management tool for law journals used to enable ‘express online deliveries to law reviews,’[183] There is a time-sensitive acceptance process and articles are published far more quickly than in the humanities and social sciences which makes the whole peer review and publication process highly efficient.

5.4.3 Finding and Selecting Reviewers

¶ 88Leave a comment on paragraph 880
A number of digital platforms have been developed to tackle the question of reviewer suitability, or ‘peerness’. NINES (Nineteenth-century Scholarship Online) offer a solution to the difficulty of finding the best reviewers, by collating ‘a peer-reviewing body for digital work.’[184]. The arXiv.org e-Print archive offers ‘a process for the review of peers,’[185] and the Research Information Network describes arXiv as ‘widely used in the physics community for the rapid dissemination of papers before they are formally published.’[186]Tallaksen argues for a model in the Arts and Humanities that echoes Open Science, but suggests that ‘the humanities show a slower development towards realising some of the central tenets of Open Science, than the natural sciences. Within the humanities we find no (significant) parallels to arXiv or ResearchGate, online communities that facilitate collaboration, discussion and dissemination of scientific research.’[187]

5.4.4 Payment

¶ 89Leave a comment on paragraph 891There have been calls for reviewers to receive appropriate remuneration for the work involved in peer review. Payment for reviewers is more firmly established in monograph publishing, although the Crossick report finds that even in this field, peer review ‘has never been well remunerated.’[188] In journal publishing, some publications have taken steps towards offering monetary reward to reviewers. For example, the ‘pay-it-forward model’ adopted by open access journal Collabra ‘directly compensates reviewers and editors for their work on the journal’ with funds from article processing charges.’[189] Several third-party platforms have also emerged which charge the author directly in order to pay reviewers. For example, variations on the Portable Peer Review (PPR) model offer a service where authors can submit their article to professional reviewers, effectively seeking feedback and evaluative commentary in advance of submission. Baverstock writes that ‘Portable Peer Review’ ‘potentially speeds up the process [of publication], shedding further light on the frustration of those trying to access appropriate vehicles for publication, particularly when these are linked to metrics for establishing the usefulness of colleagues’ contribution to their field and their readiness for promotion.’[190] However, such processes can be costly for the author. The Rubriq platform set up in 2012 offers a version of Portable Peer Review, promising ‘rigorous, double-blind peer review of manuscripts within two weeks’, but a ‘Complete Rubriq Report’ will cost the individual $650, with no guarantee of subsequent publication.[191]

Figure 11: Portable Peer Review

6. Conclusion

¶ 91Leave a comment on paragraph 911Scholarly critiques of current peer review practices often begin by conceding that there is considerable value to traditional review models, and this is certainly the case in the literature surveyed in this report. Peer review is a well-established and generally respected means of maintaining high standards in the pursuit and dissemination of academic research, and is often perceived as a hallmark of quality and fairness for academic publishing and institutional decision-making. A key argument in favour of conventional single- or double-blind peer review is the view that it enhances the rigour of scholarly work, encouraging meticulous research, clarity of expression and accurate referencing. As this report has indicated, double-blind review is often cited as a ‘gold standard’ in reviewing practice, particularly since the anonymity of both reviewer and author has the benefit of reducing the possibility of bias on the part of reviewers. This is an important distinction from single-blind review, where the author’s identity is known to reviewers and may allow judgments to be influenced by additional factors such as institutional hierarchies, nationality, or prior knowledge of the author.

¶ 92Leave a comment on paragraph 923Although double-blind review is frequently praised for quality and rigour, it shares many of the difficulties identified in critiques of single-blind review practices. Anecdotal opinions and scholarly studies tend to agree that anonymous reviewing can encourage inconsistent judgements and unhelpful comments, gives insufficient credit to conscientious reviewers who do produce careful commentary and decisions, and places undue responsibility on reviewers to identify scholarly malpractice. The problem of inconsistent or unhelpful reviewer comments is compounded by a lack of structured opportunity for authors to respond to reviews. The question of reviewer selection is particularly crucial in the case of highly innovative work, which can be more difficult to assign to suitably informed ‘expert’ reviewers and whose tendency to divide opinions may be disadvantaged by a preference for consensus among reviewers. Traditional review processes have also been roundly criticised for inefficiency, with scholars noting that anonymous peer review practices are time-consuming, and can lead to unnecessary delays in decision-making. Finally, a key concern regarding established single- and double-blind peer review practices is the possibility that they may be impractical for a contemporary academic environment, where professional pressure to publish has led to an increasing volume of submissions to journals, with escalating demand for scholars with sufficient expertise to act as reviewers (Bauerlein). Amidst claims that there is not enough evidence that peer review works (Peer Review Consortium, Meruane et al), that it lacks a core purpose (Jubb) and is less suitable for contemporary media (Fitzpatrick), there is a clear impetus to consider new approaches to peer review in the Arts and Humanities. Alternative review practices promise to develop or transform traditional approaches to peer review. However, as indicated by visualisations in this report, innovative models often involve compromising certain aspects of traditional peer review in order to emphasise or improve specific elements in the new approach.

¶ 93Leave a comment on paragraph 931This report has identified an urgent need to address issues with communication, consistency, efficiency and credit outlined by scholars who critique traditional peer review models. However, it is essential to value the advantages of established peer review practices, and to work to maintain positive elements of traditional peer review when developing new models and approaches. Most importantly, it is the task of all contemporary researchers to ensure that peer review is a sustainable, fair and effective model for future generations of academia. This can only be achieved through vigorous discussion and careful consideration of peer review’s role across the Arts and Humanities, including a wide range of topics beyond the scope of this article. There are, inevitably, aspects of peer review we have not been able to consider in detail, and other models, approaches and scholarly perspectives we might wish to include. With this in mind, we are opening this report for post-publication review, and invite you to contribute suggestions, critiques and comments on how it might be further refined to evaluate the future of peer review in the Arts and Humanities.

¶ 109Leave a comment on paragraph 1090Committee on Publication Ethics. “Code of Conduct and Best Practice Guidelines for Journal Editors,” March 7, 2011. http://publicationethics.org/files/Code%20of%20Conduct_2.pdf.

¶ 113Leave a comment on paragraph 1130Eve, Martin Paul. “Before the Law: Open Access, Quality Control and the Future of Peer Review.” Edited by Nigel Vincent and Chris Wickham. Debating Open Access, 2013, 69–81.

¶ 131Leave a comment on paragraph 1310Mole, Beth. “The Future of Peer Review in the Humanities Is Wide Open.” The Chronicle of Higher Education, August 13, 2012. http://www.chronicle.com/article/The-Future-of-Peer-Review-in/133563/.

¶ 144Leave a comment on paragraph 1440Risam, Roopika. “Rethinking Peer Review in the Age of Digital Humanities.” Ada: A Journal of Gender, New Media, and Technology, April 21, 2014. http://adanewmedia.org/2014/04/issue4-risam/.

¶ 149Leave a comment on paragraph 1490Sciullo, Nick J. “Must We All Be Beholden to Peer Review? On the Advantages and Disadvantages of the Law Review Model for Rhetorical Scholarship.” Review of Communication 15, no. 2 (April 3, 2015): 102–21. doi:10.1080/15358593.2015.1044019.

¶ 152Leave a comment on paragraph 1520Stern, Nicholas. “Building on Success and Learning from Experience: An Independent Review of the Research Excellence Framework.” Department for Business, Energy & Industrial Strategy, July 2016. https://www.gov.uk/government/publications/research-excellence-framework-review.

¶ 165Leave a comment on paragraph 1650[3] The scope for discussion of peer review in monograph publishing is indicated by the findings of the Crossick report on Monographs and Open Access (2015), which acknowledges ‘doubts about the sustainability and indeed rigour of current peer review practices for print books’ but maintains it is ‘a necessary part of academic publishing’ (27, 29). See also Levine, ‘Rethinking Peer Review and the Fate of the Monograph’ (2007), Fitzpatrick, Planned Obsolescence (2011).

¶ 166Leave a comment on paragraph 1661[4]Nicholas Stern, “Building on Success and Learning from Experience: An Independent Review of the Research Excellence Framework” (Department for Business, Energy & Industrial Strategy, July 2016), 21, https://www.gov.uk/government/publications/research-excellence-framework-review.

¶ 167Leave a comment on paragraph 1670[5]British Academy, “Peer Review: The Challenges for the Humanities and Social Sciences,” ix, accessed February 1, 2016, http://www.britac.ac.uk/policy/peer-review/contents.cfm

¶ 169Leave a comment on paragraph 1690[7]Björn Hammarfelt and Sarah de Rijcke, “Accountability in Context: Effects of Research Evaluation Systems on Publication Practices, Disciplinary Norms, and Individual Working Routines in the Faculty of Arts at Uppsala University,” Research Evaluation 24, no. 1 (January 1, 2015): 63, doi:10.1093/reseval/rvu029.

¶ 170Leave a comment on paragraph 1700[8] For example, the substantial resources developed by the ‘Peer Review Week’ initiative are largely focused on the sciences. This report draws on a considerable body of critical work devoted to peer review in the sciences, but with a focus on the Arts and Humanities where possible. To view the list of resources collated for Peer Review Week, see http://www.pre-val.org/prw/.

¶ 173Leave a comment on paragraph 1730[11] For more on digital humanities, scholarship, collaboration, and peer review see Roopika Risam (2014), who argues that digital scholarship is ‘often collaborative’, ‘rarely finished’, and ‘frequently public’ and ‘is best understood as part of an ongoing trend in academic discourse prevalent enough to require rethinking the production of academic value’.

¶ 198Leave a comment on paragraph 1980[36]Kathleen Fitzpatrick, “Peer Review, Judgement, and Reading,” 197–99, accessed February 18, 2016, http://www.mlajournals.org/doi/pdf/10.1632/prof.2011.2011.1.196. See also Planned Obsolescence, 7-8. For more on peer review, promotion and tenure, and grant funding see Lamont, How Professors Think: Inside the Curious World of Academic Judgment (2009) and Report of the MLA Task Force on Evaluating Scholarship for Tenure and Promotion (2006).

¶ 202Leave a comment on paragraph 2020[40]Roopika Risam, “Rethinking Peer Review in the Age of Digital Humanities,” Ada: A Journal of Gender, New Media, and Technology, April 21, 2014, http://adanewmedia.org/2014/04/issue4-risam/.

¶ 227Leave a comment on paragraph 2270[65]Ibid., 2. Results for the Arts and Humanities were as follows: Taylor and Francis 2015 report (Authors’ comfort out of ten): Double blind 8.6, Single blind 4.1, Open 5.7 and Post-publication 4.9. Publishing Research Consortium 2016 (Percentage likely to submit work to this review practice): Traditional 94%, Open 48%, Post-publication 50%.

¶ 241Leave a comment on paragraph 2410[79] This platform is based on scientific journal PLOS ONE’s concept of ‘peer review lite’ whereby peer review is exercised at the post-publication stage. For more on this, see Martin Eve, ‘The Future of Peer Review’ (2013).

¶ 242Leave a comment on paragraph 2420[80] For a detailed account see ‘Lessons Learned from Open Peer Review for Digital Book Publishing’ (2013) by Jack Dougherty.Examples in the sciences include Electronic Transactions on Artificial Intelligence (ETAI), Openreview.net, F1000Research, and The Winnower. Kennison (2016) writes that The Winnower ‘collapses the idea of pre-publication and post-publication entirely in an open peer review system that allows any format to be reviewed at any stage, with the author being the final decision-maker as to when a work is finished’ (71).

¶ 265Leave a comment on paragraph 2650[103] The retractions came only months after BioMed Central, an open-access publisher also owned by Springer, retracted 43 articles for the same reason. For more on these examples see Haug, ‘Peer-Review Fraud – Hacking the Scientific Publication Process’, (2015).

¶ 267Leave a comment on paragraph 2670[105]Committee on Publication Ethics, “Code of Conduct and Best Practice Guidelines for Journal Editors,” March 7, 2011, 2,4, http://publicationethics.org/files/Code%20of%20Conduct_2.pdf.

¶ 290Leave a comment on paragraph 2900[128]Rebecca Kennison, “Back to the Future: (re)turning from Peer Review to Peer Engagement,” Learned Publishing 29, no. 1 (January 1, 2016): 69, doi:10.1002/leap.1001. This viewpoint is echoed by Peter Sigal, editor of Hispanic American Historical Review (qtd. in Mole). The purpose and function of peer review, and related concerns, are also discussed by David Shatz in Peer Review: A Critical Enquiry (2004).

¶ 295Leave a comment on paragraph 2950[133] Taylor and Francis’s Peer review in 2015: A global view identified the following ‘ethical issues’ within peer review in the humanities. In descending order of what they call ‘perception of prevalence of ethical issues’ – where the most concerning is first – the report lists: Seniority Bias, Regional Bias, Reviewer Rings, Competitor Delay, Competitor Espionage, Gender Bias and False Identities’ (12).

¶ 300Leave a comment on paragraph 3000[138]Beth Mole, “The Future of Peer Review in the Humanities Is Wide Open,” The Chronicle of Higher Education, August 13, 2012, http://www.chronicle.com/article/The-Future-of-Peer-Review-in/133563/.

¶ 302Leave a comment on paragraph 3020[140]Frontiers have been criticised for their handling of peer review, with suggestion that they use only a single peer reviewer, and that the innovative ‘interactive’ model is not used in practice. See Terras, ‘Why I do not trust Frontiers journals’ (2015). This indicates the potential gulf between review processes ‘on paper’ and their practical application.

¶ 305Leave a comment on paragraph 3050[143]Kathleen Fitzpatrick and Avi Santo, “Open Review: A Study of Contexts and Practices” (Andrew W. Mellon Foundation, December 2012), 12. For more on the journal Kairos and their peer review process, see Jaschik (2012), ‘Humanities scholars consider the role of peer review’ from Inside Higher Education.

¶ 307Leave a comment on paragraph 3070[145]Nick J. Sciullo, “Must We All Be Beholden to Peer Review? On the Advantages and Disadvantages of the Law Review Model for Rhetorical Scholarship,” Review of Communication 15, no. 2 (April 3, 2015): 111, doi:10.1080/15358593.2015.1044019.

¶ 339Leave a comment on paragraph 3390[177] For more on this and GHSP as a case study see Salem et. al, ‘Process for selecting and implementing a manuscript management system: Experiences of a new peer-reviewed journal’, (2016).

Should you maybe start by defining what you mean by peer review? What sort of practices fall under the scope of what you’re investigating (and what don’t)? Isn’t part of the problem with peer review in general is that it is poorly defined?

My admittedly limited data gathering on peer review amongst management and organization studies researchers also suggests that it would be interesting to do the same sort of project focusing on the peer review of grant applications.

I read the Stern report with interest as I know many, many others will have done too. I am always intrigued by the use of metrics in judging the quality of research. I’m probably being naïve here but I wonder whether the type of citation made is taken into account – eg, when something is cited in order to criticize it. I’ve also heard anecdotes (again I suspect I am far from being alone in this) about citation cabals.

Indeed. There’s some interesting stuff on the social construction and also the psychological dynamics of peer review, which does rather cut against the notion that somehow peer review can be reliable, apolitical, objective etc.

There may be some advantages, in terms of flexibility, in having no fixed model of peer review. But are there not also serious disadvantages, namely, something can be said to be “peer reviewed” even if its assessment was neither rigorous or appropriate?

Something which also crops up on occasion is the possibility that in areas where a small number of researchers are working or where someone has a very recognizable style then it is easier to guess who has written the paper and/ or who has reviewed it :). My own field is an example of this.

I don’t think the reference to Nature’s trial is relevant. Nature was trialling (what one might call) public peer review (i.e. where review comments were unsolicited) rather than open peer review (i.e. where reviewers aren’t anonymous)

There is a lot of ambiguity here with regards post-publication review. What end is envisaged? Is the Version of Record (i.e. the published paper) changed after review? Or does post-publication review aggregate papers for readers? I think you need to define what you mean by post-publication review and what it is that these respondents are in favour of (or not, as the case may be).

There are lot of different ethical issues bundled together in this paragraph. None of them are to do with the anonymity of reviewers and there is no indication the open peer review, or post-publication review would handle these issues differently. There seems a false dichotomy in the report between “traditional” peer review and everything else. That isn’t particularly helpful.

Yes – in my discipline the growth is even more significant, so that one source – a paper by Starbuck – suggests an annual increase of publications (which I assume doesn’t include the rejections) of 11.04% between 1999 and 2009.

Again, yes, a really important point. The increasing commentary on what we might call the political economy of academia insists (and I agree) that publishers are effectively generating profit from journals by virtue of the unpaid work done by reviewers.

This paragraph notes the desire of authors to get recognised (through publication). It says nothing about the desire of readers to know what is worth reading (especially given the increase in published output).

Cascade review is starting to take off in my area too: interestingly it seems to be driven by publishers so that when a paper is rejected the author is invited to submit to another outlet in the same family. It would be really interesting to get data on how well this works. I wonder how difficult that would be 🙂

Yes, peer review is expensive even without paying reviewers. The cost of peer review could not be sustained by most journals under their current model and so inevitably the cost of paying reviewers would be passed on to authors.

I don’t say, as claimed in this paragraph, that peer review ‘lacks a core purpose’. I identify four distinct but related purposes, but argue that there is a lack of consensus as to which of those purposes are – or should be – in play when peer review is asked for and undertaken. That’s very different from saying that peer review lacks a core purpose.

I think the problem is that you have not defined what peer review is or what purpose you think it should fulfil. It is no good trying to decide what makes good peer review until you know what peer review is and it is no good trying to assess whether peer review is working until you know what it is you are trying to achieve.

You highlight in the report the tension between peer review as the gatekeeper and peer review as the collaborator. Which should peer review be?

The report lists a lot of problems with peer review but does not consider the role of editors in any of this. The role of reviewers is limited (or should be) and therefore they cannot be held accountable for all the ills of journal publishing. You mention that peer review can’t always spot plagiarism, but you don’t mention plagiarism checking software.

I think there is also a problem in assuming that single or double blind peer review must be done one way. But single or double blind only refers to one aspect of peer review (i.e. the anonymity reviewers and authors). That doesn’t tell you anything else about the model of review. It doesn’t tell you whether it is fair, objective, rigorous. It doesn’t tell you how reviewers are selected, whether they are quality reviewers, whether they do a good job or how the editor might respond if they don’t.

I think the dichotomy throughout the report between “traditional” peer review and “alternative” peer review is a false one and is misleading. Especially when “traditional” peer review is equated with single or double blind review.

Cascade review is starting to take off in my area too: interestingly it seems to be driven by publishers so that when a paper is rejected the author is invited to submit to another outlet in the same family. It would be really interesting to get data on how well this works. I wonder how difficult that would be 🙂

Again, yes, a really important point. The increasing commentary on what we might call the political economy of academia insists (and I agree) that publishers are effectively generating profit from journals by virtue of the unpaid work done by reviewers.

Yes – in my discipline the growth is even more significant, so that one source – a paper by Starbuck – suggests an annual increase of publications (which I assume doesn’t include the rejections) of 11.04% between 1999 and 2009.

Something which also crops up on occasion is the possibility that in areas where a small number of researchers are working or where someone has a very recognizable style then it is easier to guess who has written the paper and/ or who has reviewed it :). My own field is an example of this.

Cascade review is starting to take off in my area too: interestingly it seems to be driven by publishers so that when a paper is rejected the author is invited to submit to another outlet in the same family. It would be really interesting to get data on how well this works. I wonder how difficult that would be 🙂

Again, yes, a really important point. The increasing commentary on what we might call the political economy of academia insists (and I agree) that publishers are effectively generating profit from journals by virtue of the unpaid work done by reviewers.

Yes – in my discipline the growth is even more significant, so that one source – a paper by Starbuck – suggests an annual increase of publications (which I assume doesn’t include the rejections) of 11.04% between 1999 and 2009.

Something which also crops up on occasion is the possibility that in areas where a small number of researchers are working or where someone has a very recognizable style then it is easier to guess who has written the paper and/ or who has reviewed it :). My own field is an example of this.