Mechanism change in a simulation of peer review: from junk support to elitism.

Paolucci M, Grimaldo F - Scientometrics (2014)

Bottom Line:
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

ABSTRACTPeer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

Fig8: Average quality of papers for different mechanisms. Only the replication scenario drops sharply for a small amount of rational cheaters

Mentions:
For peer review, we have confirmed that unfair play (here, rational cheating), in a first approximation, is likely to impact the performance of the review process rather heavily. However, by modifying the mechanisms employed (see a comparison for all scenarios in Fig. 8), we have also been able to lessen this impact, and even to reverse it when we come to a specific scenario (see “Restrained cheaters” section). This evidence points out to a more complex role of cheating in an evaluation system, ranging from sabotaging effect, to a sort of “useful idiot” role. In turn, this suggests a multi-level approach for containment of cheaters. Indeed, cheating behavior could be directly addressed at the individual level with enforced norms, and/or it could be made harmless by using the opportune mechanism at the collective level.Fig. 8

Fig8: Average quality of papers for different mechanisms. Only the replication scenario drops sharply for a small amount of rational cheaters

Mentions:
For peer review, we have confirmed that unfair play (here, rational cheating), in a first approximation, is likely to impact the performance of the review process rather heavily. However, by modifying the mechanisms employed (see a comparison for all scenarios in Fig. 8), we have also been able to lessen this impact, and even to reverse it when we come to a specific scenario (see “Restrained cheaters” section). This evidence points out to a more complex role of cheating in an evaluation system, ranging from sabotaging effect, to a sort of “useful idiot” role. In turn, this suggests a multi-level approach for containment of cheaters. Indeed, cheating behavior could be directly addressed at the individual level with enforced norms, and/or it could be made harmless by using the opportune mechanism at the collective level.Fig. 8

Bottom Line:
Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results.In addition, we also show how this result appears to be fragile against small variations in mechanisms.These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.

ABSTRACTPeer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.