Is it time to open up Peer Review?

Peer review is arguably the keystone of academic publishing, with reviewers serving as gate keepers of legitimacy and tasked with ensuring that standards are maintained and trust in the field is sustained. It is also, for the most part, a thankless job. This might be about to change.

The practice of peer review probably kicked off when Henry Oldenburg, Secretary of the Royal Society in the mid-17th century, sent out manuscripts to experts for vetting before publication. Since then, peer review has gotten more institutionalized, but the form itself has been remarkably stable: an editor sends out a manuscript to a handful of experts in the relevant sub-domain, and if the experts green-light it as a valuable academic contribution, it is published.

Anonymity is crucial to how this system works. The peer reviewer is unnamed so that they can offer honest evaluations about the quality of a submission without fear of retaliation, particularly if the author is someone with clout. The identities of authors are also kept anonymous, so reviewer judgments aren’t influenced by personal relationships, both warm and cold. Of course, academics present their work at conferences and some sub-fields are so small that everyone knows what everyone else is working on, but for the most part secrecy is taken seriously and respected for what it makes possible.

As of late, an increasing chorus of scholars are questioning whether this accepted wisdom about the importance of anonymity in peer review is actually as wise as it might initially appear. For example, Caroline Schaffalitzky de Muckadell and Esben Nedenskov Petersen argue both that papers accepted for publication should be published along with their peer reviews, and that the reviews published “should include not only reviews from the journal accepting the paper, but also previous reviews which resulted in rejections from other journals”.

It should be conceded that this is a fascinating proposal for a number of reasons because earlier peer reviews and corrections are also vital parts of how academia works, and to simply sweep them under the rug makes this whole aspect opaque and obscure. Plus, peer reviewing is notoriously arbitrary, as is captured well by the meme of the capriciously cruel “#Reviewer2”. John Turri from the University of Waterloo who argues that maintaining anonymity keeps academic disciplines from developing open norms about what is worth publishing, frequently leading to frustration when a submission is rejected on seemingly arbitrary grounds. If we want honest discussions about the state of a field, we need to be able to see what standards are actually being employed to determine what gets published in its journals.

As great as these arguments are, a worry is that the persistence of peer review anonymity possibly undercuts all the possible benefits they advocate for. For example, one scenario de Muckadell and Petersen are keen to avoid is that of unqualified or abusive reviewing. According to this, people know their reviews are going to be made public, so they’ll take care to not review abusively. But if reviewers know they’ll remain anonymous regardless of how abusive their reviews are, it isn’t clear why they would be motivated to change their behaviour. So although the authors might still succeed in their aim to “put forward [reviewers’] arguments for public scrutiny”, this might not be quite enough to elevate the quality of reviews or decrease the incidence of abuse.

Considerations like these have led to a more radical proposal — remove peer review anonymity.

At first encounter, this might seem like a dangerous suggestion. After all, how can early career academics, especially those with vulnerable employment, critique honestly and even harshly, when it is called for? But if we look past this first reaction, two strong arguments can be found for this position.

For one, there finally will be motivation for people to write more responsibly, since they can claim credit for well-written and well-argued review but obviously cannot for abusive ones. While it is true that a reviewer who has no interest at all in using peer review history as academic credentials might still continue being abusive, this should help better things significantly.

Second, it should be pointed out that peer review still is academic work, even essential academic work. As Justin Weinberg from the University of South Carolina points out, “my sense is that the credit one gets for peer reviewing is disproportionately small compared to how important peer reviewing is for the academic enterprise”. To give people the ability to take credit for work well-done then is a matter of fairness.

An interesting model for how this might be done, without stripping anonymity from reviewers without their permission is the website Publons. This collects information on a voluntary basis from peer reviewers and verifies this with the journal publisher. This allows for the creation of reviewer profiles that each reviewer can claim claim credit for and add to their CVs.

With any solution there are those skeptical. David Roy Smith from Western admitted that as an early career academic, he simply hadn’t had the opportunity to review many papers, especially from prestigious journals, and so he wasn’t all that eager to sign-up. In addition, there’s the perpetually relevant question about whether the endless march to quantifying and comparing work done and its impact is actually good for academia.

Still, the removal of anonymity in peer review, voluntary for now, seems to be the direction we’re travelling and so we need to take it seriously. Universities and funding organizations need to incorporate the now-public data about peer reviews performed into their decision making, and choose whether people without publicly recorded reviews will be penalized or not. Publishing and publishing tech need to incorporate the ability to transfer and approve finished peer reviewers quickly to standard sites, so workflows don’t get cluttered.

The opening up of Peer Review is bound to be a momentous transformation in what is now a procedure that hasn’t changed much in centuries, so who knows where we’ll end up.

Trackback URL:

No comments yet. Be the first.