I came across this illuminating post by a former journal editor yesterday, and was reminded of a disturbing discussion I happened across on social media a couple of weeks ago. In that discussion, a philosopher on social media began by sharing an interesting proposal: that journals should ask submitting authors for a list of people they have circulated their paper to, in order to help ensure that the journal selects reviewers who do not know the author's identity (viz. ensuring anonymized review). Although a number of people expressed support for the proposal (myself included), there was some pushback, including one bit that I found concerning. At one point, one commenter mentioned that they thought the proposal could prevent journals from finding the most qualified reviewers for a given paper. Their suggestion was that in very small subfields the people most qualified to review a paper might be ones who have seen the paper before and know who its author is. That person then added that they routinely accept reviewer assignments in cases like this, estimating that approximately 50% (!) of the time they review papers, they either know or strongly suspect the author's identity.
This isn't the first time I've come across claims like this on social media. Over the past few years, I have come across multiple comment threads on social media where reviewers and even some editors have said similar things. For example, in previous threads, I've some suggest that if a reviewer knows who a paper's author is, then the editor should decide whether the reviewer should accept the assignment. I have to confess that I find this very concerning. Anonymized review exists for a reason. The entire point of it is to prevent explicit or implicit biases toward or against authors from affecting editorial outcomes–so that work is judged on its merits. Reviewers or editors deciding to let a review occur when the identity of an author is known compromises this protection against bias.
Now, as readers may know, I'm actually in favor of moving away from anonymized peer-review to public peer-review. However, as long as our current system is in place, I think anonymized review has a purpose: to lend fairness and legitimacy to the peer-review process. Given that hiring, tenure, professional recognition and engagement, and so on, can all depend on where one publishes, it seems vital for editorial processes to strive for fairness. This, I take it, is why journals typically have strong statements expressing a commitment to anonymized review. For example:
'The Journal operates with a policy of ‘triple anonymity’: referees and editors assigning papers to referees do not know who the author of a paper is, and the author does not know the referee’s identity.' (Journal of the APA)
'Nous has adopted a triple-blind review policy, according to which submissions are blinded by the managerial staff on receipt. Editors and reviewers assess submissions blind to authors' identities until a decision is made on the submission.' (Noûs)
However, for editorial policies like these to be more than words, the process must follow them. Now, in social media threads like the one mentioned earlier, people who know an author's identity often say they only accept the reviewer assignment if they 'feel they can be fair.' But again, this not only defeats the very point of anonymized peer-review, which is that it shouldn't be left for people to decide whether they can be fair. It also conflicts with a wide body of empirical literature indicating that people are often deeply biased even when they think they are not. This in turn seems to me undercut another common response I have come across, which is that editors should decide whether reviewers should accept these kinds of assignments. Yet this too seems to me to not only conflict with the point of anonymized review, which is that no one–not even editors–should be able to compromise anonymized review; it also contradicts the letter and spirit of journals' own editorial policy-statements. For as we see above, those statements don't say things like, "This journal practices double-anonymized review…well, when it is convenient; but when it's inconvenient, sometimes we let people review papers of authors whose identity is known." No, journal policies typically say very clearly that they practice anonymized review, period.
In sum, I have two related concerns here. First, if as comment threads like the ones I am referencing indicate, reviewers routinely review papers by authors whose identities they know, then journals' anonymized review policies are routinely compromised. Second, there are independent reasons–given the empirical literature on rationalization, confabulation, and bias–to think that anonymized review policies should be enforced, so long as the current peer-review system exists. Reviewers or editors to deciding whether to follow anonymized review policies, wittingly or not, permits personal biases to creep into the review process that anonymized review is designed to prevent–all in a system that plays a central role in hiring, tenure and promotion, allocating recognition and engagement, and so on.
For these reasons, I think more steps should be taken to ensure anonymized review, not less–at least as long as our current publication model is in place. In addition to authors providing a list of people they've circulated their paper to, I think it may make sense for review requests to remind reviewers that they must not accept if they know a paper's author or are reasonably confident of the author's identity, and that in no case should it be left to reviewers or editors to decide whether it is acceptable to violate anonymized review. This, it seems to me, is what fairness requires, at least if anonymized review policies are to treat every author fairly and equally.
Or so I'm inclined to think. What say you?
Leave a Reply