The Philosophers' Cocoon

I recently reviewed a paper for a journal, and was struck by a check-box in the online reviewer form that I recall encountering before. The check box asked something like, "Would you be willing to have your review forwarded to another journal if the manuscript is submitted elsewhere?" This in turn got me thinking about a broader phenomenon: reviewers reviewing the same manuscript at different journals. I've encountered this phenomenon on social media before, the typical case being a reviewer saying something like, "I rejected this paper at one journal. Then I got asked to review it again at a new journal, and because the author didn't seem to revise it in light of my feedback at the first journal, I just attached my first review advocating rejection at the second journal."

I recognize why some reviewers might want to do these kinds of things. Sometimes you get tasked to review a paper, and it just seems so obviously problematic–and you think you make a good case for that in your review–that you think you would be saving another journal (and referee) time and energy by having yourself review it again. Trust me, I'm sympathetic: I've reviewed papers where this is my own reaction. Despite this, I am extremely uncomfortable with these practices. I am uncomfortable with them, first off all, because I read a lot of history of philosophy and science, and have read case after case of reviewers confidently rejecting papers and books that later went on to enjoy great esteem. Here are just a few:

• Robert Mayer: groundbreaking work on thermodynamics desk-rejected by Annalen der Physik, had to self-publish.
• James Joule: first groundbreaking electrochemistry paper rejected by the Royal Society, and a referee demanded that he remove the entire part of his famous paper establishing conversion of mechanical energy to heat.
• Hermann Helmholtz: groundbreaking paper on conservation of energy rejected by Annalen der Physik.
• Wittgenstein's Tractatus: rejected by something like a dozen different presses.

There are also more contemporary cases ready to hand. For instance, a while back Jason Stanley shared how some of his most influential papers were systematically rejected by journals–and I've heard similar stories from others.

These sorts of stories should, I think, lead reviewers–and, I would suggest, editorial policies–to err on the side of epistemic humility. Yes, you may be convinced that the paper you just reviewed is the Worst, Most Unpublishable Paper ever…yet this is apparently how some of the most important papers in history were viewed by reviewers. In every case where you are confident a paper is unpublishable, it might seem to you unfathomable that you could be wrong about it. Still, given that there have been numerous reviewers in history in just your position who not only got it wrong but indeed spectacularly wrong, I think there is at least one reason to adopt the epistemically humble position, erring on the side of letting new referees review the paper if it is submitted at a new journal.

Further, I think there are other reasons to err in this direction. One of the things I have noticed reading the history of this sort of thing is that it often appears to be unacknowledged biases that lead experts to get things so wrong. For instance, although his papers were published, many eminent physicists strongly denounced Einstein's relativity papers, even after there was observational evidence of bending starlight! For example, then-president of the British Royal Society and 1906 Nobel Prize winner in physics dismissed Einstein, saying, "no one has yet succeeded in stating in clear language what the theory of Einstein really is." Similarly, Sir Oliver Lodge, chair of the University of Chicago's physics department, dismissed the theory as "repugnant to commonsense." (Einstein: A Life, pp. 101-2). And head of the astronomy department at Chicago? He questioned Einstein's credentials as a patent clerk adding, "The Einstein theory is a fallacy. The theory that the 'ether' does not exist…is a disgrace to our age." (Ibid, pp. 102-3) Then there is Frege's reaction to Wittgenstein's Tractatus, which I recently recounted here–where Frege essentially dismissed Wittgenstein's now-famous work because it didn't conform to standard conventions ("One expected to see a question, to have a problem outlined, to which the book would address itself. Instead, one came across a bald assertion, without being given the grounds for it…"). Whatever one thinks of Frege's view here (I know there are some that concur with him, though I don't!), the point is that all of these cases–including the cases of Mayer, Joule, Helmholtz, and many others–show that, as certain as one may be that a work is poor or misguided, sometimes our judgment can be corrupted by biases, ones that other readers/reviewers may not share. This, in my view, is a second reason to err on the side of epistemic humility–a second reason not to review a paper again for another journal or have one's review forwarded.

Finally, I think there is a third reason for the same attitude: the moral risks involved. Allow me to explain. First, it is no secret that we work in a "publish or perish" discipline (and indeed, some people publish and still perish!). Researchers need publications to receive jobs, obtain tenure, etc. Second, the peer-review process is already excessively long. While some journals have two or three month turnaround times, other journals can take many months or well over a year. Consequently, I think there is some real moral risk–in terms of harming authors' career prospects–to the mere possibility of both (A) getting things wrong with a paper (viz. advocating rejection as a reviewer), and (B) holding a particular paper to that judgment at multiple journals. For, quite realistically, if both (A) and (B) are the case–if one not only gets things wrong as a review and prevents the paper's publication at multiple journals–you have, in effect, outsized negative effects on the author's research output. Given how little one has invested in the matter as a reviewer (the time one spends reviewing has little bearing on one's career prospects), compared to just how much a researcher's career can depend on their publishing record, this seems to me yet another reason to err on the side of epistemic humility: yet another reason to give the author a chance with new reviewers.

Is it possible that allowing a paper to be reviewed by others may be a waste of their time? Sure it is. However, if a paper is really as bad as you think it is, then it really shouldn't be that hard for editors and other reviewers to spot the problems as well, in which case it won't be much skin off their back to reject the paper themselves. All in all, then–counting up the reasons for and against reviewing a paper multiple times for multiple journals–the balance of reasons seems to me firmly against the practice. Personally, I am inclined to think that journals should not only not give authors the option of forwarding their review to other journals, but that editors should also ask reviewers to disclose and recuse themselves if they have reviewed a given paper at a previous journal. Why? Again, because of the serious risks of (wrongly) harming someone's career through potential biases–risks that can be avoided by simply giving the author a chance with new reviewers (something which again, in my view, carries few if any serious counter-costs, since it is reasonable to assume that a truly bad paper will be judged as such by other reviewers).

But these are just my thoughts. What are yours?