Shay Logan (postdoc, NC State) wrote in with the following interesting query:
We've all received horrible referee reports. I'm curious if any of us has ever *written* a horrible referee report and then realized it. That is, have you ever written a referee report, then (for example) on receiving the author's responses come to realize that you'd misunderstood the paper in some completely boneheaded way?
Bonus points if you can say what made you realize you were at fault.
I think it could be very instructive to hear readers' experiences. It might help us all better understand what being a "bad reviewer" amounts to! So, then, have you ever been a bad reviewer? How? Alternatively, if you haven't been a bad reviewer but have been on the receiving end of bad reviews, what exactly was bad about them? I suppose I'll begin…
It is hard to remember most of the reviews I've written. Fortunately, of the ones I do remember, I'm not sure that I've written any that grossly misunderstand the author's argument, etc. One problem here, of course, is that it can be hard to know if one has grossly misunderstood a paper (if one knew it and one wasn't being willfully obtuse, one would avoid it!). Still, I work very hard on the reviews I complete, trying to make sure I get things right. In any case, I do recall one review I wrote a number of years ago that stands out. It was a very bad review in the sense that I essentially argued that the author's argument was a non-starter, as well as inept at a scholarly level (ignoring vast swaths of literature). Because it's the only review like that I've ever written, and the content was pretty harsh, I went back several times to temper the review's language, so that at the very least I did what I could to not come across as needlessly mean (though I wouldn't be surprised, given the review's content, if the reviewer still interpreted the review as mean).
Fortunately, as I said above, that's the only review like that I've ever written. Generally speaking, I try very hard to write "balanced" reviews–ones that, first, lay out the essentials of the author's argument; second, say something complementary about the paper (as there is usually something worthwhile even in papers that are currently publishable); and then, finally, justify my recommendation in detail by engaging with the author's argument.
Much more often, I've been on the receiving end of awful reviews. Typically, the kinds of bad reviews I've received are the 4-5 sentence sort–the sort that, after I've waited anywhere from 2-12+ months, simply dismiss the paper as unpublishable without any detailed explanation why (often simply saying things like, "This paper is too ambitious, and does not succeed in fulfilling its ambitions"). Interestingly, albeit anecdotally, my sense is that I've gotten these types of reviews much more often when submitting to highly ranked journals. I've tended to notice that most of the "good reviews" I've gotten–ones that really engage my work in detail–are from reviewers at lower-ranked journals. Although this is anecdotal, I do have a sense as to why this might be. I have heard a few times that some journal editors simply "trust the judgment" of some reviews, particularly really established figures in the field–not expecting detailed reviews from them because they are so notable. Personally, whatever is the case, I think 4-5 sentence reviews are unconscionable and should in no case be accepted by editors. If a paper is bad, a reviewer should be able to provide some actual detail as to why–not just be able to say, "Trust me, it's bad." There are many reasons for this. As established as a given person may or may not be, they are also human–with human fallibility, biases, etc.
The other kinds of bad reviews I've received are the willfully meanspirited ones–ones which go out of their way to insult the author or the work in question. And I know I'm not alone in these either. I know at least one leading figure in the field who told me a referee told him a recent paper of his was one the reviewer would expect from a grad student. Finally, the most amusing bad review I ever received was for my 2013 paper, 'A New Theory of Free Will.' When I first submitted it to another journal, it was about 21 pages long. I got a revise-and-resubmit, and the reviewer's comments were so helpful that I rewrote it so that it was 78 pages long. I figured since the journal had no official word-limit, I might be able to get away with it. Um, no. The reviewer was none too pleased, writing a single sentence to the effect of, "I'm not going to read a 78 page paper." Unfortunately, the paper was rejected–but on the other hand, I got a good laugh out of it and published it elsewhere! 🙂
Anyway, what are your experiences either writing or receiving bad reviews? What constitutes one?
Leave a Reply