The Philosophers' Cocoon

A reader writes in:

Would it be possible to open a topic on your blog about advices for those of us who start to do reviews for the first time? I am in general on the other side of the publishing process: I submit articles that are often rejected (and quite often with some nasty comments). Now that I have to do reviews, I do not know how to be critical without being offensive.

Great query! I'd love to hear from readers, but here are a few tips that I've found helpful…

I always begin my referee reports with a brief introductory paragraph containing a brief overview of the author's thesis and argument (so the editor reading my report has some idea of what the paper is about), along with a clear editorial recommendation stated in language as neutral as possible. For example, if I think a paper should be rejected, I will write something to the effect of, "Unfortunately, I believe this paper should be rejected for reasons X and Y.' If I think a paper receive a revise-and-resubmit, I will write something like, "Although this paper has an interesting argument potentially publishable in Journal's pages, in my judgment it would require the major revisions described below." And so on.

Importantly, I try to keep the language here as neutral as possible, simply describing (in a couple of sentences) the substantive grounds for my recommendation (viz. "As I detail below, the author's argument is logically invalid"). I think this is important, because all too often I've received (and heard others receive) reports with needlessly inflammatory language, such as "This paper should have never been submitted to this journal" or "This paper looks like it was written by an undergraduate" (side-note: I know one influential senior academic who received a report stating the latter for a paper that went on to be widely recognized in their area!). Look, if a paper is really bad–and you think it was a waste of your time as a referee–I understand the temptation to let the author know in explicit language that theirs was The Worst Paper in the World. But I still think you shouldn't say things like this. If a paper is bad, a referee report written in neutral language (i.e. describing in philosophical terms where the paper errs) is sufficient to demonstrate that. Anything beyond that is simply hurtful–and, as I explain below, I find it helpful to remind myself of how it feels to be on the receiving end of this kind of stuff as an author. 

After my report's introductory paragraph, I always try to begin my report with something complementary. Sometimes, of course, it is hard to find something to complement the author for–and if the paper is bad from beginning to end, one may find it difficult (or even impossible) to begin a report this way. Indeed, I think I may have refereed a paper or two where there honestly wasn't anything positive I could say. That being said, I'm inclined to think it is good practice to try one's best to begin a report this way–not simply to go out of one's way to be "nice" to a paper's author, but because (in all honestly) most papers do (at least in my experience) have at least some redeeming features, at least if one looks hard enough. Since there are usually at least some good parts of a paper, I think it makes sense to recognize them at the outset of one's review. First, it seems to me intellectually honest, as in my experience academics can have a tendency to focus overwhelmingly on what's wrong with a paper instead of recognizing a paper's meritorious features. Reminding oneself to recognize merits first is, I think, a good way of counteracting the above tendency toward negativity. Second, my sense is that beginning one's review this way can be kind, as it can function (at least somewhat) to soften the blow of a report that is otherwise very critical. Although some academics may think it is "not our job" to soften the blow of our critical comments, I again find it helpful to remind myself of what it feels like to get comments that are entirely negative. I've heard more than a few people say that it makes them feel worthless, like a bad philosophers, and so on. Even if a paper is bad, my own conviction is that it is simply unkind to make other people feel that way (especially if, as is often the case, they have invested a lot of time and effort into the paper one considers bad).

Next, after saying something complementary, I provide the critical heart of my report: the part that functions to justify whichever editorial recommendation I made in the opening paragraph. In the case where I recommend acceptance, this is easy: I briefly explain why the paper is great. In cases where I recommend revise and resubmit, I begin (if I did not explain it in the former paragraph) explain why the paper is worth reconsidering, but why it needs revision. And, of course, in cases where I recommend rejection, I lay out in detail what I think warrants that recommendation. Importantly, when I recommend an R&R or rejection, I again try to write my critical comments in neutral language, simply explaining (descriptively) which parts of the paper appear to me to have serious problems (viz. "The author's first premise appears false. Allow me to explain."). In the process, I try to avoid needlessly colorful language (viz. "The author's first premise is absolutely absurd") and intensifiers (viz. "The author's writing is incredibly unclear"). One helpful tip I'd suggest here is composing one's review and then putting it away for a few days or a week before returning to it. My experience is that this can give one an important kind of "distance" from the review, such that when one reads it again one will experience it more like it was written by a stranger (and hence, experience it more like the author would). Oftentimes when I do this, some of the things in my report will strike me negatively–as too blunt, or needlessly aggressive, and so on. And so I will then go back in and try to temper the language, making it more neutral.

Finally, at the end of the day, before submitting my report, I try to ask myself, "How would I feel about receiving this report if I were the author?" As an author, I've always been happy to receive critical reports, provided they don't seem needlessly hurtful or 'personal.' Indeed, I've been happy to learn that some of my arguments don't work, and have given up on papers on occasion due to helpful critiques. There's nothing wrong with receiving a critical report, in my experience. It all depends on how the critical review is presented. And while I have not always been successful in this regard in my role as a referee (I would be lying if I said that I always lived up to the tips I've outlined above), my experience is that it is important to try one's best and that it can be rewarding to feel like one has been a "good referee" for the author and journal.

But these are just some of my thoughts. What are yours? Do you have any helpful tips for writing good referee reports?