While I was away last week, a reader wrote in about a facebook discussion on how long referee reports should be. What is too short? What is too long? I think these are really good questions, as philosophers rarely discuss how to write referee reports. In this post, I'd like to solicit answers to the reader's query, and also offer some general tips I think may be helpful for writing good reports.
As an author, I have received referee reports that are just a few sentences or short paragraph long. While I have no idea whether this is because the report itself was so short, or because the editor chose to only forward a small part of the review, I have to confess that I find these types of reviews deeply frustrating. It is difficult to spend weeks, months, or (in some cases) years working on a paper, then wait several months patiently while the paper is under review, only to receive a very short report that merely says (but doesn't show in any clear way) that the paper is worthy of rejection. I think we owe authors better than this, even authors of papers we think are very bad. A good review–even a good review of a bad paper–should substantiate its judgments of the paper's merits. And this is takes a fair number of words.
How many words? Each case is unique of course, but for my part I try to compose really conscientious reports: reports that give a clear editorial recommendation, briefly review the parts of the author's argument I evaluate, and provide clear reasons why I think those parts of the argument warrant the recommendation I give. So, I took a look at some of my recent reports this morning. How long are they on average? My shortest report was 500 words, my longest 3,000 words, and the median length something like 1,500 words. I'm curious how this compares to other people's reports. So, let me ask: how long are your referee reports, on average?
Finally, let me offer a few brief tips based on how I write reports:
Tip 1 – provide a clear editorial recommendation. Some referee reports are confusing. The referee may say some complimentary things, then some very critical things – yet it may not be clear what they are recommending (acceptance?, rejection?, revise and resubmit with major revisions?, etc.). I always try to make my recommendation clear–with a very brief preview of my reasons for it–in my very first paragraph. Alternatively, if I am torn between more than one recommendation (e.g. I am unsure whether to recommend rejection or R&R), I'll try to be forthright about that as well.
Tip 2 – summarize relevant arguments before commentary. I expect many of us have received referee reports where a critique is given that appears to us (as authors) to misunderstand our claims/argument. Because I want to avoid this as a referee, I always try to concisely summarize the portion of the paper in question (citing relevant passages of the text) before providing my evaluation. I do this to try to make it clear to the author and editor that I read the paper carefully and tried to understand the argument charitably. This is the sort of thing I like from referees as an author. Moreover, in one case this made a big difference for the outcome of a paper I reviewed: the author received an R&R, explained how I misunderstood part of the argument, and I accepted their explanation (with the paper ultimately being accepted). Although in this case I did make an interpretive mistake as a referee (nobody's perfect!), making my interpretation clear enabled the author to make me and the editor aware of the mistake: a good result.
Tip 3 – begin with some complimentary remarks. It's true, some papers one reviews are bad. While I think some people overplay how many papers are bad or how bad they are (I've heard some people say 90% of papers are "garbage"), one nevertheless reviews papers sometimes one considers unpublishable. How should one respond to these papers? I know from online chatter that I'm not the only person who has received absolutely brutal reviews, where literally every sentence in the review makes it clear that the paper itself is a Punishable Offense Against Reason and Philosophy. I think these kinds of reviews are appalling. As problematic as a paper may be, the paper made it past the desk-rejection stage, the person wrote it is a human being who probably put a great deal of time into it, and there are usually some good things about it. Consequently, I try whenever possible to begin my reviews with something complimentary about the paper before providing a more critical analysis.
Tip 4 – critique with neutral language. Some reviews contain aggressive or unkind language (e.g. "This paper is terrible and looks like it was written by an undergraduate or first-year grad student). As an aside, I know a leading figure who recently received a report like this, despite the fact that they are recognized as one of the most important modern-day figures in their entire field and continue to publish in their discipline's highest-ranked journals (it is almost certainly not true that their paper was of the quality of an undergraduate paper). In any case, I think we owe our colleagues better than this. Instead of using colorful language, I think it is best to use purely descriptive language in composing one's report, such as "The argument is unsound for reasons X, Y, and Z." There is no reason to go beyond this, saying the argument is unsound and an Offense Against Reason.
Tip 5 – if revisions are suggested, give suggestions. Some referee reports recommend revision. Yet it is frustrating when they recommend this but give no clue what the referee might be looking for in revisions. Consequently, when I recommend revisions, I try whenever possible to provide some brief suggestion(s) of ways to move forward (e.g. "While I think this part of the argument is problematic, here is one way I think the author might revise the paper to address it"). While I think few referees do this, in the end I think it probably saves referees time – as it doesn't leave the author in the dark. After all, what does take a lot of time? Answer: the author revising their paper in ways that don't address one's concerns, leading either to a second R&R or another referee report where you have to explain why they didn't do so very well. Better, I think, to point them in the right direction from the outset, at least if you sense there is a particular direction they should go with the revisions (at least in terms of satisfying you as a reviewer).
Anyway, these are just a few things I try to do in my reviews. What do you do and why?
Leave a Reply to Marcus ArvanCancel reply