In our new “how can we help you?” thread, a reader asks:
How do you write a good referee’s report? For a rejection? For a revise and resubmit?
Good questions – what do readers think?
The Philosophers' Cocoon
A safe and supportive forum for philosophers.
Owned & Moderated by Marcus Arvan (University of Tampa)
Contact: marvan@ut.edu
Blog mission & moderation policy
Posted in Peer review
11 responses to “How to write a good referee report?”
-
Here are just my two cents.
1. Writing the report two weeks after the deadline takes as much time as writing the report the same day of accepting to review. You don’t have to do it the same day, but a quick report is, other things being equal, better than a slow report.
2. Summarise the paper to show that you have read the paper without misunderstandings.
3. Give a comment on whether the core idea is worthy, urgent, important, significant, or something like that.
4. Consider if there are things that *must* be changed. This can include major improvements. Try to give concrete guidance on whether the changes are feasible. Sometimes it’s possible to be more sceptical, e.g. “I’m not sure this can be done, but it is worth giving the opportunity to see whether this objection can be responded to.” (I think failing to cite relevant literature that makes points mentioned by the author is among these, but I think this is more up to disagreement.)
5. Consider if there are nice-to-do things, but make explicit that this should be up to the authors’ discretion, if it is anything of that level.
6. A referee report is not marking. A somewhat not so good paper can be publishable, if major enough changes are feasible. An almost good paper may be not publishable, if relevant changes are infeasible. (Of course, a good idea may not be publishable if the paper doesn’t seem to be fixable in three revisions. At this point I would recommend rejection.)
7. Have a bit of intellectual humility. Philosophical disagreements should not be grounds for rejection. Rather ask: would this paper, if published, advance the discussion? (I also worry about “not seeing originality.” My most cited paper was rejected by someone who couldn’t see the originality. The referee cited another paper, but my paper proved to be more engaged in the end.)
-
Realize you are writing, first, for the editor. It was they who asked you for your expert evaluation. So, always make clear what the final assessment is upfront (accept, minor revisions, major revisions, or reject). As noted above, it is useful to have a short three sentence summary of the paper/argument at the beginning, as well. Then the author will at least know how you understood the paper. That, too, is useful for the editor.
If it is a rejection, then state clearly the main reasons why. And in each case provide a justification, ideally, directing attention to places in the text where the problem is evident. For example, “the author presents a strawman of their opponents argument. This is evidence on page 6 where they say …” If you have a few choice serious problems, then your verdict will make sense to editor (who can in fact check some of the passages), and it may even make sense to the author. If the verdict is acceptance, it is still important to help the editor understand the value of the paper. For example, note how it advances the debate in a novel way, or … Things are most challenging with verdicts of revise and resubmit. On the one hand, the paper has problems or you would just recommend acceptance. On the other, you obviously see promise in the paper, or you would have recommended acceptance. You need to diagnose what the author really needs to get straight in order to make the paper publishable. Often the author is either (i) trying to do too much, and you have to recommend what to cut (and what to develop further), or (ii) exaggerating the scope of their claim or the significance of it. You need to tell them how they might amend this difficulty. You do not want to send an author away with a revise and resubmit without knowing what they need to do. So try to make your recommendations for amendments concrete and actionable. I seldom give a verdict of R&R. -
What the others have said is sensible. Two pieces of negative advice:
Don’t draw inferences about the author’s philosophical knowledge (or states of mind more generally). That someone failed to say X doesn’t mean they don’t know X. (Focus on what the paper does or does not do.)
Don’t copyedit the paper or take a position on whether the author is a native English speaker. (Try to distinguish between typographical errors, grammatical errors, and errors that impact the argument in substantive ways.)
-
Always keep in mind: an objection to an argument is not necessarily reason to reject the paper. Always consider whether some issue you have with the paper is reason to reject it.
-
Everything everyone has said so far is great and I agree 100% with it. Another thing to add: if you say the paper is failing to engage with important existing literature, cite the literature. Give the names of the relevant books or articles. (I even cite the relevant page numbers, especially for books.) Do not just say the name of the author, and especially do not say that a paper misses out on lots of existing work without bothering to name any.
This is an exception to the point above about you writing for the editor: this is for the sake of the author. But as someone on the receiving end of many reviews that seem to me misguided about whether some paper of mine has missed out on important work in the topic, it’s very frustrating as an author to be told there are ostensibly things out there you ought to have discussed but not to be given the list of those things. If you as a reviewer are not in a position to substantiate this claim, perhaps you ought not to be making it.
-
In terms of structure, I normally start with a paragraph summarizing the main point of the paper and what I see as its positive qualities. Then, I pick 2-4 main issues to comment on in some detail. Sometimes it’s a specific objection to the argument but it also might be something more general about the paper, like if has organizational problems or misses relevant literature. At the end I may have a section labeled “small comments” where I mention smaller things that I think the author would want to fix but that don’t impinge on the publishability of the project (examples would be an importantly unclear sentence or a place where there is clearly a missing or inappropriate citation). If I already have a lengthy report with 4 long comments than I may omit the small comments section, whereas if the paper is already very strong and my report is shorter then I may take the time to list out some small issues like wording problems. If I comment on the writing it’s never the main point of the report and it’s always from the perspective of “this was unclear to me” rather than “this doesn’t sound like a native speaker of variety X of English”; and while I agree with the point that a reviewer’s job isn’t copyediting, I will say that I do appreciate when reviewers catch typos for me.
I will write very short reports in two instances:
-The paper has already gone through revisions and is acceptable for publication
-The paper is completely unsuited for publication and I struggle to find positive things to say about it
If I have a very strong opinion about the paper I might leave more pointed comments in the “for the editor only” section. I try to always keep in mind that I’m writing the report for a real human and that my goal is to help them improve a specific research project rather than to attack their value and competence as a person (or stroke their ego).Otherwise I don’t tend to write reports very differently for cases where I recommend minor revisions, major revisions or reject. This is because I anticipate that the editor might not agree with my recommendation, and because I want my comments to be helpful to the author regardless of the decision to publish in a particular journal.
-
Some R&R reports I have received are extremely long. On the one hand I am very grateful for these because they can be some of the most careful engagement with my work that I can get. And they come from clearly very well-intentioned reviewers who are going above and beyond the call of duty. On the other hand, they make revisions really hard. Often there is just not enough space to reply to everything in the paper because of the journal word limit and I end up writing very long letters engaging with the suggestions before I resubmit. And often a lot of the suggestions are to me pretty clearly not the kind of thing that would bear on the publishability of a 10,000 word paper, they’re rather things that would make a 20,000 word paper more interesting — things like “you should engage with this really interesting thought experiment I came up with — what does your view have to say about that?” In these cases I wish the reviewers would write a short review documenting the important changes I need to make, and then save these very interesting comments for an email once the paper is out (they don’t need to out themselves as a reviewer.) Or, even a paper objecting to mine, if they’re interested enough.
I know this is kind of ‘beggars can’t be choosers’ — obviously I’m grateful to get any R&Rs and any helpful comments. I’ve just noticed this is a trend that I think is less helpful than people (who clearly aim to be helpful) realize.
On another note: many published papers have a beautiful, self-contained first 18 pages, and then a bizarre, tacked-on 4 pages responding to what are obviously referee comments. I have never thought that a paper is better for having that last section. I think it’s mostly on authors not to do this and integrate their responses into the paper more seamlessly. But I also think referees could maybe be clearer about what it would take to respond to the comments so that this doesn’t happen. (i.e., this one could be clarified in a footnote, this one requires substantive changes to section 3, etc.)
-
Especially if you foresee accepting the paper, e.g. because you’re giving it minor revisions, please make clear which of your comments are ‘optional’ i.e. that the author can take-them-or-leave-them without worrying about your decision. If you leave this unsaid, we authors will of course take pains to address all of them to ensure acceptance since our fate is in your hands.
-
This might deserve its own thread, but I’m wondering how folks, when reviewing, handle cases where you think the author ought to engage more with (or at least refer to) your own work. E.g., I’m currently reviewing a paper that seems to make the same point I’ve made (and elaborated on in 2-3 different published papers) but in a new and interesting way. I’m not a big name by any means, but I’m somewhat surprised the author didn’t come across my work or engage with it at all. How do I frame my referee report so it doesn’t come across as me just hunting for citations?
-
This is a great query and I think it does deserve its own thread! I’ll post it for further discussion next week.
-
-
I would add that referees should in general bear in mind that some topics are genuinely difficult, and the value of a contribution can well depend on the space it would help open up rather than how well-done a job it does in recapitulating or summarizes existing views (e.g., the referee’s view).
Leave a Reply