In our newest "how can we help you?" thread, Frustrated writes:
I have a question about good practices for reviewers. In the past few months this happened to me 3 times. I was asked to review a paper for a top journal in my AOS. This paper is really really bad. It's bad in the following sense: the topic is important, the thesis is interesting, but it'so poorly written and argued that, if I did something like that myself, I'd get for sure a rejection. However, I recommend major revisions. After a few weeks I got the revisions back, and nothing much has changed. I can see what the other reviewer has suggested, and it's minor revision. I then suggest rejection and the paper is accepted. From this I understand that the paper was either from an established scholar, or from a student of an established scholar. I feel I'm wasting my time by doing my diligent work of reviewer, given that the editor has already decided. Therefore, this is my question: when I have the feeling that something similar is happening again, shall I address the editor directly and ask how I should treat the paper? Or shall I just decline to review the revisions?
I'm glad Frustrated wrote in about this, as I think it's a great issue to discuss. Another reader submitted the following response:
Frustrated: is it possible that your standards are too high? Maybe this series of events has some other kind of takeaway. Three different editors have overridden your negative reports … perhaps your standards are either too stringent or just out of keeping with what the editors are looking to publish. I've had an editor override a referee on one of my papers. The referee accused me of all kinds of weird things (like I'm *clearly* just a graduate student, and some senior scholar has made some edits to my paper) and seemed really irritated. The editor contacted me with this review and asked if I would make some changes. The paper was then sent out to a third referee, since the editor judged the second referee to have an overly harsh view of the paper. I was grateful to the angry referee despite all the mean-spirited commentary, since I was able to improve the paper. I am not an established scholar, nor is anyone running around wielding influence on my behalf. Did the editor act inappropriately? I don't think so, not at all. These are recommendations you are giving, not dictates.
This is more or less the reaction that I had to Frustrated's comment. If I am reading Frustrated's comment correctly, in just the past few months they have had three (different?) editors (at three different top-ranked journals?) ignore their negative referee reports in favor of more positive reports by other referees. What should Frustrated infer from this? While it's hard to know for sure without looking at the papers themselves, four things seem to me worth noting here:
- Referees and editors at top-ranked journals tend to have very high standards (viz. high rejection rates).
- In each of the cases the OP describes, at least one reviewer thought quite highly of the 'really really bad' paper in question, judging the paper to warrant only minor revisions.
- The associate editors who handle a paper may be specialists in the area.
- When they receive conflicting reports, these editors may read the paper themselves to decide which report(s) to trust.
Given these four things and given the fact that same thing has happened to Frustrated repeatedly, it seems to me that Frustrated should seriously consider that the possibility that (A) they are writing overly negative referee reports, and (B) that the papers they judge to be obviously really really bad aren't. After all, if the papers really were that bad, one would figure that someone else–either the other referees or the editors who read the reports and then the paper–would see this as well. But that's not what's happening here. In three separate cases, it seems, the editors have read the referee reports and the paper and decided for themselves that the other referees' positive reports are the ones to trust.
Now, this read on things could be totally wrong. Maybe Frustrated is absolutely right about these papers, and in each case the referees and editors have gotten things wrong. Anything is possible! However, if I were in Frustrated's position, I guess I'd think the appropriate thing for me to do would be to adopt a stance of epistemic humility and infer something like the above explanation. Indeed, I think I'd be especially inclined to adopt this perspective given what is known about peer-review in general: namely, that referees and editors already appear to have a documented tendency to reject great papers, judging them not to be great when they are. After all, consider all of the Nobel Prize winning economics papers that were famously rejected from top journals in that field on grounds (such as referees claiming their results to be trivial) that, in retrospect, appear to be ridiculous. Or consider Jason Stanley's report from a few years back that many of his papers that were rejected from multiple top-ranked philosophy journals are now among the most cited papers in those very journals. Or consider Heesen and Bright's discussion of the empirical literature on peer review, which finds (among other things) that referees agree on the quality of papers barely better than pure chance. All of these things suggest to me that when we are serving as reviewers, we should adopt a standpoint of epistemic humility–and so if I found myself in Frustrated's position, I guess I'd think that the appropriate thing for me to do is to rethink what I'm doing as a reviewer.
But these are just my thoughts. What are yours?
Leave a Reply to Marcus ArvanCancel reply