In the comments section of a post today at Leiter reports on how much time and effort referees should expend, a commenter writing under the pseudonym 'Out to pasture' wrote:
Here's a brusque (but I hope not brutal) headline thought. Referee's reports for journals are written for editors. They should offer a judgement about whether the paper is worth publishing. They are not, primarily, tutorial reports for authors. If the judgement is negative, then it is a work of supererogation to spell out the reason in any detail: a couple of lines for the editor's eye is enough. The editor should trust your judgement if s/he has chosen you to report. It is nice if you have the time to say more, but there is no need…
If my first, second, and third-hand experience are any indication, a decent proportion of philosophy journal referees appear to subscribe to Out to pasture's general line of thought. I have not only had papers rejected with one or two lines to the effect of, "The arguments are no good", without any substantiation of the reviewer's judgment. I have also had numerous collegagues in the discipline–including some people in my social media feeds recently–recount similar experiences: as having to wait for anywhere from 3 months to over a year to get a rejection consisting of a couple sentences that amount to little more than, "Trust me, this paper is not publishable–though I cannot be bothered to give you any explanation why." Interestingly, although I do not have much experience in other fields, I do have some first and second-hand experience with one other field: psychology. I have submitted papers to psychology journals myself, and my spouse is a researcher in Industrial-Organizational Psychology. And, although my data points might not be representative, I have yet to come across a single instance of this kind of reviewer behavior there. Even the bad papers I submitted to psychology journals were met with clear explanations of what was wrong–in some cases, explanations by a reviewer, in other cases a brief explanation by the editors. And every referree report my spouse has shown me with her papers is similar: there is always a real explanation–a justification–of the referree's recommendation.
Of course, these differences between fields might be justified. Maybe philosophers have special reasons–special qualifications or abilities–that should exempt them from having to justify their editorial recommendations. Or maybe there are more general reasons to think referees shouldn't have to give detailed recommendations. This seems to be Out to pasture's thought: that in soliciting a referee's recommendation to begin with (especially, let's say, if the person has a good philosophical reportation), philosophy editors have grounds to trust the judgment of that referee. Yet this seems to me to fly in the face of everything we know, empirically–and we know a lot–about the pernicious effects of tacit biases. People tend to think that they are "objective, impartial" judges of things–and it seems, in philosophy, that we judge "the arguments on their merits." But there are so many empirical reasons to doubt this. People tend to think that they do not judge people differentially on the basis of race, gender, social class, etc.–yet, time and again, empirical research shows they do. Similarly, although a given referee might not think they judge papers in biased ways, there are all kinds of other possible ways that their judgments might be biased. For example, one recent study of doubly-anonymized review showed that,
[A]uthors often could be identified by reviewers using a combination of the author's reference list and the referee's personal background knowledge…[identifying] authors correctly 40-45% of the time. One main motivation for double-blind review is to eliminate bias in favor of well-known authors. However, identification accuracy for authors with substantial publication history is even better (60% accuracy for the top-10% most prolific authors, 85% for authors with 100 or more prior papers).
Second, many papers are shared publicly–at conferences, department colloquia, etc.–in ways that can clue a reader into the author's identity. Third, there are "graduate student" ways of writing, or so I hear–features of a person's writing which can signal a person being an early-career philosopher. And so on.
This, in brief, is why I think referees are obligated to give actual, detailed justifications for their recommendations, and why editors should expect them to fulfill that obligation. The point of anonymized review is (or should be) to mitigate bias, and ensure that papers are judged on their merits. Having to give an actual justification is one way of ensuring this: it is a way to (A) hold referees accountable for their recommendations, and (B) enable second and/or third parties (an editor or multiple editors) to judge whether the judger actually has a good case to make for the recommendation, or whether their recommendations are due to biases they (the referee) may or may not be aware of. I just have a hard time seeing how anything less is appropriate scholarship. Given the kinds of biases that can–and are known to–afflict human judgers, "trust me" shouldn't be considered good enough, especially when it comes with a 3-6+ month wait time (which, as we all know, are common wait times with journals).
Or so I'm inclined to think. What do you think?
Leave a Reply to PostdocCancel reply