This is the final of four guest posts by the Editors of Analysis. The Editors' aims have been to "give some guidance to prospective authors and to contribute to public debate about philosophy publishing." In today's post, the editors respond to some of the comments in their first three posts. Many thanks to the Editors for sharing their thoughts on the peer-review process here at the Cocoon.
By Stacie Friend, David Liggins, and Lee Walters
We’re delighted that our posts have generated so much debate. A big thank you to everyone for their comments and questions! The discussion was rich and wide-ranging; we can’t do justice to it all, but here are some thoughts on some of the points raised.
Our first post was about referee reports and whether they should be aimed at helping authors or helping editors. Since a couple of commenters seem to have misinterpreted us, we’ll start by reiterating that our main point was not that lengthy reports aimed at authors take longer for us to read, but that they take longer for referees to write; and we didn’t argue that it’s impermissible to write reports like that, but that writing more, shorter reports would help with the current crisis.
In the comments, “Sometimes reviewer” made the point that it’s important to give comments if recommending a revise-and-resubmit, to help the author change the paper so it gets accepted next time. We agree! Sometimes referee reports need to be long, because the recommendation is complicated: “I recommend that you accept this paper subject to the following changes…” or “I recommend that you ask the author to revise the paper in such-and-such ways”. We like short referee reports, but sometimes reports can be too short to be helpful to an editor.
We emphasised that we’d like referees to provide an assessment of the paper’s quality and significance. Richard Yetter Chappell expressed the worry that ‘many referees seem to fall into a default mode of assessing papers by the metric of "can I find an objection?" rather than "does this advance the debate?"’ We agree that looking for objections is not the best way to approach refereeing. When making decisions, we tend to think more in terms of quality of argument: if a paper isn’t well argued, then it’s unlikely to advance debate. Referees’ objections typically tell us something about how well argued a paper is. (But quality of argument is only one dimension of evaluation. There is more to a good paper than a strong argument.)
Andy raised an important concern: brief comments might not convince authors to stop sending out weak papers: ‘Clear and detailed comments are needed to persuade authors not to just send their papers out again … whilst brief reports may take less time to write, I am not convinced that they save time overall because they fail to take bad or half-baked papers out of circulation’. We have a couple of thoughts about this.
First, brief comments can still be specific ones: examples include ‘Author’s main argument is that (P) on p. 6 is obvious and leads to determinism, but I find (P) implausible’ or ‘This paper adds little because its argument was anticipated by Jones’s 2017 paper’. Second, what is most important in a referee report is often a judgement which may or may not convince the author, no matter how long the report is (e.g. ‘I don’t think the conclusion is significant enough for the paper to be worth publishing’). (In the comments, Noah made a similar point.)
Chris asked a couple of questions about referee reports:
‘1. I am sometimes tempted, as a referee, to note how the manuscript compares to the average article published in the journal for which I'm refereeing. Is this something that you like? dislike? encourage? discourage?’ Our answer: we don’t mind either way.
‘2. In the comments only to the editor, I sometimes remark (when it is relevant) about the level of confidence of my opinion. … is this helpful? and is this the kind of thing you'd also like me to pass on to the author?’ Our answer: yes, it can be extremely helpful. For the sake of transparency, we’d like to pass that on to the author, assuming you are willing for us to do so.
Louis asked for advice on what to put in the report if the paper is very weak. We tell Analysis referees that if the paper is very weak, only a very short report is required. Our advice is to write something like “In my judgement, the paper is too far from publishable at present to recommend a revise-and-resubmit” and then mention one or two aspects that need further development. Writing a report aimed at the editor should make the task shorter and less painful. (By the way, we see nothing wrong with saying in a referee report “This paper reads to me like an early draft”.)
It can be useful to mention the instructions given by the journal. For instance, we ask referees to recommend resubmission only if they think it’s likely that the next draft will merit publication in Analysis; it’s perfectly appropriate for our referees to refer explicitly to that instruction in their reports.
Our second post was aimed at helping authors understand verdicts from journals, and the time it takes journals to reach a decision. We concluded by asking why other journals often don’t give comments – not even brief ones – when desk rejecting a paper.
Douglas W. Portmore, the editor of Ethics, responded. He stressed that the reasons for giving a desk rejection often cannot be provided in just a couple of sentences. And, echoing Joona Räsänen’s comment, he mentioned that giving comments to authors can invite correspondence from authors, disputing the decision.
Our view is that it’s fairer to authors of desk-rejected papers to give them some indication of why the paper was rejected. This is partly based on our experience as authors: we know just how discouraging it is to receive a desk rejection with zero information about the reason. We trust authors of desk-rejected Analysis papers to understand that the explanation we give them may well be a partial one, and that much more could be said. From our perspective, a partial explanation is better than nothing. As “provide feedback” put it, a comment “doesn't have to be a full referee report in order to be helpful.”
Any explanation of a verdict, however long or short, can certainly provoke the author to write back with their own reaction to the comments received. That is surely a cost, but we do not see this as a good enough reason to stop providing any explanations of desk rejection decisions. “Disclaimers” wisely proposed that journals indicate to authors that they will not get into discussions of desk rejections. As it happens, our rejection letters have said that for some time.
One thing that helps us to provide comments on each desk-rejected paper is the involvement of our team of virtuosic Associate Editors. Although the three of us make all the decisions on papers, including desk rejection decisions, those decisions are often based on Associate Editor advice. Of course, the Associate Editors contribute in many other ways too; it’s a pleasure to acknowledge all they add to the journal. Adrian Alsmith, Eduardo Barrio, Jess Begon, Mike Caie, Roger Clarke, Catarina Dutilh Novaes, Doug Edwards, Luara Ferracioli, Max Freund, Dana Goswick, Andrea Iacona, Jess Isserow, Alex Kaiserman, Justin Snedegar, and Mike Titelbaum, we cannot thank you enough.
A topic that appeared in the comments, which we want to flag up here while leaving discussion to another time, is desk rejection rates. If they are indeed so high that the evidential value of receiving a desk rejection is low, then that seems to us a threat to the efficient working of the philosophy journal publishing system. At Analysis, our policy is to send a large majority of submissions to referees, keeping desk rejection to a minimum.
Our third post was modest in its ambitions: we admitted we have no way to resolve the crisis in philosophy journal publishing, and simply offered some suggestions for making things slightly better. Pace Matt Weiner, our suggestions were not about making things slightly better for journal editors, but for everyone in the discipline.
One paragraph of ours met with particular resistance in the comments:
“Consider whether you really need to submit your paper to a journal. Would it be preferable to publish it in an edited collection, or just put it on your website? If the paper closely resembles your earlier publications, is there sufficient new material to justify publication?”
The comments make us think that a clarification is in order: where we wrote “consider”, we really meant “consider”. We could hardly be unaware that information about journal publications informs hiring decisions, and that not everyone is given the opportunity to publish in edited collections. Many professional philosophers and many aspiring professional philosophers have excellent reasons to focus on trying to publish journal articles. But not everyone is on the job market, or planning to be; and some people are more focused on contributing to philosophy than on seeking professional recognition for everything they do. (Also, we wonder whether publishing large numbers of extremely similar papers really leads to greater employment success or enhanced professional recognition.)
We anticipated that another suggestion of ours would prove controversial, namely, the idea that referees should be willing to review a paper multiple times. In support of this, we asked readers to consider a weak paper in a small subfield of philosophy. If the author keeps on submitting it, and each expert in the subfield refuses to report on the paper more than once, then eventually every expert in the subfield will have written a report on it. That means journals will eventually have to rely on reports from non-experts, which will lead to weak papers taking up space that might have been occupied by papers that actually deserved publication.
Several of the commenters were uneasy about the idea of papers competing for space in journals. But, as we see it, that is inevitable, because publishing papers costs money. This remains true even if the journal is only published online: as Matt Weiner pointed out, underlining something we said in our second post, ‘going online does not completely solve the space/cost problem; Ergo is online-only, but our largest expense by far is still per-page costs for proofreading and typesetting.’ Unless journals abandon proofreading and typesetting altogether, financial constraints inevitably generate space constraints. That means that journal editors are right to worry about accepting bad papers, since doing so takes away space that could have been devoted to good papers, and thus detracts from one of the important functions of journals: to direct readers to the best work in the discipline.
David Thorstad asked: “Could the editors provide authors with a credible guarantee of support for standing up to reviewers? The last time I stood up to a reviewer … I lost a paper over it. I have to say I am not inclined to repeat the experience.” Shen-yi Liao raised the same issue in one of his comments.
These are important concerns. In response to them, we can only talk about our own practice as editors. What is most important to us is not the verdict recommended by the referee, but the content of their report. We adjust recommendations up or down when we judge it to be appropriate to do so. Our verdicts are not a function of the referees’ recommended verdicts; we simply try to make the best decision we can in the light of all the evidence available. That evidence often involves the insightful advice of an Associate Editor. When recommending revise-and-resubmit, we sometimes explicitly tell authors which referee comments to focus on and which ones to set aside (though in other cases it seems to us the author is better placed than we are to make that judgement); and when a revision is received, we give serious consideration to what the author writes in their response to the referees.
One theme that emerged from the comments could be summed up in the word “distrust”. “Summaries” thinks that referees should be made to write summaries since they cannot be trusted to read a paper properly otherwise. Richard Yetter-Chappell thinks that many referees are biased against excellent iconoclastic papers. According to David Thorstad, “It's well known that referees hate being argued with and like seeing their comments addressed directly in print.” And “grymes” doubts that established philosophers would be willing to help others by changing their publishing practices. We would be interested to know whether these comments are representative of a wider feeling in the profession. If they are, then that is a worrying fact – especially if the distrust is merited. We would prefer philosophers to trust each other to act professionally, and for that trust to be deserved.
The comment for which we are most grateful came from the aptly named “Truth”, who wrote: ‘Editors, it seems, are not monsters trying to frustrate the lives and careers of every young (and old) academic.’ Quite right! Journal editors do not spend their time devising ever more ingenious ways to inflict pain on authors, and ultimately ruin their lives. That said, we completely understand that to authors it can sometimes seem exactly like that. But in reality, we are just some academics, trying to do a good job, at a very challenging moment for journals. We hope this discussion has deepened understanding of journal article publishing, and of how it might be improved.
Leave a Reply