The Philosophers' Cocoon

A safe and supportive forum for philosophers.

Owned & Moderated by Marcus Arvan (University of Tampa)

Contact: marvan@ut.edu

Blog mission & moderation policy

IMPORTANT NOTE on how to ensure anonymity in commenting
Job-Market Reporting Thread (2025-26)
Job-Market Discussion Thread (2025-26)

recent posts

about

Peer-review: what works and what doesn’t?

Several months from now, I'm scheduled to take part in a discussion on our profession's peer-review practices: what works, what doesn't, and what might be improved. To prepare for the session, I'd like to run a new series here at the Cocoon soliciting readers' opinions on our peer-review practices and norms, and what (if anything) readers think might improve the process.

In today's post, I'd like to ask readers to weigh in on two initial questions:

  1. What do you think currently works well in the peer-review process?
  2. What do you think the biggest problems are with the peer-review process?

After hearing what readers think, my plan is to run a few informal polls, so as to gauge readers' overall satisfaction with the peer-review process, desire for change, and priorities if change is pursued. Then, after those polls, I hope to have a series of posts discussing some of specific issues in more detail, including more detailed debates about whether the peer-review process should change at all, if so how, and so on. 

So please do considering weighing in on questions (1) and (2). The more readers we hear from, the better discussion we are likely to have in the series overall!

Posted in

17 responses to “Peer-review: what works and what doesn’t?”

  1. Anon

    This is an issue that’s already been discussed on this blog, of course, but: the biggest problem I’m experiencing right now is time to complete review of papers.
    I have one paper that has gone eight months without an initial review (I will probably just get around to withdrawing this one soon, but no one’s replied to my inquiries, and I feel a bit worried about some report coming in after I withdraw). I have another paper getting up to six months post submission of revisions.

    Reply
  2. do I really want to wade into this

    Marcus
    I am curious to know where you will be speaking about this issue.

    Reply
  3. Sebastian Lutz

    What currently (sometimes) works well
    1. Sometimes you get a friendly reviewer who, even when recommending rejection, gives sensible, thought out feedback based on your (actual) arguments.
    2. Some journals have established deadlines for reviews. While not all reviewers meet those deadlines, I think this has improved review times a lot.
    The currently biggest problems
    3. There is little if any incentive to actually do good work as a reviewer besides moral satisfaction. Even publons.com only keeps track of the number of reviews, which is at best a motivation to not be so terrible that you are never asked again. I would love to have an evaluation system for reviewers by the reviewed authors (probably best partitioned into rejected and accepted).
    4. Something of a result of the above and the inverse of point 1: Some reviewers seem to read the conclusion first and then write their review based on their opinion of the conclusion. Sometimes this lets weak arguments get published, sometimes this lets strong arguments get rejected.
    5. A corollary of the above: The review process is very scattershot, and some journals will never accept specific papers simply because the papers will be send out to the same unfriendly reviewers.
    6. Too many half-baked papers are submitted to too many journals. I assume this is the result of publication pressure.
    7. A corollary of the above: Editors and reviewers are overworked, which increases false positives and false negatives in journal acceptances.
    8. A corollary of the previous two points: The ratio of crap to good papers increases, first, because the number of half-baked submissions increases and, second, because the false positives go up. The increased false negatives won’t make as much of a difference.

    Reply
  4. William Peden

    My own experiences have almost all been very positive. I would particularly point to detailed feedback, encouragement, and only one “This author has overlooked the brilliant X on this topic Y [obviously their work].”
    From friends, I know that there are major problems with (1) timeliness and (2) pointless negativity by some referees.
    If there was just one change I could make, it would be somewhat more detailed information for authors when an article is being delayed, e.g. “Our first choice of referee was unable to do the review.” While fundamentally insubstantial, I think this would relieve a bit of easily avoidable stress for authors, especially for when articles take 12+ months to get back.
    Also: in extreme cases, when editors don’t even acknowledge the delay, maybe we should set a norm that e.g. 12+ months under review is long enough to wait, and the author can start sending the article elsewhere? I’m not sure about this proposal, as it could end up wasting editors’ time and they’re overloaded already.

    Reply
  5. Amanda

    Problems:
    (1) Many journals are double blind, not triple blind. This means the editor can just desk reject a paper for the wrong reasons, or send the paper to a reviewer who they suspect will reject/accept a paper according to the editor’s own opinions. The editor also has authority to simply reject papers in spite of positive review reports – this is problematic when the editor is not blind to the author
    (2) Even in triple-blind review, many papers are not blind review at all, because almost all experts had already seen the paper before at a conference or read/talked about it online. May reviewers admit to reviewing a paper when they know the author, and editors often approve.I see no solution to this problem, and what bothers me most about all of it is still calling the paper ‘blind review.’ The more famous and/or connected one is, the less likely the paper is to actually go through blind review.
    (3) Everything takes way too long.
    (4) Reviewers have no incentive to do the review well, and so often they do not.
    (5)Journals are intellectually conservative, i.e. controversial papers seem more likely to get rejected. Likewise, the journal process seems to favor careful, boring papers over big, new, and interesting ideas which inevitably have mistakes.
    (6) The “top” journals tend to favor traditional metaphysicians and epistemology, and certain kinds of highly abstract ethics. There are lots of great papers in other sub disciplines that have little chance of getting published in the most recognized journals.

    Reply
  6. Anon

    “I have one paper that has gone eight months without an initial review (I will probably just get around to withdrawing this one soon, but no one’s replied to my inquiries, and I feel a bit worried about some report coming in after I withdraw). I have another paper getting up to six months post submission of revisions.”
    Here’s a suggestion: wait until reviewers have been found, then withdraw it. And submit it elsewhere in the meantime. That way, the journal gets some small token of punishment (i.e. finding refs for no reason) for their bad behavior.
    I had a journal take a year to review. They gave me an R and R but didn’t respond to my emails (such as my email asking: hey, if I do this R and R, is it going to be sent to the same lously ref who took a year to review?), so I just declined to revise. They suck.

    Reply
  7. elisa freschi

    1 As an author:
    1.1 It teaches one humility.
    1.2 At times, I got really good advice (I have to admit, however, that I also got great advise through sessions on Academia.edu).
    1 As an editor of peer-reviewed journal issues and a supervisor of PhD students:
    1.3 It is a great way to explain to students and younger colleagues that they have to be, e.g., clearer (since the reviewer did not understand X, you just have to explain it better!).
    1 As a peer-reviewer:
    1.4 At times, I reviewed really horrible papers, papers I would not have thought people would have even dared sending to a journal. One needs a filter and peer review is better than no filter at all.
    1.5 It gives a fair chance to many who do not come from prestigious universities etc. I could review, help improving and finally accept articles from young colleagues who did not have English as their mother tongue and/or lived in obscure parts of the world. Which journal would have accepted them without some reviewers vouching for the quality of their contributions?
    1.6 It made me aware of colleagues whose work I would not have ever read.
    2. As an author:
    2.1 The time needed to find referees is at times just too long (I am also among the ones waiting since months and getting no information).
    2.2 The fact that at times I get referees asking for contradictory things and the editor is not brave enough to step in and tell me that they understand I cannot do both and that I should rather follow B instead of C.
    2. As a peer-reviewer:
    2.3 The fact that at times I invest time and energy only to then see the article accepted after minor modifications, so that the process seems at least opaque.
    2.4 The fact that at times authors reply in an annoyed way as if they were artists who had not been fully understood in their genius.

    Reply
  8. Pendaran Roberts

    I started trying to publish seriously in 2014. I eventually was able to publish most of my ideas (still waiting on some papers). So, I think, from my experience, that a positive about our system is that if you’ve got a novel and interesting contribution and don’t give up that you can publish it, eventually. The biggest negative about our system is that it’s incredibly inefficient. About half of the referee reports I’ve received over the years were at best borderline incompetent and at worst borderline negligent or dishonest. This slows down the publication of good papers tremendously. Often you end up with that 1 referee that just won’t be reasonable or just can’t seem to understand the merit of your work.
    Some of my papers were accepted almost straight away, but some have taken years. In particular, an experimental philosophy paper of mine, which brings X-Phi to a new area in an interesting way, is apparently too controversial to be easily published. I first submitted it in Jan 2016. It’s still going around review land. I’ve had a total of 5 reviewers accept the paper but haven’t lucked out with getting enough reviewers at one journal. At one top 10 journal, 2 out of 3 referees accepted the paper, and it was still rejected because of one overly critical referee. So, the system is delaying the publication of this work for years, while others who wanted to engage with the work are having to wait on the system to function and publish something that should have been published a long time ago.

    Reply
  9. O.

    (1) I’m fortunate to have had really good experiences as an author, with really thoughtful, helpful, interesting reviews that have made my work better. I think I’ve felt more confident in responding to reviews and in writing reviews because I don’t know who the authors or reviewers are. I imagine more than a bit of self-doubt would creep in if I knew that I was responding to some super-famous person (I am not super-famous, to put it mildly).
    (2) Many of the problems are connected: too many submissions, too few reviewers, too many delays. Ultimately I think the solution will involve doing something about the unfortunate focus on quantity of publications as the only real measure of scholarly contribution. One small suggestion: encouraging ways to have referee work and various modes of responding/ replying to journal articles and books be counted more in assessing one’s scholarly work. These are real, valuable contributions to scholarship; publishing articles is great, but it’s not the only way.

    Reply
  10. Random job market person

    Best thing:
    1. Some editors are great. I think many editors are sensitive to the plight of both referees and authors and really do their best with what they have got.
    For me the biggest problem is shoddy refereeing:
    1. Rejection of papers for absolutely absurd/trivial reasons.
    2. Failure to even properly read the paper, to the point that papers are rejected for doing things which are very clearly and demonstrably not done (I have had reports where I think it is fair to say the referee outright lied about the paper. More often I think it is simple failure to properly read the paper though).

    Reply
  11. Peter Furlong

    I have had the same experiences as others above. Sometimes I get great feedback, sometimes it is ridiculous. I should note that this does not always correspond to how positive the referee is. I have had someone recommend accepting a paper, but some of their comments made me think they really didn’t understand the area I was working in. Of course, some of my rejections I though were for stunning reasons too. Much of the time, though, I think feedback (whether positive or negative) does reveal that the referees carefully read the paper and understood the sub-field it was in.
    One thing that I don’t think any other comments have mentioned is the phenomenon of papers getting lost somewhere in the process. I have not been submitting papers for that many years, but I have already had two papers lost. In both cases, the editors admitted that they had lost the paper. In one case, after repeatedly contacting the journal, the editor responded noting that the paper (which had been submitted more than a year earlier) had not even been sent out to review. In the other case, I contacted the journal and they responded noting that the referees had given their approval, and the decision was made to accept the paper, and somehow between this stage and informing me of the decision, the paper had fallen through the cracks.
    I appreciated the honesty of both editors in admitting the errors that had been made, but it made me wonder whether this is even more common, whether papers get lost and, when this is noticed, editors blame this on slow referees or such.
    In any case, I am not quite as pessimistic about the system as some others are. I do think that the system does sometimes operate blindly, does give non-famous authors a chance, and does help filter out some work that isn’t quite ready for publication.
    I also think that some of the problems with the system can be fixed from the inside, with changes to the policies and cultures of particular journals. Still, I am open to ideas that are a bit more radical.

    Reply
  12. Chris

    One thing I think could be improved is adopting stricter ethical norms for reviewers who seek to violate blind review by (for example) Googling the title of a paper they have received.
    There may be no feasible way to police this practice or punish those who do this, but I think we could as a profession make our ethical expectations more clear, that this is not just a type of human weakness to be tolerated, but that it is a serious transgression of academic integrity.
    For example, maybe there could be an honor pledge reviewers sign, acknowledging their commitment to blind review and the seriousness of its violation.

    Reply
  13. Peter Furlong

    I think Chris’s suggestion is fantastic: I think it would be easy to implement, has no down side, and has the potential to reduce the number of Google reviews. Obviously it would not completely solve the problem, but I think it would help, at least a little.

    Reply
  14. Matias Slavov

    (1) I feel the same way as O.: fortunate to have received many good reports. They have improved the quality of my work. Only a couple of reviews have been incompetent.
    (2) Submissions take in general too long to process. Things are changing. Good examples are journals like Ergo and Thought. I submitted a paper to Ergo, and I got very good comments in two days. Although the paper was rejected, I followed the editor’s suggestions, submitted it to a next good journal, and got the paper accepted.
    There should be a quick screening by the editor (a few weeks at most), and if the paper is likely to be accepted, it should be sent for a review (that should not take more than two months).

    Reply
  15. Amanda

    While Chris’s suggestion might help, I want to stress that what I think is the biggest problem to blind review violation (and, at least the least, a significant issue) is not people googling papers but people already knowing who wrote the paper just by looking at it, because they had seen it before. Often people then write to the editor, tell them they know the author, and the editor says to review the paper anyway. I can’t think of any real way to address this issue, other than reviewers and editors turning down the option to have a non-blind review. But it already takes sooo long for papers to get reviewed, and eliminating all the times it isn’t really blind review would make the process even longer.

    Reply
  16. Nick

    1- Works well: I find it beneficial, as a reviewer, when the journal shares its verdict with me and also sends along the reports from other referees.
    2- Works poorly: turn-around time for verdicts.

    Reply
  17. Brandon Beasley

    Works well:
    When the time-to-decision is reasonable; when you get good, helpful, and charitable comments; when it’s clear your reviewers actually read and tried to understand your paper; then the system is pretty good!
    Works poorly:
    The frequency of the lack of some or all of the above. Decision times are too long. Reviewers who clearly did not really read or try to understand your paper rejecting it. You get no comments or perfunctory/worthless ones.
    Also, relatedly both good and bad: I think the desk rejection policy of lots of journals should be adopted more frequently. It’s so much better to get a “No, sorry, not right for us/not up to snuff” rejection in a week or two then to wait 3 months and get a rejection with worthless comments which show the reviewer(s) didn’t really even read the paper enough to engage with its main points.

    Reply

Leave a Reply

Discover more from The Philosophers' Cocoon

Subscribe now to keep reading and get access to the full archive.

Continue reading