The Philosophers' Cocoon

I was recently talking to a philosopher who had to wait 17 months for an initial decision on her paper at a top journal. Another had to wait over 12 months and has now retracted her paper after the editor remained unresponsive to her inquiries on the status of her paper. In both cases, the slow journals were in the top 5. I have had many discussion with people about how awful reviewers are, how they systematically misunderstand the points we make, or try to get citations to their own work into our papers.

Here's a puzzle: Why do so many people (me included) think they are conscientious, timely referees with constructive feedback, yet do we end up having our papers reviewed by tardy, careless, uncharitable referees? Since we have no reason to assume we're special, something is going on. It might be that we overestimate our own quickness and constructiveness. Also, we might underestimate the value of reviewer comments we receive. This seems likely given that me-side bias (the bias whereby we systematically underestimate the strength of views not in favor of our prior beliefs) is not attenuated in highly educated people. 

Looking over my CV in the section "services to the profession" (which is pretty useless for career purposes), I see that I have reviewed for at least 24 journals (not all of them in philosophy), and 3 academic book publishers. Most of these requests have come in over the past few years. I have adopted a set of practices that I find useful in keeping reviewing manageable and that hopefully maximizes the benefit my reviewees get (see also here, Thom Brooks's guidelines, some of which converge with the practices outlined here)

1. When I get a request to review a paper, I accept or decline the request on the same day. When declining, I try give names of alternative reviewers. Apparently, many editors have to wait up to a week for a decision on this; in this way, valuable time is lost, especially if the hesistant prospective referee declines, and one needs to find someone else. I think this might be because we put off the decision (not wanting to be uncollegial, but not wanting to do it either).

2. To aid my decision, I immediately decline if
   2a. I don't have any time to do the review in the next 4 weeks.
   2b. I already have a review to perform, i.e., no more than one review task at the same time.
   2c. The paper is on something I would need to do extra research on to give an informed decision on its merits. This seems also fair to the author. So even if it's a paper in my AOS (philosophy of cognitive science), if it's something I'm not up to speed about, I say no. 
   2d. I am certain who the author is. I think we are less good at guessing authors' identities than we think, therefore a suspicion is not enough. With certain I mean, for instance, that I saw the author present the paper. Even in those instances, editors have asked me to review anyway since it is a very small field and I am one of the few people who could give a verdict. If that is the case, I accept the review request since I guess it's worse not finding any referees for your paper than having someone review your work who knows who you are.
   2e. For commercial publishers that impose heavy embargoes and demand high subscription prices for journal packages, like Springer and Elsevier, I cap my reviews at 3 per year per publisher. I'd love to support open access journals (unfortunately, that's not for me to choose).

3. The anonymity of the author: There has recently been discussion on unethical practices of referees (e.g., Googling titles) that might aggravate biases against some authors, especially junior people, people from smaller institutions, women and other minorities. Even when you don't google the title, the anonymity of the author is not guaranteed. When reviewing a paper, all sorts of ideas about the author involuntarily pop into my mind. I of course do not try to find out the author's identity but I found myself thinking, one of the last times I reviewed a paper, "Is the author a graduate student?" This is very troubling, since I would not consciously take this as a reason to reject a paper. But implicit biases are legion. For instance, if one guesses a paper might be written by a grad student because of its very lengthy and thorough review of the literature (older authors, so is my impression, are more succinct in their attribution of sources). Or perhaps the language suggests the author isn't a native English speaker.  So what to do here? I have no easy answers. I try to be aware of such thoughts, especially on how they might affect my decision process, and focus on the quality of the paper, but I am not sure what additional steps I could take. Hopefully Cocooners have useful suggestions? I think implicit bias is a huge problem in philosophy. 

4. My own anonymity: I usually remain anonymous, given that rejection is the most common decision and one doesn't want to make enemies for life. A few years ago, I refereed for Frontiers, which has the policy of publishing the names of referees alongside accepted papers – this way, the author finds out who the referees are *if the paper is accepted*. This is very good policy – if you know your name might appear eventually, it makes one more careful as reviewer. So I've recently taken to signing my name under comments when the paper is at its second round of revisions, i.e., when I get revisions back from author, and provide my thoughts on the revised version.  I hope by revealing my identity to the author at some point in the process I would do a better job at reviewing.

5. What is publishable? I have recently adopted a policy of positively recommending papers that make an original point, even if there are some flaws, as long as the overall structure of the argument is sound. By contrast, I tend to recommend rejection for papers that seem to meticulously pre-empty any concerns but that only have an incremental value in the debate they are situated in. Journal publishing is after all a zero-sum game, and in spite of low acceptance rates, I find there is too much incremental stuff in the major philosophy journals. There is scope for disagreement here, but reviewing gives one the opportunity to put forward what one thinks are important desiderata for the profession (and I prize innovation over sudoku-philosophy).

6. Reviewing time. I try to keep reviewing time under 4 weeks, but it is unfortunately not always feasible. So I take the journal's allotted review time as a final deadline for a paper to review. When my own writing goes well, I prioritize that. If I need to give it a rest, reviewing is a nice way to concentrate on something else (it's not the same as grading student papers or reading papers for research, and it feels like one has done something substantial at the end of the day).

7. Communicating a verdict: Years ago, I made the mistake of being invariably too nice in my reviewer reports (inspired by my own reactions to devastating review reports). So I reserved my most devastating objections to the confidential comments to the editor. The problem then was that authors got a report with lots of constructive feedback but no clear reason for why their paper was rejected. So I now try to avoid confidential comments to editors (exceptions would be if I found evidence of plagiarism or the like), in this way the author has a clear idea of my assessment of strengths and weaknesses of the paper.