In the comments section of our newest "how can we help you?" post, a reader 'NK' writes:
I think it would be helpful to hear from those who've done some refereeing about how they approach the task. For early-career people like me, that would help both with refereeing, which we're just starting to do, and with publication, since it would give some insight into what referees are thinking.
But I'm thinking it would be especially helpful if the focus were on the kinds of things you ask yourself about papers you review, the kinds of things you remind yourself to do when reviewing, and so on. (For example, how do you figure out whether the fact that *you* disagree with something an author says constitutes a [perhaps defeasible] reason to reject?)
This is a really great query. When I was first struggling to learn how to publish, I felt like I had no clue what reviewers were looking for–and I suspect a lot of early-career people are in a similar boat. So, then, when you're reviewing for journals:
- How do you approach your task?
- What do you look for in a paper (viz. making a recommendation, such as "accept" or "reject")?
- How do you figure out whether something you "disagree" with is a reason to reject?
The last of these questions, in particular, raises some interesting questions that I've wondered about for a while–among them, whether and to what extent it is "permissible" for referees to use their own particular conception of what constitutes "good philosophy" or "good methodology" as a basis for making a recommendation.
For instance, suppose you believe–as I very much do believe–that some fairly standard methods used by philosophers today (appeals to intuition, reflective equilibrium, etc.) are demonstrably problematic, while there are better methods available that philosophers should use instead. At what point does it become permissible for a reviewer to say in a review, "You know what, I realize everyone uses these methods–but they've been demonstrated to be seriously problematic, so this paper should not be accepted"? Surely the answer to this question cannot be: "never." For, if that were the case, then our entire field could never impose methodological constraints on what is considered fit for publication–which cannot be right. Must one then wait for a consensus to arise, then? Why? It's called "peer review" not "group think"! If, as a reviewer–as a peer–why shouldn't I be free to make whatever kind of recommendation I like, provided I make it clear to the editor where I'm coming from (whose job it then is to decide whether my rationale is persuasive)?
Just to be clear: I don't think I've ever done anything like this! While it is something that I've thought about (which is why I raise it now!), for my part I think I'm probably less harsh than many reviewers are. Which brings my to my own set of answers to NK's questions:
- How do I approach my task as a reviewer?: The honest (albeit probably disappointing) answer is that I try to just read the paper and have an open mind! I sometimes get the impression that when I submit papers to journals, reviewers are looking for "reasons to reject." This isn't my attitude at all. I'm hoping to read something good, and all things being equal, I'm looking to see whether the paper–either in its current form or revised form–would be worth accepting. I just read the paper then and figure out what I think, and always write up a very detailed account (between 1-3 pages single-spaced) briefly summarizing (A) what the author's thesis is, (B) how they defend it, (C) what I think works, (D) what I think doesn't, and finally, (E) what my editorial recommendation is.
- What do you look for in a paper (viz. making a recommendation, such as "accept" or "reject")?: The first thing I guess I look for is a clear thesis, and well, an argument expressed clearly enough that it is understandable. A second thing I think I really look for is originality. While I have recommended "accept" verdicts for a fair share of papers that just "move the ball a few feet down the field" (viz. some small philosophical point), as a philosopher I'm really looking for something new and original that is likely to open up interesting new avenues of discussion (rather than just be the Nth+1 paper on topic X). This is especially the case when I review for prestigious journals like Mind or whatever–which, as prestigious journals, I think should be publishing groundbreaking work, not just sound, technically proficient arguments. Indeed, on that note–and I suspect I'm in the minority here, given what appears in the pages of many journals–I tend to not be all that impressed with technical/formal complexity. Although I absolutely don't hold formal methods against philosophers, my general sense–which was also drilled into my head decades ago by my undergrad advisor Dan Dennett–is that formal stuff is often unnecessary. Finally, of course, I care about the argument! I probably don't care as much as many reviewers whether an argument is "air-tight." The way I see it, if the greatest philosophical minds in history couldn't put together air-tight arguments, it's a bit odd to insist that we do! Instead what I think I look for is an argument's being "good enough", whatever that means–but roughly, I think it means no clear fallacies, the argument being one that I take seriously, and one that I think readers will take seriously and find interesting, even if there are some problems. (On that note, I often recommend revise-and-resubmits for promising papers that have, in my judgment, serious problems that must be addressed for the argument to be "good enough." As a side-note, this is why as an author I basically try to preempt every possible objection or misinterpretation reviewers may come to–as in my experience, if there is any possible way a reviewer could misinterpret your paper, there's a pretty good chance they will. This is, I think, one of the more important things I learned in finally learning how to publish).
- How do you figure out whether something you "disagree" with is a reason to reject?: I never reject papers because I disagree with the author (or, at least I don't think I do). Instead, I read the paper and put together an argument: either for acceptance, rejection, or revise-and-resubmit. And I do it just like I do any other kind of philosophy: I make a case. If I can make a strong argument to the editor that something about the paper would not make it worth readers' time–even with substantial revisions–then that thing, whatever it is, is my reason for suggesting rejection. I can't pretend to always get things right here, any more than I can realistically claim to do philosophy right all the time more generally. What I do feel comfortable saying is that I give it my best shot, and think that as reviewers we owe that to authors: an argued case for an editorial recommendation. Honestly, I cannot tell you how many times I have submitted a paper to a journal (typically, in my experience, a very highly-ranked one) and then waited 5-6 months only to receive a vague, one or two-sentence brush-off like, "This paper is too ambitious and doesn't fulfill its many ambitions." That's just not good enough, and I have no idea how some editors think it is appropriate to let reviewers make recommendations like that.
Anyway, these are my answers to NK's queries. What are your answers?
Leave a Reply to HelenCancel reply