Readers may recall that I have dedicated quite a bit of time to arguing for systemic changes to our peer-review process–specifically, for moving away from "anonymized review" in favor of the kind of "open online" peer-review model utilized in math and physics. Why I have I dedicated so much attention to this topic?
One set of reasons (the ones emphasized in the posts linked to above) is that I think such changes are likely to have substantial benefits for everyone–for authors, reviewers, editors, and the profession at large (by, among other things, improving the dissemination of new research). I have tried to draw attention to these benefits in large part because, or so I have argued, the oft-ascribed benefits of "anonymized" review are dubious and probably illusory ("anonymized" review really isn't in the digital age). Interestingly, Remco Heesen (Cambridge) shared a draft of a paper co-authored with Liam Bright (LSE) defending similar conclusions using empirical research:
We think there are a number of clear benefits to abolishing pre-publication peer review. In contrast, while various benefits of the existing system (downsides of abolishing peer review) have been suggested, we do not think there exist any that have clear empirical support. Insofar as empirical research exists, it is ambiguous in some cases, and speaks relatively clearly against the claimed benefit of the existing system in others…[Consequently], we claim that the present state of the evidence suggests that abolishing pre-publication peer review would lead to a Pareto-improvement: each factor considered is either neutral or favors our proposal. (p. 3, my emphasis)
However, the likely benefits alluded to above–to authors, editors, reviewers, and the profession–are not the only reasons that I think we should favor change. More deeply, I have long worried that "anonymized" review–far from ensuring fairness (as often claimed)–plausibly places authors at serious and undue career risks, particularly early-career and less-established authors. Allow me to explain.
In the comments section of my recent post on brutal reviews and rejection letters of famous works, Ingrid Robeyns (Utrecht) shared a remarkable paper, "How Are the Mighty Fallen: Rejected Classic Articles by Leading Economists." In my view, anyone interesting in defending "anonymized" review should read it, as I think (following Heesen and Bright's paper) it pretty thoroughly illustrates how illusory the supposed benefits of the peer-review model probably are (including how, even pre-digital age, "anonymized" review often wasn't, see pp. 172-3 in particular). However, by my lights perhaps the most striking thing about the paper is the ways it illustrates how "anonymized" review may routinely lead authors to face serious harms. Here are a few cases:
Early in his career, [Gary Becker] submitted what became "Competition and Democracy" to the Journal of Political Economy. "The then editor Earl Hamilton agreed to publish it. He eventually withdrew the commitment because of negative comments by Frank Knight, who was one of the people who refereed the paper. I still have a copy of Knight's referee report, and I cannot say that I am any more impressed by it now than at that time." Becker "became discouraged by the report and put the article away" until he finally published it in the Journal of Law and Economics several years later. However, by that time, other articles had been published that employed the same approach. (p. 169; my emphasis)
[L]ike Becker, Krugman notes that, even if another journal eventually prints a paper, the delay that initial rejection causes may permit others to beat the paper into the intellectual market. Krugman sent his "Target Zones and Exchange Rate Dynamics" to the Journal of Political Economy. "This time I got two favorable referee reports. The paper was nonetheless rejected … by [the referee] who thought that the paper was of 'insufficient general interest' for the JPE. The paper didn't come out (in the QJE) until August 1991. By that time the target zone literature, all of which made use of the techniques first introduced in my paper, had exploded, and consisted of at least a hundred published and unpublished pieces; in fact, I had to add a postscript to the QJE version referring to subsequent literature." (p. 170)
"Only after years of rejections by four journals did Brian Arthur's "Competing Technologies, Increasing Returns, and Lock-In by Historical Events" appear in the Economic Journal…The paper began a six-year odyssey. "First it was dismissed at AER in desultory fashion. Then I submitted it to QJE, and it was turned down there. Then because Clower had left AER I resubmitted it to AER. It underwent one refereeing go-round, followed by two appeals. Finally, two years after this second submission, AER turned it down again. … I then submitted it to EJ; and it got turned down. I appealed; and finally, in 1989, EJ published it…Like other authors, Arthur suggests that delay from the rejections threatened his ideas' currency. "Because papers based on mine had started to appear in the literature," referees told him that "the idea … is already recognized in the literature." (p. 173)
"Keynes rejected Roy Harrod's article that first sketched the marginal revenue curve. Although the Economic Journal finally published the article years later, Harrod felt that the delay in publication cost him credit for the new concept…Paul Samuelson remembers, "Roy Harrod went to his grave bitter because Maynard Keynes, absolute monarch at the Economic Journal, turned down his early breakthroughs in the economics of imperfect competition. Thus, Harrod was robbed of credit for the 'marginal revenue' nomenclature. All this was on the advice of Frank Ramsey, genius in logic and mathematics. " (p. 175)
To some readers, these kinds of concerns–about authors being deprived of credit or 'priority' for a scholarly idea–may seem petty. Shouldn't "the point" of scholarship not be so much a matter of "who gets the credit" but rather the ideas themselves, as it were? What does it matter who gets the credit for a given idea? These are reactions I have encountered fairly often. But let me try to explain–using a few real life stories I have heard before–why I think these reactions are normatively mistaken.
A few times over the years, I have heard variants of the following story. The story involves a graduate student or early career person working with a less-than-scrupulous (or perhaps unintentionally obtuse) well-established member of the profession–a faculty mentor or dissertation advisor, for instance. The early-career works tirelessly on developing an idea for publication, yet has difficulty publishing it. Then, a few years down the road, the more well-established person they worked with publishes something eerily similar, 'scooping' them as it were. And unfortunately, because the person is well-established (and may be in a position of professional power), the vulnerable person whose idea was plausibly 'scooped' has little recourse. They may feel powerless to even raise the notion that a professional and scholarly wrong might have been done. While I have no idea how common these types of cases are, I've heard of at least a few of them–and they seem to me the sort of thing that professional institutions and norms should guard against.
Here's another kind of case I have encountered seemingly much more often (and indeed, I have heard others remark on this too). A less-than-well-established scholar publishes Argument X in the literature, perhaps in a lower-ranking journal. Then, a few years later, a much more established scholar publishes a similar argument, perhaps in a more highly-regarded journal–without referencing the less-well-established scholar's work (perhaps entirely innocently, not knowing that work exists). Alas, in the subsequent literature, the well-established figure becomes associated and credited with X, and the less-established author's contribution ignored or suppressed (I have at least one now-well-established acquaintance in the profession who has mentioned suffering this fate). This too, I think, is the kind of thing professional institutions and norms should guard against.
One additional reason why these things should be guarded against, I think, is the anecdotal sense (though it is just that) these kinds of cases may have a tendency to disproportionately affect vulnerable or otherwise-marginalized members of the profession–individuals who, for one reason or another, feel or are powerless to rectify whatever wrong(s) they think may have occurred (and again, it is worth emphasizing that some or even many such wrong(s) may be completely inadvertent, such as someone genuinely not remembering that it was a student of theirs who initially thought of X several years ago, or someone not aware that Argument X existed in the literature because they simply missed the article where it originally appeared).
In any case, the points I would like to make are these. First, I do not think there is anything petty in worrying about these cases. Whether the publication of an idea is delayed for years by 'anonymized' review, such that a person is (either inadvertently or intentionally) 'scooped' or otherwise denied credit for a contribution: these things can have serious and lasting negative effects on a person's career (such as whether they get a job, get tenure, etc.). Second, for reasons noted above, it is plausibly the more vulnerable members of the profession who, due to their positions of professional vulnerability, probably disproportionately face these risks and harms. Finally–and perhaps most importantly–these are yet another set of problems that the "open" peer-review model in math and physics would plausibly address. For, as I have mentioned before, in that model (1) researchers post working drafts of articles on an online repository before sending them to journals, (2) people are expected to be aware of, discuss, and cite those articles (even before acceptance by a journal), and (3) public discussion prior can catch and correct scholarly errors (including matters relating to priority and appropriate credit in citations) before new works are placed under review or published. In the math and physics model, if a graduate student uploaded a paper on Argument X years ago, it would be possible and (judging from how things work in math and physics) incentivized for people to help ensure that work is appropriately credited.
These are, I believe, very important plausible benefits to be aware of. In addition to the other plausible benefits of an open peer-review model (and lack of adequate empirical support for the benefits of "anonymized" review), one thing we should want out of a publication system is to protect authors, ensuring they are appropriately recognized in the scholarly record–particularly when it is more vulnerable, less-established authors whose careers most lie in the balance.
Leave a Reply to Marcus ArvanCancel reply