In my post sharing the results of the informal survey I ran on peer-review, I reported that a full 93.67% of respondents at least somewhat agreed with the statement that journal turnaround times are too long and inconsistent (with 67% strongly agreeing). Given that so many people seem to think this is a serious problem, I think it could be good to have a discussion about what could be done to best address it.
In order to think about how journals might address the issue, we need to know its causes. From what I have heard on social media (including remarks from various editors), long and inconsistent turnaround times appear to have the following two primary sources:
- Difficulty and delays in finding willing reviewers.
- Problems with unresponsive reviewers who fail to get their reviews in or respond to reminders.
Oftentimes, when I've heard (1) discussed on social media, it has involved editors imploring people to respond to review requests more quickly (like, within 24 hours). However, imploring people seems to me likely to be an ineffective solution–and, judging from people's responses to the survey, it has been pretty ineffective in practice. No, effective systems of compliance don't just implore people to do the right thing: they incentivize it. Moreover, good systems are also efficient. Instead of looking haphazardly for people who might say yes (but just as well might say no), an efficient system would also drive editors toward people likely to say 'yes.' All of which raises a first question: how might journals most efficiently (A) find competent reviewers likely to accept an assignment request, and (B) incentivize quick answers to review requests?
I'd like to brainstorm answers to this question, and am curious what readers think…
Here's a thought that just occurred to me. What if there were a central system–a "reviewer bank" (say, at philpapers/philpeople)–that all journal editors might use to solicit reviewers. That central system could organize potential reviewers by topic, making easier for journals to find reviewers. But, more importantly, it might rank reviewers by how quickly they respond to review requests, as well as how often they say 'yes' as opposed to 'no.' Such a system–if it aggregated those two things into a single Reviewer Score–might effectively drive editors to people who are likely to say yes, and do so quickly. "Okay", you say, "but that system would probably lead editors to always ask the same people highest in the list, which would inevitably lower their score because they would have to start saying no." Indeed, but I wonder whether this might be addressed by building a component into the system that both incentivizes saying yes, but also gives editors reasons in some cases to explicitly select reviewers lower down in the list. Allow me to explain.
Suppose that once the central system were set up, journal editors adopted a Matching System of matching authors with 'reviewers like them.' For example, suppose you are a reviewer who takes forever to respond to reviewer requests, and when you do, you say no the vast majority of the time–giving you a low Reviewer Score. Now suppose you have a low Reviewer Score and you submit a paper to a journal for review as an author! Instead of asking reviewers high in the list (i.e. those with good Reviewer Scores), the editor would instead reach out to someone with a low Reviewer Score just like you, giving you the kind of reviewer you are. Here's my thought: this would incentivize you to be a better reviewer, because as a Bad Reviewer you would be matched with Bad Reviewers yourself. This would incentivize everyone to respond to review requests more quickly, and say yes more often–or at least it would for those who want to have a good experience as authors!
Interestingly, I think this sort of system might also address problem (2): irresponsible reviewers who miss deadlines and are unresponsive to reminders. For notice: this too might be a component of your Reviewer Score (along with how long you take to respond to initial queries, how often you accept assignments, etc.). On this model, if you have a bad Reviewer Score, you would once again be matched as an author with bad reviewers–people who take forever to review your paper. Conversely, if you have a good Reviewer Score, editors would preferentially match you with good reviewers: people who get their reviews in quickly.
Maybe I'm missing something–but it seems to me that this kind of central system, if put in place (perhaps by philpapers) and utilized by editors properly, would dramatically incentivize better reviewer behavior at all stages, reducing the time that people take to respond to requests, how often they say yes, and how long they take to review papers. For it would attach real consequences to these things, improving one's experience as an author for being a Good Reviewer and disincentivizing being a Bad Reviewer.
Anyway, what do you all think? Is my proposal (which, to be frank, I just put together on the fly) a good one? Or, do you think there's a better way for journals to address these issues? I'm really curious to hear your thoughts!
Leave a Reply