Several days ago, I put up this informal survey to gauge readers' opinions about various features of peer review. I am going to leave the survey up so that people can still take it. But here are the results so far (n = 91), organized by what I will call the good, the ambivalent, and the bad.
As this is only an informal survey, the results should of course be taken with a grain of salt (and then some). Still, I found the results interesting and will be curious to hear readers' reactions.
The Good
64.1% of respondents agreed that anonymized review works well on balance, giving authors a fair chance (whereas only 23.07% disagreed).
An even larger proportion of respondents 76%(!) agreed with that the peer-review process tends to improve their work.
68.4% agreed that the peer-review process improves their knowledge of other work in their field.
Next, a full 83.3% of respondents either disagreed with or neither agreed nor disagreed with the item that journal editors desk-reject too often. This suggests, somewhat surprisingly (to me, at least), that most people don't think desk-rejection is overused. 
Let
Respondents also pretty overwhelmingly expressed support for triple-anonymization, with 67.1% of respondents at least somewhat agreeing and 53.16% strongly agreeing.
Finally, respondents overwhelmingly supported journals sharing referee reports with all referees, with 84.8% of respondents at least somewhat agreeing. I find this particularly interesting given how few journals seem to utilize this practice.
In sum, respondents were pretty positive about some features of peer-review. Let us now turn, however, to…
The Ambivalent
Despite the results reported above, only 37.08% of respondents at least somewhat agreed that they are satisfied with current pee-review practices, whereas 53.9% of respondents either somewhat disagreed or strongly disagreeing. This suggests a bit more dissatisfaction than satisfaction with current peer-review practices across respondents.
However, looking at the chart below, the results as a whole suggest to me some real ambivalence across respondents. For the results appear bi-modal, with roughly equal proportions somewhat agreeing (33.7%) and somewhat disagreeing (38.2%) with them. The results that tugged this item in the "bad" direction was the greater proportion of people strongly disagreeing (15.7%), compared to only 3.37% strongly agreeing.
Given the features of people rated positively above (viz. anonymization giving authors a fair chance), what are the sources of the moderate negativity / ambivalence we see in this item?
First, respondents appeared fairly ambivalent about the typical referee comments they receive. Although 50.6% of respondents at least somewhat agreed that their referee reports tend to be sensible and fair, a full 49.4% either disagreed or neither agreed nor disagreed.
A similar form of ambivalence was exhibited in response to the question of whether referee reports tend to be too perfunctory. Here 46.8% agreed with the item, but 53.16% either disagreed or neither agreed nor disagreed. In other words, quite a few respondents (just a little less than half) think overly perfunctory reports are a problem, but a little more than half do not.
People also seemed pretty ambivalent about whether too many bad papers are published. Roughly equal proportions at least somewhat agreed (39.7%) or somewhat disagreed (42.3%), with the remaining 18% neither agreeing nor disagreeing.
People were also fairly ambivalent about whether more should be done to ensure anonymity. Although 55.7% at least somewhat agreed, that's just a bit over half of all respondents, the others of whom either disagreed or neither agreed nor disagreed.
Respondents were also fairly ambivalent about revise-and-resubmits, with some agreeing that R&R's needlessly swamp the system but a bit larger proportions disagreeing:
Respondents were also a bit ambivalent over whether editors are too conservative. Although a 51.9% agreed that editors are too conservative (and 24% strongly agreed), the other 48% either somewhat disagreed or neither agreed nor disagreed:
Finally, respondents were fairly ambivalent on the whole to the item on whether a peer-review journal has 'lost' at least one of their submissions. Although a fairly disturbing proportion (30.4%) at least somewhat agreed with this item, nearly 70% disagreed or neither agreed nor disagreed.
Let us now turn to…
The Bad
The issue respondents focused on is not too surprising. A full 93.67% of respondents at least somewhat agreed that 'long and inconsistent turnaround times' are a problem (with 67% strongly agreeing):
A fairly large proportion of respondents (62.5%) also agreed that too many reviewer reports are incompetent, biased, or needlessly aggressive:
People also thought reviewers are too conservative philosophically (60.76% agreeing):
People also thought too many half-baked papers get sent out for review (62.8% agreeing):
Respondents also overwhelmingly agreed that journals could communicate with authors better (87.3% agree, 54.4% strongly agree):
Finally, respondents tended to agree more than disagree that the peer-review process is too opaque and should be more transparent (54.4% agreed, 29.11% strongly agreed, only 13.9% disagreeing)–though 31.6% neither agreed nor disagreed.
Again, these are only informal results. My main aim is to use them as a jumping off point to discuss these issues more in future posts in this series. But what does everyone think? Any reactions?
Leave a Reply to AndyCancel reply