The Philosophers' Cocoon

A propos our recent discussion of whether philosophy journals should have and enforce clear peer-reviewing standards, I came across this 2011 paper by Carole J. Lee and Christian D. Schunn comparing philosophy's peer-review practices to psychology's (thanks to Feminist Philosophers for drawing my attention to it). The paper contains the results of:

1. A study coding 423 reviews for 221 papers submitted to the 2007 Cognitive Science Society conference)–112 philosophy reviews of 53 philosophy papers and 311 psychology reviews of 168 psychology papers–for negative and positive comments, as well as for "inflammatory comments" (p. 358).
2. The results of surveys of psychology and philosophy journal editors (pp. 360-)

Although I am not in a position to say much about the studies' methodologies (though I would certainly invite those who can to comment!), here are some of Lee and Schunn's rather striking findings:

• Nearly 30% of philosophy reviews contained at least one inflammatory comment, compared to only about 15% of psychology reviews (p. 359).
• The total percentage of philosophy papers receiving inflammatory comments (13%) was also about twice that of psychology papers (6%) (ibid.)
• About three times as many philosophy papers contained multiple inflammatory comments (4.5%) as psychology papers (1.6%). (ibid.)
• 41% of philosophy papers were rejected, compared to only 20% of psychology submissions. (p. 360)
• "[In philosophy journals] A single negative review has more power than a single positive review on editorial determinations. About 40% of editors ‘‘never’’ or ‘‘rarely’’ accept a paper receiving a single negative review (of these editors, 80.0% report sometimes relying on a single review)." (ibid.)
• Philosophy journal rejection rates averaged 92%, compared to a mean of 78% for psychology journals and 20-40% for physical science journals. (pp. 360-1)
• "About 63% of general philosophy editors reported sometimes making determinations on the basis of a single review." (p. 357)
• "Approximately 25% of [philosophy] editors reported sometimes relying on three or more reviewers (though at least half of these editors remarked that this was not normally the case)"(ibid.)
• "In contrast, journals published by the American Psychological Association moved from a two-reviewer system in the 1950s to a three- to five-reviewer system in the 1990s." (ibid.)

Given that my wife works in psychology, I can personally attest to some of these differences. Thoughts?