I’ve previously discussed how frustrating confused referee reports can be for the author, and how the system might actually be made more efficient by allowing authors to (briefly!) respond to these reports before a verdict is reached. But I think there’s a more systematic problem, in that too many referees (seemingly) base their verdicts on bad criteria, such as whether they can think of an objection to the paper. (One otherwise-brilliant philosopher once told me that he has a deliberate policy of rejecting any paper that he disagrees with! Few would explicitly endorse this, I imagine, but many more may follow a similar rule de facto.) So I’ve been wondering what steps a journal editor could feasibly take to try to counteract this. In particular, are there particular questions that it would be worth asking referees to explicitly address in their report, that would better reveal the truth about a paper’s merits?I’d be curious to hear what others come up with. But here’s an initial stab at what I think a report should ideally address:(1a) What (if anything) is interesting and original about this paper?(1b) On a scale of 1 – 10, rate how interesting you expect this paper should be to those familiar with the existing literature on the topic.(2) Are there any egregious errors or oversights that would need to be addressed before the paper was potentially publishable?(3a) How cogent are the paper’s central arguments?(3b) Do you expect most other experts would share your verdict, or is there significant room for reasonable disagreement?(3c) On a scale of 1 – 10, rate how insightful or illuminating you expect the average reader of the journal would find this paper.* * *My own view is that (with some wonderful exceptions) referee reports in philosophy tend to systematically overweight (often idiosyncratic) judgments about whether the argument is successful (3a). But obviously, the mere fact that somebody could object to. . .
News source: Philosophy, et cetera