A reviewer examines a grant proposal (public domain)

How to choose a reviewer for an academic article or a grant proposal? Adam Day raises an interesting idea in his article about peer review, in a recent post on Medium.

He describes how he has built an AI-based system that matches a submission with other papers based on content (a methodology also used by UNSILO). But, he argues, what if I write a paper, and my submission has some crackpot idea about (say) injecting disinfectant to prevent infection by coronavirus? Won’t the AI system simply come up with someone else who has the same potty idea? The craziness would be endorsed rather than critiqued. As a result, claims Mr Day, we can’t rely on high accuracy alone for success in referee recommendations.

I’m not so sure about this. It seems to me that much of research, not just in science but in the humanities as well, is based around researchers, or groups of researchers, having opposing, or at least incompatible, ideas around a subject. I wrote in an earlier post, for example, about the two schools of AI thought: those who believe in a general, all-purpose AI, and those who believe in “narrow” AI solutions, often based around neural networks. While these two schools do not directly contradict each other (they view things from a very different standpoint), you can be pretty sure that if you are from one school, and you submit your paper for peer review by a member of the other school, the resulting review will not be particularly favourable. It’s important to realise this isn’t just, in other words, good science versus bad science: research is more sophisticated than that, and while peer review aims at eliminating poor reasoning, it is frequently examining contrasting hypotheses that may both have some validity; you realise there may not be a single truth that, once discovered, is adhered to by all. There are many examples in the humanities as well: I wrote here about differing interpretations of the English Revolution, with a Marxist approach emphasising class relations, a school grouped around Christopher Hill, and a revisionist school, centred on Blair Worden. Unsurprisingly, perhaps, a review by Worden of a book based in the other camp resulted in a very vitriolic and not, I think, a very helpful review.

Perhaps in acknowledgement of this common situation, authors are often invited when they submit a paper to state anyone who they particularly want not to review their paper. This isn’t just favouritism and personal preference, so that friends give each other good reviews (or you hope it isn’t like this). It is more that many papers consider a hypothesis within one of the currently accepted paradigms, and a good review will assess what that paper achieves within the paradigm, rather than complaining that it has not challenged that paradigm.

Peer review cannot achieve magic; the system we work with is based around small increments of knowledge (for the most part) within an existing paradigm. Trying to resolve all differences of opinion within the peer review system is unlikely to be successful. I would suggest that content matching is as good as any other system as a starting point for finding peer reviewers, with human judgement and experience applied to fine-tune the final selection.