The problem is a common one. What decision should we take? It could be a business decision, a peer-review decision, even a household decision (should we move house or stay put?). Whatever the case, there are several people involved, and the solution is not an obvious one. How do you reach a consensus? Is it different for research questions? And can you involve AI in that decision?
Consensus-making is most noticeable where research overlaps with policy, for example with decisions on best treatments, usually determined via systematic reviews. The Delphi method, developed during the 1950s, is common for reaching consensus in health and commercial projects. With this method, experts are invited to give their opinion, in two or more rounds, with a facilitator summarising their arguments between each round. The Wikipedia article describes it as being used “in areas where precise scientific laws have not been established yet”, but it’s not quite so simple. Science is not just waiting for a law that we all agree on, then moving on. The progress of AI is a case in point. Avenues of research, such as symbolic reasoning, were pursued and then abandoned, not because they were right or wrong, but because interest shifted elsewhere.
How does the scholarly community define consensus? The triumph of Eugene Garfield, back in the 1960s, was to create a machine-calculable definition of the importance of a paper: the number of citations. The fact that one article was cited many times more than another doesn’t imply that the science in the most-cited article is correct. Nonetheless, the implication behind Garfield’s idea, which is still widely used today, is a kind of power of crowds. Lots of citations for an article suggests the science is right. When you think about it, it’s an odd metric to use.
Compare that with peer review for academic articles. Peer review, and citations, act to try to provide a consensus where none exists otherwise. An essential component of the Delphi method is anonymity, and anonymity has been the norm in peer review for many years: specifically, the gold standard was the double-blind review, by which the reviewer did not know the identity of the author and the author did not know the identity of the reviewer. Two (or even more) reviews were often solicited for one manuscript. Today, there is increasing use of public review, and often, a post-publication review. That would at least be closer to consensus than an anonymous peer review, which in some ways is similar to the academic grant application – a decision with little or no consensus. You may be successful, or you may be rejected, but the decision-making process is usually arbitrary (as far as you are concerned) and the details of the judgment are not made clear. With peer review, the reviewer’s reasoning may be stated, but you do not usually enter into a discussion about it.
Wikipedia itself is a consensus. There will never be full agreement about the interpretation of events, so Wikipedia resolves it by repeated edits by often warring editors, although the only consensus is the article you read, which does not betray the struggles that take place behind the scenes. For the most part, what appears is the result of the latest intervention, right or wrong. In a few cases, if there are still disputes, it can come down to what Wikipedia terms “consensus decision-making”, similar to voting at an election: the view with the highest votes wins. Nonetheless, Wikipedia states that although “anyone can be an editor, Wikipedia’s community processes do not make it … a democracy”.
Where there is ongoing disagreement, Wikipedia uses a Requests for Comment (RfC) process. Contributors state their view in a forum, and other editors add their comments, before a neutral volunteer editor makes a final decision. However, as many as one-third of disputes remain unresolved, according to a 2018 study by MIT.
Of course, in politics, there is, instead of consensus, the device of an election. If there is unlikely to be unanimity or agreement, then an election at least provides a way forward; it is, for better or worse, an imperfect way of translating a human conflict into a machine-readable (and actionable) decision.
Citizen Juries are another decision-making methodology, Around 12 to 24 citizens, chosen to represent a relevant stratigraphic spread, are assembled for a number of days; they then reach a collective decision. During the meeting they hear experts presenting various views. The idea behind it is that decisions are better when informed. Perhaps the same methodology could be adapted for research questions. Imagine an academic conference with several presentations on an area of dispute. There is a degree of randomness about the participants, although by no means comprehensive. By the end of the conference, the participants are a kind of citizen jury; they have had informed presentations and should be able to make a collective, informed decision. If the presentations were not balanced, you could perhaps conclude it wasn’t a very well organized conference. Would science proceed better via “citizen conferences”?
Leave a Reply