A new article by Johan Chu and James Evans (Slowed canonical progress in large fields of science” suggests some major problems in the current state of scientific research. The authors, from a business school and a department of sociology, claim there are too many articles being published in many well-explored areas, with the result that genuinely innovative papers are less noticed. Here are their claims:
- “too many papers published each year in a field can lead to stagnation … The deluge of new papers may deprive reviewers and readers the cognitive slack required to fully recognize and understand novel ideas”
- “Competition among many new ideas may prevent the gradual accumulation of focused attention on a promising new idea.”
- “When the number of papers published per year in a scientific field grows large, citations flow disproportionately to already well-cited papers.”
- “These findings suggest that the progress of large scientific fields may be slowed, trapped in existing canon.”
- “Policy measures shifting how scientific work is produced, disseminated, consumed, and rewarded may be called for to push fields into new, more fertile areas of study.”
I am no statistician, and I am not qualified to judge the methodology used by the authors to validate their claims. But I am not convinced, even from the authors’ own description of the data, that they have proved their point. I think the article is fundamentally an opinion piece, with some large-scale analysis attached to it.
While an impressive number of articles has been surveyed (more than 90 million), How can an analysis of a corpus of abstracts lead to the conclusion that “too many papers published each year can lead to stagnation”? Terms such as “stagnation” have echoes of gloomy decline-of-civilization ideas that have been around for many years (Jared Diamond is a recent example). The authors use highly emotive language that is unusual for an academic article (a “deluge of new articles leads to ossification of the canon”).
Let’s take the arguments in turn. If there are too many papers, researchers may not have the “cognitive slack” to recognize novel ideas. There is indeed a proliferation of papers, but then again, there are lot of books in any academic library, and does that prevent me thinking clearly and recognizing new ideas? Certainly not. Are the authors suggesting limiting the number of papers published? In their discussion, the authors seem to suggest it (more on this below).
The second point is: “Competition among many new ideas may prevent the gradual accumulation of focused attention on a promising new idea.” Such a statement presumes a model by which the brain is incapable of focusing because of competition in other areas? If I read about one idea 25 times, does it prevent me recognizing a new idea in the 26th paper? It sounds like folk wisdom. In any case, if there are too many papers presenting existing ideas, then new ideas should stand out all the more forcefully. “A novel idea that does not fit within extant schemas will be less likely to be published, read, or cited.” How could this be measured?
Thirdly, “When the number of papers published per year in a scientific field grows large, citations flow disproportionately to already well-cited papers.” Common sense suggests that highly cited papers are more likely to continue to be highly cited. If a new paper is published, the chances are that it will cite a paper that is more highly cited than less highly cited. The authors use the analogy of grains of sand; if many grains of sand are dropped, there is an avalanche of sand. Is this an appropriate image? It’s a very evocative image, but are academic papers really like grains of sand piled up on each other?
The authors try to demonstrate this idea in figure 3, which, they claim, shows “The probability of a newly published paper ever reaching the top 0.1% most cited in its field decreases when it is published in the same year as more papers in its field”. But surely it is self-evident that if ten papers are published on a single topic in a year, the probability of any one of those papers becoming highly cited is less than a the only paper published on a topic in a year. Figure 4 claims to show that when more papers are published, the most cited papers receive proportionally more citations. This is certainly a correlation, but is it a cause? Could it not be that as more scientific papers are published, new articles tend to be less about completely new ideas, but filling in the gaps around existing research? This would produce more citations to existing literature, but it does not mean that innovation is stifled, or that (to use the authors’ colourful language) the canon has “crystallized”. As the authors themselves state, “Most papers published in the same year as many others build on, rather than disrupt, existing literature“.
Now to the authors’ most startling claims: that science may be “trapped in the existing canon”. It’s a very evocative idea, with scientists struggling to emerge from the dead weight of existing knowledge, but is it true? How could you measure such a statement? The nature of scientific progress is always that, with hindsight, scientists appeared to be stuck in tradition, but is there any empirical way of demonstrating such a hypothesis?
Finally, “Policy measures shifting how scientific work is produced, disseminated, consumed, and rewarded may be called for to push fields into new, more fertile areas of study.” Are the authors seriously suggesting that authorities, presumably the research councils, should be pushing research into “more fertile areas of study”? I would have thought they already attempt to do this, insofar as it is possible. Research grants are awarded based on the grant-funding body identifying as best they can the most fertile areas for study. The authors of this study make no suggestion how things could be done differently.
The discussion section of the paper raises very questionable conclusions. “The more-is-better, quantity metric-driven nature of today’s scientific enterprise may ironically retard fundamental progress in the largest scientific fields.” The authors themselves have few practical suggestions other than “a clearer hierarchy of journals”, yet there is a pretty clear hierarchy at present, using the citation ranking of journals. Critics of impact factor for journal rankings usually call for a move away from a hierarchy of journals. It should not matter which journal your research is published in.
As an example of ossification of the canon, the authors mention a 1976 paper that has been the most-cited article in molecular biology for many years – but that is because the article is a recipe, that describes a particularly useful methodology for carrying out experiments. Citing this article does not represent ossification but simply acknowledges best practice.
The article concludes with an unverifiable and tendentious statement: “could we be missing fertile new paradigms because we are locked into overworked areas of study?” If you saw this claim in a newspaper, I don’t think you would imagine it was taken from a scientific paper.