The traditional measure of academic significance, invented way back in the 1960s, is the citation index. If a paper is cited, then it has significance for other researchers. If we count the citations, we get a rough idea of the paper’s importance. Despite many criticisms of this simplistic methodology (I discussed some of them here), the metric has persisted. This post looks at one possible indicator of the significance of an article (or book, for that matter).
Citations are binary measures: either an article is cited, or it is not. It is well known that negative citations are counted, just like positive citations, initially because at the time there was no machine-based tool that could distinguish a positive from a negative citation. Today, there are several tools that can provide a reliable distinction: scite.ai is one of them, Scholarcy another.
But there are more valuable insights that can be gained from references to other works. You could simply look for recommendations, like the glowing comments attached to best-selling works – if so many people like the book surely it must be good?
Of course, we all know that the ecstatic, glowing praise accorded to a book we glance at in a bookstore may not be entirely disinterested. The term “bestseller” appears with such frequency that it is difficult to imagine any book not being a bestseller in one category or another. Nonetheless, we do take praise seriously, if we believe the source is credible. So while we might not take adulation on a book jacket very seriously, we might take notice of praise in the context of a scholarly article, or a respected reviewing journal. And perhaps, if it is Daniel Kahneman saying that this book changed the world, we are quite likely to believe him.
Is there a more reliable way of detecting significance for academic content? Here is one example. In a review of two books about work and its meaning for humans (the TLS dated May 27 2022, a review of The Story of Work, by Jan Lucassen):
Less satisfactory is a full section of one chapter devoted to “The Original Affluent Society”, Marshall Sahlin’s essay on hunter-gatherers … This attention to Sahlin’s essay makes sense, given that it is one of the most important and widely read cross-cultural studies of work ever produced.
Later in the review, the reviewer, Daniel Segal, notes another recent book that quotes from the same article by Sahlin:
Whereas Lucassen devotes one section of a chapter to Sahlin’s famous essay, Suzman provides an entire chapter entitled “The Original Affluent Society” …
I had never heard of this essay, but clearly two books about the world of work both devote space to the article, and the reviewer calls it “one of the most important … studies of work ever produced”. The reviewer helpfully continues to provide the main argument of this famous paper:
Sahlin’s argument, to the effect that our modern predicament of finding our means or resources insufficient for our wants – driving us to work and work and work, and driving planet-destroying “growth” – is not a universal aspect of human existence.
By reading this review, we have identified a classic paper in one domain, that has clearly been acknowledged as a key source for two recent works on the subject, and which also shows that the paper is still relevant for the contemporary debate.
There are other indicators of this paper’s classic status, first published 1966. There is a Wikipedia article entitled “Original Affluent Society”, and the entry for Marshall Sahlin states that this is his “most famous” essay.
Of course there will be no general agreement of the argument, but such statements, combined with extensive citations, establish the article’s importance. Could we identify an algorithm that would capture such significance? If we could, we would move closer to the heart of the academic study of humanities. How many such papers are there? It is tempting to imagine there is a finite number, and that they could all be identified. The gems of human knowledge would be captured for all time, the most significant statements. It’s a lovely idea, to capture the indicators of significance for us. But perhaps it is a slight paradox that we struggle to capture every great idea, when Sahlin’s original article suggested there was a time in history when we didn’t feel the need to try to do everything.