Reading Time: 4 minutes

How do you measure the success of a scholarly article? Many writers disagree that citations should be the only measurement, for example, Bornmann 2017, even if at present the number of citations is the agreed independent metric for scholarly research.

A fascinating article by Toby Green examines three of his own articles, and to what extent he was able to enhance their dissemination. One thing he didn’t point out, of course, was that his articles were not primarily original pieces of research; they weren’t describing a new way of treating kidney disease, or lung cancer, but polemical pieces engaged in the debate about open-access content. Their goal is to persuade, to encourage the move towards open-access publication. So you could say that one measure of their success would be if open-access became more widespread after they had been published. Of course, it is impossible to state the extent to which one or two articles contributed to this goal. But we need to remember that “success” is here implicitly defined as reads and downloads of these specific articles. For a scholarly article about kidney disease, the number of citations might well be the most effective measure.

Green refers to the well-known purchase funnel used to measure the effectiveness of websites. Read any book on web analytics, and the purchase funnel figures prominently. Unfortunately, this model does not work so well for websites where no purchase is involved. In fact one of the limitations of a world obsessed with start-ups and making money from websites is that there is so little agreement on what constitutes success for a web page if doesn’t involve any kind of e-commerce or transaction. Is it a success if you just read the text on the page? Is it a success if you click and download something? Is it a success if you spend more time on the site?

Green’s version of the funnel has the ultimate goal of the article being read. He created a fascinating graphic to show the process:

Figure 1 Toby Green’s scholarly article funnel

One problem is that while “buy” is an unambiguous action from a website, “read” is nowhere near so simple.

What did I notice? Interestingly, Green describes how he makes his content prominent by a tactic he calls “riding the wave” – identifying a related paper, event, or discussion, and linking it to his paper. In this case, he gained readers by effective tweeting from a conference that he didn’t attend (the COASP conference), Green was able to link his paper to an effective slide that had been presented at the conference. I tend to concentrate on conferences where I am attending in person, and to ignore conferences where I am not present.

Green presents a rather bewildering mass of data that was not always very clear. What had really made an impact? My conclusions are slightly different to Green’s:

  1. With a lot of detective work, Green was able to identify some specific examples of how his article had been noticed. But it required quite a bit of digging, and for most authors spending 50% of the writing time promoting an article is 50% too much.
  2. Use listservs, at least, the library listservs (there are thousands of listservs, but I assume Green uses the major academic library listservs). Posting to two listservs generated around twenty times more downloads than posting to Facebook, Twitter and LinkedIn combined – a remarkable statistic. Nonetheless, I am reluctant to use listservs, just as I am reluctant to use Twitter. I don’t like any service that delivers messages to me without any curation. There is very little to prevent an individual posting tens, hundreds of tweets on the same subject; likewise listservs.
  3. Find influencers: easier said than done. Green identified some influencers after the event, but you can’t just approach, say, TechCrunch, and ask them nicely to promote your article. You hope that influencers will find you.
  4. Counting citations is surprisingly difficult, and perhaps for Green’s purpose, a polemical cause, not very useful. Green reports between seven and 14 citations for one of his articles – you would think a citation would be unambiguous. Whatever the case, it is clear that a figure around 10 is almost trivial compared to 70,000 reported downloads of his paper.  I would argue that for Green’s purpose, downloads are far more important than citations.
  5. Perhaps the most telling statistics are the comparative download of articles from one issue of Learned Publishing, the journal where one of his articles appeared. Clearly, some articles are noticed more than others, and this statistic (he is diplomatic enough not to name all the other articles) is of great value to the journal editor; and does provide, for the author, some idea of which topics people find interesting. Even if it is not great literature, many more people read Agatha Christie than, say, Hazlitt (to use David Lodge’s example from Small World). That doesn’t mean one is better than the other.
  6. When I look at read statistics for my own blog posts, the results seem to be almost random. Some blog posts get ten times more reads than others, for reasons that do not seem to be connected with the quality of the post. A few of my articles get cited. But, to be honest, I’m not that bothered.