Ghost references are links that have not been validated. If an author cites an article, then many tools exist to check that link and to turn it into a hyperlink (if one doesn’t already exist). But, for whatever reason, many citation links fail – a ghost reference.
GenAI gets a very bad press for making things up, and one of the first changes I noticed when using it from the first few months was the move to providing sources and citations for any statements. A brilliant piece of sleuthing by Aaron Tay, described in a post, demonstrates how genAI may not be as culpable as many people claim when it displays citations to non-existent papers.
Of course, incorrect citations predate genAI – they have existed ever since the citation mechanism began. The problem is that AI sees a ghost reference and assumes it is correct.
Tay is clear where the blame lies:
ghost references are not primarily an LLM/GenAI phenomenon. They have always existed, propagated by entirely human mechanisms—typos, careless copying, citing without reading.
When, for example, a new article is uploaded to Google Scholar, it attempts to match the references to articles already in its database. What if the attempt is unsuccessful? There could be several reasons, such a misspelling in the reference, or it could be that the article cited is not in Google’s database. Either way, Google Scholar cannot be sure. It could mark as invalid any non-matching citation, but that would mean a huge number of missing links, many of which could be resolved fairly easily (such as matching Thomas Jones to Jones, Thomas). Google, like most aggregators, uses a set of probability-based algorithms to find a match. These algorithms, of course, will not be perfect, and if they don’t find a match, then Google adds a reference to [CITATION].
Why does the problem happen? It is an inevitable consequence of trying to adapt a human construction (academic articles) into a machine-readable database. We have set up a system – and this dates right back to Eugene Garfield – to use machines to aid the human process of scholarly review, and even though the machines might work perfectly, the corpus they are using is riddled with errors . Once you start using machines, errors are perpetuated and often become more visible.
What can we do about it? Well, the algorithms have got better over the years at attempting to interpret human inconsistency and many simple errors, just as your word processor has become increasingly able to correct your spelling for you. But a machine can only do so much; for a problem of this kind, there is a need for some human input as well. Some checks are done by scholarly publishers, although much more could be done, since even articles published in 2025 have ghost references. Authors themselves could carry out checks, but very few in practice check their own articles to see that a machine can correctly read their references.
What can we do about it? This is somewhat similar to the crisis of research integrity, which resulted in many of the interested parties (societies, publishers, libraries) all claiming they didn’t have the resources to run the additional checks required to ensure the validity and accuracy of the content.
Who is responsible for the quality of links? Google Scholar, OpenAlex, and all the other aggregators are not in the position (nor can they be) of checking the validity of every link they harvest. But there is one part of the scholarly publishing chain that is already doing this work, where checks are carried out on a continuing basis. Pretty much every university has an institutional repository, and their staff spend many hours checking and amending just these kinds of link. What is more, some of the better-funded repositories spend a lot of time and money building better algorithms to reduce the human checking required – not replacing, but making the human input more effective. Perhaps this is something the scholarly community (I have to keep reminding myself it is a community, with a shared interest in getting the corpus right) could agree to fund. Unlikely, but much needed.
Leave a Reply