Reading Time: 5 minutes
Photo by Denny Müller on Unsplash

In a provocative post on Scholarly Kitchen, Angela Cochran (VP of Publishing of one of the most prestigious scholarly societies, the American Society for Clinical Oncology) argues that societies should not be burdened with the additional expenditure of checking submitted articles for what are today called integrity errors – paper mills, evidence of plagiarism, and so on. Instead, she suggests that institutions should take responsibility. This could be described as passing the buck.

To explain her position, Cochran continues “forensic analysis of data sets and gel stains was never an expected task of traditional peer review” (and by implication, nor is checking for integrity issues). She justifies her position by stating that “journals are not equipped with their volunteer editors and reviewers, and non-subject matter expert staff to police the world’s scientific enterprise”. Well, if traditional peer review is no longer the  universal validator of the scholarly world, then what should be added to it? If volunteers and generalist journal staff cannot fix the problems, what should be done?

Is there a crisis of research integrity? I would certainly think so, when the national press runs major articles on the scale of the problem (for example, The Observer, London, February 2024, “Fake scientific papers push research credibility to crisis point”. In 2013, there were 1,000 article retractions; in 2023, there were 10,000. Cochran states this is “a relatively low number of incidents”, yet the reputation of one of the largest open-access publishers (Hindawi) has been destroyed as a result.

If we accept there is a problem, then there is the issue of who should be responsible for fixing it, and who should pay. David Crotty, editor of Scholarly Kitchen, repeatedly reminds us that adding more checks “makes publishing slower and more expensive”. It would be cheaper to eliminate all editing of papers. You could say that imposing speed limits on roads makes it more expensive to do business; but it is a limitation we (largely) accept. The analogy is not so trite, because while the majority of drivers drive within the speed limits, we still accept the need to police those who do not.

What is the issue?

  1. AI is blamed for reducing submission quality, but is this the case? There have been several articles recently (for example here) identifying to the author’s horror that AI was being used to generate and to check academic articles. Why not? If generative AI can produce a good basis for an abstract, why not use it? Surely anything that can improve quality is a good thing.
  2. The presence of obvious AI-generated wording (“Certainly, here is a possible introduction for your topic”) indicates that societies and publishers are failing to read submissions, which is a much more fundamental issue.  
  3. Peer review is a necessary but not sufficient component of research integrity. Peer reviewers can check the scholarly content, but not, for example, all the citations. Today, more and more authors are gaming the system, so checks have to be applied.

Who should pay for these checks?

Suggestions for solutions fall, fairly predictably, into camps. Cochran suggests that societies are hard-pressed, and that institutions should pay. Ten of the first 11 comments on her post had nothing but praise for the suggested solution, suggesting that the Scholarly Kitchen readership is firmly in the publisher camp. But societies (unlike institutions) are paid to publish articles; it is a reasonable expectation that editing (and checking) is part of the cost involved in publishing a paper.

The majority of institutions are not wealthy (leaving aside the handful of privately-supported prestigious schools) and are further challenged by reducing student numbers, and hence reduced income. Nonetheless, a fundamental purpose of the institution is to promote knowledge in a curated and responsible way.

Possible solutions

Most importantly, we need to acknowledge there is a crisis: we can no longer assume that a submitted paper is fit for publication, without running any checks. I am shocked by the complacency of many commentators (Cochran: “The vast, vast majority of papers submitted to the vast majority of journals are written by ethical and responsible researchers.”)

  1. The simplest remedy is to make sure the existing scholarly workflow functions as designed. Journal staff are responsible for reading and copy-editing the article, both before and after the author and peer reviewer have carried out their corrections. If they do their work properly, visible signs of AI tools would not appear in published articles.
  2. A check for “research integrity” should be expanded to cover not just consciously bad practice, such as falsification, but also unconscious errors in a manuscript, such as missing references, figures listed out of order, and so on. Most of these can be identified very quickly by mechanical means, and there is no reason for not using such tools. A machine can check within seconds that all references are cited in the article. The important principle is that we should accept the need for the machine to assist the scholarly process. This is not abnegating human responsibility; it is acknowledging that scholarship today is reliant on mechanical tools, for example for text and data mining and for discovery, which means that the academic record should be reliable, well-tagged, and hence capable of machine-based discovery and appraisal.
  3. Societies are gradually adopting machine-based checks on their submissions, to aid human evaluation, but this has taken several years. The use of machine learning, NLP and AI tools for checking submissions all have their place. Many of these checks can be carried out without a significant increase in editorial time.
  4. Authors should be using checks on their own articles, before submission; plenty of pre-submission checking tools exist. It would not be difficult to set up a certification process to indicate that an author submission has passed a number of basic checks. One benefit of checks at this stage is that they re-introduce a minimum level of checking for preprints: there is a worrying trend by which many authors use preprint servers not for preprints but for full publication. Obviously, such a process eliminates both the peer review and the checks that journal publishers carry out.
  5. Certainly, institutions can take action. The research office could take more responsibility for ensuring checks are carried out, even though academics are notoriously resistant to any additional steps that make the process of publication slower.  
  6. There should be more neutral assessments of AI submission tools, with independent critical evaluations – this is something libraries and societies can do. However, there is to date no compulsion in the use of such tools, so their adoption has been patchy.

I don’t claim that any of these solutions is perfect; but I think that a community-wide attempt to solve the problem, rather than simply passing responsibility to another agent in the workflow, is what is required to deal with the integrity crisis. And it may be time to begin looking at an element of compulsion, rather than simply requesting voluntary compliance with best standards. Maybe even speed cameras.