You don’t need telling that there is a multiplicity of views on AI in scholarly publishing. The annual NEC Publishing Conference (London, 5 November 2024) was no exception. Among the many statements about AI I heard were:
- AI – parlour trick or profit generator?”
- “Our strategy is to ride the wave …”
- “85% of AI projects fail to meet their objectives”
- “AI is the great equalizer – we ALL know nothing about it”
- “99% of AI apps are not ready”
- “AI is a “no-man’s land” – would I want my grandma to be affected by this technology?”
- “AI isn’t normal software”
- “If we behave ethically to AI, maybe one day it will behave ethically towards us”
- “AI – It’s just a slightly different search box”
- “I’m unable to process that response”.
That last comment was the title of a presentation, which I think was an implied criticism of generative AI, although rather unwittingly it revealed a possible benefit about AI tools could be better: it would be useful if the AI told you when it didn’t feel it appropriate to answer your input question. Such lack of finesse in the early releases of genAI betrayed a lack of human involvement in the development process.
Overall, the effect of hearing all the above, like the statements from some recent political candidates: any statement, true or false, repeated enough times, has a kind of reinforcing effect: people are lulled into believing it.
You will, I hope, agree that, rather than saying or hearing untested generalisation about AI, it is better to concentrate on what people have discovered from using it. Instead of saying “we are trialling it”, it is more helpful to state what you trialled, and what the result was (and, ideally, how you measured it). I’m afraid it is simply not true to say we all know nothing about AI. We have known about AI for some years, and, if you were prepared to listen, there was more to be learned at this very conference.
In any case, AI is already well-established in the academic workflow. I can’t believe there can be many researchers, at least in STEM subjects, who do not use one of the many tools to help them discover relevant content. Finding potential peer reviewers, running checks for the integrity of submissions, checking metadata for consistency, identifying missing citations – the list of widely used tools is long. Using a broad definition of AI, I would expect every publishing organization is making use of at least some AI tools in the scholarly workflow.
So, what new things did the conference tell me about AI? One finding that fascinated me was to learn more about prompts that give better results. I heard three types of successful prompts for gen-AI tools:
- Take your time before responding.
- Please respond with detailed instructions, as I don’t have any fingers and I can’t type.
- I know you don’t have access to this content, but pretend that you did.
What I learned from actual AI use included, from David Smith at the IET, using LangChain, a clever tool that breaks down the responses of LLMs into more intelligible and traceable steps. He also reported that the use of an SLM, or small-language model, using only the IET’s own content, gave good results (although it should be mentioned that the IET is an organisation with a particularly rich domain-specific content corpus – not every publisher could follow this methodology).
Andrews Sales of Bloomsbury described some interesting tools by which the machine proposed a solution, in this case markup, and presented it to the human to approve or to reject the change. It’s not perfect, but in most cases this approach will provide a dramatic reduction in time and improvement in quality.
Fiona Romeo, from the Wikimedia Foundation, provided some fascinating ideas that have proved to be successful with users. Wikipedia now includes a simple browser extension by which readers can “Add a Fact” to Wikipedia. Once installed, the user highlights some text from their reading on the Web, and the extension uses AI to identify any relevant Wikipedia articles. The user can then add the fact, if not already mentioned, to Wikipedia – although, crucially, not to the article itself, but into the article talk page. Nonetheless, the bulk of the work in identifying a fact and uploading it has been done. All that is left is for the Wikipedia editor to approve or reject it. It’s a very nice example of taking away the drudgery of the workflow to enable humans to do what they do best, which is making judgements, rather than copying out citations.
As for humans, and their often illogical and paradoxical responses to technology, there was a clear but sad example in the discussion session with current and recent PhD students talking about AI governance. One of them stated “I’m a researcher, so I want access to the largest possible pool of content for my research”. Yet a few minutes later, when asked if she would be prepared for her own articles to be used to train LLMs, she responded that her research was government funded, a condition of what that it was published Open Access. But her preference was not to have her own work made available. That strikes me as contradictory.
All in all, an enlightening and valuable day. All credit to NEC for providing this event that has been consistently valuable over more than than ten years.
Leave a Reply