If the Search Solutions conference is to be believed, there isn’t much hope for genAI. This conference is always worth attending, because of its unique relaxed atmosphere, intimacy (around 75 attendees) and range of speakers (this year from Microsoft, Amazon, and Signal AI, just to name a few); but perhaps for reasons that the organizers might not have intended, it revealed some interesting assumptions about AI.
A recent Financial Times article (“Nvidia and the AI boom are coming up against a scaling problem”) makes the interesting point that progress in AI was, initially at least, measured by the growth in the corpus pretraining, which would be proportion to the size of the corpus. The bigger the corpus, the more pretraining was required, and this produced better results. In consequence, Nvidia sold more and more chips (and the market was restricted to a handful of wealthy global organizations, such as Google, Microsoft, plus OpenAI, etc). But perhaps, and not surprisingly, simply scaling up the systems with more and more hardware and more processing produces diminishing returns, after a while. And this is where we seem to be today: the focus appears to be more on how the devices process the user input, something called “inference”. This may require computing power, but it will also require a lot of thought about what users actually input, and trying to make sense of their query.
Here is where I thought the Search Solutions conference might be able to help. It is, after all, a room full of B2B search professionals, most of them focused on enterprise or e-commerce search, and with practical knowledge of RAG. But did the conference reveal any answers? It didn’t seem to express much of an opinion on genAI. If anything, the tone was one of suspicion, summarised by the remarkable (and seemingly unironic) flowchart shown in one of the presentations about using genAI for systematic reviews. The flowchart’s first question was “Does it matter if the output is true?” and if the answer was no, of course it was OK to use genAI.
Such a one-dimensional view of generative AI is not, I think, very helpful. Google is full of unreliable results, yet we use it. There are many things genAI can do and are already doing in the scholarly world, such as assisting in the literature review, providing an indication of well-structured or poorly structured articles. I wouldn’t be surprised to know there is already a tool that can select articles that are valid examples of randomised controlled trials (RCTs).
Unfortunately, I don’t think there was a more accommodating view in the other presentations. RAG, retrieval-augmented generation, which was heralded just a year ago as the great hope for making good use of genAI, was roundly criticized. One speaker (Alessandro Benedetti), in a talk entitled “What I don’t like about RAG”, pointed out that if your initial query was good enough, you wouldn’t have any need to refine it. But this begged the question: we wouldn’t have started using RAG if the initial query-response gave us the results we wanted. We doubt that the user understands enough about the system to trust the answers.
At the end of the conference, there was a discussion on whether genAI had replaced search engines. What became immediately apparent was that this probably wasn’t the right audience for the question; few people in the room seemed to be aware that Google was providing gen-AI results (generated by Gemini) at the top of every search, a feature called “AI Overview”, introduced in May 2024. Given that the activity of these search professionals is B2B search, it was perhaps not surprising that genAI has not impacted on their everyday activity. Yet we are all general users of search engines, and we are all familiar with what works, or doesn’t work, when you search with Google. One of Ethan Mollick’s principles for using AI is “Always invite AI to the table” – get involved with it, try it out.
For what it’s worth, I think the comparison of search engines and genAI tools is very instructive. Here are some things I’ve noticed (with the proviso that what I describe here may well change in a few weeks or months):
- GenAI by default gives you one answer, while a search engine gives you many.
- Search engines by default provide more attribution with their results. For example, if you look up a well-known recipe with genAI, it gives you a generic recipe; the same search on Google gives you several names of cookery writers and their respective recipes.
- GenAI, unlike search engines, can be treated as a persona, in fact they work better treated in this way. Ask the system to provide more examples, or from a different point of view, or for a particular purpose, and genAI responds as you wish – something it is all but impossible to get a search engine to do.
If the FT article is right, and I think it is, then the pressing need is to keep developing the AI tools so they make more sense of the user query. That requires human understanding both of search engines and of AI tools – not programming expertise, but user expertise.
Perhaps we have come full circle, and we are back with the reference librarian, the smiling person at the desk who can answer all your questions. One participant told me that the fuss about genAI and the information community’s response to it was a bit like when Google first became common. He remembered a signature line that was used by several reference librarians at the time: “Google gives you 1,000 answers, but a librarian gives you one correct answer”.
Yet there is a grain of truth even in this luddite response. There may well be a helpful correct answer that can be obtained from genAI, which may derive as much from the context of the question than from what was actually asked – and that’s something librarians are familiar with. Reference librarians are fond of stating how they heard a question and then, after a brief discussion, are able to formulate that question into a more precise query or statement, matching the questioner’s actual needs. Surely that is what RAG is all about?
Leave a Reply