Search in an age of AI

There is an endearing air of disorganisation about the Search Solutions Conference: even though it has been running for over 20 years, it continues to have an unscripted feel. But that informality results in a unique atmosphere, where the hidden barrier between speaker and attendee seems to dissolve. The participants, typically 70-80 in number, feel empowered to get involved and to ask questions, and not just the more confident ones – and that is no small achievement.

One reason why there are high levels of participation is because the event combines theory and practice in a very visible way. From my own perspective, it provides updates on AI-informed search practice, use of RAG, and so on, as well as sessions about information retrieval in different domains. Even though I’ve been involved in search and discovery for more years than I care to remember, each conference provides insights into the process that I hadn’t been aware of. This year, for example, it was a presentation by Adam Roegiest, on information retrieval in the legal world, such as mergers and acquisitions. The M&A process, remarkably, does not typically involve looking at every contract currently in force with the company being bought; due diligence usually means only sampling the contracts. So here is a landscape where a sample is considered sufficient, compared with, say, systematic reviews, where not accessing a relevant document would be considered a failure of the process.

While genAI appears to move the goalposts, some principles still apply. Each year we keep coming back to the fundamental paper by Voorhees (1999), which states (and was quoted by Roegiest) that there is “a practical upper bound on retrieval system performance”, which is “65% precision at 65% recall, since that is the level at which humans agree with each other”. If you are using humans as your judge, it is somewhat pointless to expect a machine to produce better results.  

Human subjectivity is a fundamental part of information retrieval, a point picked up in the panel session, where we reminded that “you are not the user” – ultimately, it is the user who decides on what is most relevant. There was a great example of this with the Trip Database, a question-and-answer database he created for medical practitioners. But what advice is authoritative? Jon Brassey, who built the system, explained that their view of authority was based on what users expected. There is ultimately no guarantee, no external authority that because an article is published in, say, The Lancet, it is therefore correct, but there is an expectation among users that it will be trustworthy if published there. Even systematic reviews, in other words, are based on a network of trust.

What was new? Automatic generation of questions was one example, the topic of the “Dataset Generator” presentation by Alessandro Benedetti. The talk was interesting, but for me the questions generated by the system did not look like the kind of questions humans would ask. A presentation that tried to evaluate the success of a RAG-based sentiment analysis system suffered, I think, from a small number of questions in the sample. A fascinating presentation by Orland Hoeber looked at the discovery interface for large-scale academic libraries, and made some modifications to try to incorporate more active learning on the part of the student, and proposed “workspace-augmented generation” (WAG) which provided summaries of what was found.

Stephann Makri and Andrew MacFarlane looked at searching by people with mild aphasia, and concluded that improving the search process for those with disabilities often resulted in a better discovery process for all users. Sean MacAvaney talked about “adaptive reranking”, using a lovely analogy with the game Battleships, which involves locating the position of ships on a grid. Once you have identified a part of a ship, your subsequent moves will be modified – which, put more technically, is “nearest neighbour graph exploration”: taking a graph structure and resampling it to find a smaller set of related documents. It sounded good to me, and might even improve my ability to play Battleships (I never was very good).

Jon Brassey (mentioned above) looked at physicians asking patient-care questions. He referred to a remarkable 2005 paper that found physicians asked many more questions than they answered; the most common reason for not pursuing the answer is because the resource they consulted did not provide an answer. So Brassey built his own question-and-answer service,  known as The Trip Database, or Ask Trip, which, amazingly, operates independently of the many large-scale clinical decision-support tools such as UpToDate (Wolters Kluwer) and BMJ Best Practice. I found it interesting that simply providing a search tool in this area is clearly not sufficient: a clinical trial, even if successful, is not necessarily a recipe for a change in clinical practice. What is needed is evidence-based, proven solutions, another step beyond simply finding relevant texts.  

No Search Solutions conference is complete without any mention of enterprise search, neatly defined by Cedric Ulmer as “enabling companies to find all the information a company possesses, without knowing the location”. What was more important, we discovered, for enterprise search, is the need to restrict access to some parts of the content, depending on who is doing the searching – a level of complexity that genAI does not have to deal with. This session revealed a couple of other issues with implementing recent AI tools. Filtering and reranking don’t always work well together; and there is need sometimes to try to distinguish AI and search – something Google manages quite effectively with its “AI Overview” appearing above the search results (although I suspect that in practice most people already ignore the search results and just read the overview).

The day ended with a panel session, which complemented the sessions by asking some fundamental questions, notably, “Is AI making search better?”. One respondent put it nicely: “Search has fundamentally changed because of AI, but I am confused about what search is now”. This conference will, let’s hope, continue to provide some answers to that challenging question.