Search engines have not become obsolete because of genAI. They serve different purposes, and it’s worth while considering how they differ.
Benedict Evans hits the nail on the head when assessing genAI tools. He asks genAI a factual, numerical question, and he is given the wrong answer. To make it simpler for the machine, he provides the primary source, but he still gets the wrong answer. “The problem here is not so much that the number is wrong, as that I have no way to know without doing all the work myself anyway … Of course, these models don’t do ‘right’. They are probabilistic, statistical systems that tell you what a good answer would probably look like.” He identifies the problem perfectly. He calls the difference between factual questions and wider, idea-based questions “deterministic” or “non-deterministic”, but I think factual and non-factual work fine as descriptors.
Here is another interesting examination of the differences between search engines and genAI, from a fascinating workshop on using AI by two enthusiastic team members from Vanderbilt University, Josh Borycz and Alex Carroll. One exercise we were given was to ask the same question and compare the answers from a search engine and from genAI. Two of us chose a recipe: the example I used was how to cook the famous French dish coq au vin. This is a recipe that has been around for many years, so there will be many different versions of it in cookery books. On searching in Google, I was given several specific recipes from named sites. Of course, there will be many other recipes available only in print, that the engine has not (yet) captured.
In contrast, genAI (in this case, Copilot) gave me a single, generic recipe, what they term “a highly recommended recipe” – although they don’t state who recommended it. The recipe comes without attribution to any individual – no doubt due to the creators of LLMs wanting to avoid identifiable copyright infringement, although they almost certainly used the same sources as genAI tools. Of course, if we are making the dish with the recipe, our preference is often to see who wrote them. For some purposes, however, it may be more appropriate to see the genAI amalgam of several different sources.
One other observation emerged from the workshop discussion. Another participant often made sourdough bread, and did a similar comparison between the results for “sourdough recipe” from a search engine and genAI. He discovered that the genAI version was “kind of OK”, in that it seemed to have a reasonable set of ingredients and instructions, but would need to be checked in practice. This ties in with Benedict Evan’s complaint about asking certain questions of genAI: “I am in principle trying to use a non-deterministic system for a deterministic task”. If you have to check the full recipe for yourself, then you’ve lost the advantage of having tested recipes supplied by a name you have used and whose recipes you trust.
Ideally, of course, it shouldn’t be too difficult to separate out factual from non-factual questions. I remember Google introducing “featured snippets”, their term for answering factual questions, such as the height of the Eiffel Tower, or the birth date of George Washington, via a patented solution in 2015. Even though different sources may state different figures for the same fact, this patent explains how the search engine examines many documents to find a consensus view of a fact (in fact, it looks simply like the value contained in the highest number of documents). What is described is the Google Knowledge Graph. So we know Google has a methodology for identifying facts, and genAI complements this approach by typically providing generic, non-attributed answers. Even if the distinction isn’t watertight, you can see the benefits of providing both approaches. There’s your challenge, Google! Combine the search engine with genAI to provide the appropriate kind answer depending on my question, and I’ll be very happy. Give me a factual answer if I ask a factual question, and advice if I want an opinion. Instead of a proliferation of interfaces, and the need to learn which approach to use for which service, just give me one search box, and I’ll be happy.
Leave a Reply