A recent excellent webinar about the O’Reilly Online Learning site, part of the excellent Haystack Conferences, was presented by Anthony Groves, O’Reilly Media’s Technical Lead of Search, and promised a fascinating insight into how discovery should be provided. Some might be more familiar with this site under its earlier title Safari Books Online, and it was under this label that I reviewed the site back in 2013. As Anthony Groves pointed out, this online learning site about IT is run by IT professionals for IT professionals. In other words, if O’Reilly don’t know how to implement search and discovery, who does?
Anthony Groves’ presentation was remarkably candid. He began by stating the search interface had received very little attention for several years, and then he demonstrated very clearly what was wrong with the old site and why it needed changing. First, let’s look at the main display of results:
When a user searches for content, they see a list of hits, as you would expect, with star ratings for reviews. Along the top of the page are six headings, five of them representing facets: the display above shows how you can select only books (an impressive 46,000 of them) or videos, and so on.
Moving beyond this, Groves revealed a very impressive site testing routine that measured relevance of search hits by awarding points between 0 and 4, a process called “judgement capturing”. There is even an internal version of the search display where relevance scores can be displayed. And, as you might expect, since this site is not about e-commerce (users will already have subscribed to enter the site), the success metric is not click-through to any specific point such as a shopping basket, but based around content usage. This varies depending on the media, so for books, for example, he explained, we check how many pages they read; for a video, how long they spend on the page; for an online learning module, if the user registers for the training, and so on. This is excellent practice.
In another interesting comparison between the old and new sites, he revealed that the old site used to boost new content above popular content; today, the results are more balanced, with a mixture of new and popular content.
In conclusion, then, an impressive presentation, and a great insight into understanding user journeys for a specific site. The book Relevant Search was used not just as an example of a search, but was recommended more than once in the presentation – clearly, this is a book worth looking at.