This is a book that packs a punch – although it’s not quite the punch the reader might expect. Published in the MIT “Essential Knowledge” series, Michael Schrage’s Recommendation Engines moves from ancient Greece to the recommender systems we are familiar with today – and then culminates in a hymn to self-realisation through recommendations (about which more below).
Michael Schrage is commendably clear on his goal for this book. He sets out to provide “not a technical command of the underlying material but with a sure conceptual grasp of what aspects of technology make recommendation engines so powerfully effective.” I’m not expecting a full technical seminar, but some understanding of what is going on would be helpful.
As an introduction, the book contains some useful background in the development of recommender systems. But this introduction is neither sufficiently detailed to be informative, and nor is the essential information easy to find. It is surrounded by a lengthy opening chapters looking at divination in ancient times, with mention of Aesop’s Fables, the I Ching, Gutenberg, Castiglione – everyone seems to be here.
Moving on to the core of the book, there is a glossary of key terms, an excellent idea. Here are indeed the “essential” terms; but they aren’t necessarily well covered in the book. For example, “content-based recommenders” (that is, recommendations without any user input whatsoever) is clearly defined, but the text never considers the use of content-based recommenders. Content-based recommenders such as UNSILO (with whom I work) make no use of user data; the use case is that academics want to know what research articles exist on a topic, even if nobody has ever read the article.
The glossary has an entry for “cold start” where the recommendation engine has insufficient usage data to provide any suggestions. But that is contradicted by a reference on page 120: “Popularity is the … easiest and most common way to deal with the ‘cold start’ recommender problem.”. If we know something is popular, we don’t have a cold start.
The author’s standpoint is revealed by his glossary definition for “recommendation engine”, defined as algorithms that discovers similar items and predicts the user’s response to them. This is not a full definition, but describes usage-based recommendations perfectly, and this is the author’s focus: the use of recommendation engines to increase sales. Netflix is mentioned 38 times in the book, around once every six pages, and after describing in glowing terms how services such as YouTube, Tik Tok and Netflix work, he concludes his survey in the final chapter with a remarkable justification of recommender systems. I’ve never read a personal-growth justification of recommender systems before, and this one makes rather uncomfortable reading.
Schrage acknowledges criticisms of recommender systems, for example that Netflix encourages binge watching. Schrage even quotes Netflix co-founder Reed Hastings: “Netflix’s brand for TV shows is really about binge watching … it’s to just get hooked … it’s addictive, it’s exciting, it’s different.” Schrage also points out how Tik Tok has no pause button, and was forced by the Chinese government to introduce alarm warnings for Chinese users after 30 minutes of uninterrupted usage, to warn users they are getting hooked.
But no matter. For Schrage, because “recommendation implies choice” (p219), and because, if we wish, we can ignore the recommendations: “Ignoring good advice can feel even better than not following it.” But the most startling argument is how recommender engines help self-realization: “Successful recommenders promote self-knowledge, inform self-interest, nudge self-improvement and invite self-indulgence even as they encourage self-control.” As if self-knowledge is not sufficient, the author concludes: “Recommendation really does become a magic mirror reflecting imaginable … future selves.”
The author talks vaguely about providing public accountability for recommenders: (p224) “Transparency and public scrutiny are important safeguards … Nothing should be hidden or covert.” How does that square with “No computer scientist … grasps exactly how these … deep learning algorithms work” (p107).
I’m the world’s greatest fan of recommender systems, but not because they give me self-knowledge, or because they enable me to emulate what product choices celebrities make (p226). I think the algorithms on which recommender systems are based can and should be explained, and I think they need careful human curation to meet the specific use case they are designed for.
The true timescale of this book is the short period between Netflix managing DVD lending services to running online recommenders. I came away from the book feeling it is primarily a celebration of just one kind of recommender: the sort that drives us to consume stuff. Schrage loves the big numbers and the impressive statistics – and skips over lots of the details. Yet there are occasional references in the book to what could potentially be interesting angles. Katrina Lake, CEO of Stitch Fix, comments: “A good person plus a good algorithm is far superior to the best person or the best algorithm alone”. Why this should be, and how to integrate human and machine tools effectively, is not really considered.
The text is full of grand statements that, on reflection, could be stated more concisely:
To the extent past is prologue, the future of machine learning is the future of recommendation engineering. That said, it is no less true to observe that the future of recommendation engineering is also the future of machine learning. (p139)
Unfortunately, when it comes to the details, Schrage is less than informative. How does Stitch Fix create its recommendations? “Stitch Fix ensembles all kinds of algorithms – neural nets, collaborative filters, mixed effects models, naïve Bayes – to do a first pass at recommending styles for individual customers”. [p200]. According to this author, the more data, the more parameters, the better. The idea of limiting the number of parameters in the interests of computing efficiency is not mentioned. In this recommender world, all numbers are billions, and all computers infinitely powerful.
Finally, a word or two about the style and editing. No fewer than five people are credited with helping the author create this book, so it is all the more disappointing that none of them spotted elementary errors. One character is introduced no fewer than four times:
26 Greg Linden, who helped launch Amazon’s recommenders …
76 Amazon employee and recommender guru Greg Linden recalled …
175 Greg Linden, who successfully pioneered Amazon’s earliest recommendation engines …
214 Greg Linden, who led Amazon’s earlier recommender work …
A whole paragraph (about Netflix) is repeated on pages 93 and 152.
The author has an annoying habit of saying everything several times. Never give one example when you can give two – or three, or even more (the highest number I counted was 22, on p108):
The real-time recommendation and personalization that help make Alibaba, Amazon, Facebook, Netfix [sic], YouTube, and Tencent so compelling …
Why stop at six? Comparisons have to be made more exciting by providing examples, which become wearisome after a few pages:
People from Boston to Beijing to Berlin to Buenos Aires expect good recommendations. (p150)
One of the features of the Essential Knowledge series seems to be quotes from the text, displayed on a black background on a full page. When such quotes are insightful, there is a powerful impact, but several of the quotes appear to be underwhelming:
Do recommender engines deserve a book? Yes, absolutely! A book with essential knowledge about recommender engines would be very welcome – with less repetition, and without the claims to provide self-knowledge or to help us build our future selves.