Some implications of "digital" for scholarly writing and publishing

Evaluating systematic reviews

Reading Time: 2 minutes

The European Association for Health Information and Libraries (EAHIL) annual meeting included a number of fascinating workshops. These workshops were revealing perhaps for unintended reasons. What do health librarians do? On the basis of this conference, they largely carry out systematic reviews of clinical trials.

The session by Alison Bethel and Morwenna Rogers, on using aids to reviewing and setting up a systematic review, was a model of its kind, with around 35 attendees. At first, I thought it would simply be passing on some tips about how to use the major medical content collections, but it became more interesting. The presenters did not attempt to present the rules. Instead, they handed out a summary of the rules, and skilfully elicited comments from every group in the room, thanking every contributor for their comments. In this way everyone in the room got involved with the subject and participated with their opinions.

What did I learn from this session about systematic reviews? First, the search strings used by medical librarians are highly complex. A typical search, finding content about bed sores, might look like (I’m just showing the first few lines):

  1. exp Pressure Ulcer/
  2. (pressure adj (ulcer* or sore*)).tw.
  3. (decubitus adj (ulcer* or sore*)).tw.
  4. (bedsore* or (bed adj sore*)).tw.
  5. Or/1-4

This is rather cryptic, but intelligible Boolean. It’s clear what each line in the search is trying to do. A typical search might be 20-30 lines long.

The workshop was about reviewing such searches by peers to identify possible errors and improvements. Various methodologies were considered; but while the recommendations provided were very helpful, I’m not sure they provide much genuine help; one tip is “are there any spelling errors?”. I would describe this as “exhortation advice” – the idea that telling you what to do will result in a better end product. Much better would be to provide some tool that suggested additional terms for the search string creator. For example, one example we were given was about setting up a search of the literature to cover references to mobile phones. The actual Boolean was:

“mHealth” OR “m-health*” OR “mobile health” OR “mobile device” OR “mobile app” OR “smartphone” OR “mobile phone” AND…

Unfortunately, this attempt to cover all the ways in English that we refer to mobile phones missed at least one crucial term, “cell phone”. The search was perhaps set up by a British English librarian and did not take into account the American English version. As a result of one string missing from the search, many relevant articles would not be found.

How could this be fixed? Asking a peer information specialist to review your work would not necessarily find the problem – another British English librarian might not spot the missing term. The human brain is far less effective at identifying gaps than being asked to select from a list of possible additions. If a system could be found that provided semantic equivalents to any of the terms in a Boolean search, the search designer would immediately notice important terms in the search that had been missed.

My comments on this workshop are not intended in any way as a criticism. Instead, it is a tribute to the open dialogue encouraged by the course convenors that all the participants were willing to engage and to think of possible ways of improving searches for systematic reviews. This was a great session, one of the best in the Conference.


The Cochrane PICO Finder: how not to trial an interface


Misunderstanding a dictionary

1 Comment

  1. Great post Michael.

    I know we shouldn’t generalize from a sample of 1, but if you enter ‘mobile phone’ into 2Dsearch and ask for suggestions, ‘cellphone’ is #1 on the list 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Powered by WordPress & Theme by Anders Norén