I recently read about a new research report, Mind the Gap, by a team with lead author John Maxwell, about some initiatives in open-source software. It made me ponder how to appraise and to evaluate new software tools.

This year’s SSP Annual Event included quick presentations of new technology; there are similar sessions at many of the larger conferences. Each vendor from a small startup presents their big idea in the space of a few minutes. The format varies after that, but at SSP, there is a vote for the “people’s choice” (a phrase that grates in the UK), where the most popular initiative is chosen following a vote among the attendees.  

It seems harmless enough; so what’s wrong with such a format? First, the votes reflect the pitch, presentation, not the software. It’s a bit like selecting a software developer based on their interview performance rather than on the quality of their coding.

Second, if I stand at your doorstep and present to you a new solution for cleaning windows, you are pleased that a solution presents itself in front of you. You are probably not thinking about other ways of solving the problem. Critical thought is not encouraged in this kind of presentation.

So is there a better way to assess and evaluate new software? There is such a proliferation of tools that it is difficult to know where to start when reviewing. Kramer and Boesman, in their famous 101 Innovations in Scholarly Communications graphic, just showed the tip of the iceberg.

Some general stats about software innovation: as you may already know, most software apps disappear rapidly. The average retention curve for Android apps is rather shocking: the average app loses 90% of its daily uses within the first month of installation [source: Quettra, now SimilarWeb].

So a considered and well-researched report looked very welcome. Mind the Gap looks at open-source publishing tools and platforms. What does it (or does not) do?

  • Many of the software tools it considers were developed via external funding. In fact, the report itself was funded by the Andrew W Mellon Foundation, like many of the initiatives described within. For example, The Public Knowledge Project, PKP, which is based at Simon Fraser University, where the authors is based, gets six mentions in the report. Of course, a problem with funding is that by its very nature it tends to be one-off; it isn’t sustainable.
  • Unsurprisingly, more than half the projects described here are less than four years old. Even so, such is the frenetic pace of online development, the project had to check that projects were ‘still alive’.
  • Unfortunately, the descriptions of the tools themselves are so bald and lacking in judgement as to be almost useless.
  • The report is very incomplete. Zotero, for example, is included, but the authors admit it isn’t really a publishing tool. There is no mention of Dspace (now DuraSpace), perhaps the most widely used open-source software for institutional repositories. Their justification is “these systems operate in an ecosystem of their own, and while they may in some cases underlie publishing software, their scope is outside this project.” I would have thought Duraspace is a model of wide community acceptance of an open-source platform, which has reached the envious position of critical mass – enough of the community has accepted it as a platform of choice for it to be an essential part of the landscape, not just a possible enhancement. How did it get to this position? How is it funded and how can it continue? These are questions I would like to have answers for. The answers might provide some very valuable feedback for the tools considered here. It’s almost as if the author has said “Here are some recent startup software initiatives. How can they become sustainable?” without considering examples of open-source start-ups that did become sustainable.
  • There is a slight sense of closing the stable door after the horse has bolted. Geoff Bilder of Crossref claimed “Everything we have gained by opening content and data will be under threat if we allow the enclosure of scholarly infrastructures”.  But scholarly publishing has for years been highly enclosed in subscription-based publishing and highly regulated access. Try getting access to scholarly content if you are not a member of an institution, and you rapidly see that scholarly infrastructures have almost never been anything but enclosed.
  • It seems very unsatisfactory to evaluate open-source tools without any idea of the landscape in which it is situated, that is, the commercial tools it is pitching against?
  • Some indicators that might have been useful would be some indication of how well maintained the software is. When was the last release? When was the release before that? How big is the community that maintains it?

What does the report reveal?

  • The lack of coordination between the many initiatives described. It’s relatively easy to dream up a technical solution and get some one-off funding to build it. Much harder is to produce a concerted approach to the academic workflow, that integrates this solution with other initiatives. That’s because, as the report states “there aren’t obvious incentives for collaboration between projects”.
  • The arguments for open-source software are well-known and not new. But not stated by the report is how any platform, open-source or otherwise, creates a degree of lock-in, since the cost of moving elsewhere is always high.

In conclusion, there is a great need for independent evaluation of software tools (open source and otherwise), so this initiative is welcome, but by no means complete in many of the sectors it considers, which limits its value. If you know the sector within which each initiative operates, you will find this report useful. If you don’t know the sector, check elsewhere for other potential tools and for some idea of the landscape.