How to assess the societal impact of open science

Can we assess societal impact at all? Or are we asking the wrong question?

An interesting article on the LSE Impact of Social Science blog describes an attempt to measure the effect of open science (or Open Science, as the authors have it). The post summarizes a large-scale research project that involved a systematic review of the literature to try to find if the impact of Open Science can be assessed. The paper, with five authors, uses the traditional methodology common for medical best practice reviews: you search in a corpus of research articles using a large-scale Boolean search, and then narrow down the results by hand, to create a core of full-text papers that you believe are relevant to your question. In this case, the researchers used Web of Science and Scopus, two paid-for collections of scholarly articles. The elaborate Boolean search they created revealed some 14,000 papers, which they then reduced to just 196 full-text articles that they (the humans) believed were “in scope”, that is, relevant. But the results were pretty paltry. Of the papers they identified, 83% were related to Citizen Science, which is not the same as Open Science, but concerned with involving the public in projects of a scientific nature – something that is relatively easy to measure, for example, by checking the participants’ knowledge of birds after an experiment to identify birds in your garden. The remaining 33 papers from the scoping review related to their three areas of interest, Open Access, Open Science, or Open Code.  You could say at this point that 33 articles was hardly sufficient data to state anything meaningful. But in any case, was it the right question to ask?

Let’s think about this in a common-sense way. Research papers might be published OA or claiming allegiance to Open Science, and I am in full agreement with the principles of both. Yet availability or methodology does not mean that the research papers will be intelligible to the general reader, nor that this is the kind of research that would be useful for the general public to read. So does it make any sense to try to measure impact? It is unrealistic to think that all research content should be intelligible to the general reader.

It may be heretical to suggest, but while providing access to the general reader is a good, yet we should not expect the general reader to spend their time reading specific articles of interest only to researchers in that space. There seems to be an almost innocent idea that adhering to the principles of open science means that every citizen will dutifully scan the major A&I databases (and it seems to me ironic that this experiment was based around two collections for which there is no public access outside the academy) in order to improve their scientific knowledge and awareness. Providing access to the articles would be a start, but it is necessary, not sufficient. Open Science is a good, but it does not convert every member of the public into a scientist.