The Center for Open Science, co-founded by Brian Nosek, has been responsible for many of the big innovations in science practice over the last ten years, including the Open Science Framework and the Reproducibility Project. In this interview I talked with him about his background and how it led to his work on science practice.
Nosek studied psychology, and for his doctorate, looked at the gap between what we intend to do, and what we actually do in our everyday behaviour. You can see the parallels with Open Science: for many researchers, it illustrates the gap between what they aspire to practice as good science (to make content as accessible as possible), and what they actually do (submit to highly-cited subscription journals in the hope of career progression) .
Nosek studied at the research process: how do we try to make inferences about what we know? How do we strengthen our methodology or rigour? He identified a gap in research culture between scientific values and scientific practices. “What do we think are the most important features of science that help us to discover things?”. What he discovered was:
a reward system that benefits exciting novel discoveries, and not necessarily rigorous, reliable, credible discoveries, and the limitations of the fact that humans are the ones doing science.
Scientists themselves agree:
few scientists say that the culture of science is highly functional. Everybody has complaints about the dysfunction in that environment. Nobody likes the fact that we have to chase papers in the high impact factor journals as the mechanism of advancing our career or apply for grants over and over again, just to have an active laboratory.
In response to this, the Center for Open Science was founded (2013). One of the earliest projects the Center carried out was the Reproducibility Project, to see if experiments described in papers could be replicated, with more than 270 collaborators.
Nosek recalls how central Twitter (now X) was in its early days for researchers: “Twitter really deserves credit for helping foster that discovery of others who are working on similar problems in different spaces.”
Another COS project, Registered Reports, looked at the rewards system for research. In practice, research has many false starts and things that don’t work, yet the publishing incentives are to make the work as tidy as possible. Instead, using the Registered Reports model, the researcher develops their research question and proposed methodology, and publishes it. This, of course, promotes research integrity, since it is not influenced by the results.
the reward is gained if I ask an important question, and if I design really rigorous, effective methods to test that question. That’s what we should be rewarding researchers for.
One incidental benefit of this approach is the publication of negative results. Another is the serendipitous nature of research in the lab: “we had one idea going in, and coming out, something totally different occurred.”
The nature of science research is dealing with uncertainty, and here Nosek is very clear:
I love when that uncertainty is part of the social discussion of science, because I think that’s such an important part for the general public to understand and wrestle with.
There’s too much of an orientation in any popular press reporting of science to say “here is the new discovery. Turns out wine is good for you.” And then two years later, turns out wine is bad for you. And then two years later, no, wine is good for you. What’s a reader to think?
Nosek’s vision is for a fundamental change in how researchers are evaluated, on their work, not on the publication:
We’ve conflated publication and evaluation into a single process that is opaque to readers because they just assume the journal went through some mechanism to decide that this was legitimate research that deserved being in the publication.
Instead of the single-researcher model, Nosek praises a “horizontal contribution” model, by which other researchers participate in a research project. The model is there, but the reward system has yet to catch up.
The Center for Open Science is perhaps best known for the Open Science Framework, an infrastructure that supports the research lifecycle and is used by many journals. Using it,
researchers can share their data, their materials, all the different components of research that occurred along the way to make that part of the scholarly record.” It enables what Nosek terms “lifecycle open science” by the researcher.
Preprints should be regarded as a statement of research, rather than the accolade of publication, and treated in that way.
Overall, it’s remarkable that one group has not just identified failings with current science practice, but at the same time carried out real contributions to good science. Long may it continue.
Leave a Reply