Not a day passes without someone suggesting a way to reduce the inefficiency of the scholarly publication cycle. There is an interesting article in Scholarly Kitchen by Tim Vines, about Michael Eisen, the new Editor-in-Chief of eLife, suggesting ideas for the future of Peer Review. It raises some interesting ideas, but at the same time threatens what I think is one of the key principles of peer review: authority.
It is interesting to compare the editorial system for Wikipedia with the scholarly article workflow. Once upon a time, encyclopedias were compiled by teams of experts (I should know, I used to manage such a team). These experts supplied updates and new content to a team of publisher editors. The in-house editors had the final say in what was published. The resulting text had some level of editorial integrity, even though of course over time it would require updating.
In contrast, Wikipedia can be updated at any time. Anyone can make changes to any article. The result is often a startling level of up-to-dateness – for example, the article on Leven station in Wikipedia was updated soon after the Scottish Government’s decision to reopen the rail link to the station (Wikipedia is quite up to date on railways).
What are the downsides? What you read might be completely wrong, hopelessly biased, or just unreadable. Wikipedia has tried to implement quality control measures, but the number of volunteers willing to police all 4m+ Wikipedia articles is limited. Hence there is a level of pot luck whether the article you read is correct, up to date, unbiased, or full of spelling errors.
Now let’s look at academic article. Anyone can in theory write an article, but for it to be published, it has to be both peer reviewed and approved by a journal editor. That editor will obtain the approval or two or more peer reviewers, experts in the field, who state the article is sufficiently high quality to be published. Here already is a difference: scholarly publishing has a kind of catalyst, the publisher’s editor, who manages the process. Nobody manages the process for Wikipedia. If you want to create a new article right away, you can.
The benefits for scholarly publishing are that articles are authoritative. It’s not just that my words that are published, but at least two other experts have approved it. I have to follow some rules when I write; for example, there are scholarly citations for most (ideally all) of the claims in an article.
The downside is an appalling waste of labour. To meet the requirements for publication in a journal requires a lot of work. The Scholarly Kitchen article estimates as few as 8% of manuscripts submitted to a publisher are approved for publication. Some of these articles will be resubmitted, but nobody (as far as I know) keep statistics on resubmitted articles. Often the reason for rejection is trivial – it can be as simple as having too many words in the abstract. Each journal (and there are 24,000 different journal titles in medicine alone) has its own style rules, its own spelling rules and its own system for showing citations. If you fail to meet these criteria, the onus is on you, the author, to change the article. As a rule, journals do not make changes to a submitted article, the author does. So the onus is on the author to keep changing the article until it is correct.
The changes proposed by Michael Eisen are two-fold. Firstly, “anyone, or any group, should be able to review a paper, so long as they do so constructively. It’s silly to leave something as important as evaluating the validity and contributions of a work based on the judgement of 3 people”.
Wikipedia demonstrates the problem of this approach. Just look at the “revision history” page for any article in Wikipedia on current affairs, and you can see warfare between opposing factions. Often a statement is corrected, then uncorrected, several times in sequence, leaving Wikipedia’s reputation for authority in question.
Secondly, Eisen suggests a kind of ongoing review: “There’s value in assessing a paper at the time it’s published, but also in looking at a year, 5 years, 100 years later”. This sounds great, but here is another fundamental difference between Wikipedia and scholarly articles. Scholarly content represents a view composed at one point in time. A scholarly article, for better or worse, is date-stamped. It is not updated; you write another article. In contrast, encyclopedia and Wikipedia articles are continuously updated. The content has to be taken to pieces and put back together, as, for example, when plate tectonics became a new consensus for the history of the earth. In science at least, the discussion about an individual article at some point becomes superseded; the discussion moves elsewhere.
So while Wikipedia and scholarly publishing have some similarities – they are both keen to reduce the delay from authorship to publication – it would be worrying if scholarly publishing were to start to drop its authority and try to change itself to continuous publication.