July 17, 2018

the preprint dilemma: good for science, bad for the public? A discussion paper for the scientific community

By Fiona Fox

This blog contains the thoughts of the author rather than representing the work or policy of the Science Media Centre.

There is a quiet revolution taking place in scientific publishing and almost everyone I speak to thinks it’s long overdue.  One aspect of the revolution is ‘preprints’, the practise of making scientific papers available to any experts for scrutiny before journal peer review and publication. Others explain the benefits far better than me but suffice to say enthusiasts argue that – if adopted widely – preprint could herald the end of the perverse effects of the dominance of journal impact factors over scientific careers and speed up the publishing process which can sometimes see scientists waiting for months or years as they work their way down the journal hierarchy.

All this sounds great to me and I am a big fan of bold and disruptive changes which can lead to fundamental culture change. My reading around work on reproducibility, open access and preprint make me proud to be part of a scientific community intent on finding ways to make science better. But I am concerned about how this change might affect the bit of science publication that we are involved with at the Science Media Centre. The bit which is all about the way scientific findings find their way to the wider public and policymakers via the mass media.

This essay is an appeal to the scientific community – researchers, publishers and communicators – to take stock and engage in a discussion of the wider impacts of preprint.

Now at this stage I feel the need to throw in a few caveats and limitations.When I sit down to write I usually know exactly what I think about a subject which is pretty clearly defined.  Neither is true of preprint. No one yet knows whether this will be widely embraced and – if so – how things will look at our end of the chain. Depending on who I speak to we might have nothing to worry about or lots to worry about. So why write this essay now before our opinions are more developed and the landscape more certain?

For two reasons.  First, because I think many of those who are driving preprint have not yet considered the impacts of this exciting development on the wider public and so I hope there is still all to play for in terms of these discussions.  And second, because more and more people are telling us that preprint is now coming to new subject areas including biology, medicine and climate change. As many people have pointed out, preprint has been around in subjects like physics and maths for many years now and the sky has not fallen in. Very true. But studies in physics and maths don’t tend to make the front pages of the Sun and Daily Mail, don’t tend to influence personal behaviour or decisions about new technologies and are therefore not the core focus of the Science Media Centre. It may be rather self-regarding but we care about this now because it is about to affect what we do. Potentially dramatically.

So what exactly is the SMC worried about and what would we like to discuss with those driving and shaping the rules of preprint?

The system works (relatively) well today

The daily diet of stories for the average science and health journalist working in national news typically comes from new findings published each week in the peer-reviewed literature including journals like the BMJ, Lancet, Science, Nature, PLoS etc. At present they report these studies primarily from press releases issued by the journal and the authors’ universities/research institutes; or from a press conference (at the SMC or elsewhere) where authors present their findings and highlight limitations and caveats.

The press releases are issued under embargo to the time of journal publication, allowing the journalists a few days to read the paper, call up the authors directly, maybe attend a media briefing and to seek reactions from independent experts in the field. For the last 15 years these journalists have also received reactive quotes from other scientists, via the Science Media Centre. We proactively issue these comments with the agreement of the journals to help ensure that time-poor journalists have easy access to the wider context of research in this field as well as any important caveats and limitations that the wider public should be aware of.

When you add these routine activities together and throw in a group of specialist science journalists who generally strive to produce accurate copy you have the situation we enjoy in the UK today: a system which includes checks and balances aimed at helping ensure that what is reported to the wider public is broadly accurate and reliable. Journalists and scientists both feel this system improves things on their side of the fence.

Of course it doesn’t always work perfectly and we still see sensationalism in both science and news coverage. But as someone absorbed in media reporting of science for 16 years I can tell you it works better than it used to. There is a good reason why people still like to hark back two decades to the pill scare or MMR for their go-to examples of where it all went wrong.

How preprint could disrupt that

Preprint may disrupt this system in the following way.  Instead of new findings being seen first by journalists when they have been peer-reviewed and are about to be published in a journal, research may be placed on a publically accessible server at the same time as it is submitted to a journal – sometimes before. This potentially means a dramatic increase in the amount of non-peer reviewed findings available to journalists, should they choose to seek them out; and through them to the wider public.

It’s true of course that every busy science reporter will not be able to scour thousands of papers on preprint servers looking for a story; and why would they want to when they already have 50 ready-made stories in their inbox.But only one journalist has to find and report on new findings for it to be ‘old news’ to competing titles, and even older news when it finally makes it to peer-reviewed publication. Nature News reporters have already proved pretty adept at finding interesting preprints and we’ve already seen journalists on some nationals reporting on preprints. And what’s to stop media-hungry authors who are excited about their findings from alerting journalists to their newly-posted, unverified findings?

Try as I might I cannot find much about this prospect to calm my nerves. Surely we have to entertain the possibility of many more new scientific claims being reported in the news media before they have been subject to any peer review, without the fact-checking time provided by an embargo, without a measured and cautious press release from the journal or university, without the benefit of the third party comments gathered by the SMC. Does any of this worry you too?

The critical point is this: that once these findings have been reported in one or two national newspapers they cannot be unreported. If the Times or the BBC report a preprint which claims that ecigs cause similar harms to cigarettes, or that statins have severe side effects, then the news editors on the Daily Mail and Telegraph are unlikely to report a less exciting and toned-down version of the same story when it’s published. They will either run the same story, or not at all. That means that the claim that appeared in the press and did the rounds on twitter may well be the only one that captures public attention. If the research is ever revised or rejected for publication – as many are – the public will already have been misinformed. Journalists will not revisit the corrected version; the damage has been done. We could perhaps live with that when the subject is gravitational waves. But can we risk that damage to public understanding on vaccines, climate change, diet or fertility?

Peer review may not be perfect, but most people agree that one of the good things it tends to do is to tone down findings and filter out overclaiming. It also sometimes means that papers that are especially weak and flawed never make it to publication and the public have been saved a scare story based on flaky evidence.  I understand the peer review process will in many ways be enhanced by preprint and that does excite me. New research will enjoy the benefit of scrutiny by hundreds of peers, not just two or three. Terrible papers will hopefully sink under the assault of an army of scientists pulling them apart while the best papers will rise to the top. I like all that. But the problem is that the public, who are not qualified to pull new findings apart, may be hearing about them before any of this beautiful self-correcting stuff takes place. The stable door will be more firmly shut on bad research than ever before, but for the public the horse will have long bolted.

The answer I get when I raise these kinds of concerns is that this kind of reporting already happens. Poor quality research already makes it into peer-reviewed publications and then into the press; non-peer-reviewed abstracts from conferences are already routinely reported as big news because of unscrupulous over-claiming for preliminary research. All this is true, but it’s hardly reassuring is it? After all it’s not often that I lie awake at night thinking that the answer to these problems is to inject a whole load more non-peer-reviewed research into the system. The last thing we need is a race to the bottom.

Preprint may also mean less coverage as well as less quality coverage. Do authors know this?

Another thing that should focus the minds of scientists and science press officers is the potential impact on the quantity and impact of the media coverage of new findings. We already know of cases where unsolicited media coverage of a newsworthy preprint adversely affected the amount of coverage when it was finally published. As well as the positive impact of embargoes on quality of reporting, the embargo also plays the equitable function of allowing all the media to run an important story at the same time. I am conscious that there are critics out there who see the embargo system as a horribly conformist and controlling system that excludes journalists working on Sunday papers and discourages original and investigative reporting. Well maybe. But if we get back to the public interest I would say that the embargo system wholly benefits the public interest, entitling readers of The Sun and listeners to Today to learn of important findings at the same time. Embargoes create a level playing field.

I have long reminded scientists and journal editors not to take to take it for granted that important scientific findings on everything from climate change to drug safety to mitochondrial donation are conveyed to a mass audience by responsible science journalists on a daily basis. The Lancet’s network meta-analysis on anti-depressants earlier this year made several front pages including the Sun, Times and Guardian. As a result few people in the UK do not now know that the weight of good quality evidence on anti-depressants indicates that they work better than placebo. But what if that study had been found on a preprint server by an intrepid Guardian reporter 6 weeks before publication? If they had splashed it on the front page it’s unlikely that other papers would follow up a scoop by a rival; even less likely they would wait patiently for the peer-reviewed version. Readers of other newspapers and people who rely on TV news may well miss the story altogether. I wonder if journal editors and authors posting on preprint have thought about this. Have they been encouraged to consider that by putting their findings in the public domain they may also inadvertently be limiting the amount and impact of media coverage when their findings are published? Strong new evidence in the public interest deserves wide, loud coverage; I am worried that a preprint model may jeopardise that.

Embargoes fulfil another important purpose of course: they provide journalists with breathing space to weigh up, summarise and seek critique to new findings. In a world of 24 hour rolling news where journalists are rewarded for breaking stories even minutes before the competition, the pressure to rush to publish is stronger than ever. The embargo system keeps that in check. Journalists generally find out about new research several days before they may publish. They can break those embargoes of course; but they don’t. The real reason for that is because they like them. Far from curtailing their investigative instincts, they recognise that embargoes give them the opportunity to seek the thoughts of experts in the field. Every week at the SMC we see attractive stories tempered by 3rd party comment and sometimes sinking without trace – precisely because they have been critiqued by the scientific community. Journalists don’t want to write stories built on bad science, and the public shouldn’t be fed dodgy claims presented as fact. Embargoes allow this system to flourish. Without them stories will be hurriedly written, and scientists and journalists will have no chance to tackle weak or false claims before they are presented to an unsuspecting public.

Some ideas for discussion

Embargoes may yet have a place in a preprint world – but not if we throw them out prematurely

The received wisdom on embargoes in our discussions on preprint is that no science press officer will be able to apply an embargo to a published paper when it is already in the public domain as a preprint. Some have even suggested that it is morally untenable or unethical to impose an embargo restricting journalists from running a story which is ostensibly already out there. This is being repeated as though it’s a self-evident truth – but I want to challenge it.

I can clearly see the arguments as to why embargoes will be difficult to enforce with preprint but there are strong moral and practical arguments for ensuring that new findings with a significant public impact are managed in a way that maximises good quality reporting. Before abandoning the entire embargo system with a somewhat fatalistic shrug – ‘what else can we do?’ – I suggest we spend some time thinking about our options.

Already I can see some discussion points here.  Could we have different criteria for publications and save the embargo system for papers with clinical or public health implications? Or could we argue to journalists that embargoing a final paper that has now completed peer review and been accepted by a journal is still legitimate because it will differ in meaningful ways to the paper that is sitting on preprint? We should remember here that journalists enjoy benefits from embargoes too. If the BMJ or JAMA treat their published papers as qualitatively distinct from preprints then the journal, the authors and the media may benefit from us not giving up on the embargo quite so casually. At least it’s worth discussing?

Best Practice guidelines

As I said, one of the arguments made by preprint fans is that most of what I worry about already happens. Science gets into the news before it’s published all the time for a variety of reasons, and poor science makes it into journals far too often. Moreover, I am telling you that we routinely minimise the damage from these through the combined efforts of responsible science press officers, good science journalists and the SMC’s ‘Roundups’ service. Surely we just need to adapt our current approaches to the new kid on the block?

Maybe. But I still think we need to use this time to thrash out best practice and agree what the new rules should look like. One discussion point would be to agree that good practice would mean universities not issuing press releases at preprint stage. That would acknowledge that the aim of preprint is to open up the process to allow experts to scrutinise and learn from their peers’ work – and not to seek publicity or promote it to the wider public as accepted findings.  We could play with terminology here and start using terms that clearly differentiate preprints from accepted papers. We could also discuss whether press officers should routinely issue guidelines to their scientists or authors, urging them not to promote preprint findings to the media before they are accepted for publication (some journals are already taking steps in this direction).

Are there other ways?

Or is there something we have not thought of that could get us round these new realities with minimum adverse effects?  Maybe journals can no longer embargo new studies but the press officers would still know when they were planning to issue an ‘immediate release’ and could prepare additional materials to help journalists get round the lack of time. We have already asked journal press officers whether they might also be able to give us advance notice so that journalists could continue to see third party comments from leading scientists at the same time as they see the news release. Some certainly seem open to considering that. And it’s possible, of course, that journalists would be content to persist with a system of embargoed press releases – even in the full knowledge that an early, preprint version of the work may exist on a public website somewhere. After all, an embargo will hold as long as everyone agrees to hold it.  At the very least we must seek their views.

Conclusion

What has really struck me in my discussions about preprint is that the changes being made to a part of the system that was not working are set to have profound knock on effects on another part of the system that works and serves science well. The challenge here is to fix one end without losing the gains we have made in reporting findings to the public in an accurate and measured way. Science is rightly proud of its reputation for informing the public based on strong evidence. Let’s not destroy that in pursuit of preprint – however great the prize.