select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
The report published in Philosophical Transactions of the Royal Society A looks at the effectiveness of non-pharmaceutical interventions for COVID-19.
This Roundup accompanied an SMC Briefing.
Prof Christophe Fraser, Moh Family Foundation Professor of Infectious Disease Epidemiology, Pandemic Sciences Institute, University of Oxford, said:
“Infectious disease epidemics and pandemics are unfortunately likely to occur in the coming years. Non pharmaceutical interventions will be needed to keep infection in check, whilst vaccines and treatments are developed. Learning the lessons from COVID-19 is critical for improving our responses next time, as is sustained investment in public health and research. Contact tracing apps could increase the chance of containing epidemics, and this authoritative review makes the case for a sustained investment in these novel tools.”
Dr Michelle Kendall, Research Fellow, Department of Statistics, University of Warwick, said
“It is challenging to evaluate the efficacy of non-pharmaceutical interventions (NPIs) for COVID-19, particularly as interventions were typically used in combination, in time-varying ways, and – for understandable reasons – without the ideal research structure for ease of evaluation, for example roll-out via randomised controlled trials. Nevertheless, such evaluations are much-needed because of the wide societal impacts of NPIs, and to learn lessons for the future. The findings of this report are important, as are its practical recommendations towards enabling more rapid and robust cost-benefit evaluations in future.
“We are pleased to see the report’s acknowledgement of the effectiveness of digital contact tracing apps. Their inclusion in the evaluation is especially welcome because these tools – particularly the NHS COVID-19 app for England and Wales – were shown to have had important epidemiological impacts, but their significance in the toolkit has sometimes been overlooked since the initial hype. More recent research by our group, in addition to that considered in the report, strengthens the conclusion that digital contact tracing helped to reduce the spread of SARS-CoV-2 in England and Wales.
“This review discusses communication and adherence to NPIs as critical considerations for these interventions. This is a welcome reminder of the fact that, during challenging times, the significant efforts of individuals were key in fighting the virus and should be acknowledged. Special thanks should be reserved for all the people who engaged with these NPIs, particularly those targeted NPIs like testing and tracing which helped relieve our reliance on blanket emergency measures like social distancing and lockdowns. There should be a long-term focus on fostering public trust as a key ingredient of most NPIs presented in the review.”
Dr Luca Ferretti, Senior Researcher in Statistical Genetics and Pathogen Dynamics and Pandemic Sciences Institute Fellow, University of Oxford, said:
“This authoritative review confirms the effectiveness of many well known public health measures against infectious diseases. Crucially, it also supports the usefulness of some novel approaches such as contact tracing apps when embedded in a strong public health framework and in a context of public trust and support. As emphasised in the review, continuous study and development of improved interventions is key for effective management of future outbreaks.”
Dr Julian Tang, Clinical Virologist, University of Leicester, said:
“This brief summary is nothing surprising really:
“These interventions are actually all just common sense – even if you are not quite sure exactly how the virus transmits predominantly – because this has been known for various infectious organisms for hundreds of years.
“The only situation where they don’t apply so much is if the infectious agent was vector-borne – like via mosquitoes; or if it is predominantly food- or water-borne.
“In fact, the behaviour of this coronavirus, as a specialist in respiratory viruses and aerosol transmission, has not really surprised me throughout the pandemic. But, rather, the reaction of some people to some of the interventions proposed above – including what some believed about the nature of the virus – was more surprising (masks don’t work, don’t buy them; it’s not airborne – just wash your hands; asymptomatics don’t transmit, children don’t get infected and don’t transmit, etc.)
“So I welcome this summary and agree with its main findings – and hope that we will all heed these for the next pandemic – which is again, likely to be a respiratory RNA virus of some sort – as previously, SARS-CoV-1, pandemic influenza, and SARS-CoV-2 – because:
Prof Kevin McConway, Emeritus Professor of Applied Statistics, Open University, said:
“This report from the Royal Society is certainly major in terms of length. The main report and the evidence-led reviews that accompany it come to well over 200 pages, and the supplementary documentation for the reviews come to nearly double that number of pages again. I freely confess to not having read every word. A lot of work has gone into producing it. But was that work worthwhile? I think it was, but only if we take into account what was actually done and, more importantly, what wasn’t done.
“We all know there was a pandemic and that there were lots of non-pharmaceutical interventions (NPIs), which were applied in different ways in different countries and at different times. And it’s a natural reaction to wonder how well it all worked. The overall covering report does state the aims clearly – to consider what the current scientific evidence tells us about the effectiveness of NPIs in preventing the transmission of SARS-CoV-2. So it’s only about preventing (or reducing) transmission, not about reducing illness and death, let alone any other outcomes. Clearly if transmission is reduced, you’d expect illness and death directly caused by the virus to reduce too, because if someone doesn’t catch Covid-19, they can’t get ill with it or die from it. But that doesn’t necessarily mean that the best policy to reduce illness and death is to reduce transmission as far as possible. And the report makes it clear that the aim of NPIs wasn’t necessarily to reduce transmission as far as possible anyway – the aim varied between countries and with time, but was often to keep illness and death to levels that could be coped with, without causing healthcare and other systems to be overwhelmed.
“The report does point out explicitly that NPIs can impose a great number of costs and burdens, in terms social and economic impacts, and indeed of increasing ill health (physical and mental) not directly caused by the SARS-CoV-2 virus. It also says (rightly) that understanding all these other impacts is very important, but makes it very explicit that this piece of work isn’t going to consider any of that. So the work was only ever going to deal with only a rather narrow part of the question of whether the ‘right’ things were done during the pandemic, and I think that limits quite severely its effectiveness in helping decisions on what should be done in the next pandemic, whenever it arises.
“I’m absolutely not saying that this report should not have been done – only that the work is aimed at answering a rather specific question, and must definitely not be taken to be the last scientific word on NPIs in the context of Covid-19. If nothing else, it does provide useful summaries of the work that has been done on evaluating NPIs during the pandemic. But it shows many other things too.
“In a way, one could argue that this collection of reports shows up the work of scientists on NPIs, during the pandemic, in a rather unfavourable light. Measures to reduce transmission, whether they are NPIs or pharmaceutical, can’t be tested properly in the real world when transmission isn’t occurring. The great majority of the studies of NPIs carried out during the pandemic were observational. Observational studies are much more likely to be subject to biases than are, for instance, randomised controlled trials (RCTs). Here, I’m not talking about political or personal biases on the part of researchers, but about the difficulty of working out what causes what in an observational study. That difficulty arises because people, and groups of people, will always differ in ways other than the way that is being researched. I’m certainly not claiming that RCTs of NPIs would be easy to do in the middle of a pandemic, far from it. But surely it’s important to make an effort to carry them out at a time when they can be carried out, that is, during a pandemic or epidemic. Several good RCTs of pharmaceutical interventions (drugs, and also vaccines) were carried out during the pandemic, after all, and under difficult conditions and pressures. RCTs of NPIs would probably have been even harder to do – but did we (scientists and research funders) try enough? The report points out many important difficulties of conducing RCTs of NPIs during the pandemic, including that different NPIs were generally introduced in combination with one another, so that picking out the effects of individual interventions would be really difficult. But could we have tried harder? And could more and better collaboration on designing good observational studies have helped understand what worked and what didn’t? I think so.
“Even if doing more RCTs would have been very difficult or impossible, the report is forthright in proposing that lessons should be learned about the future. Some of the lessons that are proposed are for policymakers and their advisers, on how to choose and implement NPIs, but what interests me more is lessons for researchers and for research funders. The researchers write that “One key lesson to researchers is to ‘be prepared’.” (Page 65 of main report.) The idea is that, in advance of a future pandemic, national and international collaborations should be set up, with standardised ways of data collection. They do give an example where this had happened before the Covid pandemic, namely ISARIC, the International Severe Acute Respiratory Infection Consortium, which was founded in 2011, and did contribute to rapid collaboration during Covid-19. Setting up new collaborations from scratch when a new pandemic is arising is just too time-consuming.
“In relation to what needs to be done in the future, the main report asks for much more, and more coherent, collection of data on the adverse effects of NPIs, including social and economic aspects as well as health and wellbeing. Doing that properly would also requires serious planning and collaboration. The researchers point out, rightly, that having better information on the costs (in every sense) of NPIs would help policymakers “to make the best policy decisions on NPIs, which are in the main complex social interventions.” Is there an implication here that the best policy decisions were not made during the Covid pandemic, because the relevant information was not available at the right time? The researchers point out that this requirement for better information has important implications on research collaborations, which need to take these aspects into account, and to plan for them for the future.
“In that context it looks rather ironic that this report, and the evidence reviews that accompany it, chose explicitly to leave consideration of adverse effects largely out of the picture, particularly social and economic effects. Are they saying that so little was known about these adverse effects that they just can’t be considered? Surely not, even though they are right in saying that collecting evidence on them should be done better in future. So they seem to be saying, simply, that this kind of assessment is out of scope of this piece of work. If not the Royal Society, who is actually going to do it for SARS-CoV-2? The writers say (page 67) that “there is a strong case for the development of another report, complementary to this one, based on a series of evidence reviews, examining what has been learnt during the pandemic about the full range of social and economic impacts.” Absolutely. But that’s a lot of work, and would need to involve experts from many fields. Who will do it? They mention (page 25) that the consideration of all the health, social and economic impacts “is a key question for inquiries being conducted around the world.” But will those inquiries, most of which have strong political aspects, get to the bottom of that question, and how should scientists (and social scientists) contribute?
“The structure of the report and its accompanying evidence reviews is quite complicated. There are six evidence reviews, to support the main report, together with case studies of NPIs in Hong Kong, New Zealand and South Korea. The evidence reviews aim to summarise previous research, rather than to do any new primary research. To a considerable extent, the teams working on the evidence reviews for each of the specific types of NPI (masks, social distancing, and so on) used a common methodological framework for what they did, and that that framework is based on international statements of good practice in reviewing other research, that have been developed and refined over time. That’s good. But no such framework, however clear it is, can make the whole process of reviewing previous studies entirely objective. For example, in each case the review team had to deal with studies carried out in different ways, with different levels of quality (many of them not of very high quality), and had to make decisions within the overall framework on how to deal with that.
“And there were wider differences between the size of the tasks the review teams were faced with, and aspects of how they approached them. The team reviewing communications found only 13 studies to review, whereas the team reviewing social distancing NPIs (of which there were a great range, from near-complete lockdowns with stay-at-home orders, to school closures, to advice to keep two metres away from others where possible) had to cover well over 300 studies. The social distancing reviewers do spend some time discussing the effects of NPIs on deaths, which arguably doesn’t fit so well with the stated aim to consider effects on transmission – but many of the studies they were reviewing did concentrate on death and/or serious illness. The team reviewing international border controls had somewhat over 100 studies to summarise, but their review is mostly based on five previously published reviews rather than directly on the original studies themselves. Nothing immediately wrong with that, but it does make the review, and the conclusions that it could draw, look rather different from the others. I’d be surprised if members of any one evidence review team would agree with everything all the other teams did. That’s just the nature of carrying out complicated reviews such as these.
“Doing a review like these doesn’t magic away the uncertainty (and the researchers aren’t pretending that it does). And there does remain rather a lot of uncertainty. That arises from many aspects, including differences in detail of the types of previous studies that were reviewed, differences in context in different countries and at different times in the pandemic, to some extent differences in reviewing methods, and more. It also arises, importantly, because NPIs were almost always carried out in combination with one another, not singly, so separating out the effect of one single NPI is at least very difficult, and often impossible. Because of all this, there isn’t much quantitative information from these reviews on how much a given NPI would reduce transmission by. That’s another reason why trying to use the information from this report to plan detailed interventions in a future epidemic, where the pathogen is likely to behave differently from SARS-CoV-2 anyway, could be pretty futile. It’s the general approach that will be important, not the detailed numbers.
Communication
“Uncertainty is always prominent in a statistician’s mind, and it raises its head in a different way in the evidence review on communication about NPIs (which I have read more closely than the others). Unlike the other evidence reviews,, which considered worldwide sources of evidence, this is based only on previous studies that focused on, or included, the UK. Also unlike the others, it is not looking directly at the effect of communication on reducing transmission of the virus, but generally at the effectiveness of communication in encouraging people to take part in the NPIs that were proposed to them or imposed on them. (Effective communication of a measure that in fact doesn’t reduce transmission is, in a wider sense, not really effective at all.) The researchers did find a few themes emerging. Low trust in government was associated with low adherence to NPIs (with the implication that information should be conveyed by trusted sources – which does assume that we know which they are and how their trustworthiness arises and is perceived). Too many messages, particularly if they conflicted with one another, were a barrier to adhering to NPIs (so mixed messages should be avoided). Communications that struck the right balance between being authoritative and being controlling seemed to work best (but is it clear what the right balance is, and how to ensure it?).
“In quite a strong sense, none of this is new, and the reviewers aren’t claiming that it is. There has been a huge amount of research on effective communication of messages encouraging or exhorting people to behave in particular ways, including in relation to epidemics, and it would have been surprising if research on communication during Covid-19 led to radically different conclusions. But it concerns me that the research that was reviewed here is mostly short-term, and that there is not much discussion of uncertainty. There was a great deal of uncertainty about the effects of the virus and of different interventions, particularly in the early part of the pandemic, which is not surprising given that the virus was new and its properties were initially largely unknown. The reviewers mention that one of the studies they reviewed pointed out that inadequate communication of scientific uncertainty led to confusion in the recipients of the communication. Other previous research, for example carried out at Cardiff University and at the Winton Centre for Risk Communication in Cambridge, has established that communicating uncertainty properly does not hinder understanding and, in the longer term, can help. Has that lesson been learned? I do wonder about the effect on communication and on compliance with NPIs in a future epidemic, and on trust in science generally, when uncertainty was glossed over so often in some of the communications during Covid-19.”
‘COVID-19: examining the effectiveness of non-pharmaceutical interventions’ a report by the Royal Society was published in Philosophical Transactions of the Royal Society A at 00:01 UK time on Thursday 24th August.
Declared interests
Prof Kevin McConway “I am a Trustee of the SMC and a member of its Advisory Committee. My quote above is in my capacity as an independent professional statistician.”
For all other experts, no reply to our request for DOIs was received.