select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
A preprint, a non-peer reviewed unpublished paper posted on medRxiv, looked at a possible blueprint out of lockdown based on analysis of effects of various measures used in several European countries.
Prof Kevin McConway, Emeritus Professor of Applied Statistics, The Open University, said:
“This study is interesting. But is it a good guide to choice of future policies on lockdowns and other restrictions? I’m not at all sure that it is, and certainly it has important limitations and issues of interpretation that need to be taken into account. We certainly should take account of policies, and their effects, in other countries in coming to decisions about what to do here, but these finding are far from being the last word on that.
“My first concern is that this is an observational study. The authors describe it (appropriately) as a quasi-experimental study, but ‘quasi-experimental’ here describes an observational study of a reasonably elaborate and careful sort. As with any observational study, it can be difficult or impossible to establish what is causing what. To put it simply, there are differences between countries, that can or may affect the numbers of COVID-19 cases and deaths there, which aren’t explained by differences in policy on the five interventions that were considered. Perhaps these differences contribute, at least in part, to what’s causing changes in cases and deaths. The researchers took into account overall differences between countries that did not change over time (by including terms for ‘country’ in their models), but not differences that might have changed over time. That does not give a strong basis for concluding that the associations between the interventions and the rates of cases and death are largely due to the interventions.
“Then I have concerns over the data. The researchers have obtained their data on numbers of cases and deaths from a reputable standard source, but that source collects and collates data from individual countries. There have been many concerns that deaths and cases are being counted in different ways in different countries, and in some countries, also concerns that the basis for the counts has changed over time. This could affect the conclusions from the modelling. The researchers did take into account the number of tests for the disease that were used in different countries each day, but that’s far from the only reason for differences in case and death counts.
“But perhaps my main concerns are about the meaning of the research findings, if they are to be applied to guide policy on removing restrictions on normal life. One obvious issue is that the findings are entirely based on the introduction of restrictions. The researchers acknowledge that each restriction would have been implemented, and enforced (if it was enforced), differently in different countries. But I think it’s not appropriate to make the assumption that what happens after a restriction is removed would be the same as the position before the restriction was put into place. What happens after the removal of a restriction depends on how people choose to behave afterwards. It seems very unlikely to me that what goes on in schools, or how often people leave their homes, would be the same after a period when schools were closed and people were told to stay at home. So perhaps the research can tell us something in hindsight about which measures were more useful than others – subject to the important issues I’ve already described – but it tells us much less about the process of removing restrictions.
“A point that is very important, though rather technical, is as follows. The researchers included all five interventions that they considered in their statistical models at once. That’s fine, because that’s the only way one can take into account their combined effects. But the consequence is that each measure of the effectiveness of one of the interventions is actually reporting its additional association with cases or deaths, after the other four interventions have been taken into account. So these measures do not really show the effect of an intervention taken on its own. For example, in a country where there have already been restrictions on mass gatherings, and business and schools are closed, people may be staying home anyway. This could mean that it isn’t worth imposing stay home measures as well, because people are already staying at home – or it could mean that the effect of stay home measures is hard to estimate properly because of the side effects of the other measures. We just can’t tell. It certainly tells us rather little, not directly anyway, about the effects of imposing stay at home measures in the absence of business and/or school closures, or indeed business closures in the absence of stay at home measures – and, unfortunately, it tells us even less about the possible effects of removing any of these measures. The researchers clearly point out that this kind of association or overlap between the effects of different measures “makes it hard to separate out individual intervention effects.” They are absolutely correct on that, but it hasn’t stopped them making rather explicit statements about the effects of individual interventions.”
Prof Keith Neal, Emeritus Professor of the Epidemiology of Infectious Diseases, University of Nottingham, said:
“The main limitation of the study is that in many countries various measures were put in place at the same time making it difficult to disentangle which effects were having an impact. The strength lies in looking at data from 30 countries.
“The more of the lockdown restrictions that have a minimal, if any, benefit in controlling spread that are then relaxed the better the physical and mental health of the country will be.
“They found stay-at-home orders and closure of non-essential businesses appear to have minimal effects on transmission prevention. Relaxing these would therefore likely to have minimal adverse consequences on the epidemic.
“In controlling spread their main finding was that school closures were important. As many schools have now gone, or soon will go, back in Europe the impact of school closures and the re-opening will become apparent and help the UK’s exit strategy with real data. Next of importance was the banning of mass gatherings comes as no great surprise and also has little impact on the economy overall if it needs to be continued.
“Overall the paper helps those planning the exit strategy with some more science.”
Prof Trish Greenhalgh, Professor of Primary Care Health Sciences, Nuffield Department of Primary Care Health Sciences, University of Oxford, said:
Commenting only on the face coverings aspect:
“The authors are to be commended for emphasising that, in relation to face coverings, their findings are “preliminary” and should not be used to inform policy. It is worth considering just how preliminary the findings on face coverings in this study are. The authors looked at data up until April 24th. It takes a month or so until the impact of an intervention can be seen on deaths (since the intervention first has to impact transmission rates, then those newly infected have to get sick, and then eventually die). So we need to look at countries that introduced mask requirements before March 24th to analyse impact on deaths. But there are only two countries in Europe that did so – Slovakia and Czech Republic, 14th and 18th March respectively – so they are looking at just two data points for this sub-analysis.
“An alternative analysis might have considered Covid-19 cases (which would show more quickly than deaths, allowing more countries to be included), but countries changed their testing level significantly throughout this period, which may be why the researchers largely ignored this variable (they only include a single testing number: the number of tests as at 16 April). Another way of strengthening the study would be to look outside Europe, where there is a lot more data from countries which introduced policies on face coverings much earlier.
“Two other studies, both with more data points, should be taken into account. A group led by Christopher Leffler from Virginia Commonwealth University in USA, undertook a similar cross-country study including Slovakia and Czech Republic plus Thailand and Vietnam, as well as studying the experience of several countries where mask wearing was not mandatory but had become widespread (Hong Kong, South Korea, Malaysia, Taiwan, Japan, Mongolia). Lanjing Xiang’s team from Rutgers University, USA, analysed both cases and deaths across the USA between 1st March and 20th April. They tested the hypothesis that the stay-at-home order (SAHO, introduced across various US states between 19th March and 7th April, and removed in some states a few weeks later) and a requirement to wear face masks in public places (introduced in USA on 3rd April) was associated with a reduction in subsequent new cases and deaths. In both of these larger studies using different modelling methods, both teams came to the opposite conclusion to Hunter’s: that the widespread wearing of face coverings by the lay public had a statistically significant impact on Covid-19 (deaths in the former study and both cases and deaths in the latter study). Importantly, none of these papers has yet been subject to expert peer review.”
Dr Joshua Moon, research fellow in sustainability research methods in the Science Policy Research Unit at the University of Sussex Business School, said:
“The press release itself does accurately reflect the findings of the study, and the analysis is robust.
“There are, however, a couple of points where perhaps there are flaws in the interpretation or conclusions drawn by the paper. First and foremost, the paper does not seem to control for differences in testing rates and testing strategies in each country which will also impact the number of cases. This is particularly important when you consider the stay-at-home findings or the mask findings which in many countries happened in a similar time-frame as increases in testing for covid-19. This would make it unsurprising that cases around these tended to increase.
“That being said, there is the fact that other measures demonstrated decreases when those were likely concomitant with increases in testing. There is a lot to be deciphered still. What would be particularly interesting is seeing how different countries followed these trends and looking more qualitatively at the outlying cases. This would allow a much more causal analysis, as opposed to looking at correlations.
“When it comes to what this means for actual policy, this could indeed be a useful piece of evidence for policymaking. This study would indicate that mask-wearing and stay-at-home orders could be the first to be relaxed, then others come later in a more staged approach. This should be coupled with intense testing, tracing, and isolating to ensure that transmission is minimised.
“However, we have to remember that decisions like this cannot and should not be made on a single finding. Nor should policy be made based solely upon science – there are many social, economic, political, and moral factors to consider that science simply cannot answer. When it comes to this pandemic, caution is paramount, otherwise we could tip too far and risk a second wave and a return to lockdown.”
Preprint (not a paper): ‘Impact of non-pharmaceutical interventions against COVID-19 in Europe: a quasi-experimental study’ by Paul Raymond Hunter al. is on medRxiv. This work is not peer-reviewed.
https://www.medrxiv.org/content/10.1101/2020.05.01.20088260v1
All our previous output on this subject can be seen at this weblink:
www.sciencemediacentre.org/tag/covid-19
Declared interests
None received.