October 1, 2020

expert reaction to preprint with the latest interim data from the REACT-1 study on COVID-19 spread across England

A preprint, an unpublished non-peer reviewed paper from Imperial College London, has looked at interim data from the REACT-1 study on COVID-19 spread across England.

This Roundup accompanied an SMC Briefing.

Dr Mike Tildesley, Associate Professor, University of Warwick, said:

“The REACT study is a large scale study examining infection across the UK, whereby 150,000 randomly selected participants are tested each month.  The results suggest that R may have reduced significantly from early to late September, though it is important to stress that there is uncertainty around this mean value, with confidence intervals suggesting R is somewhere between 0.75 and 1.4.  Their study suggests that R is most likely still above one, so this is definitely not a cause for complacency, but it may be an early indication that the introduction of measures in late September are having some effect.”

An anonymous disease modeller, said:

“REACT is a well-structured and statistically sound study.  However, care should be taken to not over interpret the slight downturn in positivity seen in the tail of the fifth round especially as this round is currently only partially complete.

“Epidemiologically the story is in the jump in prevalence from 0.13% to 0.55% (Table 1, P10)  over the period of a couple of weeks.  This, crudely, would place us at about the beginning of March on the first wave timeline, albeit with a slower growth rate.

“I believe the putative ‘slowing’ in the growth rate comes from the fits to individual round data (shown in Figure 2B, P12) where the slope of round 5 is less than that of round 4.  I feel that this is misleading as we should consider the pandemic as continuous (the virus doesn’t pack up it’s bags and go on holiday between rounds) and so we should be more concerned with the models that have been fit across all rounds (Figure 2A, P12) where if anything there is a speeding of the epidemic growth.”

Prof Matt Keeling, Professor of Populations and Disease, University of Warwick, said:

“I have huge admiration for the REACT study, it’s been massively influential in the past months for giving early indications of the scale of COVID infection in the UK.

“The statistical approach taken by the REACT team is first-rate, and I would therefore trust the numbers.

“The key issue is the messaging and the head-lines around R dropping to 1.1.  The value of R=1.1, is based on just 9 time points in round 5, and has large confidence intervals (0.74-1.45).  I would be much more inclined to look at the change including both rounds 4 and 5, which captures the large rise in cases over recent weeks and leads to R=1.47 (confidence interval 1.40-1.53) — which is much more in keeping with recent estimates from SPI-M.”

Prof Kevin McConway, Emeritus Professor of Applied Statistics, The Open University, said:

“It’s not surprising that results from the REACT-1 study have shown a large increase in the prevalence of SARS-CoV-2 infection in comparison to their previous report.  Their previous report already showed strong signs that infections were increasing, and the ONS infection survey and data from Test and Trace and other community testing since then have confirmed the increasing pattern.  REACT-1, and the ONS survey, are particularly important in tracking the course of the pandemic.  That’s because they both take data from samples of people across the English population, simply with the aim of estimating patterns of infections, so they aren’t affected by the general availability of swab tests in Test and Trace or on changing conditions on who can ask for a test.

“While it’s very concerning that the infection rate has gone up so much since their last report (which covered tests up to 8 September), obviously the most interesting news is that the rate of increase seems to be slowing.  That’s good news, but I’d urge a certain amount of caution about it.  This is an interim report, not based on the full data for round 5 of the study (because that is ongoing).  So there’s quite a lot of statistical uncertainty about the R number that they estimate from the most recent data.  That estimate is based only on 9 days of data (18-26 September).  The central estimate of R from that data is 1.06.  That’s still bigger than 1, indicating that each 100 infected people will on average infect 106 other people.  If that continues, infections won’t fall; they will grow more slowly than was previously estimated.  The tide may be advancing more slowly, but it hasn’t turned yet.  The inevitable statistical uncertainty means that there’s a wide margin of error around this estimate, from 0.74 to 1.46, so it’s not even absolutely clear that the R number has fallen much (the estimate from REACT-1’s round 4 and round 5 so far, put together, was 1.47).  Since that range goes below 1, the REACT-1 team are not certain that R is actually above 1 now, and that infections are still increasing.  But they estimate the probability that R is above 1 at 63%, so again an indication that the tide (probably) hasn’t turned.  We do still need to be very careful, and restrictions and interventions continue to be needed, even though research like this can’t be clear on exactly which interventions might have been most effective.

“We’ll get more information about levels of infection in England when this week’s results from the ONS infection survey are published tomorrow.  Tomorrow’s ONS bulletin should also cover the period up to 26 September, so we’ll get a good indication of whether these two important surveys are giving similar pictures (as indeed they have generally done in the past).”

Dr Simon White, Senior Investigator Statistician at the Medical Research Council Biostatistics Unit, University of Cambridge, said:

Is this robust data; what are the strengths and limitations?

“The report details preliminary results from the fifth wave of an on-going nationwide surveillance programme, called REACT.

“The current (5th) wave is only partially complete (80,000 out of the planned 150,000).  It is not clear why the authors are releasing this preliminary report now?  Have they released preliminary reports for earlier waves, was this preliminary report part of a pre-defined protocol or have they decided to release this early for another reason?

What does this say about R, and what does or could that mean?

“The paper, as released, does not include complete details on how the authors derive their R number.  In an earlier REACT report they cite a method of estimating R from trajectory matching, that original paper discusses some of the serious biases that can be introduced by failing to account for latent periods in the infection.  It is unclear if the REACT method is accounting for the issues raised in that paper, however it appears a consistent method has been used across all REACT reports, so comparisons between waves are sensible.

What does this say about viral prevalence, and what does or could that mean?

“A key feature of the regional estimates, and the deriviation of ‘clusters’ (the paper has a specific definition for a ‘cluster’ that may not be inline with the common understanding of the term, take care) is using the address of the volunteer.  I suspect, given the drastic change in movement patterns within the 18-24 age group (due to the return to higher education) that the fifth wave may have issues with regional estimates.

Is it important to consider both prevalence and R?

“Prevalence, both types (weighted and unweighted) is distinct from the R number.  The R number is giving information about the likely future, crudely it is ‘the number of people an infected person is expected to infect’, hence an R number less than one means the epidemic will eventually stop.  However, the derivation of R is based on more assumptions.

“The unweighted prevalence is purely a count of the number of positives among all the volunteers, as such the key questions are about how representative the sample is and whether the test results are reliable.  The authors address both these issues, using a nation-wide volunteer sample and account for the non-perfect swab test.  The weighted prevalence estimates correct for the sample not being representative of the whole population, by adjusting for certain charactersitics within the population.

“Prevalence is an instantaneous measure whereas R indicates something about the epidemic process.  Both are essential for understanding an infectious disease.

Is it good news that R is estimated to be 1.1 – does that mean things have got better; has it gone down?

“As mentioned above, the calculation of R requires more assumptions than the prevalence estimates, there is greater uncertainty.  Further, the R number includes the impact of policy decisions, changes in how the population are mixing, and various other social effects.  Hence, the ‘true’ R is highly uncertain and changes on a daily basis.

“Within the paper they estimate the probability of R being greater than one as 63%, that’s really the key number.  Bigger than one means the epidemic continues, less than one means it will (eventually, though that could take a long time) end.

Does a lower R mean current measures are working, or is it too early or not possible to say?

“As the paper calculates R, it is a combination of many factors – changing measures/policy is one, but also changes in seasonal effects (flu being the obvious, but also changes in working and studying patterns).  So it’s hard to separate out the specific impact of the current measures – in this paper.

“So you cannot say the either way – from this paper.

What does the data show about different geographical regions and/or different age-groups and/or different ethnicities?

“The paper provides regional prevalence and R estimates.  Based on smaller samples they are more uncertain.

“The paper also considers a logistic model to provide odds-ratios for having a positive swab, accounting for various characteristcs, specifically age and ethnicity.  There are some issues with this model, for instance the authors divide people into: “other worker” (reference level), “key worker (other)”, “HCW/CHW” and “Not PT, FT, SE (Not part-time, full-time, or self-employed)”.  That last category changes its label on the graph, being “Not FT, PT, SE”.  It is unclear whether the authors mean ‘not FT, not PT, not SE’ or ‘not FT, or PT, or SE’.  The main thrust of this ambiguity is, where are students?  There is no student or child group, only types of worker. Since the data includes under 18s, where are they?

“Further, the authors state they adjust for deprivation, however this is not mentioned in Table S2.  Deprivation has a strong geographical and ethnic effect (I won’t provide citations for this, but there is a lot of work on deprivation and ethnicity).  I feel this puts a question mark on their ethnicity and geographic odds-ratios, as well as the working group and age group odds-ratios.

Any other comments?

“REACT is a very important study.  It is entirely separate from the ‘test and trace’ system, and any hospital-level surveillance, meaning it provides a better national-level view of infections.  It has multiple waves with a consistent methodology (again, contrasting to the changing ‘test and trace’ availability).

“The main messages in the paper are robust.  However, there are a few issues of concern (why publish an interim report, and whether the odds-ratios are adjusted for deprivation and how under 18s are grouped).”

Dr Julian Tang, Honorary Associate Professor in Respiratory Sciences, University of Leicester, said:

“The value of these REACT studies is that they are community surveys conducted on randomly selected individuals – unlike the Pillar 1 and 2 testing which are still focused mainly on symptomatic individuals.

“This means that there is a larger base denominator of both symptomatic and asymptomatic individuals being tested which will usually give an overall lower, cross-sectional incidence figure for the infection – but also gives a useful indicator of what proportion of cases are symptomatic or asymptomatic at the time of testing.

“All nine regions of England are now showing significant increases in the number of COVID-19 cases – emphasising that we all need to comply better with the various social distancing measures to reduce the number and frequency of contacts that we have with others, together with the masking and hand-washing.

“More worryingly, the numbers of cases in the older 65+ population (7-fold increase) is also increasing and this may continue into the more vulnerable elderly population – if we don’t act to curb the spread of this virus now.  The study also reported higher infection rates (doubled compared to the white population) in the BAME population, as previously.

“The alternative is that we just learn to live with COVID-19 as we do for seasonal influenza – which also causes significant morbidity and mortality each year.  But with influenza, we already have vaccines and antivirals that are effective across a wider spectrum of clinical disease severity – which are not yet available for SARS-COV-2/COVID-19.

“So I don’t think we are yet ready to go this route – unless we are willing to accept an even higher morbidity and mortality across the most vulnerable elderly and BAME groups.”

Dr Simon Clarke, Associate Professor of Cellular Microbiology at the University of Reading, said:

“The latest round of the REACT study indicates that over 400,000 people in England are likely to be infected by the coronavirus, an increase on their previous report which is broadly consistent with that seen in the Test and Trace programme.  The fact that these two different studies, which use different approaches correspond with each other, means that we can be confident in their findings.

“The increasing prevalence in all age groups is of concern, but the rise in infections in those aged 65 and over is particularly striking and of grave concern; it demonstrates that the infections recently seen in younger people should not casually be written off as inconsequential.  Taken with the increase in infections reported by Test and Trace, hospitalisations and deaths, this suggests the virus is now spreading rapidly and existing measures to control the virus are not effective in reducing transmission.

“Despite an apparent slowing in the rate of increase, which appears to have occurred fairly recently and around which there is some uncertainty, the numbers continue to head in the wrong direction and based on what we understand about the transmission of the virus, we now seem likely to see a second wave of infections spread across the UK.  A second wave which may be made worse than what we saw in the spring as the NHS may also be stretched by the additional demands of the annual flu season, making it even more important that anyone at high risk gets this year’s flu jab.”

Prof Sheila Bird, Formerly Programme Leader, MRC Biostatistics Unit, University of Cambridge, said:

“Very substantial increase (2.6-fold) in the number of swab-positives when only mid-way through the round 5 of REACT’s swab-test surveillance of persons aged 5+ years in England makes abundantly clear why REACT “reacted”.  Hence, their issuing of an interim report so that the public – all of us – but REACT’s participants in particular, and not just Ministers, were informed.

“Re-weighting is necessary because REACT’s response rate is around one-third of those invited to take part.  Notice that re-weighting increases the estimated prevalences so that those more likely to be positive are less likely to take part.

“REACT asks for brief information about its participants.  For example, REACT’s data from rounds 1 to rounds 4 and 5 (interim) show that SARS-CoV-2 prevalence is 20 times higher for person who have had contact with a confirmed/tested COVID-19 case.  The implications for members of the household of confirmed/tested COVID-19 cases and their external close contact – those whom Test & Trace asks to self-isolate – is clear.

“Equally clear is that, after 17+ weeks, Test and Trace has failed to report anything about the swab-positive rate for those whom it quarantines.  The Royal Statistical Society’s (RSS) recommendations on how to glean intelligence from Test & Trace were designed to help out.

“REACT could help too – by asking its participants an extra question to find out if they are in quarantine at behest of Test & Trace when they take their swab-test.

“ONS Infection Survey could help too – by deploying its field-force to conduct the RSS-recommended random home-visits to offer swab-testing to Test & Trace households.  Based on REACT, this extra stratum of visits would be expected to yield at least 10 to 20  times as many positives per 1000 visits as ONS Infection Survey currently reports, and hence be hugely cost-efficient!”

Preprint: ‘High prevalence of SARS-CoV-2 swab positivity in England during September 2020: interim report of round 5 of REACT-1 study’ by Steven Riley et al. was posted on Thursday 1 October 2020.  This work is not peer-reviewed.

https://www.imperial.ac.uk/media/imperial-college/institute-of-global-health-innovation/REACT1_12345_Interim-(1).pdf

All our previous output on this subject can be seen at this weblink:

www.sciencemediacentre.org/tag/covid-19

Declared interests

Prof Kevin McConway: “I am a Trustee of the SMC and a member of the Advisory Committee, but my quote above is in my capacity as a professional statistician.”

Prof Sheila Bird: “SMB is a member of the Royal Statistical Society’s COVID-19 Taskforce which, on 23 July 2020, made recommendations on statistical methods for gleaning intelligence about the effectiveness of Test & Trace.”

None others received.