October 21, 2020

expert reaction to new risk tool developed to predict a person’s risk of being admitted to hospital and dying from Covid-19

A study, published in the BMJ, looked at a new tool developed to predict a person’s risk of being admitted to hospital and dying from Covid-19.

Prof Derek Hill, Professor of medical imaging science, UCL, and expert in medical devices, said:

“The authors have developed a computer model called QCovid that is designed to predict risk of any person being hospitalized or dying from COVID-19, using the sort of information your local doctor would have about you such as your age, ethnicity, how deprived you local area is, whether you smoke, and medical history.  They used a huge amount of data – medical records of more than 6 million to develop the model, with a separate 2 million to test it.  And their testing shows their model works well – predicting nearly three quarters of hospitalizations and deaths in wave 1 of COVID-19.

“So can this model help manage the second wave of COVID-19 by working out where serious COVID-19 cases are most likely to happen, and helping target resources to parts of the counties and individuals where it is most needed?  Unfortunately not.  Despite the very impressive amount of data and sophisticated data science that has gone into building QCovid, it has a fundamental flaw that dramatically reduces its practical usefulness.  Your chance of being hospitalised or dying from COVID-19 is critically dependent on factors that the authors did not include in their model.  The model does not know who actually was exposed to or was infected by the virus that causes COVID-19.  The model has no information about individuals’ behaviour, such as whether they self-isolated, or worked in a high risk job, washed their hands properly, or wore a mask, nor of whether there was lots of COVID-19 infections in an individual’s neighbourhood.  These missing data are arguably more important than the data included in the model at predicting who will get serious COVID-19.  And we are already seeing in the second wave different patterns of spread and infection, as well as big differences in behaviour (such as widespread mask wearing).  So data collected during the first wave is now out of date.

“The authors are aware of this limitation in their model, and propose that a solution is that it can be updated as new data arrives.  But to do this well is much harder than it sounds, and doesn’t ultimately solve the problem that some of the most important data is simply impossible to obtain at national scale.

“This model is a fascinating demonstration of what can be achieved by linking data from different parts of the NHS, but the omissions from the model ultimately mean it is not a useful practical tool for managing the second wave of COVID-19.

“Using this model to predict which individuals in the general population are going to die of COVID is a bit like predicting who is going to get pregnant without including in the model any information about whether they had sex and whether they used birth control.”

Prof Mark Woolhouse, Professor of Infectious Disease Epidemiology, University of Edinburgh, said:

“In a paper published today in BMJ a team led from the University of Oxford describe QCOVID, an algorithm that predicts an individual’s risk of being hospitalised or dying from COVID-19.

“The algorithm was constructed using data for more than 6 million people and was validated using data for more than 2 million people, making this one of the largest COVID-19 risk studies to date.

“The authors report a very striking result from the validation exercise: the 5% of people predicted to be at greatest risk accounted for 75% of the deaths.  This implies that we can now identify in advance a small proportion of people at greatly increased risk (over 50x that of the remaining 95%).

“This result provides support for the concept of targeted shielding.  The contribution of shielding to reducing the public health burden of COVID-19 so far is thought to have been quite low.  This is because only a small fraction of people are in the ‘extremely vulnerable’ category and they account for only a small fraction of cases.  To lower the death rate substantially on the basis of age more than 20% of the population would need to shield.  This new study suggests that could be reduced to just 5% using more complete knowledge of risk factors.  If ways could be found to protect this 5% – such as regular testing of their closest contacts – then this could make a substantial difference to the public health burden of COVID-19.

“There are some important caveats to these results.  The study is based on a combination of risk of exposure and risk of a severe outcome if infected.  This makes it a study of absolute, not relative, risk; the absolute risk is, of course, highly dependent on the level of infection among the contacts of each individual, which varies greatly over time and from place to place.  Also, as pointed out in an accompanying editorial, both people’s behaviour and the quality of care they received are inextricably tied up in the risk calculations.  If behaviour or quality of care changed for any reason then so would the risk.  Finally, the authors were also able to predict the risk of hospitalisation, but not with the same degree of accuracy.

“This paper is an important landmark in our understanding of COVID-19 risk and paves the way for more emphasis on targeted, risk-based responses to managing the public health threat over the coming months and years.”

Dr David Strain, Senior Clinical Lecturer, University of Exeter, said:

“There is a desperate need for an accurate risk assessment tool to facilitate the protection of key workers that are at very high risk as we move into the second wave of COVID.  This needs to be objective, reproducible and accessible.  This is excellent research that has undoubtedly achieved the first two of the requirements.

“It has caveats though.  These data are derived from hospitalisations and deaths during the first wave.  Behaviour (and thus risk of transmission) has changed significantly this time, some for the better (e.g. the widespread acceptance of masks) some for the worse.  Notably, the shielding scheme that supported the extremely vulnerable has been discontinued.  Any tool that was derived during this time of artificially lower exposure may under-estimate the risk of this population.  The manuscript acknowledges this, and future iterations will be able to account for this, however, until this has been evaluated in the second wave (assuming the government continues it’s position of not supporting those previously shielded) those at previously shielded should be regarded as ‘high risk’ irrespective of the recommendations of the tool.

“The second is a far more pragmatic caveat.  Any risk assessment tool is only as good as the actions that are prompted by it.  Although incredibly detailed in preparation and validation, this manuscript does not indicate how this can be translated into practice.  The HSE recommends that risk assessments are completed with a line manager and then appropriate action is taken.  If this was to be used by means of an ‘app’ calculator, it requires disclosure of a significant amount of personal information.  There is extensive evidence that some individuals, particularly from ethnic minority groups, that are already more prone to bullying and harassment in the work place, that they do not disclose all to their direct manager.  The alternative would be to integrate this into primary care records, however that raises the question as to who will actually perform and implement the risk assessments.  General practice is already overwhelmed trying to maintain a primary care service in a COVID secure manner, providing an enhanced vaccination service to protect vulnerable adults from influenza, and managing the long term conditions that are highlighted by this tool as significant risk factors.  Without a clear implementation plan this tool does not as yet have a purpose.

“Finally, the actions that a ‘high risk’ status stimulates differ widely around the country.  Without consistent mitigation recommendations (based on either local prevalence or tier level) the risk assessment tool becomes little more than an academic exercise in which geographic areas have the highest risk based on population.  Given that the single strongest risk factor is age this can be accomplished looking at basic demographics in any region.”

‘Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study’ by Ash K Clif et al. was published in the BMJ at 00:01 UK time on Wednesday 21 October 2020.

DOI: 10.1136/bmj.m3731

Declared interests

None received.