November 17, 2022

expert reaction to study of a clinical risk score prediction tool for dementia

A study published in JAMA Network Open looks at the development of a clinical risk score prediction tool for dementia.

Prof Mika Kivimaki, Professor of Social Epidemiology at University College London, said:

“I think this is an interesting analysis, although the claim that the score has almost 100% predictive accuracy seems unrealistic.

“There is information on two important metrics missing: the detection rate, which denotes the proportion of test-positive individuals among people who developed the disease at follow-up and the false positive rate which is the proportion of individuals with a positive test result among those who did not develop the disease on follow-up.

“These are particularly important metrics when evaluating prediction of an incurable and feared illness by individuals, such as dementia. A false positive test result in dementia risk assessment can elicit psychological distress for many of the affected individuals. Receiving a false negative result, in turn, may discourage a person to take up preventive measures.”

Prof John Hardy, Professor of Neuroscience, UCL, said:

“This study appears to show a remarkably high prediction capability for analysis to predict dementia.  As the authors’ note “dementia” covers several diseases: Alzheimer’s disease, vascular dementia, frontal temporal dementia among others.  Given this mix of diseases, it is surprising that there should be a “one size fits all” prediction algorithm.  If this algorithm replicates in a separate cohort, then it will be a significant step forward.”

Dr Sara Imarisio, Head of Research at Alzheimer’s Research UK, said:

“A person’s risk of developing dementia is an interwoven mix of their genetics, age and lifestyle. Because the diseases that cause dementia can start long before symptoms appear, identifying people whose lifestyle puts them at greater risk could open up ways to prevent dementia or delay its progression.

“For this study, researchers used data from a large UK-based repository of biological data called the UK Biobank to try to develop a robust way to identify people at greater risk of dementia. 

“This study is well-designed, and since it involved data from a large number of people, it allowed the researchers to take into account any differences between men and women. And because it involved the sort of lifestyle information that’s readily available to health care professionals, the resulting prediction tool seems to be practical to use in the general population. Crucially, however, as it only looked at data from UK Biobank participants, it will be important to carry out further studies with other populations to confirm its findings.

“Prediction tools like the one developed in this study, if validated in further studies, could one day help doctors identify people at greater risk of dementia, and help them take steps to keep their brains healthy, reducing the risk of developing dementia later in life.”

Prof David Curtis, Honorary Professor, UCL Genetics Institute, UCL, said:

“This is a very poor quality report and the fact that it was published in its current form reflects badly on the editorial processes of the journal concerned.”

“The claim that ‘the risk score model yielded nearly 100% prediction accuracy of 13-year dementia risk’ is extremely misleading. The score does not accurately predict whether or not one will develop dementia in 13 years, rather it provides the probability that somebody will develop dementia. This would be like me claiming that I can predict the risk of getting heads when I toss a coin with 100% accuracy – the risk of getting a head is 0.5. The score is a poor predictor of whether somebody will get dementia or not, it only predicts their chances of getting dementia.”

“In spite of what the authors claim, there is very little clinical value in being able to predict somebody’s risk of dementia. If there were, we would already be using tests which are actually quite strong indicators such as APOE genotype. But in fact, as in much of medicine, simply having an assessment of risk is of little or no practical benefit. If drugs to prevent Alzheimer’s disease become available then we will start routinely testing for preclinical disease using blood tests being specifically developed for this purpose but we certainly will not be relying on crude scores based on readily available information such as age and presence of cardiovascular disease.”

“The study design is weak and the results mostly simply recapitulate already well- established risk factors such as age, low education and vascular disease along with a number of features which may indicate that the subject already has early dementia but that it is yet to be formally diagnosed. It does not make a helpful contribution to our knowledge about risk factors for dementia and does not contribute to the effort to develop effective preventive strategies and treatments.”

Dr Tom Russ, Director of the Alzheimer Scotland Dementia Research Centre; Consultant Psychiatrist & Honorary Clinical Reader, University of Edinburgh, said:

“This study used UK biobank, which is a very large, high-quality study. However, the authors don’t clearly describe how dementia was identified, which is crucial to interpreting the value of their research. Dementia has been ascertained for many UK Biobank participants through linkage to medical records, but it is unclear if these researchers used those high quality data. In any case, their findings are not very surprising – increasing age is most important for an individual’s risk of dementia. In addition to this, a number of socioeconomic factors and cardio-/cerebrovascular disease are also important, which we already knew about. These have been previously highlighted in great detail, in particular by the 2020 Lancet Commission article, which – importantly – highlighted the different points in the lifespan when particular risk factors might be most important. It is interesting to see that a different amount of dementia risk is accounted by these risk factors for in men and women – about a third of dementia risk in men and half in women – but the research doesn’t really show anything new apart from this.”

Dr Ivan Koychev, Senior Clinical Researcher at the University of Oxford, said:

“This is a paper that describes a dementia risk prediction tool with a 5-13 year horizon based on the UK Biobank study. It has sounds methodology and has the benefit of working with one of the largest datasets currently available.

“The authors have accounted for confounders within the dataset. The limitations are that UKBiobank participants are not fully representative of the general population and some of the measures, for example sleep, rely on participants’ self report instead of objective measures of sleep quality that are available within the dataset. 

“The implications are that models such as the one described are in good position to be rolled out in clinical practice to complement screening methods for dementia such as blood tests. The cost-benefit of such screening programmes for dementia is yet to be established.”

‘Development of a Clinical Risk Score Prediction Tool for 5-, 9-, and 13-Year Risk of Dementia’ by Lina Ren et al. was published in JAMA Network Open at 16:00 UK time on Thursday 17^th November.

DOI: 10.1001/jamanetworkopen.2022.42596

Declared interests

Prof Mika Kivimaki: “I am studying risk prediction, so I have academic interest in the topic but no commercial interests.”

Prof John Hardy: “No conflicts.”

Dr Sara Imarisio has no interests to declare.

Prof David Curtis: “I have no conflict of interest to declare.”

Dr Ivan Koychev is a medical advisor to Five Lives, a digital technology company developing a solution to offer dementia risk prediction and lifestyle modification to ageing adults. 

Dr Tom Russ: “I don’t have any conflicts of interest.”

For all other experts, no reply to our request for DOIs was received.