August 1, 2019

expert reaction to a study looking at an AI algorithm to identify atrial fibrillation

A study, published in The Lancet, reports the use of an AI algorithm for use in identifying patients with atrial fibrillation.

Dr Malcolm Finlay, Consultant Cardiologist, Barts Heart Centre, said:

“This study used patients who were already under investigation for atrial fibrillation for the testing and algorithm input dataset. This really limits the applicability of the findings to real world situations, where we would like to use ECGs to determine which patients will benefit from certain treatment for AF to decrease their risks of having a stroke.

“One of the problems with the AI approach is that the detection may be good, but it cannot tell you why it makes a detection, so the detection may be based on something on the ECG which is already known about. This means simpler methods than an AI algorithm could be used to detect AF using this marker. It’s also hard to know if medication already used which might have affected the ECG was being used for AF treatment or hypertension treatment.

“Additionally, in this group of patients, AF was common, much more so than in the general population, so false positives (a result saying AF is present when it is not) will be far more common and frequent when this algorithm is applied to the general population. The data is taken from a US Mayo clinic population, which is very different from the diverse multicultural population we have here in the UK. These details really matter in AI based studies.

“The idea is a good one and using deep learning to pick out hidden features in the ECG might be highly informative, but it has to be highly reliable to be worthwhile and useful. There is a significant extra piece of work to be done where the algorithm is tested on data from the general population to show us it would be possible to allow UK doctors to diagnose or treat patients before a stroke or heart failure occurs. Data from the UK biobank may be ideal to test this type of method.”

Dr Franz Kiraly, UCL and Turing Fellow in the Department for Statistical Science, Alan Turing Institute, said:

“Some quick calculations about the algorithm in its tested settings finds that about 10% of the patients in the dataset the algorithm was tested on are sick, which is a lot higher than the proportion of those with AF in the general population. When the algorithm identifies AF, it’s a false result 75% of time. In healthy patients, the algorithm detected 20% of cases with AF and in sick patients (those known to have AF), it detected AF 80% of the time. However, since there were a lot of healthy patients, there will have been a lot of false alarms, perhaps too many for the algorithm to be useful right away. However, the authors do suggest some of the 20% of ‘healthy’ patients the algorithm detected may have AF which the doctors had not detected.

“There are multiple technical problems in the study, including no reporting of how simpler methods which requires less complex and cheaper technology might perform. The authors do not provide a data summary of the ECGs so we cannot tell if the performance comes from a simple feature, e.g., signal variation, which could be used by a simple, cheaper method rather than from a complex AI algorithm. Additionally, the description of the algorithm is not precise enough to check the claims in a reproduction study; the authors provide no code. Therefore, the research may need to be checked and tested more, and we need to be aware that it is currently in its early stages.”

Prof Charalambos Antoniades, Professor of Cardiovascular Medicine, University of Oxford, said:

“This is an exciting and high quality study with a large number of participants, demonstrating that artificial intelligence can see what is not visible to the human eye. Detecting atrial fibrillation before doctors can by an algorithm reading normal ECGs could enable patients to be treated early with anti-blood clotting medication, and may save lives. However, before doctors can start to use this technology regularly, we need to see if it works correctly on the general public, and to understand how many false positive results it will generate and whether it is cost-effective to use in clinical practice. It should be noted that this algorithm does not predict atrial fibrillation before it happens, but detects it when it is already present. A question that remains unanswered is whether this technology can predict atrial fibrillation before it happens, at a stage where it may still be preventable.”

“Currently we can detect people with high risk of having AF at the time of an ECG on sinus rhythm, either by some changes in the ECG but also mainly by imaging of the atria. If this process can be automated using AI analysis of ECG, it is a major step forward, as it will remove the “clinical interpretation” of these ECGs that brings personal bias to the diagnosis of AF. However, further research needs to be done to see if and how this technology can be translated into a clinical application. Will it lead to a widely available software application that will give the prediction at the time of the ECG or will it need the ECGs to be uploaded in a cloud service for the analysis to take place using a powerful computer system? I see some difficulties in introducing the technology in practice.”

Prof Kazem Rahimi, Deputy Director, The George lnstitute for Global Health, University of Oxford said:

“This is a great study which could certainly change the way we screen for patients with AF. The results are quite striking. However, the findings do need further validation in the general population and over a long time period to monitor for any mistakes the algorithm may make before they become ready for routine clinical use.”

Prof Tim Chico, Professor of Cardiovascular Medicine and Honorary Consultant Cardiologist, University of Sheffield, said:

“This is a very important study that begins to reveal the promise of Artificial Intelligence for improving medicine. The study examined a million ECGs from people who were known or not known to have intermittent AF and trained the algorithm to recognise the differences between the ECGs of these two groups when the person was in a normal rhythm. This is not something that cardiologists are able to do by eye, and the accuracy was impressive, especially as it is possible that some of the people believed not to have AF did have this and were picked up by the algorithm. This AI-based approach could provide a revolutionary advance, although it’s important to note that this research is still in early stages and we need to see replicated results and how the algorithm responds when tested on the general population.

“Atrial fibrillation (AF) is very common, affecting up to a quarter of elderly patients, and places patients at a much higher risk of stroke. AF is easily diagnosed on an ECG, but unfortunately, many people go in and out of AF without any symptoms, and we do not have a good way to diagnose this if the patient is in a normal rhythm at the time the ECG is recorded. This means we often record patient’s ECGs for days at a time trying to find evidence of AF. It’s important to detect AF as people with AF who are at higher risk of stroke are benefitted by blood thinning medication. If it is confirmed that the AI algorithm can predict who will suffer AF based on a “normal” ECG with enough accuracy to know if we should start blood thinners immediately, then this would be likely to prevent a large number of strokes, as well as saving time and the cost of recording ECGs for several days.”

* ‘An artificial intelligence-enabled ECG algorithm for the identification of patients with atrial fibrillation during sinus rhythm: a retrospective analysis of outcome prediction’ by Attia et al. will be published in The Lancet at 23:30 UK time on Thursday 1^st August, which is also when the embargo will lift. 

DOI: 10.1016/S0140-6736(19)31721-0

Declared interests

Prof Tim Chico: No conflicts

None others received.