April 21, 2020

expert reaction to new preprint on antibody testing for SARS-COV-2

A preprint, posted to MedRxiv, reports on antibody testing for SARS-COV-2.

Prof Sylvia Richardson, University of Cambridge & President Elect of the Royal Statistical Society & co-chair of the Royal Statistical Society Task Force on COVID-19, said:

“The paper by the National Testing Scientific Advisory Panel on the evaluation of antibody testing for SARS-CoV-2 infection in blood plasma provides eagerly awaited information on the operational characteristics of 9 candidates for use as a rapid and scalable antibody test, Lateral flow immunoassays (LFIA).  It is clear from the results reported in this paper that none of these 9 LFIA tests is acceptable for point of care clinical tests. It is regrettable that the manufacturers did not allow full disclosure, making it difficult to answer the question of whether one or more of the 9 candidates would be useful for population surveillance studies, as discussed in more detail in the recent Royal Statistical Society statement on Covid-19 antibody testing issued on 14th April 2020 (https://rss.org.uk/RSS/media/Policy-and-campaigns/Policy/Statement-Antibody-testing-14-04-2020.pdf).”

Prof Mala Maini FMedSci, Wellcome Trust Senior Investigator at University College London, said:

“Reliable antibody tests for past infection with SARS-CoV-2 are urgently needed for a number of important reasons: to help with tracking the pandemic in communities and populations; for research studies examining how long these antibodies persist and whether they confer protection; and to evaluate the effectiveness of new vaccines.

“We’re not yet sure if these antibodies indicate protective immunity against SARS-CoV-2 but preliminary data suggest they may be a reasonable proxy for this – so they are being considered to inform release from lockdown etc.

“Results from this new study show that their own new lab-based serology test look more reliable when compared to existing ‘point-of-care’ tests.  Point-of-care (LFIA) tests did well on specificity (only finding real cases) but poorly on sensitivity (identifying all the real cases). So their point-of-care test data suggest that if you test positive you are likely to have been infected whereas if you test negative you may or may not have been infected.

“This study only reports a relatively small sample size, and their data will be updated once extended.  Another weakness of this study is that they did not test any samples from cases infected with other related coronaviruses which are most likely to give false positives and are needed to be sure about the specificity of a SARS-CoV-2 test.

“Their data reinforce the message that antibody testing is only reliable if carried out more than 10 days after the onset of SARS-CoV-2 symptoms.”

Dr Simon Clarke, Associate Professor in Cellular Microbiology, University of Reading, said:

“This is the formal report on the testing of the promised 3.5 million at home “pregnancy test” type kits for antibodies that Public Health England assured us were due for imminent distribution by companies such as Amazon and Boots.  While this work is yet to be independently peer-reviewed, it shows quite clearly that several different types of test kit were insufficiently accurate to be used by the general population and would in no way have been sufficient to determine whether people had been exposed to the coronavirus or if they were immune.  At this important time, reports like this underscore the importance of independent verification of new technologies before the claims of their manufacturers are taken at face value.”

Prof Alison Sinclair, Professor of Molecular Virology at the University of Sussex, said:

“The new lab-based Elisa test is a step forward. The tests show that it reliably identifies people in the UK who have been infected with SARS-CoV-2. It takes time for our bodies to make specific antibodies to a virus and this report shows that, after 10-days of symptoms starting, this test able to detect all of the people who were infected.

“Current LFIA devices are designed to deliver a simple yes or no answer to whether someone has antibodies to SARS-CoV-2.  In this report, these were compared to each other and to the Elisa test.  If the current LFIA devices were used for community screening this report shows that many people who had been infected would be identified, but some others would be missed. A few false positive results were also found with most of the current LFIA devices which could give a false sense of security to people who had in fact not been exposed to the virus yet.”

“The research shows that the new lab-based Elisa test is a step forwards in both sensitivity and reliability compared to the currently available LFIA tests. The study has not been peer-reviewed by experts yet and as the authors point out themselves a more robust comparison would require the analysis of samples from a much larger group of patients and controls, but the cost and manpower of doing this may not be justified. The new lab-based Elisa test described here is being optimised for a high-throughput format and both the details of this Elisa and the Biobank of serum samples can be used to help the development of improved Elisa tests and LFIA devices.”

Prof Richard Tedder, Visiting Professor in Medical Virology, Imperial College London, said:

“The Oxford group study compares the performance of nine un-named point-of-care tests (PoCT) against an in-house conventional indirect immunoassay (ELISA) for IgG and IgM antibody using a tri spike antigen. This ELISA has a claimed 85% sensitivity in detecting antibody in known cases.  It also has a claimed 100% specificity on a panel small panel of 50 pre-Covid 19 samples.

“In comparison the PoCT sensitivities ranged for 70% down to 55%. The PoCT specificities ranged from 100% down to 95%. However, the samples used for this latter determination appear to be different and of larger numbers than those used for the in-house Elisa determination, making objective comparisons of specificity somewhat difficult and at best inexact.

“Interesting though the data are, they are simply of no value at this time as there is no way of relating the Oxford findings of an in-house assay to those PoCT assays which are currently known about in this country. It is also presented in an unfortunate manner which fails to recognise the effort and expertise which the scientific community is putting into the development of PoCT kits at this time.”

Prof Lawrence Young, Professor of Molecular Oncology at the Warwick Medical School, University of Warwick, said:

“This study compares the reliability of different tests for SARS-C0V-2 antibodies. It describes the establishment of a new ELISA test as a gold standard laboratory test and used this to compare the sensitivity and specificity of a panel of nine different commercially available lateral flow immunoassays (LFIA). These LFIA devices are similar to pregnancy testing kits and could be used for point-of-care testing by individuals in their own homes. But these devices need to be both sensitive and specific if they are to be reliably used to make individual decisions about release from lockdown. This study presents a comprehensive analysis which clearly demonstrates that the currently available lateral flow devices are not accurate enough for individual testing. The gold standard ELISA test is being further optimised and could be used in hospital laboratories for antibody testing.

“Worth mentioning the definitions of sensitivity and specificity. Sensitivity is the proportion of true positives that are correctly identified by the test.

Specificity is the proportion of true negatives that are correctly identified by the test. The UK MHRA have set a minimum of 98% specificity for the LFIA devices on the basis that if used at the individual level to inform release from lockdown, high specificity is essential as false-positive results would return non-immune individuals to risk of infection.

“So the current lateral flow devices could be used for developing a broad understanding at the population level, but their performance is inadequate for individual testing. The ELISA test is extremely useful and the authors have established an important bank of serum samples as a resource for future validation of new tests.”

Prof Sheila Bird, Formerly Programme Leader, MRC Biostatistics Unit, University of Cambridge, said:

“Today’s report on the first stage of evaluation of rapid lateral flow immunoassay (LIFA) devices by the National COVID Testing Scientific Advisory Panel offers welcome encouragement that some LIFA devices will pass muster for use in serosurveillance studies.

“The authors remark: ‘Restricting to 31 samples collected ≥10 days post symptom-onset (all ELISA IgG-positive), LFIA sensitivity ranged from 61% (95%CI 39-80%) to 88%(68-97%)’.

“I assume that a second stage of evaluation will follow, for selected LIFA devices, the purpose of which will be to narrow the uncertainty intervals which qualify estimates for both sensitivity and specificity. For example, to reduce the width of confidence intervals by a factor of 3 requires 9 times as many samples to be studied in any second-stage evaluation. Greater precision about the performance of the LIFA devices selected for further evaluation should follow, greater than the initial-stage evaluation could deliver.

Prof Eleanor Riley FMedSci, Professor of Immunology and Infectious Disease at the University of Edinburgh, said:

“This is a really useful paper. It shows that the problem with the commercial rapid antibody tests is that they are not sensitive enough – they fail to pick up antibodies in over a third of people who do in fact have antibodies. However, these tests do have acceptable levels of specificity – that is, they are only picking up people who have genuinely been exposed to the COVID-19 virus. This means if your test is positive, you can be confident that you have been infected and have antibodies. But if your test is negative, you can’t rule out that you might have been infected.

“Just as importantly, this paper shows that we do have a very good assay for use in the lab. The ELISA assay showed excellent sensitivity and specificity. This means that public health labs now have a test that they can use to begin to understand what percentage of the general public have already been infected.”

Prof Stephen Evans, Professor of Pharmacoepidemiology, London School of Hygiene & Tropical Medicine (LSHTM), said:

“While this paper is not yet peer-reviewed it is of much higher quality than many of the preprints posted on medRxiv.

“The statistical analysis is clearly described and well-carried out. The conclusions are well-based on the data, with some limitations clearly stated.

“The study uses a good laboratory measure of antibody status (the ELISA test). The performance of various lateral flow immunoassay (LFIA) devices is then studied in comparison with the more accurate test. As the authors say, ‘LFIA devices are cheap to manufacture, store and distribute, and could be used as a point-of care test by healthcare practitioners or individuals at home, offering an appealing approach to diagnostics and evaluating individual and population-level exposure.’ The problem is that ‘cheap’ does not mean either good or even adequate.

“If the LFIA devices are to be useful, their usefulness depends on the context in which they are to be used. If it is to detect an infection with SARS-COV-2, they must be really good at finding people with the infection- this is called “sensitivity”.  A sensitivity needs to be close to 100%, certainly better than 90%, if it is used on its own to detect an infection. With values less than this, it means that 1 or more in every 10 people who have the infection will be declared to not have the infection. This will mean that instead of isolating themselves they will feel free to go about as normal, both potentially infecting others and not getting appropriate healthcare for themselves. If the test were to be used in conjunction with another type of test then it might provide an additional help to diagnosis, especially if the person acquired the infection some time ago.

“The other context is in checking whether people have had the infection previously and have antibodies that remain, that would be expected to give them immunity against infection. If someone has such antibodies and are immune, they could be allowed to go about essentially as normal and know that, even if they encountered someone who was infectious, they would be immune.

“If the test is to be used to show that a (hopefully) large percentage of the population is immune, enabling people to return to work, especially in healthcare, it is important that it does not give a positive result when someone does not have the antibodies and would therefore definitely not be immune. They could go into situations thinking they were immune but would actually be vulnerable to infection. This would be not only bad for the individuals, it could lead to a major resurgence of the disease in the population. In this context the test must be highly ‘specific’ and a minimum value of 98% has been set (sensibly) by the regulatory authority before it approves a test for general use.

“Although the numbers of patients whose infection status was known precisely was small, and hence there is some uncertainty in the actual values of sensitivity and specificity for the LFIA tests, the results show that none provide the test performance that is necessary for their stand-alone use in any context.

“This means that there is a need to rely on more accurate tests, which are more expensive and in most cases slower and less simple to carry out. Results from such accurate tests will be needed, with clear results showing levels of immunity, before restrictions on keeping distance from other people can be relaxed.”

Prof Gary McLean, Professor in Molecular Immunology, London Metropolitan University, said:

“This is a great study even though it is not yet peer reviewed.

“The major limitation is the relatively low number of SARSCoV2 samples tested for antibodies, just 40, but a good number of confirmed negative samples to calibrate their assay criteria for the positive/negative cut-off.

“What they found was that 100% of the later stage samples (convalescent) were IgG positive for SARSCoV2 by a specific laboratory ELISA test. Some of the early stage samples (acute phase) were also IgG positive and most were IgM positive. This shows that ELISA can detect the different stages of antibody responses and kinetics of antibody appearance in the infection course. They also show that IgG levels tended to rise over a 3-week period and importantly (and in contrast to other studies) there was no association of antibody levels to disease severity – others have suggested that high antibody levels could mean more severe symptoms.

“The most critical aspect of the study was the testing of a panel of rapid test devices – these are the lateral flow immunoassay (LFIA) rapid tests, somewhat like a pregnancy test type device. The results obtained from these were less reliable with false positives emerging and lower levels of sensitivity. These devices would be useful for monitoring antibody levels “at home” and would be a simple way to evaluate the population that has been exposed to (or might be immune to) SARSCoV2 – however these devices were not as reliable as the lab ELISA test unsurprisingly.”

Dr Al Edwards, School of Pharmacy, University of Reading, said:

“This study reveals long-awaited findings that have been used to suggest that rapid antibody tests (lateral flow) purchased and promised by the government are not suitable for home testing e.g. via Amazon.

“It’s clear that within this sample set and study design, the tests evaluated didn’t meet the exacting standards outlined by MHRA who have indicated they would prefer tests to achieve 98% sensitivity. Some of the headline sensitivities have to be taken with a pinch of salt, it seems like for only 31 of their 40 patients, samples were available at >10 days post symptom onset. For these 31 patients each individual person contributes 3% to a sensitivity value. This reflects the difficulty of rushing this type of study.

“The limited patient sample size must be compared to other studies from countries that have had SARS-CoV2 for longer- a more detailed study of neutralising antibodies from China measured not just antibody but also virus inhibition for 175 patients with wide spectrum of COVID-19 infection (https://www.medrxiv.org/content/10.1101/2020.03.30.20047365v2). That study showed IgG antibody against S protein (the parameter chosen by the Oxford study as Gold Standard) may not correlate with neutralisation. Other recent studies explore immunity to many more viral targets (https://www.biorxiv.org/content/10.1101/2020.04.15.043364v1.article-info).  Only by integrating all this emerging data together can we move forwards.

“The study was carefully designed to measure ‘individual protection from reinfection’ rather than simply diagnose infection vs not infected. These uses need different performance and validation study design. As WHO has recently emphasised, it’s very hard to develop a simple test that guarantees personal immunity.

“But rapid home antibody tests aren’t usually intended for this application- they are more normally used alongside other tests for confirmation or in screening programs. For example, home HIV tests that you can buy from online pharmacies can help people decide to visit a diagnostic lab for full confirmation testing. These rapid COVID-19 tests might still be extremely useful alongside other tests- let’s not throw them out on the basis of this one small study.

“The report raises many new questions and illustrates the complexity of clinical diagnostics. Perhaps some early government plans, whilst made with the best of intentions, failed to comprehend what is required to roll out a new clinical diagnostic service. The NHS is well known to be slower than many other countries in adopting near-patient testing, although we do have leading experts and pioneers in this area. But although lab testing can in many cases be more reliable than rapid near-patient tests, the challenge remains that it’s incredibly difficult to collect patient samples at scale – just making bigger diagnostic labs clearly hasn’t helped with swab testing because very few healthcare staff have been able to access such testing.

“Denying healthcare staff rapid tests- especially given that these tests do appear to have very high specificity when screened against pre-outbreak samples- may turn out to be a mistake, given how hard it appears to be to get NHS swab testing. At the moment, a very high proportion of NHS staff tests (pillar 2) are testing positive (for 19^th April over 30% of tests were positive) – confirming that only tiny proportion of key workers are able to be tested. There are many reports that NHS staff find it extremely difficult to access swab testing, so they go untested and never find out if they have already been infected at work. Surely some of these people might benefit from access to highly specific, cheap and accessible, rapid antibody tests, as long as they understand that a negative result doesn’t rule out past infection, and that a positive test does not guarantee protection from reinfection?

“Not only are many NHS staff and key workers lacking basic protective equipment to protect them from infection, they are being denied testing to find out if they’ve been infected during their vital work.

“Once again, diagnostic testing turns out to be more complicated than first hoped. From within the diagnostics field this is no surprise.”

https://www.medrxiv.org/content/10.1101/2020.04.15.20066407v1.full.pdf  

All our previous output on this subject can be seen at this weblink: www.sciencemediacentre.org/tag/covid-19

Declared interests

None received.