select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
An observational study published in BMJ Nutrition, Prevention & Health looks at the effects of dietary supplements on the risk of testing positive for SARS-CoV-2, using data from the COVID-19 Symptom Study app.
Prof Kevin McConway, Emeritus Professor of Applied Statistics, The Open University, said:
“It’s very important to realise that the researchers on this study aren’t claiming that their research is a basis for any recommendations to take, or not to take, the dietary supplements they studied. They say clearly that large randomised controlled trials would be needed to confirm the results, before any such recommendations could be made – and, if those trials were indeed carried out, it’s perfectly possible that they wouldn’t produce similar results at all. And this isn’t just the researchers being inappropriately modest – Professor Sumantra Ray points out (in the press release) the lack of convincing evidence for any therapeutic value of taking nutritional supplements, and also that this study wasn’t primarily designed to answer questions about the role of supplements in Covid-19.
“So why can’t we be any more definite than that about a study that involved the best part of half a million people? Several reasons, actually. The main one, I’d say, is that it’s an observational study. People weren’t told by the researchers to take, or not take, nutritional supplements. The people involved had signed up to use the Covid-19 Symptom Study app for different reasons, and the collection of data on supplements wasn’t added until sometime after they had signed up. But then the researchers did ask the participants about whether they had been using certain dietary supplements regularly. They were asked only once, and about their past use of supplements. The researchers also recorded various other information about the participants. This included whether they had had a swab PCR test, or a test for antibodies, that would indicate if they had had either a current infection with the virus that causes Covid-19 (from an swab test)of had had an infection in the past (from an antibody test – this was long before vaccination began so they would not have had a positive antibody test because of being vaccinated). People who had had no test at all weren’t included in the data that were analysed for this research. Then the researchers compared the numbers who tested positive on people who had and had not reported that they took the supplements. For some of the supplements, in some groups of people, they found that those who had taken a supplement regularly were less likely to test positive for the virus that those who hadn’t taken the supplement.
“The big problem with any study like this is that there will be a great number of other differences between people who take and don’t take a supplement, apart from the difference in supplement use. One or more of those other differences might have been the cause of the difference in the rate of testing positive, and not the supplements at all. In the jargon, these other differences are called ‘confounders’. For instance, it’s known from a lot of other data about Covid-19 that having certain pre-existing illnesses (comorbidities), such as kidney disease, seem to be more likely to get infected, and there are several other diseases which affect the chance of having severe Covid-19. The researchers on this study did collect data on some pre-existing diseases, and indeed found that supplement users were rather more likely than non-users to have each of the diseases they considered (see table 1 in the research paper). So maybe any difference in the risk of testing positive between supplement users and non-users is, in part at least, due to whether they had these diseases or not. It’s possible to make statistical adjustments to try to take into account these other differences between supplement users and non-users, and the researchers did make some adjustments of this sort. They adjusted for the most obvious confounders – age, sex, body mass index (BMI) and an overall measure of what people said their health was when they first signed up – in all the comparisons they made, but they adjusted for other possible confounders, including comorbidities, and also the level of deprivation where people lived, ethnicity, smoking and diet quality, only in some of their analyses (and not in the analyses that looked at men and women separately, for instance). So, while it’s possible that the differences in infection risk they found between supplement users and non-users are caused, in part anyway, by the supplements, it’s also possible that they are caused by other things entirely. A study like this simply can’t tell us how cause and effect works in this situation, and that’s why the researchers say that large randomised controlled trials would be needed to confirm their findings, or otherwise. In a randomised trial, some people would be chosen at random to take the supplement, while others would be chosen at random not to take it. The randomness ensures that, on average, the two groups don’t differ at all except in whether they take the supplement or not, so if the infection rates did turn out to differ, it would be pretty clear that the difference was actually cause by the supplements.
“So, if it isn’t the supplement use that explains the difference in the risk of testing positive, what else might be going on? The researchers themselves do give a range of possible alternative explanations. The data are all self-reported, and biases can occur in self-reported data for a number of reasons. There are issues about why people choose to take supplements, and why people are tested. (Remember that the data used here aren’t for everyone who used the app, only for those that had a test (positive or negative). The researchers mention the possibility that people started taking supplements because they had had symptoms that might have been Covid-19 – that would make supplement use higher in people who had symptoms, and people who had symptoms are more likely to test positive than people who don’t have symptoms, so that supplement use would be associated with positive test results whether or not the supplements themselves affected infection risk. The people who use the app are not representative of the general population on the countries concerned in several ways, and the people who actually had a test (and so were included in the data for this study) aren’t necessarily representative even of all the people who use the app. This could have introduced biases in several ways. The researchers mention one possibility in particular – that people who have had symptoms, or even who have already definitely had Covid-19, might be more likely than others to sign up for the app, and also people who take supplements might be more likely to sign up (perhaps because they are more health conscious), and that could lead to an apparent association between testing positive and using supplements, even if supplements don’t actually affect infection risk. It’s possible that people who take supplements are more likely to carry out other measures that are known to reduce infection risk, like keeping social distance or wearing masks, and it’s these other behaviours that reduced their infection risk and not the supplement taking at all. We don’t know what doses of supplements people took, or what exactly was in the multivitamin supplements that some took. I could go on – the researchers are assiduous in listing yet more possible limitations of their findings.
“And is there an explanation for the finding that the correlations between supplement use and the chance of testing positive really only show up for women? Again, the researchers themselves list several different possible explanations, including the following. First, perhaps there is a real biological difference between the immune systems of men and women that means that the use of some supplements causes lower infection risks in men and women. As the researchers point out, though, if that’s the case, it’s strange that the differences between men and women showed up in different ways in the three different countries included (UK, US, Sweden), because sex differences in the immune system are likely to work in similar ways in those three populations. Second, it’s possible that the effect of supplement use really is one of cause and effect, but that it doesn’t actually work differently in men and women. The researchers point out that the effect might depend on the dose of supplement in relation to body weight, and maybe the women took higher doses, on average, in relation to their body weight. I’d add that, for some of the differences between men and women, the statistical uncertainty about the strength of the association means that we can’t even be sure that there is a sex difference at all, whatever causes it. Third, some of the reasons for the overall correlation between supplement use and the risk of testing positive, that have nothing to do with the supplements actually causing the effect, might apply different in men and women. In the comparisons between men and women, the researchers did not make so many adjustments to allow for covariates as they did in some of their overall analyses – so perhaps the difference between men and women has something to do with pre-existing illness, or levels of smoking. Or, even more plausibly, the associations between supplement use and other healthy behaviours (like social distancing, use of masks, cancelling visits and so on) might differ between men and women, and that might in part of entirely be the reason for the differences that were found between men and women.
“So, while the data are certainly compatible with the possibility that using supplements causes a reduction in the risk of testing positive for the virus, they are also compatible with a lot of other explanations that have little or nothing to do with the supplements themselves changing the risk. But how big is the change in risk anyway? The title of the research paper (“Modest effects…”) makes it clear that the researchers aren’t claiming a huge effect. The largest reduction in risk that they report is for taking probiotics, where they say that the risk of infection is reduced by 14%. Actually that figure is from the analysis where only a few (important) adjustments were made to allow for possible confounders – in the analysis that was fully adjusted, the reductions in risk were 9% for probiotics, 10% for omega-3 fatty acids, 12% for multivitamins and 8% for vitamin D, all rather less than for the analysis with fewer adjustments for possible confounders. But these are all relative figures, and it’s important to see how they might change the overall risk of infection. The new study covers only the first wave of the pandemic last year, and using data from the study itself and from the ONS infection survey data on antibodies a reasonable estimate is that about 7% of people who were not taking supplements would have been infected. The most optimistic estimate from the study of the reduction in risk associated with a supplement (for probiotics, without full adjustment for confounders) would reduce that risk to about 6%, and for reasons I’ve given, we can’t know whether that reduction is actually caused by the supplement use or by something else. The other estimated risk reductions in the new study wouldn’t even get the risk down as low as 6%.
“One more point: could the large randomised trials that the researchers suggest actually happen? Well, that could be difficult. Large randomised trials cost a great deal of money to run, and tend to take a long time. The success of vaccinations, and other interventions, mean that infection rates are now pretty low. So a trial to compare the infection rate in people who do and do not take supplements would have to be very large indeed, given that this study has indicated that the reductions in infection from using supplements are not likely to be very large, and given that you would need reasonable substantial numbers to become infected to estimate the size of the reduction properly. So I wouldn’t expect to see clear results from randomised controlled trials of these or other supplements any time soon, if ever.”
‘Modest effects of dietary supplements during the COVID-19 pandemic: insights from 445 850 users of the COVID-19 Symptom Study app’ by Louca et al. was published in BMJ Nutrition, Prevention & Health at 23:30 UK time on Monday 19th April.
Declared interests
Prof Kevin McConway: “I am a Trustee of the SMC and a member of its Advisory Committee. I am also a member of the Public Data Advisory Group, which provides expert advice to the Cabinet Office on aspects of public understanding of data during the pandemic. My quote above is in my capacity as an independent professional statistician.”