select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
An analysis, published in BMJ Evidence Based Medicine, looked at clinical trial data on cholesterol lowering drugs and targets for ‘bad’ cholesterol levels.
Prof John Cleland, Professor at the Robertson Centre for Biostatistics, Institute of Health and Wellbeing, University of Glasgow, said:
“DuBroff and colleagues cast doubts on the efficacy of cholesterol lowering therapies on cardiovascular events. Challenging preconceptions should be encouraged in scientific debate as it forces people to check if the evidence supporting their preconceptions is robust. For cholesterol lowering therapies there seems little doubt about their efficacy. For many other preconceptions the evidence is less robust (eg:- treating mild-to-moderate dysglycaemia or obesity, coronary revascularisation for stable coronary artery disease to reduce mortality, long-term aspirin for cardiovascular disease, calcium and vitamin D for osteoporosis) (reference 1-3). Unfortunately, expert opinion has a dreadful inertia; it does not like to change prior recommendations.
“Of course, it is important to protect the public from being misled by scientific debate. Long-winded official recommendations often don’t help lay-readers, either causing confusion or somnolence. Short, clear recommendations are required but alternative opinions should be recognised and rated rather than ignored. Ignoring alternative opinions stimulates conspiracy theorists. Transparency, which stimulates debate and creates more evidence, is usually the best course of action.
“DuBroff make an important point that a systematic review and meta-analysis are not the same. Before putting trials in the ‘blender’ of meta-analysis, it is important to look at their quality. Where no single trial provides adequate evidence, the meta-analysis should be considered hypothesis generating rather than decisive evidence. Nowhere is this better seen in cardiovascular medicine than for long-term aspirin for secondary prevention, where the largest trial conducted showed an excess mortality, but the results of the meta-analysis were driven by improbably large effects in the smallest trials (3).
“Choices of words, statistical analyses and perspectives matter.
“Words: The authors talk about “negative” trials when they mean neutral. A negative trial is one that is uninformative due to poor design or conduct. A neutral trial is one where there is no difference in the primary endpoint amongst interventions. Where the control group is superior to the intervention, this is a positive trial; it’s just in the wrong direction.
“Statistical Analyses: The authors compare absolute reduction in the risk of cardiovascular events with relative-risk reduction in LDL cholesterol. Other authors have found a strong relationship when they have compared the relative-risk reduction of events with the absolute reduction in cholesterol (4). For patients and clinicians, absolute reduction in the risk of events is the more important metric but from a statistical and scientific perspective relative-risk reduction is the correct analysis, since absolute risk reduction is highly dependent both on background risk and on the duration of follow-up.
“Perspectives: Genetics make a substantial contribution to blood cholesterol. Treating a life-long risk factor for a few years would be expected to have only a small effect on the incidence of disease or prognosis. Viewed from this perspective, it is remarkable that trials of lipid-lowering therapies have shown any benefit. The West of Scotland Coronary Prevention Study (WOSCOPS), randomised patients to pravastatin 40mg/day or placebo for 5 years and showed substantial benefit including a 20% reduction in all-cause mortality (p=0.051). The survival curves were diverging at the end of the trial. On trial completion, the patients’ primary care doctors decided whether or not to prescribe statins without knowing the results until they were presented many months after trial completion. Interestingly, after a further 10 years of follow-up, the benefits observed in the trial had not dissipated (5). How far would the survival curves have diverged had the trial been continued for longer?
“Clinical trials require a sufficient number of events to demonstrate a treatment effect and the annual rate of serious cardiovascular events is relatively small in primary prevention trials (for instance about 1% per year for people with Type-2 diabetes (6)). There are at least four strategies to deal with this. Inclusion criteria can be added to increase risk; this usually means switching to secondary prevention. Surrogate outcomes can be used to provide more events but, if they don’t matter to patients or the public, they are of questionable value. Trials could be longer but the patent-life of new medicines is only 25 years and it can easily take 10-15 years to get a new medicine licensed, making it difficult to make a return on investment and driving up what companies need to charge for new treatments. Unfortunately, enrolling large numbers of patients is an inadequate substitute for longer exposure to treatments. Regulators need to incentivise trials of a longer duration. This could be done in several ways. It is probably essential to extend patent-life (artists get 70 years intellectual property protection), which could be linked to a large reduction in the costs of new medicines creating a win-win situation for industry and consumers (1,2). Trial design could be simplified, with patients simply prescribed a treatment, or not, without the expensive complexity of a placebo arm. Companies might also share the costs of conducting long-term trials with organisations, such as the National Health Service in the UK, in innovative partnerships. The current system is breaking and we need to fix it.”
References
1. Cleland JG, Atkin SL. Thiazolidinediones, deadly sins, surrogates, and elephants. Lancet 2007;370(9593):1103-4. doi: 10.1016/S0140-6736(07)61488-3
2. Cleland JG, Witte, K, Steel S. Calcium supplements in people with osteoporosis. BMJ 2010;341:c3856. doi: 10.1136/bmj.c3856.
3. Cleland JG. Is aspirin useful in primary prevention? Eur Heart J. 2013;34(44):3412-8. doi: 10.1093/eurheartj/eht287
4.Cholesterol Treatment Trialists’ (CTT) Collaboration. Efficacy and safety of more intensive lowering of LDL cholesterol: a meta-analysis of data from 170 000 participants in 26 randomised trials. Lancet 2010; DOI:10.1016/S0140-6736(10)61350-5.
5. Ford I, Murray H, Packard CJ, Shepherd J, Macfarlane PW, Cobbe SM; West of Scotland Coronary Prevention Study Group. Long-term follow-up of the West of Scotland Coronary Prevention Study. N Engl J Med. 2007;357(15):1477-86. doi: 10.1056/NEJMoa065994
6. Wiviott SD, Raz I, Bonaca MP, Mosenzon O, Kato ET, Cahn A, Silverman MG, Zelniker TA, Kuder JF, Murphy SA, Bhatt DL, Leiter LA, McGuire DK, Wilding JPH, Ruff CT, Gause-Nilsson IAM, Fredriksson M, Johansson PA, Langkilde AM, Sabatine MS; DECLARE–TIMI 58 Investigators. Dapagliflozin and Cardiovascular Outcomes in Type 2 Diabetes. N Engl J Med. 2019;380(4):347-357. doi: 10.1056/NEJMoa1812389.
Prof Alun Hughes, Professor of Cardiovascular Physiology and Pharmacology, Department of Population Science & Experimental Medicine, UCL, said:
“In their abstract DuBroff, Malhotra and de Lorgeril claim that ‘the negative results of numerous cholesterol lowering randomised controlled trials call into question the validity of using low density lipoprotein cholesterol as a surrogate target for the prevention of cardiovascular disease.’
“The authors base this conclusion on a flawed analysis of published data.
“In contrast to the authors’ conclusion, I think there is convincing evidence that statins reduce total mortality and cardiovascular events,1 while evidence for beneficial effects of ezetimibe2 and PCSK9 inhibitors3 is less certain.”
*To take one example from Table 1 for cardiovascular (CVD) benefit. The authors categorise WOSCOPS 1995 A study as ‘yes’ based on an odds ratio of 0.69 with a 95% confidence interval of 0.57 to 0.83, whereas they classify the St Francis 2005 A study as ‘no’ on the basis of an odds ratio of 0.68 with a confidence interval from 0.43 to 1.07. The point estimate of the odds ratio is almost identical in these two studies the only difference between them is the precision of the estimate (the 95% confidence interval which in WOSCOPS doesn’t include zero and does in St Francis) – both studies are entirely compatible with one another, but according to the authors’ analytical approach one is ‘yes’ and the other is ‘no’!
References
1.Taylor F, Huffman MD, Macedo AF, et al. Statins for the primary prevention of cardiovascular disease. Cochrane Database Syst Rev 2013; 2013(1): Cd004816.
2. Zhan S, Tang M, Liu F, Xia P, Shu M, Wu X. Ezetimibe for the prevention of cardiovascular disease and all-cause mortality events. Cochrane Database Syst Rev 2018; 11(11): Cd012502.
3.Schmidt AF, Pearce LS, Wilkins JT, Overington JP, Hingorani AD, Casas JP. PCSK9 monoclonal antibodies for the primary and secondary prevention of cardiovascular disease. Cochrane Database Syst Rev 2017; 4(4): Cd011748.
Prof Kevin McConway, Emeritus Professor of Applied Statistics, The Open University, said:
“Some of the conclusions from this research just don’t properly follow from the data that the researchers considered. If conclusions are unsupported by the data, there’s no way to tell whether they are correct. The researchers took the results of a considerable number of randomised controlled trials (RCTs) of drugs intended to reduce the levels of ‘bad’ (LDL) cholesterol. (The drugs being trialed were usually statins, but another two classes of drugs were also included.) They compared the average amount of reduction of LDL cholesterol in each trial with the targets for reduction in LDL cholesterol, recommended by the American Heart Association and the American College of Cardiology for clinicians treating individual patients. Then they looked at whether each of the RCTs had found statistically significant evidence of a benefit of the cholesterol-reducing drug, either in terms of a reduction in death rates, or in the rates of cardiovascular disease (CVD) events (such as heart attacks or strokes), and they compared this with the average LDL cholesterol reductions in the trials.
“So what’s wrong with doing that? Several things, statistically. First, in comparing the average reduction in LDL cholesterol in the trials with the guidelines for individual treatment, they are simply not comparing like with like. For individuals at moderate risk of cardiovascular disease (defined in a specific way), for example, the American guidelines recommend reducing LDL cholesterol by at least 30%. One of the RCTs these researchers looked at, where the individuals in the trial would generally have been at moderate risk, reduced LDL cholesterol in those trial participants by 26% on average. Now 26% is obviously less than 30%, so the researchers on this new study declare that that particular trial did not meet the target for LDL cholesterol reduction. But the trial didn’t have a target for LDL reduction in individual participants. And the 26% figure is an average. Some individuals in the trial would have had an LDL cholesterol reduction by close to 26% – others would have had a much greater reduction, and others much less than 26%, or perhaps even no reduction at all. So some of the individuals in the trial would have met the individual guideline target for LDL cholesterol reduction, and others would not have met it. The average 26% reduction does not describe the individual experiences of the trial participants, except as an average, and averages never tell the whole story. The new study found very little association between the average LDL cholesterol reduction in RCTs and the reduction in risk of cardiovascular disease, again measured as an average for each trial. But that doesn’t mean there is no association between LDL cholesterol reduction in individual patients and their disease risk. There might or might not be such an association. To claim that an association, or lack of association, at the level of averages in groups or populations of people (such as the participants in an RCT) tells you anything direct about the association in individual people is such an important type of statistical fallacy that it even has a name (the ‘ecological fallacy’). It can happen that there’s no association at the level of group averages, but there is an association at the individual level, or that there is an association at the group level but not at the individual level, that the association goes in different directions at the group and individual level, or indeed that the two associations do match. You can’t tell without looking at both, and these researchers couldn’t look at both because generally the data on individual patients, looked at in the appropriate way, isn’t available from the RCT reports they considered.
“Also, there’s a considerable differences between the average reduction in LDL cholesterol in a trial, where participants were given a fixed daily dose of a statin, or a placebo, and a target LDL cholesterol reduction for an individual patient, where, if the target isn’t reached, the clinician involved should take an individual approach and discuss with the patient whether they are taking any drugs at the right time and regularly enough, should discuss other possible changes in behaviour and lifestyle, like weight reduction, changes in diet, changes in alcohol consumption and smoking, and might also consider changing the dose of statin-reduction drugs. This looks completely different to the average reduction in an RCT.
“The researchers make a point of reporting that they did not carry out a formal meta-analysis, which is the statistical process by which the results from several different RCTs (or other studies) of more or less the same treatments and outcomes can be combined to give a clearer picture of the effect of the treatment on the outcomes, based on more data. They say this is because the RCTs are too varied, in terms of the patients involved, and the details of the treatments, for it to make sense to combine them in a meta-analysis. There is something in this view – but the trouble is that, instead, the researchers carry out what amounts to an extremely crude, statistically inappropriate, meta-analysis anyway. They simply categorise each of the RCTs according to whether the average LDL cholesterol level was above or below a figure in the targets for individual treatment, whether it reported a statistically significant effect of the drug treatment on mortality rates, and whether it reported a significant effect on CVD risk. Their main analysis is simply to count up how many trials showed a significant effect on mortality or CVD risk according to how their average reduction in LDL cholesterol compared with the target for individuals, and calculate a couple of percentages. Just dividing the RCT results into two categories (on LDL cholesterol reduction, on mortality, and on CVD risk) is a notoriously misleading method of summarising results, because it does not take account of how far the cholesterol reduction was above of below the target, or how large or small any mortality of CVC risk reduction was. Even small effects that aren’t statistically significant might contribute to the overall picture, and help move the balance in terms of whether a treatment is effective – that’s why they are included in (statistically appropriate) meta-analyses – and I’ve already explained why their LDL cholesterol reduction measure doesn’t match the targets in the guidelines. Putting these crude two-category summaries together by just counting them up takes no account of the fact that some RCTs provided much stronger information than others, generally because some had many more participants than others. It’s true that the researchers did one statistical analysis (shown in Figure 3 in their report) that took account of the actual average percentage reductions in LDL cholesterol, not just whether they were above or below a target, and also a measure of the actual effects on CGD risk – but it’s only one analysis, and again it does not take into account that different RCTs provide different amounts of information because of their different numbers of participants.
“The way this all looks to me is that the researchers rejected a conventional statistical meta-analysis, but instead carried out what amounts to a statistically inappropriate, very crude, and potentially very misleading meta-analysis instead. If, as they state, they did not do a proper meta-analysis because the RCTs they looked at involved different types of drug classes and patient populations, how can it be appropriate to do calculations that compare the trial results in much cruder and even less appropriate ways?
“The researchers do point out that the American treatment guidelines, that they quote, emphasise lifestyle changes and an individualised approach to any LDL-cholesterol-lowering drug therapy, which does sound rather different from the concentration on drug trials in the rest of their report. The European guidelines, that they also mention, also stress these points, as do the UK’s NICE guidelines, which they don’t mention, but which also refer to individual approaches and lifestyle changes as well as mentioning risk scores and targets for change in cholesterol measures. The one point where I do sympathise with this new research report is that, despite all the research, there are still gaps in evidence for the best approach – but that’s already pointed out in the American, European and UK NICE guideline reports.”
Prof Stuart Pocock, Professor of Medical Statistics, London School of Hygiene & Tropical Medicine, said:
“In my opinion this article by DuBroff et al in BMJ is an extraordinary deception. It is well known in many randomized clinical trials and meta-analyses that statins and other lipid-lowering drugs are effective in reducing the risk of major cardiovascular(CV) events and deaths in a wide range of patient populations.
“So how did this article manage to twist this overwhelmingly positive evidence into a negative conclusion? Their first trick is classify each trial into 1) whether it showed a benefit (yes or no) and 2) whether it met the LDL-cholesterol target(yes or no). Such all-or-nothing classification of trials is not how evidence is assessed and combined. Instead each trial provide estimates of the treatment effects on mortality and CV events, and their statistical uncertainty expressed as confidence intervals. Meta-analyses then combine these estimates to reach an overall conclusion, in this case that such treatments are very effective.
“Their second deception is to plot each trial’s LDL-C reduction against their absolute risk reduction of CV events, claiming that the association is too weak to matter. Such meta-regression techniques are well known to be flawed especially if they ignore the markedly different patient populations across trials. In this case the absolute risk reduction in each trial depends heavily on whether its patient population is high-risk or low-risk. So is their plot meaningful? The answer is NO.”
Prof Stephen Evans, Professor of Pharmacoepidemiology, London School of Hygiene & Tropical Medicine, said:
“This paper claims to be a systematic review without a meta-analysis. The reasons given for not performing a meta-analysis which would summarise the available data numerically are not good reasons to fail to provide numerical summaries.
“They use a standard method to assess risk of bias noting that “the majority of these studies were of excellent quality and had a low overall risk of bias” but then go on to add their own unvalidated reductions in assessed quality.
“They compare “numbers needed to treat” (NNT), which they regard as “particularly helpful in assessing treatment effects” over different time periods, which is invalid, illustrating why NNT should always be treated with caution- they are not numbers which can be interpreted on their own.
“The authors suggest that reductions in LDL-cholesterol are not a good guide to cardiovascular benefit and question the benefits of statins. They quote a US AHA paper as saying “cardiovascular deaths appear to be on the rise”. In fact, the key paper they cite states ““From 2006 to 2016, the annual death rate attributable to CHD declined 31.8%”
“Almost the only clear presentation of data they give is in figure 3. They note that this does not show a clear relationship between reduction in LDL-cholesterol and absolute risk reduction in cardiovascular event rates. Their analysis of these data is flawed as it does not take uncertainty in estimates into account and misses the most obvious conclusion to be drawn from the figure! That is, all the trial results show a reduction in LDL-cholesterol and a reduction in cardiovascular event rates. The notion that this undermines the idea that statins and other cholesterol-lowering drugs bring cardiovascular benefits is clearly not true.
“Overall this opinion piece is selective in its presentation of evidence from the randomised trials and has misleading or misrepresenting use of other literature.”
Prof Robert Storey, Professor of Cardiology, University of Sheffield, said:
“There is a huge amount of evidence showing that LDL or ‘bad’ cholesterol is responsible, to a large extent, for the build-up of fat in the blood vessels supplying the heart, brain and other parts of the body. People who have developed furring of these blood vessels (known as ‘atherosclerosis’ or ‘plaque’) benefit greatly from treatment to lower cholesterol, such as statins, and this has contributed to a big fall in risk for patients who have had the most common types of heart attack and stroke. On this, there is no controversy and it is really important that people continue their statin and/or other cholesterol treatment long term after such an event, usually for the rest of their lives. Where the evidence becomes less clear is for the use of cholesterol-lowering treatment in people who do not have any evidence of furring of the arteries. This is because people who do not have ongoing furring of the arteries will not benefit in a meaningful way from cholesterol treatments over the few years that it takes to do a clinical trial, although this does not mean that they won’t benefit over a longer period of time if they are at higher risk of cardiovascular disease. Consequently, clinical trials that recruit too many people who do not have furring of their arteries may fail to show significant benefit of statins and other treatments. This should not be interpreted as meaning that these drugs aren’t effective when given to the right people on a long-term basis.”
‘Hit or miss: the new cholesterol targets’ by Robert DuBroff et al. was published in BMJ Evidence Based Medicine at 23:30 UK time on Monday 3 August.
DOI: 10.1136/bmjebm-2020-111413
Declared interests
Prof John Cleland: “I am conducting research on the WOSCOPS trial data.”
Prof Alun Hughes: “No Conflicts”
Prof Kevin McConway: “I am a member of the SMC Advisory Committee, but my quote above is in my capacity as a professional statistician.”
Prof Stephen Evans: “I was appointed by the British Medical Journal as an independent member of a panel to look into whether a previous paper by one of the authors of this paper should be retracted. I am also an member of an “Independent Oversight Panel for the Cholesterol Treatment Trialists’ (CTT) Collaboration meta-analysis of the effects of statin therapy on adverse events”.
“I am funded (1 day/week) by LSHTM. They get funding from various companies, including Astra Zeneca and GSK but I am not funded by them, I have no involvement in obtaining funding from them and I am not an investigator or any grants obtained from them. I am the statistician to the “meta-Data Safety and Monitoring Board” for CEPI. I will probably be paid for my attendance at meetings and expenses for travel.”
Prof Robert Storey: “Consultancy work with numerous pharmaceutical companies, including manufacturers of PCSK9 inhibitors, but no disclosures relevant to statins which are low cost and produced by many generic drug manufacturers”
None Others Received