January 16, 2017

expert reaction to acupuncture to treat colic

Scientists publishing in Acupuncture in Medicine claim acupuncture appears to safely reduce crying in infants with colic. A Before the Headlines analysis accompanied these comments.

Prof. David Colquhoun FRS, Professor of Pharmacology at University College London, said:

“It is truly astonishing that, in the 21^st century, the BMJ still publishes a journal devoted to a form of pre-scientific medicine which after more than 3000 trials has still not been able to produce convincing evidence of efficacy¹. Like most forms of alternative medicine, acupuncture has been advocated for a vast range of problems, and there is little evidence that it works for any of them. Colic has not been prominent in these claims. What parent would think that sticking needles into their baby would stop it crying? The idea sounds bizarre. It is. This paper certainly doesn’t show that it works.

“The statistical analysis in the paper is incompetent. This should have been detected by the referees, but wasn’t.  For a start, the opening statement, ‘A two-sided P value ≤0.05 was considered statistically significant’ is simply unacceptable in the light of all recent work about reproducibility.  Still worse, Table 1 uses the description ‘statistical tendency towards significance (p=0.051–0.1)’.

“Worst of all, Table 1 reports 24 different P values, of which three are (just) below 0.05. Yet no correction has been used for multiple comparisons. This is very bad practice. It’s highly unlikely that, if the proper correction had been done, any of the results would have given a type 1 error rate below 5%.

“Even were it not for this, most of the ‘significant’ P values are marginal (only slightly less than 0.05).  It is now well known that the type 1 error rate gives an optimistic view. What matters is the false positive rate – the chance that a ‘significant’ result is a false positive.  A P value close to 0.05 implies that there is at least a 30% chance that they are false positives.  If one thought, a priori, that the chance of colic being cured by sticking needles into a baby was less than 50%, the false positive rate could easily be greater than 80%².  It is now recognised that this misinterpretation of P values is a major contributor to the crisis of reproducibility.

“Other problems concern the power calculation.  A priori calculations of power are well-known to be overoptimistic, because small trials usually overestimate the effect size.  In this case the initial estimated sample size was not attained, and a rather mysterious recalculation of power was used.

“Another small problem: the discussion points out that ‘the majority of infants in this cohort did not have colic’.

“The nature of the control group is not very clear. An appropriate control might have been to cuddle the baby – this was used in a study in which another implausible treatment, chiropractic, was shown not to work³.  This appears not to have been done.

“Lastly, P values are reported in the text without mention of effect sizes. This is contrary to all statistical advice.

“In conclusion, the design of the trial is reasonable (apart from the control group) but the statistical analysis is appalling.  It’s very likely that there aren’t any real effects of acupuncture at all. This paper serves more to muddy the waters than to add useful information. It’s a model for the sort of mistakes that have led to the crisis in reproducibility.  The BMJ should not be publishing this sort of stuff, and the referees seem to have no understanding of statistics.”

¹ Colquhoun & Novella. 2013. Anesthesia & Analgesia, 116, 1360 – 1363 (reprint)

² http://rsos.royalsocietypublishing.org/content/1/3/140216

³ Olafsdottir et al 2001, reprint

Prof. George Lewith, Professor of Health Research at the University of Southampton, said:

“This looks to me to be a good sized fastidious well conducted study (Cochrane A for an acupuncture trail in which the therapist cannot be blinded). The outcome is clear and the power of the study seems reasonable which suggests that MA is a reasonable and as far as we know safe intervention for infantile colic. Three out of 4 similar studies come to this conclusion so a systematic review would be a good idea and even better an individual patient data (IPD) meta-analysis. The evidence of acupuncture’s safety is conclusively established in adult medicine so suggesting this intervention is safe is reasonable in spite of the limited safety data in children. It’s too small a study to be conclusive on its own but as there is no proven conventional treatment for infantile colic one could argue there is more evidence for acupuncture than conventional best practice.”

Prof. Edzard Ernst, Emeritus Professor of Complementary Medicine at the University of Exeter, said:

“This study shows almost the opposite of what the authors conclude. Both minimal and traditional Chinese acupuncture seemed to reduce the symptoms of colic compared to no acupuncture at all. This confirms previous research showing that acupuncture is a ‘theatrical placebo’.

“The study was designed without an adequate placebo group; therefore, conclusions about specific therapeutic effects of acupuncture are not permissible.

“We know that colicky babies respond even to minimal attention, and this trial confirms that a little additional TLC will generate an effect. The observed outcome is therefore not necessarily related to acupuncture.”

* ‘Effect of minimal acupuncture for infantile colic: a multicentre, three-armed, single-blind, randomised controlled trial’ by Kajsa Landgren and Inger Hallström published in Acupuncture in Medicine on Monday 16^th January.

Declared interests

Prof. Ernst: I have no conflicts of interest. I have been an acupuncture researcher for over 25 years; I have published over 100 papers on the subject.

Prof. Colquhoun: No conflicts of interest.