March 22, 2018

reanalysis of the PACE trial

A reanalysis of the PACE trial is published in BMC Psychology.

A roundup accompanied this analysis.

This review was edited on 23rd April 2018.

Title, Date of Publication & Journal

Rethinking the treatment of chronic fatigue syndrome – a reanalysis of findings from a recent major trial

To be published: Thursday 22 March

BMC Psychology

Study’s main claims – and are they supported by the data

The paper reports that claims about the effectiveness of graded exercise therapy (GET) and cognitive behaviour therapy (CBT) in the original PACE trial may have been exaggerated.  The authors reanalyse data from the ‘PACE’ trial by following the analysis plan that was set out in the original protocol; they do this in a transparent way and make sensible assumptions with regards to missing information.

The results of this reanalysis raise uncertainty about the original analysis of the PACE trial because it exposes the fact that some conclusions are sensitive to the way the primary outcome measure is defined. It would be inappropriate to use these new results to make any news claims about the effectiveness of CBT and GET treatments or class them as ineffective because improvement was observed on some measures but not others, and there is imprecision in the estimates. The justification for the changes in the way that the primary outcome was defined between published protocol and trial report are not reported in this paper and are unclear from the published trial report. (Fuller reasons for the changes are given elsewhere by the PACE triallists, in a paper in BJPsych Bulletin given as a reference in the new paper but not discussed there in detail.)

The press release seems accurate and gives a good overview of the paper.

When planning a randomised trial, it is good practice to have a carefully planned protocol and analysis plan that are published in advance on a trial registry and/or in a research journal.  One goal of trial registries is to avoid the selective reporting of trial results which can lead to exaggerated or false claims about the effectiveness of treatments. However, it is worth noting the PACE trial was planned at a time when trial registration was in its infancy and the problems with selective reporting were less well known. Differences between protocol and analysis were, and indeed still are, not uncommon, and the original analyses should not be invalid if the changes were well justified and planned in advance of the data analysis.

Reanalysis of trial data like this is rare; however as journals enforce data sharing rules, analyses will be increasingly exposed to external scrutiny in this way.

Strengths/Limitations

Strengths

The authors appear to have carefully followed the original PACE protocol and have presented their own analyses in a reasonably clear and transparent way. Sensible assumptions have been made about missing data and the data has been analysed in a number of different ways to show that results are robust. This reanalysis shows that had the original trial authors followed their own planned protocol, they would have reached substantively different conclusions. The authors of this paper appropriately use their results to raise concerns about the robustness of claims made previously about the effectiveness CBT and GET on the basis of the PACE trial.

Additional unplanned analyses were performed on long term data, collected at least 2 years after treatment. These analyses were repeated after excluding patients who received additional treatment after the trial, which suggests that conclusions about lack of long term effect are reasonably robust and not the result of post-trial interventions.

Limitations

There are points of the original PACE protocol that were either inconsistent or open to interpretation. The authors in this paper seem to have selected the most extreme analysis to make their point: for example by making adjustment for 6 comparisons where 3 or 5 comparisons are also described, and focusing on 52 week data only.

The authors rely heavily on p-values and thresholds for statistical significance when reporting results; this is quite an out-dated approach and information about the magnitude and precision of treatment effects is missing in most cases.

The authors have made little attempt to uncover the reasons for protocol deviations in the PACE trial or the point at which they were made; trialists could have been invited to comment.

No adjustment was made in the re-analysis for patient characteristics, even though this was planned in the original trial. Sufficient data to do this was unavailable. Including appropriate demographic variables could improve the precision of results.

Only the 52 week data was analysed where data was collected at 12, 24 and 52 weeks. An analysis that takes into account the measurements over time could have lead to more precise results.

The new paper may give the impression that all, or almost all the evidence on CBT and GET comes from the PACE study, and it says that “it seems unlikely that further research based on these treatments will yield more favourable results”. In fact, CBT and exercise therapies have been investigated in several other studies, and these have been reviewed in Cochrane reviews. The latest such Cochrane review (of exercise therapies, from 2017) includes eight studies other than PACE, and does come to positive conclusions about some aspects of effectiveness of exercise therapies.

Glossary

CBT- Cognitive behavioural therapy

GET- Graded exercise therapy

Any specific expertise relevant to studied paper (beyond statistical)?

Experience in the design and analysis of randomised clinical trials. Statistician for Cochrane Collaboration.

Before The Headlines is a service provided to the SMC by volunteer statisticians: members of the Royal Statistical Society (RSS), Statisticians in the Pharmaceutical Industry (PSI) and experienced statisticians in academia and research. A list of contributors, including affiliations, is available here.