July 11, 2014

nutritional content of organic and conventional foods

A meta-analysis published in the British Journal of Nutrition looked at 343 studies into compositional differences between organic and conventional crops, reporting differences including higher levels of certain anti-oxidants and lower levels of cadmium in organic crops. Roundup comments accompanied this before the headlines analysis.

Title, Date of Publication & Journal

Higher antioxidant concentrations and less cadmium and pesticide residues in organically grown crops: a systematic literature review and meta-analyses

July 15th 2014

British Journal of Nutrition

Claim supported by evidence?

This study provides limited evidence that organically grown crops contain, on average, higher concentrations of some antioxidants, lower concentrations of cadmium, and are less likely to contain pesticide residues than conventionally grown crops; however, there could be other reasons for these apparent differences.

The study found no evidence for differences in concentrations of a number of other antioxidants.

It does not prove that organic crops are higher in antioxidants.

There was also evidence that organic crops contain, on average, lower concentrations of fibre and protein and higher concentrations of carbohydrates, but these differences are small.

Summary

• This was a review and meta-analysis of a large number of existing studies (all peer-reviewed). This allows evaluation of all the published evidence but does not include any unpublished data.
• The results from the studies were combined to produce an overall average difference between organic and conventional crops with respect to levels of nutrients and contaminants.
• For many of the outcomes considered there was evidence for moderate or strong publication bias. Publication bias can lead researchers to find effects that are not real; this is exacerbated when using a random effects meta-analysis (as here) – even more so if the meta-analysis is unweighted (also used here).

Study Conclusions

This review includes a variety of types of study (field experiments, farm surveys, and basket studies, which compare organic and conventional products collected from shops) comparing organically-grown to conventionally-grown crops with respect to a number of different potentially beneficial or harmful constituents/contaminants. The main conclusion of the review is that there are higher levels of some antioxidants and a lower likelihood of pesticide residues in organic crops.  This is supported by the evidence provided, although it is impossible to ascertain the likely impact of publication bias.

The authors have carried out weighted meta-analyses based on standardised mean differences, both of which are appropriate methods.  However, they have also carried out unweighted meta-analyses, which are generally not considered appropriate because they give equal weight to all studies, regardless of their size (a study with a sample size of 10 would be considered on a par with a study with a sample size of 100, for example) or type (field trials are likely to be very different from basket surveys in terms of precision). In addition, they present unweighted mean percentage differences which would, for the same reasons, be considered inappropriate and potentially misleading. Unfortunately, these unweighted percentage differences are highlighted throughout.

The study found evidence that the concentration of some antioxidants may be higher in organic foods. However, they also found no evidence for differences in concentrations of quite a number of other antioxidants. This is not highlighted in the conclusions.

The study supports the finding of a previous meta-analysis suggesting that organic crops are less likely to contain pesticide residues than conventional crops, but no claims are made about adverse effects on health.

Strengths/Limitations

Strengths

This is a large study which considers all available published evidence and appropriate statistical methods have been used to combine the results from the different studies (although see previous comments about the unweighted analyses).

Limitations

Publication bias:

The authors highlight the problem of publication bias but do not suitably discuss what impact this may have had on the results. Generally publication bias is likely to result in an overestimate of the effect size – or, in some cases, finding an effect that is not real.  See glossary for more detail about this.

Heterogeneity (different studies found different things):

There was a large amount of heterogeneity between the studies for most of the outcomes considered; in other words, differences between organic and conventional crops were inconsistent between studies. The authors discuss various factors which are likely to have contributed to this heterogeneity and highlight the importance of investigating these factors in future studies. Because of this heterogeneity, the average difference between organic and conventional crops for these outcomes may be fairly meaningless, since the actual true difference (as indicated in the paper) varies from country to country, from crop type to crop type and from species to species.

Multiple comparisons:

The authors have carried out a large number of comparisons, only some of which are presented in the main part of the paper. This leads to an increased likelihood of obtaining “false-positives”- in other words, the more differences you look for, the more likely you are to find one just by chance. Some scientists advocate adjusting p-values to take account of this which the authors have not done in this case. At the very least, results that are borderline should be regarded with greater caution; the authors do not discuss this.

Additional comments

Some statements are particularly misleading:

The study does not prove that organic crops and crop-based foods are higher in a number of antioxidants. Statistical tests never allow you to prove something, only to provide degrees of evidence for or against it. In this case there is a strong possibility that the evidence is tainted by publication bias and, as such, should be treated with additional caution.

“Numerous studies have linked antioxidants to a reduced risk of chronic diseases, including cardiovascular and neurodegenerative diseases and certain cancers.” This is incorrect.  There is certainly evidence that a diet rich in fruit, vegetables and whole grains (which are a good source of antioxidants) is associated with lower rates of chronic diseases. However, although a few trials of antioxidant supplements have shown a benefit for specific conditions, most have produced negative results and some have even indicated that taking antioxidant supplements may be harmful. In addition, a recent meta-analysis provided evidence that taking antioxidant supplements may increase overall mortality.

Glossary

Meta-analysis: an analysis in which results from a number of different studies are combined to obtain an overall effect size; this combined result is (generally) a weighted average of the estimated effect in the different studies, such that larger, more precise studies are given greater weight than smaller, less precise studies.

Publication bias: this is bias resulting from the tendency for positive results (i.e. those finding a “statistically significant” result) to be more likely to be published than negative ones. Thus, all the unpublished negative findings are excluded from any meta-analysis. Publication bias cannot be addressed by reviewing larger numbers of papers, since the unpublished results will always be missing, but it is made worse if all possible studies are included in a meta-analysis rather than selectively including studies on the basis of their quality.

Before The Headlines is a service provided to the SMC by volunteer statisticians: members of the Royal Statistical Society (RSS), Statisticians in the Pharmaceutical Industry (PSI) and experienced statisticians in academia and research.  A list of contributors, including affiliations, is available here.