select search filters
briefings
roundups & rapid reactions
before the headlines
Fiona fox's blog
A study published in the Lancet Public Health looks at calorie labelling, obesity and deaths from cardiovascular disease (CVD).
Prof Sir David Spiegelhalter, Emeritus Professor of Statistics, University of Cambridge, said:
“There are many assumptions in this modelling exercise, and the estimated effects seem very small. Without any labelling, the authors expect around 830,000 cardiovascular deaths over 20 years. With the current proposals, they estimate this would reduce to 829,270. If labelling were extended to out-of-home food businesses, they would expect 820,800 deaths. These effects would not be measurable from future data.”
Dr Katarina Kos, Senior Lecturer in Diabetes and Obesity, University of Exeter, said:
“The method section of the modelling-based study explains that ‘We (the authors) assumed that the implementation of menu energy labelling would reduce energy consumption by 47 kcal (95% CI 15–78) for each out-of-home meal, estimates from Crockett and colleagues’ Cochrane review and meta-analysis of randomised controlled trials.’ The authors build on this assumption to model the meaning of these reductions for health outcomes including obesity. This study was not set out to test whether assumptions from the said 2018 review are correct or wrong, or whether there is new evidence on the impact of calorie intake or weight change.
“The next assumption is that energy content of menus had reduced following labelling as reported in 2018. However, these data/assumptions are pre-pandemic and from prior to the economic downturn.
“Also, we learn that ‘Due to no evidence to the contrary, we (the authors) assumed that the effect menu energy labelling has on consumer behaviour is consistent over time’. With the pandemic we learnt that this is not the case. It follows that: ‘We (the authors) assumed no differential policy effects by sex, age, or socioeconomic position on the basis of the current literature.’ In contrast, it is now well documented that food poverty affects the choice and amount of food intake to more calorific and poorer quality foods. Whilst the calculations may not be wrong, these assumptions they are based on appear outdated and thus their conclusions will not be meaningful.”
Dr Duane Mellor, Registered Dietitian and Senior Lecturer, Aston Medical School, Aston University, said:
“It is vital that potential public health policies be thoroughly tested through modelling prior to considering their implementation, and the authors have undertaken a detailed theoretical estimate of the effect of calorie labelling on population average bodyweight and body mass index (BMI). This reports the potential impact of implementing energy (calorie) labelling on all menus where food is eaten out of the home. Currently, large chains of restaurants and food outlets have to label the calories on their menus, and this has concerned eating disorders charities as it can be a barrier to eating out with family and friends for people recovering from an eating disorder. Should the policy of calorie labelling on menus be extended to all out of home food businesses this could add further to this risk.
“Additionally this modelling assumes – based on a meta-analysis – that menu calorie labelling leads to a reduction in energy consumption, however there are studies which suggest calorie labelling in some groups can lead to increased consumption as it is seen as better value for money (Fernandes et al., 2016 https://academic.oup.com/nutritionreviews/article/74/8/534/1751896)! This highlights that calorie labelling on menus in isolation could have the potential to have the opposite effect – it is important to look at the overall nutritional balance of meals and how they fit into an overall dietary pattern. It is not sensible to focus on foods solely on their energy content to assess how healthy or not they are.
“One big omission from this paper is the cost of implementing calorie labelling on all menus for small food businesses – this is before considering how the accurate the information on calories actually is. For example a black coffee made from beans one fast food restaurant contains 6kcal per cup, but an Americano from a restaurant (basically a black coffee) contains 15kcal. Given the modest amount of calories per meal that providing calorie information on menus is reported to save, this could, if this information is not accurately calculated enough, simply reflect errors in how the amount of calories was calculated. However, for small businesses to accurately estimate the nutritional composition and calorie content of menus, along with other essential safety information e.g. allergens, this could make the costs prohibitive. So, when considering implementation of universal calorie labelling on menus, the costs to business and accuracy of data need to be reviewed as part of the overall assessment of how useful such an approach is to improve health.”
Prof Kevin McConway, Emeritus Professor of Applied Statistics, The Open University, said:
“This study is entirely based on statistical predictive models of what might plausibly happen in the future. The inputs to the modelling process include data on things that have already occurred, but also projections of future population, and the results of other modelling exercises and investigations. But the modelling inevitably makes assumptions about how things will change in the future, over the period from 2022 to 2041 which the new study covers.
“You might wonder why this is worth doing. Statistical models, particularly predictive ones, often get a bad press, because things don’t turn out as the model said they might. However, many statisticians (including me) are fond of quoting an eminent 20th century Anglo-American statistician, George Box, who said that “All models are wrong, but some are useful.” Models inevitably simplify what goes on in the real world, so the will misrepresent some aspects, just as a map won’t exactly represent what’s there on the real ground. But models can give us some idea of what has happened, or might happen, in ways that can be very helpful, just as a map can be very helpful. Here, policy-makers do have to make decisions on whether to change or increase the requirements for compulsory calorie labelling for food provided and eaten outside the home. They can’t observe what will happen under their policies over a twenty-year period in the future. A good, useful, model can help them see what might work and what might not work, and can allow them to compare the impact of different policy choices, in ways that are essentially impossible to do using other methods1. One can be pretty confident in saying that the predictions from this modelling study will not happen exactly, and with hindsight, might turn out to be quite inaccurate. But, if nothing else, the exercise will indicate what features appear to be important and where the biggest uncertainties lie.
“Maybe this particular use of modelling looks particularly strange, given that one of the scenarios for policy change that it assesses, mandatory calorie labelling by larger out-of-home food businesses, was already implemented in 2022, so that policy decision is already made. Also, it’s now 2024, so why does the new model start from 2022? Well, the researchers need to look at the whole impact of that policy change so they need to go back to when it was implemented. And the main health outcome that is being investigated, deaths from cardiovascular disease, generally would take some time to respond to changes in people’s intake of calories, so that outcome of the 2022 legislation isn’t anywhere near clear yet. Indeed the future impact would still not be clear even if we wait until 2041 – there will, by then, be data on changes in cardiovascular deaths, but figuring out how much of those changes is due to mandatory calorie labelling won’t be possible even then, without a lot more statistical analysis and, indeed, modelling.
“Further, the modelling approach allows some assessment of the impact of a different, alternative, policy – mandatory calorie labelling for all out-of-home food outlets.
“I’ll clarify one point that is not mentioned explicitly in the research paper, though it is mentioned in the press release. The policies that are being investigated are not about all calorie labelling, but on making the labelling compulsory, either for larger outlets with at least 250 employees, or for all outlets. Calorie labelling is estimated to have already been adopted voluntarily by about 59% of the larger outlets, according to work carried out in 20192. Some of the smaller outlets were also using calorie labelling, though far fewer as a percentage. This is a major part of the reason that the impact of the current policy on mandatory labelling looks relatively small in this study – the model is assuming that more than half of them were already labelling their menus with calories before they were compelled to do so by law (even though the evidence for that assumption seems somewhat problematic to me2). One reason why the impact of a different policy, of introducing mandatory labelling for all outlets, is much bigger, according to the model, is simply that far fewer of the smaller outlets are using calorie labelling voluntarily, so that the change of requiring them to do the labelling is much greater than for the larger businesses.
“The point of the modelling is to estimate the effect of the two policies that are being assessed on obesity in the English population, and through that effect on obesity, on deaths from cardiovascular disease (CVD). In general, the modelling assumptions that are made in the new research seem very broadly plausible to me, though they are obviously and inevitably a simplification of what might actually happen. And there are gaps where the researchers did not model some aspect that is likely to exist, often because they have no evidence for how big its effect might be. But some of the assumptions are subject to considerably more uncertainty than others. The researchers do take the uncertainty seriously, and investigate it by calculating ‘uncertainty intervals’ that take into account plausible range for many of the inputs, and also by sensitivity analyses where they change some of the assumptions of the model to investigate how much that changes the health outcomes. That’s all good practice – since no modelling of health and nutrition issues can ever be without uncertainty, it’s important to get a handle on how much uncertainty there is.
“The researchers estimate that the current policy (compulsory labelling by large food businesses only) would prevent or postpone 730 CVD deaths in England between 2022 and 2041, but with an uncertainty interval that runs from 430 to 1,300 deaths prevented or postponed. Now perhaps that does sound like quite a lot of deaths, but you have to bear in mind that CVD is a very common cause of death in countries like England. The researchers estimate that, on current trends, and without any compulsory calorie labelling at all, there would in total be somewhere between 600,000 and 120,000 CVD deaths over that 20-year period (with their best estimate being 830,000) – that’s an average of between 30,000 and 60,000 a year. Saving somewhere between 21 and 65 CVD deaths annually, on average, by the current policy of compulsory labelling, is almost lost in the noise amongst those huge annual numbers of CVD deaths. If you take into account that the uncertainty intervals will not take into account all plausible aspects of uncertainty, it looks feasible that there may be no saving in CVD deaths at all. Or maybe the annual number could be considerably larger than the 65 deaths per year average – but it would take some major problems with the modelling or its assumptions to make the number of deaths prevented or postponed reach a substantial proportion of the total CVD deaths each year. And, as the researchers point out, these figures do not include deaths from other causes that are also known to be impacted by obesity, such as some cancers, or the consequences of diabetes.
“The other policy that the researchers considered, of making all out-of-home food businesses publish calorie counts on their menus, is estimated to prevent or postpone many more CVD deaths. The central estimate, for the same period from 2022 to 2041, is 9,200 CVD deaths, or on average 460 a year. The uncertainty interval goes from 5,500 to 16,000 deaths, or on an average annual basis, 275 to 800. That’s still not a huge number compared to the total CVD deaths, but it’s much greater than the annual average of between 21 and 65 for the policy only involving larger businesses.
“To emphasise again the uncertainty in these estimates, it’s interesting to consider a slightly different version of the model, for which the researchers give the results in their Appendix. In their modelling, they need to estimate the share of large food businesses in the out-of-home sector. In the main models, they use the number of outlets to estimate the share, instead of the turnover. They give reasons for this choice in the Appendix, but without giving many details. How they do this is clearly likely to make a difference to the results, because large businesses account for 18% of all the outlets in England, but account for 47% of the turnover. (That is, they have much larger average turnover per outlet than do the smaller businesses, not surprisingly.) From Appendix Table 3, over the 20 year modelling period, if the turnover rather than the number of outlets is used to estimate the share of the large businesses, the estimate of deaths prevented or postponed with the current policy of mandatory labelling only for large businesses is 2,000 (with uncertainty interval from 1,200 to 3,500), and for the policy of requiring labelling by all businesses, it is 7,200 (uncertainty interval 4,300 to 13,000). Using their main method of using number of outlets to estimate the share of large businesses, the corresponding numbers were, for current policy, 730 (uncertainty interval 430 to 1,300) and 9,200 (uncertainty interval 5,500 to 16,000). The big differences between the two sets of estimates indicate one major aspect of uncertainty, for a data input (the share of large businesses in the sector) that’s clearly every important but not known. And the uncertainty intervals for the main estimates don’t include the effect of this particular uncertainty.
“Now preventing or postponing any death, other things being equal, is worthwhile, but other things never are equal. If money and resources are not spent on this policy, they could be available to spend on some different policy that would have more impact. And, as the researchers point out, they did not take into account possible unintended negative effects of labelling calories on menus, particularly in triggering or reinforcing eating disorders.
“The researchers describe several limitations of their study. One that I consider important is as follows. The driver through which calorie labelling prevents or postpones CVD deaths is by reducing how many calories people consume, and hence reducing obesity – and that means that the model has to make assumptions about the effects of the calorie labels on reducing calorie consumption. To do that, they mainly use data from a Cochrane review of the evidence, their reference 10. The researchers point out (as the Cochrane review itself did) that the evidence behind that review comes from just three randomised trials, all of which were assessed as being at high risk of bias. In fact (it seems to me) none of those randomised trials addresses what must be a relevant issue here, that the response to labelling probably is different in a small scale context of a trial to what it would be in the context of having calorie labels on all food from big suppliers (now the status quo, of course), or on all out-of-home food from all suppliers. Will people just ignore the labelling if it’s not new any more and is on everything, or will they pay more attention, or what? We really can’t be sure, I’d say. And if the modelling results on reductions in calorie consumption aren’t well-evidenced, then those issues feed through to the estimates on obesity reduction and hence to the results on CVD deaths.
Further information
“One more point. The researchers say that they could not carry out a proper cost-effectiveness evaluation to compare the two policies that they consider, and that that is because they don’t have empirical data on several important aspects. It’s interesting to note that that did not deter the Department of Health and Social Care (DHSC) carrying out an impact evaluation in 20202, before the current policy was adopted. This evaluation did include cost estimates, though I can’t judge how good they are and there must be considerable uncertainty involved. The evaluation considered 5 different policies, with different options for which businesses to exempt from the requirement to put calorie labelling on their menus. The options were not to require any mandatory calorie labelling at all, to require it for all out-of-home food businesses (corresponding to the second policy considered in the new study), to require it only for businesses with 10 or more employees, to require it only for businesses with 50 or more employees, and finally to require labelling only for businesses with 250 or more employees (corresponding to the first policy considered in the new study, which is the policy that was actually adopted by Government). In fact the policy that was preferred by the DHSC evaluation was the final one, the one that was eventually adopted.
“However, part of the evaluation involved calculating a Net Present Value (NPV) for each policy, a tool that often plays a part in economic evaluation of policies. The policy that rates highest on this criterion was not the final option, but the one that involved making all businesses, of whatever size, display the calorie counts on their menus – that is, the second policy considered in the new research study. I do not think this means that the Government were completely illogical in their policy choice. The DHSC report points out that they could not take some costs into account because of lack of information, and also that there were some doubts about the feasibility of requiring labelling for smaller outlets. The report also points out that the option that was adopted was not initially part of the impact evaluation, but was added only after a consultation exercise. What all this says to me is that the relatively simple comparison of just two policy options in the new study is possibly just too simple, and misses out the important aspects of costs, and indeed of politics.”
1 Good points about the use and importance of this type of modelling are made in the Comment piece that accompanies this research report.
2 The 59% figure for the proportion of large business who were already using calorie labelling voluntarily is reported in the Appendix to the main research paper, though there is considerable lack of clarity. The Appendix cites as the source for this figure a Department of Health and Social Care report from 2020, https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/992872/calorie-labelling-impact-assessment.pdf , that assessed the impact of various options for calorie labelling. That report in turn cites a 2019 research paper, https://doi.org/10.1186/s12889-019-7017-5 , for which two of the five authors are also authors of this new research paper. But the 2019 paper and the 2020 report do not say that 59% of the large businesses that they examined were already putting calorie information on their menus – only that 17% of them were doing that and another 42% or so provided the calorie information on request. Further, the figure assumed for the percentage of smaller outlets that were voluntarily using calorie labelling seems to have come from the same DHSC report, although that report itself says (paragraph 63) that there is not really any evidence at all to support that estimate, because none was available.
‘Effect of calorie labelling in the out-of-home food sector on adult obesity prevalence, cardiovascular mortality, and social inequalities in England: a modelling study’ by Zoé Colombet et al. was published in the Lancet Public Health at 23:30 UK time on Wednesday 28 February 2024.
Declared interests
Prof Sir David Spiegelhalter: “No COI.”
Dr Katarina Kos: “I have no competing interests.”
Dr Duane Mellor: “No conflicts of interest.”
Prof Kevin McConway: “I am a Trustee of the SMC and a member of its Advisory Committee. My quote above is in my capacity as an independent professional statistician.”