• This is why policy should be evidence based

    I think I need to make a macro for this: I’ve been skeptical about menu labeling, not because I think the idea is bad, but because I‘m unconvinced that it will work in practice. All of those links tell you why. But there’s a new study in the American Journal of Public Health that adds to my concerns. “Supplementing Menu Labeling With Calorie Recommendations to Test for Facilitation Effects”:

    Objectives. We examined the effect on food purchases of adding recommended calorie intake per day or per meal to the mandated calorie information posted on chain restaurant menus.

    Methods. Before and after New York City implemented calorie posting on chain restaurant menus in 2008, we provided daily, per-meal, or no calorie recommendations to randomized subsets of adult lunchtime customers (n = 1121) entering 2 McDonald’s restaurants, in Manhattan and Brooklyn, and collected receipts and survey responses as they exited. In linear and logistic regressions, with adjustment for gender, race, age, and day, we tested for simple differences in calories consumed and interactions between variables.

    Researchers approached 1121 McDonald’s customers both before and after calorie posting began in New York City. Each had a random chance of being handed (1) information that described the recommended calories a man and woman should eat each day, (2) information that described the recommended calories a man or woman should eat each meal, or (3) nothing. The hypothesis was that giving people information about recommended intake would help them to make better choices about how much to order. After all, the whole menu labeling thing is based on the idea that giving people calorie information will reduce obesity.

    The results are depressing. Giving people calorie recommendations didn’t change what people offered. In fact, although the result was not statistically significant (p=0.07), people who were given more calorie information ordered more calories. They also did lots of sub-analyses. For instance, looking only at those who were ordering full meals (instead of just drinks) yielded the same non-significant result. They also whether weight changed the result. If information caused overweight people alone to order fewer calories, that would still be a win. But that model still showed that more information led to more calories ordered (p=0.06).

    Look, I don’t think that providing calorie information is evil. But it’s not an unequivocal good. It has costs for businesses, and compliance can be tricky. Studies like this add to evidence that it can also backfire. I have been, and remain, skeptical about its power as national policy. That belief is based on the evidence, not my gut. Policy should be made with the former, not the latter.

    @aaronecarroll

    Share
    Comments closed
     
    • Suppose McDonald’s offered calorie packages: the 300-calorie meal, the 500-calorie meal, etc?

      Re the experiment, does it indicate that calorie info posted w/ each item wouldn’t be helpful? I definitely factored it in when choosing a donut at Krispy Kreme some months ago. If I saw two alternative hamburgers and one was significantly fewer calories, the info might tip my decision.

    • I’m not aware of any research on the impact of nutrition labels on foods we buy in the grocery store. However, it’s safe to say that obesity has gone up in the population since they were introduced. But nobody is advocating to get rid of that regulation. Many people rely on those labels to make informed choices and to educate people on making informed choices (including as part of programs, such as SNAP-ED, with an evidence base showing effectiveness).

      Over time, I see the same happening with menu labeling. I don’t believe there will be a huge immediate impact. But, at the same time, the study cited above does not reflect real-life decision making processes.

      Policy cannot always afford to wait for a full evidence base. The recent RWJF report indicates people want menu labeling; that may be good enough for some. (http://www.rwjf.org/content/dam/farm/reports/reports/2013/rwjf406357)

      And there’s also some evidence (I’m searching for the article on my desk, but can’t find it right now; I’ll keep looking) that menu labeling results in restaurants offering healthier options. But that may take some time before it trickles down to consumer behavior.

      • I’m going to disagree with you here. People “wanting” it isn’t close to enough.

        Food bought in grocery stores with labels is packaged and standardized. Restaurant food isn’t like that, and often the printed counts are insufficient. Also, there is nutritional information for the food, not just calorie counts, and that’s what you get in restaurants.

        I also linked to many studies showing this may not, or isn’t working. If there’s a great study that shows it does, let me know. I’m waiting…

        • Restaurant food in chain restaurants is very much standardized. That standardization, X lbs of beef bought results in Y steaks and/or Z hamburgers, is how they manage their costs and make their profits.

        • Can we realistically expect every individual strategy to have evidence of impact? How are NYC’s and Philly’s childhood obesity rates falling, if there’s no evidence that menu labeling, corner stores, and walkability are effective? Nobody believes there’s a silver bullet; how do we examine the impact of comprehensive efforts within the contexts of their communities? And how do we pull lessons for policy from such studies?

        • Why isn’t wanting it enough? These are companies that exist because of certain legal fictions. There is no particular reason we can’t impose mandates on them, especially when they impose external costs on society.

          Second, what exactly does this study tell us? That there is no statistical significant indication that calorie counts alone change what people order at two McDonalds at one point in time in NYC? Not very useful. How useful would it have been even if it had been significant?

          Finally, maybe we should provide more than calories. I don’t find that useful. I actually use the data on occasion when they provide information on carbs/fat/sodium etc. Many restaurants do.

    • While I’m in the same boat as my colleague above, in terms of making choices based on such info, I think both of us represent that statistically insignificant portion!

      What studies like these seem to be showing is a different intuitive claim, namely: people who eat at McDonald’s (or similar places) probably aren’t concerned about calories in the first place!

      To my mind, though, that’s not enough to stop the policy. I don’t think the practice is all that burdensome – at least not for large scale places. The have research kitchens which spend countless hours perfecting the special sauce. Telling us the calories in a Big Mac, and then posting it where it is easily accessible (doesn’t have to be the menu – a pamphlet to the side, like Wndy’s and Panera Bread did was fine) isn’t really a burden.

      • You’re only looking at the potential upside. What about the economic cost? What if it backfires? If this doesn’t actually work, then why would you support it?

        • That’s why I proposed making the info available in other formats.

          Most fast food companies already had their info available online. And the time and energy to figure this stuff out is minimal. It’s not that big a cost overall. I don’t see how chain-casuals, like Fridays, etc, would be harmed by making the info accessible, even if only online. Sure, some might be surprised that this or that app packs 2000 calories. But a health-conscious person wasn’t going to order that anyway.

    • I think this type of study can be ripe with selection bias. For example, the labeling of the calories could have caused a consumer to simply no longer eat/not order from McDonalds, instead of just order something with less calories. Therefore, the study could potentially be weeding out all of the individuals who would take caloric value into consideration when adjusting their health behavior, by not measuring the fact that these consumers just left the restaurent, or no longer purchase their food there.

      Anyway, my personal thoughts are also that caloric value without nutritional value is not very valuable.

      • I’m not sure you’ve got “selection bias” right. The RCT design should overcome that…

        Your concern of getting people to leave McDonald’s would have been noted by the design. It didn’t happen.

    • Do you happen to know if they broke down the effect on people who were trying to lose or maintain their weight or who had some other reason to specifically focus on the caloric content of the food? Some of the other commenters (and I’m among this group) have mentioned that they find labeling helpful, and I’m wondering if there’s a subgroup that finds the info helpful and whom it benefits. Specifically, maybe we shouldn’t be looking at whether the labeling causes people to change their behavior, but rather whether it helps them to change once they’ve made the decision to do so.

    • Great post! Think the response to it will separate out those who believe in liberal policy positions primarily because of mood affiliation (bad!) versus those who genuinely believe in them because of empirical data.

      • As a corollary, one often finds that people embrace a single study which reinforces their policy preferences while requiring significantly more evidence when their preferences are undermined.
        So maybe it’s my policy preferences showing, but my first response is to want more data and different experiments. There’s something interesting going on here, and surely we can get more from it than “dont hand out calorie info to people going into fast food establishments”. Maybe there’s a way to do this that gets better results.

    • This is complete speculation, but I bet calorie counts on menus in chains paradoxically acclimates people to high calorie counts.

    • Im wondering if the control group (ie no pamphlet) might differ from the test groups in being, er, “less irritated at do-gooders telling them what to eat”. Or maybe “less depressed because they’re thinking about how crap their diet is compared to what they think it should be”. Etc.
      Point being that, controls are tricky. I think a better control here might’ve been a generalized pamphlet about eating healthy- or both controls, to see if there’s any effect from being stopped by a stranger and handed something effectively condemning what you’re about to eat (and perhaps your lifestyle and level of fitness as well) versus the specifics of receiving calorie info.