The following originally appeared on The Upshot (copyright 2015, The New York Times Company)
Just a few weeks ago, a study was published in the Journal of Nutrition that many reports in the news media said proved that honey was no betterthan sugar as a sweetener, and that high-fructose corn syrup was no worse.
This shocked people on all sides of the sweetener debate. It has become an article of faith among many that natural sweeteners like honey are better for you than engineered sweeteners like high-fructose corn syrup, especially for people concerned about diabetes.
Not so fast. A more careful reading of this research would note its methods. The study involved only 55 people, and they were followed for only two weeks on each of the three sweeteners. Sure, glucose and insulin levels and measures of insulin resistance were no different for honey, sugar and high fructose corn syrup. But should we really place so much faith in such a small, short-lived trial?
The truth is that research like this is the norm, not the exception. I’ve written about nutrition quite often here at The Upshot — about weight loss,dietary guidelines, healthy food choices, the role of exercise in weight loss, the potential benefits of coffee — and a fair amount of the time, it’s to counter conventional wisdom, for example about milk, red meat orartificial sweeteners. Just a short while ago, I argued that while more recent nutritional guidelines are, perhaps, more evidence-based, they may still be straying from what we can glean from studies. Readers often ask me how myths about nutrition get perpetuated and why it’s not possible to do conclusive studies to answer questions about the benefits and harms of what we eat and drink.
Almost everything we “know” is based on small, flawed studies. The conclusions that can be drawn from them are limited, but often oversold by researchers and the news media. This is true not only of the newer work that we see, but also the older research that forms the basis for much of what we already believe to be true. I’m not ignoring blockbuster studies because I don’t agree with their findings; I’m usually just underwhelmed by what I can meaningfully conclude from them.
The honey study is a good example of how research can become misinterpreted. It’s among the stronger studies we have available because it’s a randomized controlled trial. As I’ve noted before, we often make inferences not from clinical trials, but from epidemiological research, which is limited in many ways. Randomized controlled trials, on the other hand, allow us to make much stronger arguments about causality, and can account for many other confounding factors that might bias cohort or case-control studies.
A 2011 systematic review of studies looking at the effects of artificial sweeteners on clinical outcomes identified 53 randomized controlled trials. That sounds like a lot. Unfortunately, only 13 of them lasted for more than a week and involved at least 10 participants. Ten of those 13 trials had a Jadad score — which is a scale from 0 (minimum) to 5 (maximum) to rate the quality of randomized control trials — of 1. This means they were of rather low quality. None of the trials adequately concealed which sweetener participants were receiving. The longest trial was 10 weeks in length.
Think about that. This is the sum total of evidence available to us. These are the trials that allow articles, books, television programs and magazines to declare that “honey is healthy” or that “high fructose corn syrup is harmful.” This review didn’t even find the latter to be the case. They could find no harms from high fructose corn syrup, and noncaloric sweeteners were found to lead to reduced caloric intake and small, but statistically significant decreases in body mass indexes. That hasn’t stopped many people from arguing the opposite.
Even when we manage to pull off higher-quality studies in this area, we often fail to reproduce reality. A study published this year in the journal Frontiers in Nutrition looked at eight different meta-analyses on the effect of fructose consumption on cardiometabolic risk. They found that the average dose of fructose given to people in those trials was more than two to three times what people really consume in the United States. So even if they found positive results, it isn’t terribly helpful because it doesn’t apply to what most people are actually eating.
My point is not to criticize research on sweeteners. This is the state of nutrition research in general.
Although it’s easy to point fingers and make a case that there are huge gaps in our evidence when it comes to food, it should be kept in mind that it’s incredibly hard to do this kind of work. The reason that we have to rely on small, poorly designed trials is because that’s often all we can get.
Study after study has shown that people, even those trying to lose weight,cannot stick to diets for long periods of time. And that’s the research looking at highly motivated people who have taken it upon themselves to change what they eat. If they can’t stick to a certain regimen, how can we expect study participants, who aren’t as invested, to follow strict instructions for months at a time?
This is especially true when people aren’t seeing immediate results, which happens often when it comes to food changes. If they think something isn’t working, they’re unlikely to continue.
Moreover, unlike drugs, which people can get from only a limited number of places, food can be obtained from all over. It’s impossible to control people’s intake of sugar, for instance, when it’s added to so many foods today — including pasta sauce, crackers and soy milk. You can try to get them to replace sugar packets with honey or high fructose corn syrup, but when sweeteners are in so much of what we eat, it’s nearly impossible to control intake as closely as we need, for as long as we’d like.
This is why some of the most powerful research on nutrition comes fromprisons or mental hospitals, where we can control what people eat more directly. But this comes with its own ethical concerns.
The outcomes people care about most — death and major disease — are actually pretty rare. Detecting differences in rates of those occurrences requires huge numbers of people, especially if you choose not to focus on older people.
Finally, the research we require — long-term randomized controlled trials with many participants — is very expensive. Most organizations lack the deep pockets of the pharmaceutical industry when it comes to that kind of work. Further, many food companies don’t feel as if they will see great return on investment from this type of research. They can sell their products without “proving” that they are healthy, so why bother? The study on honey I discussed earlier was funded by the National Honey Board, and I’m guessing they weren’t thrilled with the result.
When industry does fund research, people tend to view it with great skepticism anyway, making it a losing proposition.
Because of this, we will probably continue to see results mostly from small, sometimes-flawed, short-term studies of nutrients and additives. Treat the results of that research with the respect they deserve, but ignore the grandiose proclamations.
