Anyone who’s been reading this blog for any amount of time knows that I’m a pretty big fan of the work of John Ioannidis, and that I like writing about the challenges of nutrition research. Thus, you can imagine my excitement when I saw that JAMA had published this opinion piece from him called “The Challenge of Reforming Nutritional Epidemiologic Research“. The whole article is quite good, but for those who don’t feel like wading through it, I thought I’d pull together some of the highlights. Ready? Let’s go!

1. Everything’s a problem (or maybe just our methods) Ioannidis starts out with an interesting reference to a paper from last year called “Food groups and risk of all-cause mortality: a systematic review and meta-analysis of prospective studies“. This meta-analysis looked at the impact of various food groups on mortality, and reported the significant associations. Ioannidis points out that almost every food they looked at had a statistically significant association with mortality, even at relatively small intakes. Rather than get concerned about any one finding, Ioannidis raises concerns about the ubiquitousness of significant findings. Is every food we eat really raising or lowering our all cause mortality all the time? Or are we using methods that predispose studies to finding things significant?
2. Reported effect sizes are large aren’t necessarily cumulative The second thing Ioannidis points out is exactly how large the effect sizes are. The study mentioned in point #1 suggests you get  1.7 extra years of life for eating a few extra hazelnuts every day? And that eating bacon every day is worse than smoking? That seems unlikely. The fundamental problem here is that most food consumption is heavily correlated with other types of food consumption, making it really difficult to tease out which foods are helping or hurting. If (hypothetically) vegetables were actually bad for us, but people ate them a lot with fruit (which was good for us) we might come to the conclusion that vegetables were good merely because their consumption was tied to other things. As Ioannidis puts it “Almost all nutritional variables are correlated with one another; thus, if one variable is causally related to health outcomes, many other variables will also yield significant associations in large enough data sets. “
3. We focus too much on food itself Speaking of confounders, Ioannidis goes on to make another interesting point about how food consumption is always assumed to be beneficial or risky based on properties of the food itself, with potential confounders being ignored. For example, he cites the concern that grilling meat can create carcinogens, and the attempts to disentangle the cooking method from the meat itself. Drinking scalding hot beverages is known to increase the risk for esophageal cancer, separate from what the beverage itself actually is. It’s entirely plausible there are more links like that out there, and entirely plausible that various genetic factors could make associations stronger for some groups than others. Teasing those factors out is going to be extremely challenging.
4. Publication methods encourage isolation of variables One of the other interesting things Ioannidis points out is that even very large long term studies (such as the nurses health study) tend to spread their results out over hundreds if not thousands of papers. This is a problem that we talked about in the Calling Bullshit class I reviewed: researchers are more rewarded for publishing in volume rather than for the quality of each paper. Thus, it makes sense that each nutrient or food is looked at individually, and headline writers magnify the issue. Unfortunately this makes the claims look artificially strong, and is probably why randomized trials frequently fail to back up the observed claims.
5. Nutritional epidemiology uniquely impacts the public So what’s so bad about an observational study failing to live up to the hype? Well, nothing, unless clinical recommendations are based on it. Unfortunately, this study found that in 56% of observational studies, the author recommended a change to clinical practice. Only 14% of those recommendations came with a caveat that further studies might be needed to corroborate the findings. This is particularly concerning when you realize that some studies have found that very few observational studies replicate. For example, this one looked at 52 findings from 12 papers, and found that none of them replicated in randomized trials, and 5 actually showed a reverse in correlation. Additionally, headlines do little to emphasize the type of study that was done, leading to a perception that science in general is unreliable. This has long term implications both for our health and for our perception of the scientific method.

Overall I enjoyed the piece, and particularly its link to promising new recommendations to help address these issues. While criticizing nutritional epidemiology has become rather popular, better ways of doing things have been more elusive. Given the level of public interest however, we definitely need more resources going in to this. Given that the NUSI model appears to have failed, new suggestions should be encouraged.