Make me some science I can’t refuse

In case you missed it, in a recent article published in the American Journal of Preventive Medicine entitled Overstatement of Results in the Nutrition and Obesity Peer-Reviewed Literature (not making this up), the authors found that a lot of papers published in the field of obesity and nutrition have, shall we say, issues.

Well–as they say down South– I never!

The authors looked at over 900 scientific articles on nutrition or obesity published either in 2001 or 2011 in leading journals. They found that about 1 in 11 include “overreaching statements of results.” 

Here’s how the authors described statements that would be coded as “overreaching”:

  • reporting an associative relationship as causal
  • making policy recommendations based on observational data that show associations only (e.g., not cause and effect)
  • inappropriately generalizing to a population not represented by the sample studied

Frankly, I am totally offended. Someone needs to let these folks know that, in nutrition epidemiology, correlation actually does equal causation.

What’s more, nutrition policy recommendations are supposed to be based on observational data. Hello? Dietary Guidelines? (Seriously. You don’t expect public health nutrition people to do actual experiments now, do you? I mean, unless you are talking about our population-wide, no-control-group, 35-year experiment with low-fat diet recommendations, but that’s different.)

And we don’t mind generalizing conclusions to Everyone in the Whole Wide World based on data from a bunch of white health care professionals born before the atom bomb because, honestly, those are the only data we really care about.

Equating correlation and causation, over-generalizing observations, and then using these results as the basis of policy is the bread (whole wheat) and butter (substitute) of nutrition epidemiology of chronic disease (aka NECD – pronounced Southern-style as “nekked”). NECD has a long proud tradition of misinterpreting results this way, and dammit, nobody is going to take that away from us.

Early NECD researchers have in the past tried to tentatively misinterpret results by obliquely implying that observed nutritional patterns might perhaps have resulted in the disease under investigation. Wusses.

In 1990, Walter Willett and JoAnn Manson came along to show us how the pros do it. These mavericks were the ones who made bold inroads into the kind of overreaching conclusions that made NECD great. Their data come from an observational study of female registered nurses from 11 states in the US, born between 1921 and 1946, who were asked to remember and report what they ate 4 whole times between 1976 and 1984, plus remember and report what they weighed when they were 18 years old. From this dataset, which is clearly comprehensive, and this population, which is practically every female in the US, Willett, Manson and company naturally conclude that “obesity is a major cause of excess morbidity and mortality from coronary heart disease among women in the United States” (emphasis mine). None of this wimpy “associated with increased risk of” bullshooey, obesity CAUSES heart disease, they tell us, CAUSES IT!!!! BWHAAAHAAAAA!!!!!!!

It is on this foundation of intrepid willingness to misinterpret data that the science of NECD was built. This is why Walter Willett is the Big Kahuna at the Harvard School of Public Health. He has demonstrated the courage to misinterpret data in innovative and comprehensive ways, publishing articles throughout his career that indicate that even small increases in BMI—including BMI levels that are currently considered “normal”–cause chronic disease.

In 1999, in what is considered a landmark article in overstatement, one with which all NECD acolytes should familiarize themselves, he states unequivocally, in a review of observational data:

“Excess body fat is a cause of cardiovascular diseases, several important cancers, and numerous other medical conditions . . . “ (my emphasis). Hmmmm. Observed associations reported as causal? Ding!

The rest of that sentence reads: ” . . . and is a growing problem in many countries.” His data is once again gathered mostly from American white health care professionals born before the atom bomb. Generalization from specific populations to the rest of the world? Ding ding.

And what should we do with this conclusion, according to Willett? “Preventing weight gain and overweight among persons with healthy weights and avoiding further weight gain among those already overweight are important public health goals.” Using observed associations to make policy recommendations? Ding ding ding. In one fell swoop, Willett dexterously manages to use all three designated methods of overstatement and misinterpretation in the nutrition epidemiology NECD toolbox, demonstrating why he is considered by most researchers to be “the ‘father’ of nutrition epidemiology.” This man overstates and misinterprets in ways that the rest of us can only dream of doing.

Sadly, some epidemiologist have failed to follow in Willett’s footsteps. In January 2013, Katherine Flegal, an epidemiologist at the Centers for Disease Control and Prevention and the woman who first noted the remarkably rapid rise in obesity that began in the decade following the release of the 1977 Dietary Goals for Americans, published results that concluded that being overweight (or even mildly obese) is associated with a lower risk of death. At no point in her article does she suggest that overweight or obesity results in increased lifespan.

The response from Harvard? Walter Willett calls Flegal’s article ” a pile of rubbish” and insists that “no one should waste their time reading it” and rightly so. Why would anyone want to hear about “associations”? What kind of nonsense is that? Obviously Flegal lacks the professionalism it takes to make the leap from observation to causation.

But that’s okay. Willett and the Harvard Family know how to deal with this sort of thing.

“Someday, and that day appears to have come, I will call upon you to ignore the work of other scientists when their results contradict my own.”

Let’s face it, in the world of NECD, you can’t just have people like Flegal refusing to infer causation from observed results, just because they don’t want to. When that sort of thing happens, well, let’s just say, if she won’t do it, the Harvard Family will have to do it for her. And so they did.

In February 2013, Willett and company convened a Harvard Family gathering to, in their words, “elucidate inaccuracies in a recent high-profile JAMA article [i.e. Flegal's] which claimed that being overweight leads to reduced mortality” (emphasis mine). Which it didn’t–except now, voila, it does. It’s not personal, Dr. Flegal. It’s strictly science.

The Family get-together was held at the Harvard School of Public Health, a “neutral convening space” that is also ground zero for the Nurses’ Health Study I and II, the Physicians Health Study I and II, and the Health Professional Follow Up Study, three datasets that have generated many NECD articles that, unlike Flegal’s article, brilliantly illustrate the powers of misinterpreting observational data. That Flegal herself was invited, but “could not attend” tells us just how ashamed she must be of her inability to make over-reaching conclusions–or perhaps she was temporarily “incapacitated” if you know what I mean.

The webcast from the meeting show us how NECD should be done, with dazzling examples of overstatement and marvelous feats of misinterpretation.

In the world of NECD, PowerPoint arrows are a scientifically-acceptable method of establishing causation.

In her shining moment, Dr. JoAnn Manson, demonstrating that she has learned well from Willett, points to the slide above and asks: “How is it possible that overweight and obesity would cause all of these life-threatening conditions, increase their incidence, and then reduce mortality?” How indeed???

The panelists highlighted the importance of maintaining clear standards of overstatement and expressed concern that Flegal’s research could undermine future attempts of more credible researchers to misinterpret data as needed to protect the health of the public.

Because that’s what it’s all about folks: protection. Someone needs to protect the science from renegades like Flegal, and someone needs to protect the public from science.

We should be thankful that we have Willett and the Harvard Family there. They know that data like Flegal’s can only confuse the poor widdle brains of Americans. Allowing us to be exposed to such “rubbish” might lead us to the risky conclusion that perhaps overweight and mild obesity won’t cause all of us to die badly, or to the even more dangerous notion that observational data should remark only upon association, not causation. And we sure don’t want that to happen.

As Don Dr. Willett says, “It is important for people to have correct information about the relationship between health and body weight.” And when he wants us to have the correct information about the relationship between health and body weight, he’ll misinterpret it for us.

Take the science, leave the cannoli.

39 comments on “Make me some science I can’t refuse

  1. Peggy C says:

    Great post. My sentiments exactly. I do wish Dr. Willett would read it. He deserves to be taken down 20 pegs or so. I’ve disliked his brand of science for a long time.

    • Adele Hite, MPH RD says:

      Thanks so much for the kind words. I’m nutrition epidemiology in general–and Dr. Willett in particular–are in an increasingly untenable position. The rise of the information age is such that the general public no longer necessarily wants or needs to be “protected” from contradictory information about nutrition and health. What we would really like–certainly what I would like–is an authoritative source to be honest about the limitations and uncertainties in nutrition science.

  2. David Boothman says:

    I wonder what Willet thinks of Ioannidis?
    http://seedmagazine.com/content/article/dirty_little_secret/

    • Adele Hite, MPH RD says:

      I would bet, “not much.” I’ve heard one of our esteemed epidemiologist at UNC-Chapel Hill refer to Taubes, Ionnidis, and [S. Stanley] Young (of the U.S. National Institute of Statistical Sciences) as “3 poisonous peas in an ideological pod.” I wouldn’t expect Willett’s opinion to differ considerably from this, although who knows?

  3. Very much enjoyed your post, and could not agree more with many of your points. By the way, have you seen the following interview? It makes some very excellent points about the “obesity mafia” are responsible for propagating “a moral panic over fat, which has transformed the heavier than average into folk devils, to whom all sorts of social ills are ascribed.” I hate to be lumped in with this group….and I hate to think that all of us are so morally misguided, but certainly interesting food for thought! I don’t understand why, as scientists, we don’t openly embrace the questioning of key assumptions (i.e. obesity is bad) and welcome more dialogue debating different viewpoints. Isn’t that how science is advanced? If it weren’t, we’d still think the world was flat and the sun revolved around the earth!

    http://m.theatlantic.com/business/archive/2009/07/americas-moral-panic-over-obesity/22397/

    • Adele Hite, MPH RD says:

      Thanks so much for the perceptive comments & the link! I’ve been reading a lot recently about “healthism” and how responsibility for health matters has shifted away from the government and corporation to become the exclusive property of the individual. Currently, being fat not only signals poor health, but all sorts of other moral failings and personal shortcomings (because, of course, if you didn’t want to be fat, you could just “eat less and move more” and you wouldn’t be). As Charlotte Biltekoff says, we’ve made fat people into “the unhealthy other,” and when we create an “other”–based on body size, skin color, religion, nationality, etc.–not only can we blame them for the downfall of Western civilization as we know it, we can justify all sorts of inhumane behaviors and attitudes towards them because of the threat that they pose to social order.

      Lindsey, you are certainly not part of any “morally misguided” group at all (for my readers, Lindsey is a nutrition epidemiologist herself), and your comments here prove it. You welcome dialogue and discussion. You think we should question key assumptions. Whether people agree with you or not, you will change the nature of the conversation in all professional circles you encounter, and in doing so, the future of nutrition epidemiology. Hang in there. The world needs you.

  4. Fabulous post again Adele

    I have also wondered how epidemiologists come up with the advice they give, considering that it’s based purely on population data.

    A few years ago the Harvard group published a study that advised what women could do to reduce CHD risk. (Dietary fat intake and the risk of CHD in women
    NEJM 1997; 337: 1491-9) They reported that substituting 5% energy from sat fat for an equal amount of energy from unsat fat would reduce CHD by 42%.
    Or another option, that replacing 5% energy from sat fat with carbohydrate will reduce CHD by 17%.

    Taken fairly literally, this translates into saying that if a woman replaced 2 tsp butter with 1 tablespoon sugar ( approx 5% energy substitution in a 1500 cal diet), risk of CHD is reduced by 17%. Or 2 tsp olive oil for 2 tsp butter gives a massive 42% reduced risk. So simple!!!

    I was curious as to how they came up with this without ever conducting a RCT or even without proof of the role of cholesterol, saturated fat or anything else in CHD causation. None of that matters in the world of statistics apparently. The starting point is the correlation between what women ate in the cohort and who developed heart disease. As you pointed out, these correlations do not not mean cause. But if you pretend that it’s a causative relationship and you have the right computer program, you can do lots of things. I’ve been told that in this type of ‘work’ they statistically hold constant total calories, protein, and the other sources of fat, then mathematically model what the hypothetical effect of heart disease would be if you move calories up for carbs and down for saturated fat holding everything else constant. That’s apparently what they did and hence the conclusion. Nothing more than computer generated guesswork masquerading as something useful.

    I’d like to know how this stuff get published?

    Keep writing. I love your posts!

    http://www.babyboomersandbellies.com

    • Adele Hite, MPH RD says:

      Thanks for the kind words! As to how this stuff gets published? Well, it’s sort of like that old Hee Haw song “if it weren’t for bad luck, I’d have no luck at all.” If it weren’t for overstated, misrepresented science on nutrition and obesity, they’d have no science at all (except for what is coming out of biochemistry, and that’s mostly animal model and cell model info). Nice to know that there are recent calls to publishers of biomedical journals to just call the whole thing off.

      And Jennifer, speaking of Edward Archer (see link above–he now qualifies as the Johnny Depp of my nutrition world), if you only knew how the “data” are collected that get fed into these computer programs . . . The primary methodology is to use FFQs. This stands for Food Fantasy Questionnaire, where everyone (not just the fat people) pretends to remember what they ate for the past year, a system based mostly on questionable rationalizations and wishful thinking on the part of both the participants and the researchers.

      • Kenny says:

        That is a great article! Thanks for pointing it out. I had not seen it. I’d seen the journal article and wondered why it didn’t get more attention. I applaud Mr. Archer, but I have a hard time seeing how the big ship can get turned right.

      • Adele Hite, MPH RD says:

        People like you & me, scientists like Archer, many others. The trick will be righting the course without just pointing in another wrong direction :)

    • Kenny says:

      “I have also wondered how epidemiologists come up with the advice they give, considering that it’s based purely on population data.”

      I’ve wondered the same thing. Here’s the best answer I’ve found from this insightful article:
      Communicating statistics and risk
      http://www.scidev.net/global/health/practical-guide/communicating-statistics-and-risk.html

      Take away message: Be extra careful to ensure your readers understand that a general population estimate of risk, exposure or probability may not accurately describe individual situations. Also, provide the important information that explains variation in individual risk. This might include age, diet, literacy level, location, education level, income, race and ethnicity, and a host of other genetic and lifestyle factors.

      Applying population risk to individuals is just wrong and invalid. Risk factors and the whole prevention paradigm we live in has serious problems.

      • Adele Hite, MPH RD says:

        Thanks for the reference. You are so right. The whole prevention paradigm has problems. You’ve pointed out one major one, but another is that this idea of prevention of chronic disease through the application of lifestyle changes that may have little or no relationship to disease seems to have created a culture of fear-mongering and food obsessions. I think I hate that almost as much as the dippy science.

      • luanabee says:

        Yes. The fear-mongering and food obsessions are taking over. I’m starting to think the paranoia about food kills people quicker than actually eating the food they think is poison.

      • Adele Hite, MPH RD says:

        Agreed. It’s like our entire nation has turned into a bunch of fussy preschoolers. “Oh no, my food was once in the same room with some food that Walter Willett said maybe possibly might cause some disease somehow.”

        Sometimes I feel like starting my own movement called “Just eat the damn food.”

  5. dulcimerpete says:

    Well played, Adele! Most enjoyable.

    Perhaps this is the quality of research we need to really advance agriculture … ?
    :-)

    “If the animal is overly finished (fat), it didn’t exercse enought … “? :-)

    • Adele Hite, MPH RD says:

      Thanks Peter! An honor, coming from you.

      But you know, those cows would not be so fat if they weren’t such gluttons in the first place, right?

      • dulcimerpete says:

        Yeah, and we *all* know that you are what you eat … Wait, and cow is beef. They eat grass … Hmmmm.

        Okay, but if you put a basset hound on a treadmill you can turn it into a greyhound, right?

      • Adele Hite, MPH RD says:

        Vision of basset hound on treadmill, tripping over its own ears :)

  6. Love this article, can’t help but think The Jaminet’s should also read it basing the entire health of the world on about 150 people living on a blissful tropical island.

    • Adele Hite, MPH RD says:

      Thanks for the nice words. I actually like the Jaminet’s Perfect Health Diet, although I think it could have been named The Very Reasonable Approach to Diet and that would have been, perhaps, more accurate. Essential nutrition is essential; past that, no diet is “perfect” (assuming you exclude a diet of red wine, dark chocolate Reese’s cups, and Thai food). Some diets are better than others at providing essential nutrition without providing a lot of unessential nutrition, but I’m not convinced that we can draw any firm conclusions about diet-health relationships based on NECD data. If your personal characteristics are a close match to the surveyed population, maybe you can glean a few insights. Otherwise, fuggedaboutit.

    • Charlie says:

      I am with you Danny to me he is just like those vegan doctors using scare tactics with a few anecdotal “evidence”, very irrelevant rat research and isolated tribes to sell his “perfect” diet. Just the title should be enough to make anyone weary.

      I love your blog Adele and your sense of humor. Your writing style make my day every time.

      • Adele Hite, MPH RD says:

        Thanks so much for the kind words. I am the funniest person I know.

        I agree with the weariness-inducing title. However, on their behalf, I want to say that the Jaminets have also signaled (at least as far as I am concerned) a commitment to addressing changes in the overall system in a way that may benefit all, even those who do not read/buy their book or follow their plan. There are plenty of other folks in the alternative nutrition world (folks in the paleoverse, I’m looking at you) whose commitment to changing the world for the better is limited to those who will pay for those services.

        My biggest concern as I watch this story unfold is that there is a very real tendency to want to replace the old (admittedly sucky) system, with a “new improved” system that folks in the alternative nutrition movements like better. I think this is BS. Meet the new boss, same as the old boss. What is needed is a different way of understanding, communicating (and probably studying) food-health relationships–not replacing one system of top-down, one-size-fits-all guidance with another. Nutritional anarchy? Why not? To quote Noam Chomsky, who I may or may not agree with regarding other contexts: “the burden of proof is always on those who argue that authority and domination [i.e. rule-bound approaches to food-health] are necessary. They have to demonstrate, with powerful argument, that that conclusion is correct. If they cannot, then the institutions they defend should be considered illegitimate. How one should react to illegitimate authority depends on circumstances and conditions: there are no formulas.”

        I don’t know what this new future should look like, but by golly it should look DIFFERENT.

  7. Jay Wortman MD says:

    Hey, great post. Really enjoyed the humour (Canadian spelling). Gary Taubes has been on their case for awhile, too. Keep up the good work!

    • Adele Hite, MPH RD says:

      Thanks! That means a lot coming from one of the original rockstars (American spelling) of the paradigm shift.

      For the uninitiated, Dr. Wortman is the man behind “My Big Fat Diet.” If you are not familiar with it, check it out (literally & figuratively). Well worth the watch. Everybody has to do a shot of tequila when Dr. Eric Westman comes on screen in an ugly sweater.

      My husband’s a musician & this is like having Bruce Springsteen come to a gig & say “Nice tone.” :)

  8. Christine says:

    I just found your blog today, and am SO glad I did! Fantastic stuff!

  9. Todd B says:

    Thank you, a lesson in science for me, along with a reminder to have a good laugh.

  10. Elizabeth says:

    And to think, I was just starting to feel like a broken record for repeating the “correlation does not imply causation” line to everyone I know. I started to feel like it’s just something everyone knows at this point; surely we don’t need to keep belaboring the issue! Apparently I was wrong…(which, OK, I probably should have guessed from the continuous flow of “red meat is associated with _____________” studies that keep landing in my inbox with accompanying notes of “concern” about my diet.)

    Your reporting is awesome and thank you for injecting that little bit of humor into my general nutritional grouchiness.

    • Adele Hite, MPH RD says:

      Oy – I think I have a permanent case of nutritional grouchiness. Thanks for the nice words. Sometimes it’s better to laugh than gnash my teeth.

  11. Kenny says:

    Uh-oh. I did not find this goofy at all. :-)

  12. Kenny says:

    Awesome. Nice to have you back. This is my second favorite of your posts, right after NaCl Debacle Part 2 which is one of the best nutrition posts I’ve ever read anywhere.

    Great to see you calling out Walter Willett. He needs it and deserves it, but I doubt he’ll hear it in his ivory tower. He’s an expert and experts know *everything* right?

    I was just thinking this morning before reading this about how all nutrition and health reporting always says something like “linked to” or “associated with” or “may cause”. Why don’t we *know* anything? How do we get out of the NECD frame?

    • Adele Hite, MPH RD says:

      We have to start by calling out all of those things that those researchers identified as misrepresentations of NECD data–because that’s what they are.

      This is why I’ve taken up communication & rhetoric, because I think what may be needed is not to teach people how to understand the science (such as it is), but to help people understand how the science is being used. I don’t know that we need to critique science as much as we need to critique scientists and their messages. Science is going to go about doing what it does, but we sure can learn to be critical consumers (emphasis on “critical”) of it.

      One of my favorite Facebutt friends, Steward McLean, asked me how to wade through all of the arguments about nutrition on the internet. Here’s a paraphrase of my response:

      First, pay attention to context: What are the motives, interests, and expertise of the writer? You have to evaluate this in order to decide to go further. When I read a young white male with no science background and a significant personal investment in “fitness” tell me that CICO rules, I’m done (although others may wish to keep reading). This does not mean he’s wrong, but that he simply cannot place the matter that he is talking about in a larger clinical or scientific context, which is what I am interested in, and which is the next step.

      Second–does the writer acknowledge and explore contrary points of view in a balanced manner? If not, be very wary. Every single journal article on PubMed is published within a larger academic “argument.” It works like this: a theory is a “story” that tries to explain reality. The theory helps to form hypotheses that get tested. The bits and pieces accumulate as evidence, some for, some against, the posed hypotheses–and the story either continues to explain reality pretty well or a new story is needed.

      If the story does not explain reality very well, and the holders of the theory explain the discrepancies by pointing their fingers at the humans whose stories do not fit the explanation, saying “This person is lying or deluded because their story does not fit our theory”–this is a problem. For me, denying someone else’s lived experience can NEVER be the first explanation (which isn’t to say that people never lie or are never deluded, but that as a scientist, when confronted with evidence that does not fit your story, you must consider alternatives to your story BEFORE you consider the evidence to be “cooked”–that’s just how science works).

      If the response to competing theories is that those other scientists are fringe scientists, in the pocket of industry, unqualified, etc. etc., you are dealing with a holder of a theory who doesn’t want to address the real issue, which is that the theory no longer adequately explains reality.

      The truth is that the story is always unfolding. Which is point 3: If someone seems to indicate that the “science is settled,”–especially when they use data from 15-20 years ago to say it–then you know they are wrong. Period. The science is never settled. We are continually revising our stories to better fit reality.

      I’m sure there’s lots more ways to ask questions of what we read (readers? got more suggestions?), but that’s a start in how we need to deal with NECDness.

  13. You’re back! What a great way to start the day…with your acerbic wit, uncensored attitude, and straight-shooting. You have been missed, Adele! I wish the average American had the time, desire, and BRAIN POWER to read and understand posts like this. It would give them a lot of insight into just how insanely wrong the headlines and sound bites (bytes, these days?) they hear are.

    Willett kills me. I have no doubt he’s a brilliant man. Imagine how much *more* brilliant he’d be with an open mind, rather than one that’s sealed shut, lined with lead, and then buried in concrete for good measure, lest anything new and challenging to his deeply held beliefs seep in through the cracks.

  14. Ray Weer says:

    Excellent Response.
    The Science spin is becoming deeply rooted in the Policy to protect the public Spin.

    • Adele Hite, MPH RD says:

      And I for one feel in deep need of protection from science. Science is Very Very Scary. If it weren’t for science, you know, the earth would still be flat. And we wouldn’t have global warming because, duh, no globe?

  15. BawdyWench says:

    Hmmmm. It appears your keyboard was stuck in SM (sarcastic mode) for your posting today.

    Bravo!

    • Adele Hite, MPH RD says:

      Thanks! Hubs gave me the “are you really going to be this goofy?” review, but I think WW & co have earned this one.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s