After nearly two years of observing the chocolate news cycle, I’ve come to expect that scientific studies linking chocolate with health, wealth, and/or happiness will be widely cited in the media, most often with great enthusiasm and little skepticism. This past week proved no exception, with a study that made for catchy headlines like “Eat chocolate, win the Nobel Prize?,” “Secret to Winning a Nobel Prize? Eat More Chocolate,” and “Chocolate Consumption Directly Related To Nobel Prize Wins, Says New Study.”
These news pieces spread like wildfire through the chocolate world as chocolate companies and afficionados Facebooked and tweeted them with gusto. I often enjoy such stories myself — they can be fun and, hey, any evidence quantifying chocolate’s many virtues is welcome, right? There is a problem with this study’s viral path through the media and the chocolate world, though: its data does not support its claims.
I find it disconcerting that this kind of reporting on this kind of study so often goes unchecked by any broadly available dissenting response. Fortunately, I live with Trevor Bass, an expert on data and its many abuses. His take on the study is below.
I managed to score last week’s issue of absurdist scientific humor publication The New England Journal of Medicine, which includes a hilarious note on “Chocolate Consumption, Cognitive Function, and Nobel Laureates.” As I continued reading the issue and failed to see the humor in such knee-slappers as “Fibulin-3 as a Blood and Effusion Biomarker for Pleural Mesothelioma” and “Evaluation and Initial Treatment of Supraventricular Tachycardia,” I quickly came to the realization that NEJM is not intended as a satirical magazine. It is, in fact, among the world’s most prestigious peer-reviewed medical journals.
Inspired by recent findings that compounds in chocolate improve cognitive function, cardiologist Franz Messerli’s note questions whether there is “a correlation between between a country’s level of chocolate consumption and its population’s cognitive function.” Using the number of Nobel laureates per capita as a “surrogate end point” for a population’s percentage of wicked smahties, the study finds a “surprisingly powerful correlation between chocolate intake and the number of Nobel laureates in various countries” (23 in all). While he concedes that correlation does not imply causation, Messerli writes “since chocolate consumption has been documented to improve cognitive function, it seems most likely that in a dose-dependent way, chocolate intake provides the abundant fertile ground needed for the sprouting of Nobel laureates.”
Reportedly, when contacted by the Associated Press, “Sven Lidin, the chairman of the Nobel chemistry prize committee, had not seen the study but was giggling so much when told of it that he could barely comment.”
Indeed, one doesn’t require a doctorate in statistics to find serious flaws in the study. It was clearly intended as tongue-in-cheek to some degree by Messerli (who has according to NPR published around 800 peer reviewed papers) and NEJM (which also according to NPR has a history of occasional tomfoolery), though to what degree I can’t quite ascertain. Scientists’ riotous senses of humor aside, I would have expected dozens of more subtly troubling logical leaps to be followed by winky faces.
Given the absence of sufficient semicolon close parentheses, I worry about the misinformation generated by this study. The media has run wild with it in the past week, citing it widely with often far too little skepticism – an excellent example of a phenomenon I’ve recently started calling quantitative exceptionalism. A comment cardiologist Sanjay Kaul provided to CardioBrief sums up the dangers well: “This article highlights, with a touch of whimsy, caveats that challenge the interpretation of findings of observational studies. From the use of surrogate endpoints (based on biological plausibility and the results of preclinical studies) to the distinction between correlation and causation, confounding (whether the effect size is too large to be explained away by confounding), and the hypothesis-generating nature of the inferential process. Careful consideration of these issues is likely to help navigate through the labyrinth of misinformation and disinformation these types of studies are particularly prone to generating.”
Messerli is no stranger to the harmful effects scientific misinformation can have. Last year, he was quoted in a Wall Street Journal article about mistakes in scientific studies as one of a large number of doctors who (understandably) fell prey to an erroneous paper in the Lancet, another highly respected medical journal. Hundreds of thousands of patients were affected, and Messerli argued that the Lancet had a “moral obligation” to withdraw the paper. Granted, doctors around the world aren’t likely to begin writing prescriptions for dangerously high doses of chocolate based on Messerli’s note in NEJM any time soon, but the difference is one of magnitude rather than direction.
A few examples of things I found more troubling slash hilarious about Messerli’s note:
- The use of the number of Nobel laureates as a surrogate endpoint for cognitive function is…how do I say it?…strange. In fact, the number of Nobel laureates probably has a lot more to do with a country’s wealth. As Nobel laureate Eric Cornell told Reuters, “National chocolate consumption is correlated with a country’s wealth and high-quality research is correlated with a country’s wealth…therefore chocolate is going to be correlated with high-quality research, but there is no causal connection there.”
- Messerli writes: “Obviously, these findings are hypothesis-generating only and will have to be tested in a prospective, randomized trial.” Considering that countries in the study have at most a few Nobel laureates per million population, imagine the enormous expense, financial and otherwise, of such a trial. A properly controlled study would deprive millions of the joys of chocolate.
- While the note warns in multiple places that causation has not been proven, its language repeatedly justifies causation based on tenuous logic. For example, Messerli writes that “it would take about 0.4 kg of chocolate per capita per year to increase the number of Nobel laureates in a given country by 1” and even refers to a “minimally effective chocolate dose.” He justifies such remarks only with references to prior studies linking cacao consumption and cognitive function, which are many leaps-of-faith removed from these conclusions.
- Messerli writes but has no justification for this statement: “it is difficult to identify a plausible common denominator that could possibly drive both chocolate consumption and the number of Nobel laureates over many years. Differences in socioeconomic status from country to country and geographic and climatic factors may play some role, but they fall short of fully explaining the close correlation observed.”
- The study appears to use chocolate rather than flavanol or cacao consumption figures, and the types of chocolate consumed in the studied countries varies significantly. Another gem from Cornell’s interview in Reuters: “It’s one thing if you want like a medicine or chemistry Nobel Prize, ok, but if you want a physics Nobel Prize it pretty much has got to be dark chocolate.” I wonder how considering less economically correlated forms of flavanols like green tea would change the results.
Is Messerli deserving of an Ig Nobel Prize for this gem? According to the Annals of Improbable Research, which awards the prizes annually: “Every Ig Nobel Prize winner has done something that first makes people LAUGH, then makes them THINK.”
Regardless, I’m left wondering what foods predispose you to becoming an Ig Nobel laureate. Foods that leave a funny taste in your mouth? Personally, I’m going to stick with salad.
Trevor Bass is a quantrepreneur and data scientist who lives in Cambridge, MA. Visit his professional site here to learn more about his work.
Trevor Bass is a helper at Bittersweet Notes, and deserves the blame for technology and humor related failures. Follow him on Twitter.