Tuesday, 13 November 2012

Flaky chocolate and the New England Journal of Medicine

Early in October a weird story hit the media: a nation’s chocolate consumption is predictive of its number of Nobel prize-winners, after correcting for population size. This is the kind of kooky statistic that journalists love, and the story made a splash. But was it serious? Most academics initially assumed not. The source of the story was the New England Journal of Medicine, an august publication with stringent standards, which triages a high proportion of submissions that don’t get sent out for review. (And don't try asking for an explanation of why you’ve been triaged). It seemed unlikely that a journal with such exacting standards would give space to a lightweight piece on chocolate. So the first thought was that the piece had been published to make a point about the dangers of assuming causation from correlation, or the inaccuracies that can result when a geographical region is used as the unit of analysis. But reading the article more carefully gave one pause. It did have a somewhat jocular tone. Yet if this was intended as a cautionary tale, we might have expected it to be accompanied by some serious discussion of the methodological and interpretive problems with this kind of analysis. Instead, beneficial effects of dietary flavanols was presented as the most plausible explanation of the findings.

The author, cardiologist Franz Messerli, did discuss the possibility of a non-causal explanation for the findings, only to dismiss it. He stated “as to a third hypothesis, it is difficult to identify a plausible common denominator that could possibly drive both chocolate consumption and the number of Nobel laureates over many years. Differences in socioeconomic status from country to country and geographic and climatic factors may play some role, but they fall short of fully explaining the close correlation observed.” And how do we know “they fall short?” Well, because the author, Dr Messerli, says so.

As is often the case, the blogosphere did a better job of critiquing the paper than the journal editors and reviewers (see, for instance, here and here). The failure to consider seriously the role of a third explanatory variable was widely commented on, but, as far as I am aware, nobody actually did the analysis that Messerli should have done. I therefore thought I'd give it a go. Messerli explained where he’d got his data from – a chocolatier’s website and Wikipedia – so it was fairly straightforward to reproduce them (with some minor differences due to missing data from one chocolate website that's gone offline). Wikipedia helpfully also provided data on gross domestic product (GDP) per head for different nations, and it was easy to find another site with data on proportion of GDP spend on education (except China, which has figures here). So I re-ran the analysis, computing the partial correlation between chocolate consumption and Nobel prizes after adjusting for spend per head on education. When education spend was partialled out, the correlation dropped from .73 to .41, just falling short of statistical significance.

Since Nobel laureates typically are awarded their prizes only after a long period of achievement, a more convincing test of the association would be based on data on both chocolate consumption and education spend from a few decades ago. I’ve got better things to do than to dig out the figures, but I suggest that Dr Messerli might find this a useful exercise.

Another point to note is that the mechanism proposed by Dr Messerli involves an impact of improved cardiovascular fitness on cognitive function. The number of Nobel laureates is not the measure one would pick if setting out to test this hypothesis. The topic of national differences in ability is a contentious and murky one, but it seemed worth looking at such data as are available on the web to see what the chocolate association looks like when a more direct measure is used. For the same 22 countries, the correlation between chocolate consumption and estimated average cognitive ability is nonsignificant at .24, falling to .13 when education spend is partialled out.

I did write a letter to the New England Journal of Medicine reporting the first of my analyses (all there was room for: they allow you 175 words), but, as expected, they weren't interested. "I am sorry that we will not be able to print your recent letter to the editor
regarding the Messerli article of 18-Oct-2012." they wrote. "The space available for
correspondence is very limited, and we must use our judgment to present a
representative selection of the material received."

It took me all of 45 minutes to extract the data and run these analyses. So why didn’t Dr Messerli do this? And why did the NEJM editor allow him to get away with asserting that third variables “fall short” when it’s so easy to check it out? Could it be that in our celebrity-obsessed world, the journal editors think that there’s no such thing as bad publicity?

10 comments:

Arg, we also just got a rejection letter too. Here's our response where we find a correlation between chocolate consumption and serial killers:http://replicatedtypo.com/chocolate-consumption-traffic-accidents-and-serial-killers/5718.html

Wait a second, this is actually a published paper in a serious journal? I thought it was a joke being passed around on facebook, as a reminder that "correlation is not causation".In itself, this parody of a study might be funny, but how can a serious journal publish this? Some editor must have thought he was being really funny...

Sean: I'm hoping that the rejection of our letters just means that they had a shedload of correspondence and that they will publish other letters making similar points. At least for my letter, everything I was saying was pretty blindingly obvious. Of course, if that is the case, it would have been nice to be told so, instead of getting a generic impersonal response. But, in my experience, NEJM doesn't go in for individualised replies to authors.Zinemin - well, the other thing that could happen here is that NEJM publish a comment to say that it was all a bit of fun and they didn't intend it to be taken seriously. I somehow doubt that they'll do that. But they certainly have confused a lot of people by publishing something quite so flaky without any clear signal that they are aware it's flaky. If it had been 1st April,that would have made sense.

As this article clearly demonstrates, you need to engage your own critical faculties when reading articles even from the top medical journals. You also need to engage your sense of humour.

The article is obviously a joke, and is actually very funny in parts. To quote (on possible reverse causation) "..That receiving the Nobel Prize would in itself increase chocolate intake countrywide seems unlikely, although perhaps celebratory events associated with this unique honor may trigger a widespread but most likely transient increase."

I'm not sure what's funnier, the article itself or the fact that anybody actually bothered to write in to try correct the science.

George: Glad to have caused you merriment and I'll be v pleased if you are right. As I explained, like most people I had my suspicions this was not serious, but the problem is that the message was confusing - one of the main comments in media and blogs was "is this for real?" Messerli has also given mixed messages in interviews - as if guarding himself against criticism while wanting people to pick up on the idea. If we were not supposed to take it at all seriously, it would be nice if NEJM would come out and say so, now that the media storm has passed. Two questions: Does anyone know of any previous example of them publishing a joke? And did they have a press release for this piece?

We need an open peer review system in academics to eliminate bad editor decisions. It's horrible that the editor seems to want to cover up their mistake. And it should not be so difficult as it currently is to correct this mistake as a scientific community.

Pierre André and I also sent a comment the 25th of october to the NEJM. Our response article was 2000 words. The NEJM told us that letters may number up to 175 words in length. Hence, we wrote a shorter document of less than 175 words and sent to the NEJM the short version the 29 of October. We were also rejected at the end because of the "lack of space". The paper of 2000 words can be found at https://9d558120-a-62cb3a1a-s-sites.googlegroups.com/site/fabgouret/research/messerlinote.pdf?attachauth=ANoY7cq4gh6p7OHjV34w8EkbTG6tY-SOhNsRH84knWAj8o5ptojxEXkAce226Kf_LdehylieZ5UCcBmozG6Vq2IEroOhfoCtXoytEYLZlQXw-UgRR1kP_FUws0WE_fO5-u_k4h43mly-CLCQCDmut88ctCzm0_LprO0H8YG2IHpjBATiqEPr3hmCcpjiRfjfbR-UeFBWKfJNara2TDyVW1e8z9wL4k-85KJJWc-FJD7J-mwXB2RxjF4%3D&attredirects=0