This was weak observational data. That’s just the start of our story, but you should know exactly what the researchers did. They sent an email inviting students to fill out an online survey, and 219 agreed.

The survey is still online (in all its time-consuming glory, I just clicked answers randomly to see the next question). It asks about caffeine intake in vast detail, and then uses one scale to measure how prone you are to feeling persecuted, and uses another, the “Launay-Slade Hallucination Scale“, 16 questions designed to measure “predisposition to hallucination-like experiences”.

Some of these questions are about having hallucinations and seeing ghosts, but some really are a very long way from there. Heavy coffee drinkers could have got higher scores on this scale by responding affirmatively to statements like: “No matter how hard I try to concentrate on my work, unrelated thoughts always creep into my mind”; “Sometimes a passing thought will seem so real that it frightens me”; or “Sometimes my thoughts seem as real as actual events in my life”. That’s not seeing ghosts or hearing voices.

And of course, this was weak observational data, and there could have been alternative explanations for the observed correlation between caffeine intake and very slightly higher LSHS scores. Maybe some students who drink a lot of coffee are also sleep deprived, and marginally more prone to hallucinations because of that. Maybe they are drinking coffee to help them get over last night’s massive marijuana hangover.

Maybe the kinds of people who take drugs instrumentally to have fun and distort their perceptions also take drugs like caffeine instrumentally to stay alert. You can think of more, I’m sure. The researchers were keen to point out this shortcoming in their paper. The Express and many others didn’t seem to care.

Then if you read the academic paper you find that the associations reported are weak. For the benefit of those who understand “regression” (and it makes anybody’s head hurt), 18% of the variance in the LSHS score is explained by gender, age and stress. When you add in caffeine to those three things, 21% of the variance in the LSHS score is explained: only an extra 3%, so caffeine adds very little. The finding is statistically significant, as the researchers point out, so its unlikely to be due to chance, but that doesn’t affect the fact that it’s still weak, it explains only a tiny amount of the overall variance in scores on the “predisposed-to-hallucinations” scale.

Lastly, most newspapers reported a rather dramatic claim, that 7 cups of coffee a day is associated with a three times higher prevalence of hallucinations. This figure does not appear anywhere in the paper. It seems to be an ad hoc analysis done afterwards by the researchers, and put into the press release, so you cannot tell you how they did it, or whether they controlled appropriately for problems in the data, like something called “multiple comparisons“.

Here is the problem. Apparently this 3 times greater risk is for the top 10% of caffeine consumers, compared with the bottom 10%. They say that heavy caffeine drinkers were three times more likely to have answered affirmatively to just one LSHS question: “In the past, I have had the experience of hearing a person’s voice and then found that noone was there”.

Now this poses massive problems. Imagine that I am stood facing a barn, holding a machine gun, blindfolded, firing off shots whilst swinging my whole body from side to side and laughing maniacally. I then walk up to the barn, find three bulletholes which happen to be very close together, and draw a target around them, claiming I am an excellent shot.

You can easily find patterns in your data once it’s collected. Why choose 10% as your cut-off? Why not the top and bottom quarters? Maybe they have accounted for this problem. You don’t know, I don’t know, they say they have, to me, in emails, but it wasn’t in the paper, and we can’t all see the details. I don’t think that’s satisfactory for a headline finding, and the first claim of a press release.

And there is another problem: putting a finding in the press release but not into the paper is a subversion of the peer review process. People will read this coverage, they will be scared, and they will change their behaviour. But the researchers’ key reported claim, with massive popular impact, was never peer reviewed, and crucially the technical details behind it are not in the public domain.

++++++++++++++++++++++++++++++++++++++++++
If you like what I do, and you want me to do more, you can: buy my books Bad Science and Bad Pharma, give them to your friends, put them on your reading list, employ me to do a talk, or tweet this article to your friends. Thanks!
++++++++++++++++++++++++++++++++++++++++++

58 Responses

jsymes said,

Amanda/emen – Usages such as “I am stood” could more usefully be lumped into the general category “dialect idiosyncrasies”, along with “I were sat there” and, since we’re coming up to Burns’ Night, “Wee, sleekit, cow’rin tim’rous beastie”. The “I am stood”-type usage derives mostly from the north (I could go into all sorts of linguistic drivel here about the Danelaw and its effect on British dialects, but life’s too short). Such usage has gradually spread south over the centuries as people gravitated south, and so is now widespread, though as Fowler would no doubt say, avoid for formal speech or writing – but unsurprisingly, he makes no ruling on whether websites count as formal or informal. It’s Dr Goldacre’s site: let him set the tone.

No, what I think reveals Dr Goldacre to be a true polymath is his use of “unblameless”, which would suggest a knowledge of the rules of Newspeak, as set out by Orwell in the Appendix to Nineteen Eighty-Four. Most people – including one or two here – obviously skip this bit, but as a pointer to how politicians, health pressure groups such as nutritionists and homeopaths, media commentators etc twist and degrade language to suit their purposes and hoodwink us, it is unbeatable. Perhaps unknowingly, but certainly ironically, Dr Goldacre gave us an excellent example of these charlatans’ “goodthinkfulness”.

hatter said,

On falling and parachutes. There does seem to be some evidence that decelerating a rapidly moving human body by impacting it with a solid object has negative side-effects. And some evidence that these side-effects are proportional to the rate of movement. But are there carefully controlled studies to show that parachutes are better than placebo? Probably not.

On cannabis and hangovers. I can imagine that cannabis users objecting to claims that the drug causes a hangover may be due to the word being primarily associated with alcohol and the specific after-effects of consumption of fairly large quantities of that particular drug. Similarly most alcohol users will probably strenuously object to suggestions that even smaller quantities of alcohol have hangover effects (they certainly tend to get bent out of shape when they’re informed that they too are recreational drug users – seems quite common to pretend otherwise). There may also be other factors at play like for instance do users stay up later, and hence get less sleep, when they’re using their chosen drug? I would also surmise that the arbitrary persecution to which cannabis users are subjected tends to make them sensitive to any, even valid, criticism of their chosen drug or references to potential dangers thereof.

Back to caffeine. According to The Independent a cup of instant coffee has 45mg and tea has 40mg. Either they’re making very watery coffee or very strong tea. The former I think. Then again every article I have looked at has different numbers reported. With caffeine I have twice in my life suffered a reversal of tolerance and currently cannot consume even a single mug of filter coffee without suffering from caffeine overload. I have so far had no luck re-developing a tolerance for it. Annoying as I really like coffee.

mikewhit said,

To combine the Finland and coffee/guarana points, plus an ‘anecdotal effect’, our subsequently-diagnosed ADHD son was noticeably well-behaved during our drive through central Finland for the couple of hours after he drank a can of a local drink called ‘Battery Acid’ among which ingredients were caffeine and guarana … wonder if the DM would propose its use instead of methylphenidate ?