Tuesday, January 12, 2016

One "Johannes" Bohannon describes
how he, "fooled millions into thinking chocolate helps weight
loss". He fibs only a little, helped as he was by an army of
scientifically illiterate -- and incredibly lazy --
"journalists".

I am Johannes Bohannon, Ph.D. Well, actually
my name is John, and I'm a journalist. I do have a Ph.D., but it's in
the molecular biology of bacteria, not
humans. The Institute of Diet
and Health? That's nothing more than a website.

Other
than those fibs, the study was 100 percent authentic. My colleagues
and I recruited actual human subjects in Germany. We ran an actual
clinical trial, with subjects randomly assigned to different diet
regimes. And the statistically significant benefits of chocolate that
we reported are based on the actual data. It was, in fact, a fairly
typical study for the field of diet research. Which is to say: It was
terrible science. The results are meaningless, and the health claims
that the media blasted out to millions of people around the world are
utterly unfounded.

Among his tricks was the misuse of
statistics:

... We didn't know exactly what would pan out
-- the headline could have been that chocolate improves sleep or
lowers blood pressure -- but we knew our chances of getting at least
one "statistically significant" result were pretty
good.

Whenever you hear that phrase, it means that some
result has a small p value. The letter p seems to have
totemic power, but it's just a way to gauge the signal-to-noise ratio
in the data. The conventional cutoff for being "significant" is 0.05,
which means that there is just a 5 percent chance that your result is
a random fluctuation. The more lottery tickets, the better your
chances of getting a false positive. So how many tickets do you need
to buy?

...

With our 18 measurements, we had a
60% chance of getting some "significant" result with p <
0.05. (The measurements weren't independent, so it could be even
higher.) The game was stacked in our favor.

It's called
p-hacking...

There has been quite a bit of soul-searching
in the scientific community regarding statistical methods lately. I am
not sure if Bohannon's work helped lead to -- or was inspired by --
this, but I am glad to see that there is at least some backlash
against the misuse of statistics in reporting about science.