Sabermetric Research

Tuesday, October 06, 2015

Does vaping induce teenagers to become smokers?

Do electronic cigarettes lead users into smoking real cigarettes? In other
words, is vaping a "gateway activity" to smoking?

A recent study says that, yes, vapers are indeed more likely to become
smokers than non-vapers are. In fact, they're *four times* as likely to do
so.

The study looked at a sample of young people aged 16 to 26 who said they
didn't intend to become smokers. When they caught up with them a year later,
only 9.6 percent of the non-vapers had smoked in the past year. But 37.5 percent
of the vapers had!

Except ... here's an article from FiveThirtyEight that suggests that, no,
this is NOT strong evidence that vaping leads to smoking. Why not? Because the
sample size was very small. The vaping group comprised only 16 participants,
compared to 678 for the control group.

Vaping: 6/ 16 (37.5%)

Non-Vaping: 65/678 (9.6%)

FiveThirtyEight says,

"Voila, six out of 16 makes 37.5 percent — it’s a big number that comes
from a small number, which makes it a dubious one.

So because six people started smoking, news reports alleged that e-cigs
were a gateway to analog cigs."

Well, I have some sympathy for that argument, but ... just a little.
Statistical significance does adjust for sample size, so, in effect, the data
does actually say that the sample size issue isn't that big a deal. To argue
that 16 people isn't enough, you need something other than a "gut feel"
argument. For instance, you could hypothesize that 16 vapers out of 694 people
is a lower incidence of vaping than in the general population, and, therefore,
you're getting only "out of the closet vapers" self-identifying, which makes the
16 vapers unrepresentative.

But, the article doesn't make any arguments like that.

------

The FiveThirtyEight story tries to make the case that the study, and the
press release describing it, are biased, because they're too overconfident about
a sample that's too small to draw any conclusions.

I don't agree with that, but I DO agree that there's bias. A much, much
worse bias, one that's obvious when you think about it, but one that has nothing
to do with the actual statistical techniques.

What's the actual problem? It's that the whole premise is mistaken.
Comparing vapers to non-vapers is NOT evidence for whether vaping entices young
people into smoking. Not at all. Even with a huge sample size. Even if you
actually counted everyone in the world, and it turned out that vapers were five
times as likely to become smokers as non-vapers, that would NOT imply that
vaping leads to smoking, and it would NOT imply that banning vaping would
"protect our youth" from the dangers of smoking real cigarettes.

It could even be that, depsite vapers being five times as likely to take up
smoking, vaping actually *reduces* the incidence of smoking.

How? Well, suppose that vapers and smokers are the same "types" of people,
those who want to send a signal that they're risk-takers and nonconformists.
Before, they all took up smoking. Now, some take up smoking and some vaping.
Sure, some of the vapers become smokers later. But, overall, you could easily
have fewer smokers than before you started.

"What do I think? A vaper is in denial. It’s not the vaping itself that
causes you to become a smoker, but simply that a smoker is a closet-vaper.

"This is likely true of most vices. It won’t act as a gateway, but simply
that you will try it because you were going to try to harder stuff anyway. Even
if you didn’t want to admit it.

" ... There’s a dozen ways to get from Chinatown to Times Square. Manhattan
then adds a direct bus line that goes up Broadway. Does that bus “cause” people
to go from Chinatown to Times Square? Or, does it simply become a stepping stone
that they would have otherwise bypassed?

"Basically, do the same number of people end up going Chinatown to Times
Square?

"Do the same number of people end up smoking the real stuff anyway? All
vaping is doing is redirecting the flow of people?"

------

If that sounds too abstract in words, it'll become crystal clear if we just
change the context, but leave the numbers and arguments the same.

"Ignore The Headlines: We Don’t Know If Suicide Hotlines Lead Kids to Kill
Themselves.

"After a year, 37.5 percent of those who had called a Suicide Hotline had
gone on to end their own lives. That's a big percentage when you consider that
the suicide rate was only 9.6 percent among respondents who hadn’t called the
hotline.

"Our study identified a longitudinal association between suicide hotline
use and progression to actual suicide, among adolescents and young adults.
Especially considering the rapid increase and promotion of distress lines, these
findings support regulations to limit suicide hotlines and decrease their
appeal."

It's exactly the same thing! Really. I edited a bit, but most of the words
come exactly from the original articles on vaping.

Now, you could argue: well, it's not REALLY the same thing. We know that
suicide hotlines decrease suicide, but, come on, can you really believe that
vaping reduces smoking?

To which I answer: absolutely. I *do* believe that vaping reduces smoking.
If you believe differently, then, study the issue! This particular study doesn't
provide evidence either way.

And, more importantly: "can you really believe?" is not science, no matter
how incredulously you say it.

------

Logically and statistically, the relevant number is NOT what percentage of
vapers (hotline callers) go on to smoke (commit suicide). The relevant number
is, actually, how many people would go on to smoke (commit suicide) if vaping
(suicide hotlines) did not exist.

Why is this not as obvious in the vaping case as in the hotline case?
Because of bias against vaping. No other reason. The researchers and doctors
start out with the prejudice that vaping is a bad thing, and, because of
confirmation bias, interpret the result as, obviously, supporting their view. It
seems so obvious that they don't even consider any other possibility.

I bet it's not just vaping and suicide hotlines. I suspect that we'd be
eager to accept the "A leads to more bad things than non-A" if we're against A,
but we see it's obviously a ridiculous argument if we approve of A. Here are a
few I thought of:

"37% of teenagers who play hockey went on to commit assault, as compared to
only 9% who didn't play hockey. Therefore, hockey is a gateway to violence, and
we need to limit access to hockey and make it less appealing to
adolescents."

"37% of teenagers who use meth go on to commit crimes, as opposed to only
9% who didn't use meth. Therefore, meth is a gateway to criminal behavior, and
we need to limit access to meth and make it less appealing to
adolescents."

"37% of patients who get chemotherapy go on to die of cancer, as opposed to
only 9% of patients who don't get chemo. Therefore, chemotherapy leads to
cancer, and we need to limit access to chemo and make it less appealing to
oncologists."

"37% of men who harass women at work go on to commit at least one sexual
assault in the next ten years. This shows that harassment is a precursor to
violence, and we need to take steps to reduce it in society."

If you're like me, in the cases of "bad" precursors -- meth and harassment
and vaping -- the arguments seem to make sense. But, in the cases of "good"
precursors -- hockey and chemotherapy and suicide prevention lines -- the
conclusions seem obviously, laughably, wrong.

In that piece, they give several reasons for why so many scientific
findings turn out to be false. They mention poor peer review, "p-hacking"
results, and failure to self-correct.

Those may all be happening, but, in my opinion, it's much less complicated
than that.

It's just bad logic. It's not as obvious as the bad logic in this case,
but, a lot of the time, it's just errors in statistical reasoning that have
nothing to do with confidence intervals or methodology or formal statistics.
It's a misunderstanding of what a number really means, or a reversal of cause
and effect, or an "evidence of absence" fallacy, or ... well, lots of other
simple logical errors, like this one.

Regular readers of this blog should not be too surprised by my diagnosis
here: most of the papers I've critiqued here suffer from that kind of error, the
kind that's obvious only after you catch it.

FiveThirtyEight writes:

"Science is hard — really f*cking hard."

But, no. It's *thinking straight* that's hard. It's being unbiased that's
hard. It must be. There were hundreds of people involved in that vaping study --
scientists, FiveThirtyEight writers, doctors, statisticians, public policy
analysts, editors, peer reviewers, anti-smoking groups -- and NONE of them, as
far as I know, noticed the real problem: that the argument just doesn't make any sense.