Wednesday, June 3, 2009

Wednesday Math, Vol. 74: samples and rare events

Earlier this year, I was reading Nassim Nicholas Taleb's The Black Swan, and I spent several blog posts taking the BIG ugly stick to it. While some of his ideas are interesting, his writing style exhibits a nasty chip he has on his shoulder, and he often makes broad, sweeping statements that are easily refuted by anyone who is paying attention to the real world. One such statement was this.

Assume that a drug saves many people from a potentially dangerous ailment, but runs the risk of killing a few, with a net benefit to society. Would a doctor prescribe it? He has no incentive to do so.

This weekend, I was talking to my friend Jeremy Smith, father of my adorable niece Holly, while sitting together on a short bus ride and I brought up this statement. A fellow in the seat in front of us who was politely eavesdropping came to Taleb's defense about this. He had spent some of his career in medical research, and brought up this point. While we know about drugs that have serious side effects, some of which are even fatal, whether or not a drug company will release such a drug depends on when they discover this damaging information. Often, it happens after the original clinical trial takes place and the drug becomes readily available to the general public.

Let's say we have a clinical trial with 500 test subjects. Let's also stipulate that the drug has life threatening side effects for some small proportion of the population. It is possible that no one from the small sub-population will be included in the sample. Here's the math of it.

If 1 person in 200 will experience the possibly fatal side effect, then 199 in 200 will not. In decimals, these numbers are .005 and .995, respectively. If 500 people are chosen at random, the probability that we get zero people from the threatened sub-population is found by raising .995 to the 500th power.

.995^500 = 0.081571861...

This says that about 8.2% of the time, a 500 person trial will miss a serious side effect if that side effect shows up in only 1 person in 200. Here are the numbers for other levels of rare event.

Once the drug has been approved, the number of people who will use it will likely be in the tens of thousands, and the probability that no one in that larger sample will be from the threatened population shrinks down to effectively zero. For example, if we have that bad thing occuring in 1 person in 200 and 10,000 people are taking the drug, the chance of no one having the side effect is .995^10,000, which is less than 2 chances in ten billion trillion, 1.7 divided by 10 to the 22nd power. The most likely thing to happen in this case would be to see about 10,000/200 or 50 cases or so.

Another situation that happens in real life usage that is often missed in samples is drug interaction. Perhaps new drug x is meant to replace drug y on the market, and researchers have worked to make sure x has no bad effects when used with drug z, which has shown problems when used in concert with y. Even if x passes this test with flying colors, there is no promise that x won't have other drugs that are counter-indicated, and the side effects of using y and z together are replaced by different but still bad side effects of using x and some previously innocuous drug w together.

So I owe Nassim Nicholas Taleb an apology about his statement about drugs with dangerous side effects. On the other hand, what Taleb said about investing in books vs. investing in movies is still massively idiotic, and my knowledgable eavesdropper on the bus is in complete agreement on this.