Tabar argued in the BMJ that the Malmo trial shouldn't be used to determine the value of mammography screening because between 70% and 74% of women invited to the trial actually had a mammogram and some 24% of women who were not invited had a mammogram.

Fik dig!

The researchers who conducted the Malmo trial actually determined both the compliance rate for the women who were invited to the trial and separately for the women who were not invited. For the second group the researchers surveyed 500 women and determine the number of mammograms that they had.

The Malmo study, randomized women between 45 and 69 living in the city of Malmo between an invitation to have a mammogram and a control group. The study found that after about 9 years of follow up 63 of 21,088 in the invitation group and 66 of 21,195 in the control group passed away from breast cancer. Again, these are very small differences and we have the issue that it is an intent-to-treat analysis. But now have estimates of the mammography rates in the two groups.

If we assume that the take up rate of mammography of those receive the letter is 70% and the take up rate for those who didn’t is 24% and we also make the "exclusion restriction" that the letter has no impact on the probability of dying from breast cancer, we can calculate the average effect of mammography on survival from breast cancer for women in Malmo.

We can write down the following system of equations, where A is the probability of dying from breast cancer conditional on getting mammography screening and B is the probability of dying from breast cancer without mammography screening.

While the intent-to-treat analysis suggest a 4% reduction in death from breast cancer associated with mammographic screening. Accounting for the actual take up rate suggests an 8% reduction. Note that none of this accounts for sampling variation and is only presented for illustrative purposes.

Now this analysis also assumes that everyone who took up mammography screening is the same as those that didn't take up the screening, excepting for the fact that they took up mammography screening. This is plainly not true.

The trial, conducted some thirty years ago, randomized women in two Swedish counties (thus the name) to receive either an invitation to receive a mammogram or no invitation. These invitations were given over the next several years and the trial participants were followed for the next twenty plus years.

The results were that 339 women of 77,080 passed away from breast cancer in the invitation group and 339 women of 55,985 passed away in the control group. The claim is that this result proves that general mammography screening reduces death from breast cancer. The issue is that this doesn't tell us that mammography reduces death from breast cancer.

We know that these women were randomized into two groups and we know the average death rate from breast cancer in the two groups, but we don't know how many people actually got a mammogram in each of the two groups. It was this number that I was searching for. How many people actually got mammograms?

After trudging through the many papers written on the study I found one of the two numbers I needed. Approximately, 85% of women in the invitation group accepted the invitation and received a mammography. As far as I can tell the researchers never determined the take up rate of mammography in the group that didn't receive mammography.

So what can we say from this trial?

As long as the mammography rate in the invitation group was higher than in the control group then the analysis implies that mammography causes a reduction in death rates from breast cancer. If we are willing to make a behavioral assumption - that Swedish women are motivated to get mammography by receiving invitations, then this large randomized controlled trial can be used to infer causality. Our prime suspect must have had an accomplice.

We can't really say much more about the value of mammography without knowing how many women in the control group received a mammogram.

The concern is that a number of people left the trial and so we do not observe if and when they passed away. Moreover, these patients may left from the two trail arms at different rates, biasing our results. The authors seem to be aware of the issue when they appeal to an "intent-to-treat" analysis. The problem is that even though the patients were randomly assigned to the two treatment arms, we don't know who left the trial and why they left. Moreover, the trial was open label. That is, patients knew exactly which arm they were in when they made the decision to leave.

In the post I suggested that it may not be possible to determine causality with intent-to-treat analysis.

My colleague, Matthew Chesnes, suggested that it may be still possible to determine causality even if not everyone accepted the random assignment that they were given. His intuition is that if almost everyone accepted their assignment wouldn't we get pretty close to showing causality?

The intuition is correct.

If it was the case that everyone in the EMILIA trial accepted their random assignment then we can use the observed probabilities to determine causality. We see that 35% of women in TDM-1 (Kadcyla) trial arm passed away within the first two years, while 48% of women in the X+L arm (the alternative treatment) pass away in the first two years (see chart). From these numbers we can determine that TDM-1 causes women to have greater survival. In fact, we can determine that for at least 12% of women in the study would have lived less than 2 years on X+L but survived over 2 years on TDM-1.

The problem is that not everyone did accept their random assignment. From clinicaltrials.gov we learn that 38 women in the TDM-1 arm left the trial and 52 women in the X+L arm left the trial. No information is provided about when these people left the trial. And because they left, we don't know what happened. Still, there are only two possibilities. They may have passed away within 2 years or they may have lived longer than 2 years.

We can use an idea developed independently by the econometrian, Charles Manski, and the epidemiologist, James Robins. The insight of Manski and Robins was that when we don't observe probabilities we may still be able to bound the probabilities using information about the proportion who leave the trial and the fact that probabilities lie between 0 and 100%.

For the 495 women assigned to TDM-1, 457 stayed with their assignment and had a 35% probability of passing away within two years. For the 38 women who left the trial the lowest probability is 0% and the highest probability is 100%. From the law of total probability we can determine that the lowest probability of passing away within two years given the assignment to TDM-1 is (457/495)*35 = 32% and the highest probability is (457/495)*35 + (38/495)*100 = 40%.

For women originally assigned to the TDM-1 arm, their probability of surviving at least two years lies between 60 and 68%.

We can do the same thing for the X+L arm. The lowest probability of passing away within two years is (444/496)*48 = 43% and the highest probability is (444/496)*48 + (52/496)*100 = 54%.

For women originally assigned to the X+L arm, their probability of surviving at least two years lies between 46 and 57%.

Note that these two bounds do not overlap. That is, it must be the case that women assigned to the X+L have a lower probability of surviving two years than women assigned to the TDM-1 arm. As there is no other explanation for the difference, we can assign the difference to the drug treatment.

As, at least 60% of women in TDM-1 survive more than 2 years and at most 57% of women on X+L survive more than 2 years, it means that at least 3% of women live longer on TDM-1 than X+L.

Certainly, 3% is not huge, but it is positive.

Kadcyla causes at least some women to survive longer than they would have if they had taken the combination of Lapatinib and Capecitabine.

Despite the potential for bias from attrition, EMILIA may still provide proof of causality.

Thursday, January 8, 2015

In the large Swedish study on the effectiveness of mammograms the researchers couldn't force people to get mammograms but they could force them to receive letters. Swedish women in two counties were randomized into two groups. One group received an invitation to get a mammogram and the other group did not. The EMILIA study of the effectiveness of Kadcyla on advanced breast cancer suffered from biased attrition. In both studies, the researchers resorted to "intent-to-treat" to save their study, get published and claim a causal relationship.

Intent-to-treat refers to the idea that while the patients were not randomly assigned to the treatment groups they were randomly assigned an observed characteristic (they received a letter or not) and that observed characteristic MAY be associated with the treatment assignment. It is like doing one stage of a two-stage instrumental variables analysis.

The problem with relying on intent-to-treat is that it may not provide evidence of causality.

To see this, think about what happens if we just observed two groups, one group received regular mammography and the other group did not. We also observe their breast cancer rates and survival rates. In fact, assume that we observe higher survival rates among the women who received regular mammography. From this information, and only this information, can we determine the causal effect of mammography on survival from breast cancer?

We cannot.

The problem is that we don't know anything about how the two groups were selected. Even if we are able to account for differences in the observable characteristics like age, there still may be differences in unobserved characteristics such as the women's genetic profile.

Now what if I told you that the group who received a mammogram was much more likely to have received an invitation to get the mammography than the group who did not receive a mammography? Moreover, the invitation was randomly assigned. Can this information determine the causal effect of mammography on survival from breast cancer?

It cannot.

The problem is the same. Despite the random assignment of the invitation we still do not know the make up of the two groups. In particular, we do not know things about the unobserved characteristics of the women such as genetics or a family history of breast cancer that would make them more likely to get a mammography (with or without the invitation).

The same problem occurs in the EMILIA trial. Women left the trial at different rates depending on the treatment arm that they were assigned. Because they left we do not know when or if they passed away. The women that remained in the trial may be different across the two arms, so we can no longer assign the difference in the treatment outcomes to the different treatments. We can no longer remove the possibility that the different outcomes were due to other differences between the women in the two trial arms. We cannot use the trial to determine the causal effect of Kadcyla on breast cancer survival even though the women were randomly assigned to treatments.

Sunday, January 4, 2015

Austin Frakt at The Incidental Economist has a review on Angus Deaton's critique of randomized control trials as the "gold standard" of science. Frakt suggests that one major advantage of RCTs is that they are "conceptually simple" requiring less mathematical or statistical training in order to understand the results.

Ideal randomized control trials provide two pieces of information:

1. An unbiased estimate of the average treatment effect, and

2. An unbiased estimate of the minimum proportion of the population who benefit from the treatment over its tested alternative.

I argue here that learning the average treatment effect is not terribly useful. The question is whether we can learn that there exist people who benefit from the treatment from settings as conceptually simple as RCTs. Here I suggest two.

The first is the case where a treatment becomes available at a certain point in time and we can look at what happened before and after. The chart below shows the survival rate of HIV-infected patients before and after the introduction of various drugs that became the AIDS cocktail or HAART regime. This chart was published in the New England Journal of Medicine in 1998 and was one of the first pieces of evidence that HAART was enormously effective in reducing deaths from AIDS.

The second setting is perhaps more controversial. It is the case where we are willing to assume that the people in our study are selected into the treatments that are generally going to make them better off. Technically, we need people to be selected into the treatment that "first order stochastically dominates." Some writers call this assumption the minimal requirement for rational decision making (Hadar and Russell, 1969).

The chart below is from the BLS and reports the familiar result that people who attend college earn more money. If we are willing to assume that people choose the education level that is more likely to give them the higher earnings then the result presented in the chart shows that for at least some people a college education increases wages.

But haven't we been told over and over again that this chart suffers from "selection bias"? People who attend college may simply be those who would have earned more money anyway. If college doesn't do anything to people's future income then these people are certainly spending a lot of money to play beer pong and attend college football games. If they are spending money for nothing, then claims that college attendees are smarter than the average Joe are pretty suspect.

To be clear the claim is that college increases incomes for some people not necessarily for all people or even many people. The chart and the assumption that college students are not idiots suggest that college has a causal effect on income. No RCT needed.