This is Steven Salzberg's blog on genomics, pseudoscience, medical breakthroughs, higher education, and other topics, including skepticism about unscientific medical practices. Here's where I can say what I really think about abuses and distortions of science, wherever I see them.

Serious doubts about new study of statins and heart disease

The news this past week was filled with reports of a new study in the New England Journal of Medicine (NEJM), which reported a “dramatic risk reduction” in heart attack risk for men using Crestor, a statin drug made by AstraZeneca. This was all over the news, getting headlines on the Sunday (Nov 9) major TV networks and front page reports Monday in The New York Times, the Washington Post, and other major papers. What was striking about this study was that it claimed that people with normal cholesterol levels could get significant health benefits. If true, this new study implies that millions more people should start taking statins to protect themselves against heart attacks.

Wow. Should we all rush out and get some statins? Should we all buy AstraZeneca stock (which went up 20% on the news of this report)? Both?

Hold on a minute. This new finding is rife with problems, despite the breathless news reporting about it. Actually, there are two studies: one published in NEJM, and the second published in Circulation. I’ve read them both. Study 1, in NEJM, got most of the headlines. Study 2 reports on a new diagnostic test that looks at levels of C-reactive protein (CRP), a marker of inflammation. Study 2 found that use of a test called hsCRP – for high-sensitivity C-reactive protein – improved the predictions of cardiovascular risk in men. In other words, the study said, using this test would let you predict more accurately who’s going to have heart problems. Let’s go over the problems one by one. (This is a long blog post, so if you want to know the REAL problem with the study, scroll down to Problem 5 below.)

Problem 1 (raise an eyebrow): Both studies were funded by AstraZeneca, the drug company that sells Crestor. Obviously, AstraZeneca must be pleased that the results suggest that millions more people – those currently considered at low risk for heart disease – should start taking Crestor. However, the funding was disclosed in the reports, and AstraZeneca did not interfere in the analysis, so the funding source does not invalidate the results – not at all. It just makes me a bit more skeptical.

Robert Bazell, a journalist at NBC, was much more credulous. He reported that the study was “squeaky clean.” Well, it’s awfully nice of Mr. Bazell to give his stamp of approval, but disclosure alone does not mean the study had no bias. We’ll get to that in a minute.

Problem 2 (raise the other eyebrow): Both studies also say that high levels of C-reactive protein (CRP) are linked to heart disease, even in men with normal cholesterol levels. The lead author of both studies was Paul Ridker. Paul Ridker owns the patent on the hsCRP test for CRP. Another consequence of these studies is that millions of people are now likely to get tested for CRP, using Ridker’s test. Clearly, Ridker has an interest in the results coming out the way they did. NBC’s Bazell gives him a pass: “As for Dr. Ridker, he says flat out that the financial interest in the test had no effect on the outcome. I certainly believe that. Dr. Ridker has spent most of career working on c-RP and this study validates all his work.”

So let me get this straight: because Ridker has spent his career working on CRP and this study validates his work, we shouldn’t question it? I don’t think so. What this meant to me was that the parties conducting and funding both studies had a very strong interest in the results coming out the way they did. That doesn't mean the results are wrong - again, it just makes me more skeptical. But that’s why we have placebo-controlled trials: to eliminate the effect of bias. So I read the papers, carefully, to see what the data actually said.

Problem 3: the Circulation study didn’t report separately on the effect of CRP and family history of heart disease. In this study, Ridker and colleagues looked at 10,724 men retrospectively (over a 10-year time period), and used a “traditional” model to predict the risk of heart disease. The traditional model had 5 variables: age, blood pressure, smoking status, total cholesterol, and HDL cholesterol. They then added two more variables to the model: (a) the hsCRP test and (b) family history of a heart attack before the age of 60. The report shows that the new, 7-variable model is somewhat more accurate. The study has several methodological problems that I won’t try to describe here, but the biggest problem is that the fail to report the separate effects of the two new variables. In other words, they report only that both variables should be used to measure risk, which means (of course) that patients should be getting the hsCRP test. But what if the entire effect of the study is due to the family history of heart disease? The study doesn’t say. We simply can't tell if the hsCRP test has added value or not. And the leader of this study - Ridker - has the patent on the hsCRP test.

Problem 4: the NEJM study actually reports a very small benefit. All the glowing press reports emphasized the “44% reduction in risk” in those taking Crestor, making it sound very dramatic, but they neglected to report the absolute risk. What I mean here is that if the risk is very, very small, then a relative reduction of 44% is not so significant. Here are the actual numbers: this was a large study, with 17,802 subjects, 8901 getting Crestor and 8901 getting a placebo. The placebo group suffered 251 “events” (one of five cardiac problems, including heart attack), and the Crestor group had 142 events.

This looks pretty significant – and statistically speaking, it is. But the clinical significance is different: you’d have to treat 95 people for 2 years with statins to prevent 1 heart attack. Is that worth it? And if we put millions of people on statins for the rest of their life, which might indeed prevent some heart attacks, will there be other consequences that we can’t yet foresee?Having an NNT (number needed to treat) of 95 might not sound so bad, but that’s a very high number. An article in Business Week a few years ago pointed out that such high NNT numbers might just represent statistical noise. That article quoted Dr. Nortin M. Hadler, professor of medicine at the University of North Carolina, who said, "Anything over an NNT of 50 is worse than a lottery ticket; there may be no winners.” The article goes on to point out that “an estimated 10% to 15% of statin users suffer side effects, including muscle pain, cognitive impairments, and sexual dysfunction.” Furthermore, it is highly likely that lifestyle changes – getting more exercise, for example – will have a better NNT than 95. So rather than prescribing Crestor, perhaps physicians should explain the greater benefits – possibly much greater - that patients will have from changes in diet and exercise.

Problem 5: This one is the biggest problem of all. The patients in the NEJM study were randomly divided into two groups, treatment (Crestor) and placebo. Table 1 in the paper describes the groups according to a long list of features, and the groups are virtually indistinguishable for most of these – average age, blood pressure, LDL cholesterol levels, body mass index, etc. However, there are 3 critical variables where the two groups are not identical. Presumably this happened by chance, but when you have such a small effect as the one found in this study, small differences can have huge consequences. Let’s look at these 3 variables and at the number of patients in each group (Crestor vs. placebo) with these factors:

Numbers of subjects in Crestor vs. placebo groups

Treatment group:

Crestor

Placebo

Current smoker

1400

1420

Family history of premature CHD

997

1048

Metabolic syndrome

3652

3723

Notice that the Placebo number is higher in all 3 cases. There were 20 more smokers in the placebo group, 51 more people with a family history of CHD (coronary heart disease), and 71 more with metabolic syndrome. All 3 of these variables are risk factors for heart disease – in fact, 2 of them were used by Ridker in his Circulation paper as part of a test to predict risk!

Even if these differences are accidental, all 3 of them put the placebo group at higher risk of heart disease. We don’t know if these totals represent separate people (one person might be a smoker and have a family history of CHD), but if they were, we have as many as 142 more “at risk” people in the placebo group.

Remember, the total number of excess events in the placebo group was only 109 (252 events versus 141 in the placebo group). How many of those events occurred in people with the 3 “bad” variables above? It is entirely possible that these differences in the two study groups could dramatically reduce – even eliminate – the supposed benefit observed in the study.

So what does this all mean? Well, I am not convinced that Crestor has a clinically significant benefit for patients with normal cholesterol. My guess is that further studies, if done properly, will show that the benefit is smaller than that reported last week, and perhaps the benefit is nonexistent.

Finally, on a lighter note, I was pleased that one of my favorite “fake news” reporters, Stephen Colbert, wasn’t fooled at all – he made fun of the study on his show (The Colbert Report) on Wed 12 November. (Scott Hensley over at The Wall St. Journal blog has a nice post about this.) Colbert reported the study in his “Cheating Death with Dr. Stephen T. Colbert, D.F.A.”:

“This is a great breakthrough in the battle to find things to prescribe to people who don’t need them. True, the drug costs $100 a month, but that is a small price to pay to not have the heart attack that there’s no way of knowing if you would have had it.”

Colbert then showed a video clip of Stanford cardiologist Mark Hlatky. Hlatky was interviewed on PBS, where he said “we need to be cautious before we expand the numbers of patients so drastically.” Colbert responded: “sounds like someone hasn’t gotten enough free Crestor pens.”

I guess I haven’t gotten enough free Crestor pens either.

10 comments:

Do you know if they are posting all the data from these trials online somewhere? Another thing I have trouble with with lots of clinical trials, in addition to all the potential conflicts of interest, is that it is frequently impossible to really completely reanalyze the data. Not that making the data completely available removes all potential biases (e.g., if the placebo group has a slightly higher risk of heart disease) but having the data is better than not.

That's a great question. I've looked into this, and the answer I got back is no - they do not post all the data anywhere, and it's impossible to get it unless the investigators give it to you. You're right that this would help to remove biases - for example, in the statin study, many patients dropped out over the course of the study, and it would be valuable to see what they looked like - for example, if the statin group had a few more dropouts (maybe they got ill?) that could alter the findings. We can only speculate unless they will make such data available.

When I have asked about such things in the past, a common answer I get for not releasing all the data is that it would jeopardize confidentiality. I can see how it might be complicated to release ALL the raw data but this sounds like a cop out to me.

This makes me wonder why the peer review process fails to catch these flaws. Is there a systemic "let's give this the benefit of the doubt" bias among reviewers? If so, is that bias stronger when the claim being made is more dramatic? Are the reviewers just missing these flaws? Are they influenced by some of the same potential conflicts of interest?

Dear Steven. I didn't find seperate data regarding CV mortality. They put it together with CV events that certainly is not the same. Regarding total mortality it was no statistically significant difference. I was also wondering about the same frequency of severe adverse events in both groups ? did you understand what was the upper limit of age because they mentiones 91 year old male that develop rhabdomyolysis. I am also curious to see whar is the division of recruited partcipants between the countries- Russia. Poline etc - different regulations and Different Cv outcomes as compared to western Europe.What is TNT number in life style intervention? Rita

Rita, these are good questions, but for some of them, the study simply doesn't provide the data. And unfortunately they are not likely to release their source data for others too analyze - although they should (anonymized, of course). You asked about CV mortality. Here is what you can read from the data they present - total mortality was only slightly higher in the placebo group, 247 deaths vs. 198, with a p-value of 0.02. If you look at the difference between "nonfatal" and "any" heart attacks, you can determine that there were actually MORE fatal heart attacks in the Crestor (rosuvastatin) group, 9 vs. 6. These numbers are too small to be significant, but still, this is another detail that they neglected to mention.

The JUPITER trial was stopped early for benefit. This makes me less confident in the reliability of the risk reduction achieved. I would be surprised, however, if there were no risk reduction, given the results of other statin trials. I think elevated hsCRP was only one of the reasons the patients in this study were higher risk than the average bear. The majority had hypertension, they all had age as a risk factor, many of them had metabolic syndrome.

I agree, the absolute benefit in this study, and in other studies of primary prevention with statins, is fairly small. In addition, rosuvastatin is still on patent and is a fairly expensive drug. That said, it is not clear to me that primary prevention in people with the characteristics of the people in the JUPITER is any better or worse of an idea than primary prevention in people who are at elevated CVD risk for other reasons.

Fundamentally, I think each person has to look at their *individual* risk (usually over a 10 year period) and ask themselves whether the reduction in absolute risk from taking a statin is worth it to them. Some people are at high baseline risk, so their absolute benefit is higher. In addition, people have different opinions with respect to how they value any given risk reduction, whether they want to take a pill every day, and other factors.

Full disclosure: my husband is on a statin for primary prevention. He is considered very high risk for several reasons, and the NNT for people like him would not be that high, although I can't give you a number.