I believe this is biology's century. I've covered science and medicine for Forbes from the Human Genome Project through Vioxx to the blossoming DNA technology changing the world today.
Email me, follow me on Twitter, circle me onGoogle Plus, or subscribe to my Facebook page.

New Cancer Data Shine Spotlight On The Secret Committees That Make Medicine's Toughest Decisions

On Saturday, a researcher from the University of California, San Francisco, stood in front of a crowd of doctors at the year’s biggest cancer conference and presented stunning results from a company-funded study of a new Johnson & Johnson drug, Zytiga. The study tested the pill, already approved for men who have received chemotherapy for prostate cancer, in those with an earlier stage of the disease. It slowed the spread of their cancer by 57%. Even more tantalizingly, the results hinted, but didn’t prove, that Zytiga was extending the men’s lives by a startling 33%.

Immediately after the data were presented, statistician Susan Halabi from Duke University took the stage to discuss the study – a job she’d been given by the American Society of Clinical Oncology, which runs the conference. She raised real concerns about how confident doctors could be about Zytiga’s benefit. The Zytiga clinical trial had been stopped too early, she said, and that meant that right now it might be giving an impression that the drug is more effective than it actually is. It would also make it difficult to prove Zytiga extended these men’s lives once and for all. In an interview, she said that the decision to give placebo patients the drug “sacrificed” the trial. “This decision,” she said, “is irreversible.”

Welcome to the controversial world of data safety monitoring boards (DSMBs), the secretive bodies that help decide when clinical trials stop – and when they can continue. DSMBs are charged with keeping patients safe. While both a drug company and the academic investigators it hires do not see data from a clinical trial in real time, DSMBs meet periodically to watch out for emerging side effects so dangerous that the study must be stopped. They also watch out for evidence that the trial could never succeed, or, as in the DSMB recommended with Zytiga, that the evidence for a treatment is so strong that it is no longer ethical to give patients placebo. This role has made these committees, which by their very nature avoid the spotlight, key players in understanding the safety and effectiveness of new medicines.

“They’re supposed to be secretive. That’s the point,” says Jeffrey Drazen, editor of the New England Journal of Medicine. “It’s probably the toughest job in clinical medicine. Being on a DSMB requires real cojones.”

But lately DSMBs have been deciding to stop trials early an awful lot, causing some statisticians to worry that they are too eager to deliver early results. Stopped-early studies proved the benefits of Viread for preventing HIV infection, Afinitor in breast cancer, Viread for preventing HIV infection, Crestor for preventing heart attacks and strokes, Revlimid for use in multiple myeloma after a stem cell transplant, Sutent in treating the rare form of pancreatic cancer that afflicted Steve Jobs, and studies of Lotrel and Norvasc, both blood pressure drugs, just to name a few.

Because early results are more likely to be due to chance, statistical rules say that in order for a study to be stopped early, a result must be far more statistically significant than normal. Usually scientists will bank on a result that could have occurred by chance only five times out of 100. But in the Zytiga study, the result for overall survival at the early analysis needed to be so unlikely it could have only occurred 8 times out of 10,000. It wasn’t even close, at 1 out of 100; Halabi, the Duke statistician, argues that the DSMB should have waited until the next scheduled analysis of the data, which was probably only a few months away. If the study were continued, it is possible there would be what scientists call “regression to the mean” – literally, the results would become more average.

There is at least one case where stopping a trial early for benefit appeared to give an incorrect result, says Sanjay Kaul, a cardiologist at Cedars-Sinai Medical Center who is a frequent adviser to the Food and Drug Administration. A 1999 study of giving heart drug called beta blockers before vascular surgery was stopped because it showed a 91% reduction in risk, resulting in changes to some medical guidelines. But the results were never replicated, Kaul says, and another large trial indicated the results might cause harm. Kaul believes studies should only rarely be stopped because a drug is so effective. Worse, he worries that sometimes DSMBs, which are often recruited by companies themselves, are making decisions “driven by marketing considerations” and the desire to get drugs to market faster and not their obligation to society.

Could concerns about marketing have been a factor with Zytiga? Both Johnson & Johnson and its lead investigator say the DSMB made its recommendation independently. But the patent on Zytiga in the U.S. could expire as soon as 2017, allowing generics on the market that would curtail sales, according to Lawrence Biegelsen, an analyst at Wells Fargo Securities. The statistical hubbub doesn’t look like it will hurt J&J. Several Wall Street analysts, in notes to clients this morning, trumpeted the data as a positive for J&J and a negative for rival Dendreon.

Not everyone agrees that the Zytiga study was stopped too soon. Donald Berry, a cancer statistician at M.D. Anderson Cancer Center, notes that the measure of tumor growth that was seen in the study was unlikely to change, and that it’s okay to analyze overall survival later. A lot will depend on how the study was designed, and what particular agreements exist between J&J and the FDA about the decision to stop the study.

Nor does everyone agree that stopping clinical trials is a bad thing. In a 2010 letter to the Journal of the American Medical Association, three researchers wrote that stopping trials early “is a cornerstone of efficient and ethical trial design” and “does not lead to substantive bias.”

Susan Desmond-Hellmann, the chancellor of University of California, San Francisco, and the former head of cancer drug development at Genentech, says she “strongly” believes in early stopping, both to protect patients from having to take placebo when there is a treatment that works and to get drugs to patients faster. She says that when it comes to making sure a DSMB does the right thing, there is no replacement for picking the right experts and giving them clear guidance at the outset.

Stopping a trial early because of benefit can also sometimes hurt a company. With GlaxoSmithKline’s breast cancer pill Tykerb, early termination of a clinical trial meant that it was not possible to prove the drug had a survival advantage. “At the end of the game it created a problem for us,” says Paulo Paoletti, head of cancer drug development at Glaxo.

Even the ground rules of how the DSMB should work can be fuzzy, determined as much by the culture and experience of doctors who run clinical trials as anything else. An example: Michael Meyers, the J&J executive in charge of Zytiga, said that the DSMB makes only recommendations, and that the sponsor – the company – makes final decisions. Steven Nissen, chairman of cardiology at the Cleveland Clinic, protested “strongly” that the DSMB should report to the independent steering committee of doctors who run the study, further limiting the company’s involvement.

Drazen, the New England Journal editor, thinks the solution is to make the committees even more secretive, from companies, other scientists, and the public. He has criticized GlaxoSmithKline and Merck for being too involved in the deliberations of DSMBs for studies of the diabets drug Avandia and the cholesterol drug Vytorin, respectively. He has proposed making DSMBs answerable to a public body, not drug companies, in order to further guard public health. For the system to work, the DSMBs need to be left alone while a clinical trial is going on, he says.

Post Your Comment

Post Your Reply

Forbes writers have the ability to call out member comments they find particularly interesting. Called-out comments are highlighted across the Forbes network. You'll be notified if your comment is called out.

Comments

The debate over stopping clinical trials early based on promising interim results seems like a red herring to me. What worries me more is the mid-course relaxation in criteria for early stoppage. Does it mean the initial criterion for early stoppage was wrong? If so, why was it adopted (I.e.who screwed up?). If the initial criterion was appropriate, then what’s the justification for the change? I can see many good medical and ethical reasons for early stoppage criteria in a trial, but when the criteria are changed with flimsy justification partway through for a drug that’s going to face stiff competition and a short patent life, it seems crazy NOT to consider less honorable factors like market pressure.

4.4.3. Other Responsibilities 4.4.3.1. Making Recommendations A fundamental responsibility of a DMC is to make recommendations to the sponsor (and/or, as noted in the Introduction, a steering committee or other group delegated by the sponsor to make decisions about the trial) concerning the continuation of the study. Most frequently, a DMC’s recommendation after an interim review is for the study to continue as designed. Other recommendations that might be made include study termination, study continuation with major or minor modifications, or temporary suspension of enrollment and/or study intervention until some uncertainty is resolved.

Because a DMC’s actions potentially impact the safety of trial participants, it is important that a DMC express its recommendations very clearly to the sponsor. Both a written recommendation and oral communication, with opportunity for questions and discussion, can be valuable. Recommendations for modifications are best accompanied by the minimum amount of data required for the sponsor to make a reasoned decision about the recommendation, and the rationale for such recommendations should be as clear and precise as possible. Sponsors may wish to develop internal procedures to limit the interim data released by a DMC after a recommendation until a decision is made regarding acceptance or rejection of the recommendation, to facilitate maintaining confidentiality of the interim results should the trial continue. We recommend that a DMC document its recommendations, and the rationale for such recommendations, in a form that can be reviewed by the sponsor and then circulated, if and as appropriate, to IRBs, FDA, and/or other interested parties. Sections 5 and 7.2.1 address implications for reporting to FDA of DMC recommendations for major study changes such as early study termination.

Further, the study was not that far away from the next interim look:

“The DMC’s decision to terminate the trial early has led to the trial’s failure to show a statistically significant survival advantage. Would it have required long to wait to show overall survival? Probably not. The interim analysis that Ryan presented was based on 43 percent of the expected overall survival events (333 deaths of trial subjects). This took place in the fourth quarter of 2011. The next interim analysis was planned for the second quarter of 2012 based on the expectation that 425 deaths (55 percent of the expected death events) would have occurred. Halabi told me after her presentation that it was highly likely this interim analysis would have crossed the O’Brien-Fleming boundary, and shown a statistically significant overall survival advantage for abiraterone in chemotherapy-naïve patients.”

Someone needs to go to the FDA, read the minutes of the Board’s deliveratiions, and report. This whole issue, somehow, just doesn’t pass the ‘smell test.’ What did JNJ know, and when? How much influence did they have on the Board’s deliverations? The trial appeared to be ‘that’ close to evidencing stat sig OS? Why not wait? The next ‘look’ would have been in June, 2012.

Even more interesting will be the FDA’s deliverations. Will the Cancer Czar bend to JNJ’s will and approve a treatment that doesn’t meet his credo for statistical purity, much less the agency’s ‘gold standard’ for stat sig OS?

It should be noted that unlike the usual environment around a placebo-controlled trial where nothing else is available aside from the being tested drug, there are options here outside of Zytiga such as Provenge and Taxotere with proven survival benefits. In fact, Provenge with its side effect profile (extremely light) and excellent median survival benefit would make a competitive treatment to Zytiga. And, patients who have progressed under either Zytiga or placebo in the trial could elect to take any of these treatments. Most did.

What does that mean? First it means that the usual “compassionate argument” to stop a trial does not really apply in this case. Second and more importantly, it means that unless Zytiga is a wonder drug with survival benefit far exceeding the others (little evidence of that so far), longer observation time would not favor the survival benefit trend seen at the interim look because most patients would be taking other drugs. Dr. Halabi was correct in stating that regression to the mean was a real danger. The Hazard Ratio of 0.75 seen at the interim could regress closer to a run of the mill 0.80 or even worse at the final look.

Such a worsening of the HR would raise some potentially embarrassing questions such as: Why with a median treatment time of only about 15 months, it took nearly 18 months to see any survival benefit? And, then why Zytiga took nearly 18 months just to show some survival benefit then the benefit immediately wanes in just a few months?

So, letting patients cross over to Zytiga at a failed interim look on OS only a few months from the end of the trial could be looked at in a cynical way. Over the course of the next few months, the HR will likely regress but the claim can be that the reason was because of a compassionate decision to let patients on the control arm take Zytiga instead of because of the concerns raised above.

The members of the DSMB are often publicly disclosed, and the minutes are given to the FDA. It does strike me as a pretty hair-raising job, and I agree with Drazen’s comment. I also think agree with his proposal of making DSMBs accountable to some outside organization, such as the Institute of Medicine. I heard repeatedly in conversations this weekend that DSMBs “are not as independent as people think” and that’s a real problem — even if it’s just a perception.

This article describes the challenges facing DSMBs in a balanced way though the term “secret Committees” is a bit dramatic since there is no secrecy surrounding the DSMB. As one who established and liaised with a number of DSMBs for medical device clinical trials, I would like to share my “on the ground” experiences.

First, DSMB members I work with performed their duties and fulfilled their responsibilities with the utmost scientific and ethical rigor that one would expect from an independent and unbiased advisory body. Second, DSMB members are highly accomplished and well respected experts in their fields and, preferably, with past experiences as DSMB members. Third, deliberations on clinical trials are conducted in closed doors with only the DSMB members present. Fourth, contractual agreement with DSMB members and clinical trial specific DSMB charters provide legal guardrails for the protection of clinical trial participants. Both the DSMB charter and the clinical trial protocol spell out clearly the study stoping rules by which the DSMB abides. Stopping rules are established after deliberations on the need to reasonably protect safety and wellbeing of study patients and maintaining the scientific rigor of the the clinical trial, including statistical considerations. At least one of the DSMB members is a biostatistician who reviews the statistical rational for recommendations made by the DSMB. Lastly, It is ultimately the FDA’s purview to review and accept the DSMB charter. Hence, early stopping of a clinical trial due to futility, or to superb treatment benefits by the test article are ethical and scientifically valid. In my humble opinion the existing FDA guidance document provides a balanced framework for the selection and operation of DSMB. Why change something that works well?

The whole point is to be secretive, at least while they’re going on. I wish I’d thought of “Secret Heroes” before I wrote it, which more accurately reflects my opinion of them. But I also think there’s not enough scrutiny of when DSMBs mess up — as may have happened in the Avandia and Vytorin examples. Thanks for your comment.

My idea of secretive may be different than yours, Matthew. Therefore, it is worth clarifying what you conceive as being a secretive approach. For me, the connotation of “secretive” is too much of a move to one extreme. If DSMBs were secretive how would they be able to probe the scientists developing the test article with clarifying questions? DSMBs, as you know, provide highly valuable insights to drug/device manufacturers especially during the enrollment phase. The value of these insights is more acute when trials are blinded. As of now, the combination of an open session that allows for discussions between the DSMB, trial sponsor and other advisors, followed by a closed session, attended only by the DSMB members, is a well balanced process. There are probably hundreds of DSMB meeting taking place each year and, I dare to say, the majority are conducted as expected. Are a few questionable experiences constitute a trend? Do we, as society, have to respond by ratcheting up the pressure on ALL DSMBs out there? I am more in favor of a smart targeted approach and I believe that the FDA serves us well when it comes to reviewing the work of DSMBs as part of the review and approval process.

Lastly, I do not consider DSMB members heroes, at least as their actions pertain to the role they play on the DSMB. My view is more mundane– these are experts who are there to do a job for which they get paid. Fundamentally, they conduct a risk to benefit analysis of aggregate clinical data. Similar types of analyses are being done in many other industries. When they accept the job they know what they are up to. I reserve the word “heroism” to different kinds of acts.

Well, there is the necessary secrecy while the trial is ongoing. Without it, the study would remain blinded. Then there is the fact that, although DSMBs are making some of the key decisions in medicine, they tend to remain silent. Unlike PIs, they are rarely available to comment. And although I’ve heard suggestions that their minutes might be made available after the fact, I haven’t seen them. All of this strikes me as highly secretive.

The past decade has elevated the importance of DSMBs for large outcome studies far above the level you suggested. In this study, I only noted the early stoppages. But there are also cases like this and the decision to unblind the RECORD trial during the Avandia trial.

I do think being in the position to protect patients and make ongoing clinical research possible is a pretty heroic thing — especially when the circumstances are extraordinary.

rickwo, looking at the published summary statistics I woudn’t call it ‘flimsy justification’. I would trust Don’s (Prof Don Berry – MD Anderson) statement that the decision was sound – and from the phrase ‘its OK to analyze the survival data later’, I take that though the trial has ‘stopped’ (stopped recruiting perhaps) its not done and dusted, any regulatory approval on the current results would be provisional on the reuslts from following up the currently treated patients. One of the issues with complex trial designs is anticipating what the ‘rules’ of the design will mean in practice. Simulation of the trials beforehand helps enormously as individual cases can be considered and the clinical team can check whether they are happy with the behaviour the predefined rules prescribe. Because with group sequential designs simulation is not necessary it does seem as though there have been a number of cases where DSMBs have been uncomfortable when faced with a particular dataset with what the rules say they should do. Simulation doesn’t guarantee you’ll have considered something like what turns up of course, but you are much more likley to have done than if you don’t do simulation at all.