Objective Overestimation of treatment effect sizes—termed optimism bias—in research protocols can lead to underpowered clinical trials that fail to demonstrate clinical benefits. We compared hypothesized vs. observed treatment effects to determine if there is evidence of optimism bias in contemporary NCTN phase III trials.

Design We queried PubMed for National Cancer Institute (NCI)–sponsored phase 3 randomized cooperative group clinical trials from January 2007 to January 2017. We identified 185 published trials. Trials with missing protocols (n = 56), equivalence or noninferiority trials (n = 5), trials that accrued less than 40% of their intended sample size (n = 14), and trials that pooled their data with other studies (n = 2) were excluded. For trials reporting time-to-event outcomes with hazard ratios (HRs) (n = 81), we compared the proposed effect size from the sample size calculation in the research protocol with the observed effect size in the published article to calculate the ratio of observed-to-proposed HRs overall and for trials that did or did not report statistically significant effect on primary end points. All HRs were standardized for a reduction in adverse events such that HRs less than 1 indicated a benefit to therapy. We also compared findings with those previously reported for NCI trials conducted from 1955 to 2006 and tabulated studies that provided a reference, evidence, or other specific rationale for their proposed effect size in the research protocol.

Results Data on 98,200 patients from 108 clinical trials were evaluated. The most common cancers were breast, gynecologic, gastrointestinal, brain, and genitourinary malignant neoplasms. The most common primary end point was overall survival (40.7%). The median ratio of observed-to-proposed HRs was 1.26 (range: 0.33-2.34). The median ratio of observed-to-proposed HRs among trials that observed a statistically significant effect on the primary end point was 1.09 (range: 0.33-1.29) vs 1.30 (range: 0.86-2.34) for trials that did not, compared with 1.34 and 1.86, respectively, for NCI trials conducted from 1955 to 2006. Twenty-four trials (22.2%) observed a statistically significant effect on the primary end point favoring the experimental treatment, compared with 24.6% previously reported. The majority of trials (76.9%) provided no rationale for the magnitude of the proposed treatment effect.

Conclusions Although most NCI-sponsored clinical trials conducted between 2007 and 2017 failed to establish statistically significant benefits of new therapies, the magnitude of optimism bias appears to have decreased compared with that in trials conducted between 1955 and 2006. Better rationalization of proposed effect sizes is needed in clinical trial protocols.