Review methods We included randomised controlled trials of current tobacco users of adult age comparing use of varenicline with an inactive control and reporting adverse events. We defined treatment emergent, cardiovascular serious adverse events as occurring during drug treatment or within 30 days of discontinuation, and included any ischaemic or arrhythmic adverse cardiovascular event (myocardial infarction, unstable angina, coronary revascularisation, coronary artery disease, arrhythmias, transient ischaemic attacks, stroke, sudden death or cardiovascular related death, or congestive heart failure).

Results We identified 22 trials; all were double blinded and placebo controlled; two included participants with active cardiovascular disease and 11 enrolled participants with a history of cardiovascular disease. Rates of treatment emergent, cardiovascular serious adverse events were 0.63% (34/5431) in the varenicline groups and 0.47% (18/3801) in the placebo groups. The summary estimate for the risk difference, 0.27% (95% confidence interval −0.10 to 0.63; P=0.15), based on all 22 trials, was neither clinically nor statistically significant. For comparison, the relative risk (1.40, 0.82 to 2.39; P=0.22), Mantel-Haenszel odds ratio (1.41, 0.82 to 2.42; P=0.22), and Peto odds ratio (1.58, 0.90 to 2.76; P=0.11), all based on 14 trials with at least one event, also indicated a non-significant difference between varenicline and placebo groups.

Conclusions This meta-analysis—which included all trials published to date, focused on events occurring during drug exposure, and analysed findings using four summary estimates—found no significant increase in cardiovascular serious adverse events associated with varenicline use. For rare outcomes, summary estimates based on absolute effects are recommended and estimates based on the Peto odds ratio should be avoided.

Introduction

Tobacco use accounts for 440 000 deaths in the United States every year, killing nearly one in two long term smokers, with the leading cause of death being cardiovascular disease.12 Quitting smoking has immediate cardiovascular benefits,3 reducing the risk of recurrence of coronary events to that of a non-smoker within three years and reducing mortality after a heart attack by up to 50% over three to five years.456

US clinical practice guidelines for treating tobacco dependence recommend the use of cessation pharmacotherapy for all smokers interested in quitting, unless contraindicated.7 The US Food and Drug Administration has approved three first line classes of cessation pharmacotherapy: nicotine replacement therapy; bupropion (Zyban, Wellbutrin), an antidepressant; and the most recently approved option, varenicline (Chantix, Champix), a partial agonist to nicotine receptors. Varenicline binds with high affinity and selectivity to α4β2 nicotinic acetylcholine receptors in neurones. The partial agonist activity induces modest receptor stimulation that attenuates the symptoms of nicotine withdrawal. In addition, by blocking the ability of nicotine to activate α4β2 nicotinic acetylcholine receptors, varenicline inhibits the surges of dopamine release that are believed to be responsible for the reinforcement and reward associated with tobacco use.8 Meta-analyses and comparative trials have demonstrated the effectiveness of varenicline for quitting smoking and sustaining abstinence relative both to placebo and to bupropion.9

For smokers with cardiovascular disease, cessation offers critical health benefits and yet has been a challenge to achieve, particularly in the long term.10 A recent placebo controlled trial of varenicline in 714 smokers with stable cardiovascular disease reported sustained abstinence of 47% at the end of treatment and 19% at one year follow-up in the varenicline group, compared with 14% and 7% in the placebo group, respectively; the group difference was significant at both time points.11 Incidence of cardiovascular serious adverse events in the varenicline group during the 52 week trial was low (7.0%) and the 95% confidence interval ruled out an excess greater than 5% (1.4%, −2.3% to 5.0%). Nonetheless, the FDA has called for systematic review of all randomised clinical trials of varenicline for tobacco cessation to determine its association with cardiovascular risk.12

Singh and colleagues conducted a meta-analysis of the safety of varenicline and concluded that the drug increased the risk of cardiovascular serious adverse events by 72%.13 However, doubts about the researchers’ conclusions have been raised,141516171819 owing to several methodological issues, namely, the inclusion of adverse events well beyond the treatment period; exclusion of trials with no events, thereby biasing findings against the null; and use of the Peto odds ratio, which has shown bias under conditions of imbalanced design and rare events, which were present in most of the reviewed trials. Inclusion of the adverse events beyond the treatment period is of concern, because of the elevated risk for cardiovascular disease among study participants associated with their chronic use of tobacco and because of differential drop out by condition. In all but one20 of the 14 studies reviewed by Singh and colleagues, retention was lower in the placebo group than in the varenicline group, thereby reducing the likelihood of detecting adverse events in the placebo arm.

We conducted a systematic review and meta-analysis of treatment emergent, cardiovascular serious adverse events in all published, randomised controlled trials of varenicline for tobacco cessation. We defined these events as occurring during drug treatment or within 30 days of discontinuation. We chose a 30 day window because it was biologically relevant for detecting a drug toxicity effect, while still being conservative; the half life of varenicline is 24 hours, and any direct pharmacological effect should be gone within seven days. Other reports of serious adverse events to the FDA for drugs with a similar half life have used a discontinuation period of seven days.21

In consideration of the low event rate and imbalanced study designs, we summarised the study effects with the risk difference, an estimate of absolute effect. Summaries based on the risk difference are easily interpretable and ideal for conveying the clinical effect of a treatment. The risk difference provides an unbiased estimate of treatment effect, has been shown to be particularly appropriate in examining rare event data, and can accommodate trials with no events.2223 For comparison, we also summarised the study effects with the relative risk, Mantel-Haenszel odds ratio, and Peto odds ratio.

Methods

Literature search

Our systematic review entailed computer based searches of Medline, the Cochrane Library, and online clinical trials registries (ClinicalTrials.gov and the industry sponsored Clinical Study Results registry) to identify randomised controlled trials evaluating varenicline for tobacco cessation. The search covered January 2005 (the year when articles on varenicline were first published) to September 2011, and included articles available online ahead of print publication. The search strings were “varenicline and randomised” in Medline and “SR-tobacco and varenicline” in the Cochrane Central Register of Controlled Trials. We manually searched bibliographies of relevant research and review articles.

Studies included in the meta-analysis met the following criteria: randomised controlled design, study sample of current tobacco users of adult age, comparing use of varenicline with that of an inactive control, and reporting adverse events. Exclusion criteria included use of a quasi-experimental or crossover design, laboratory studies with no follow-up, studies with adolescents, studies of non-smokers, studies in which all participants received varenicline, and comparisons of varenicline with another active drug (for example, nicotine replacement). Study inclusion criteria, data extraction, and methods of the analysis were specified in advance and documented in a protocol.

We obtained details on timing of the cardiovascular serious adverse events from the study publications; the online clinical trials registries; the Chantix product label; and for three trials, for which the information was not publicly available, we contacted the medical adviser for varenicline at Pfizer1124 or the study lead author directly.25 For one additional study, we contacted the lead author to confirm that no serious adverse events had occurred in the trial because only adverse events were reported.26 Discrepant findings between reviewers were settled by discussion, further review of the article, and (if necessary) consultation with a third reviewer.

We assessed study quality using a three item method developed by Jadad and colleagues27 that evaluated adequacy of randomisation, concealment of randomisation, and completeness of follow-up. We assessed two additional items, relevant to the area of interest: adequate reporting and adjudication of the cardiovascular serious adverse events. No quality scoring system has proven to correlate consistently with treatment outcomes, and it is recognised that general quality scales often need to be supplemented with more problem specific items for each particular meta-analysis.28

Statistical analysis

We described trial characteristics in terms of publication date; sample size and allocation; and participants’ exposures to tobacco, cardiovascular disease, and study treatments. For each trial included in the meta-analysis, we cross classified study participants by treatment group and by any occurrence of a treatment emergent, cardiovascular serious adverse event. For trials that examined multiple doses of varenicline, we combined the active treatment groups.

Using Comprehensive Meta-Analysis (version 2 professional edition; Biostat) and fixed effects estimation, we summarised evidence of an increased risk of cardiovascular serious adverse events associated with varenicline use via four summary statistics: the Mantel-Haenszel versions of the risk difference, relative risk, and odds ratio; and the Peto odds ratio.2930 For each statistic, we reported the mean effect, 95% confidence interval, P value testing the null hypothesis of no effect, and I2 statistic estimating heterogeneity across trials.31 We confirmed all results using the “metan” and “funnel” routines added to Stata version 11. The three relative statistics excluded trials in which event counts were zero in both arms.32 For trials in which one event count was equal to zero, both software packages (Stata and Comprehensive Meta-Analysis) added 0.5 to each of the four cell counts before estimating the relative risk or Mantel-Haenszel odds ratio.

We planned to do random effects meta-analyses and subgroup analyses if levels of heterogeneity exceeded 50%. We plotted the cumulative evidence by date of trial publication33 and examined asymmetry in a funnel plot as an indicator of publication bias. Finally, we compared the four summary statistics at the trial level grouping the studies by presence (v absence) of events and equal (v unequal) numbers of events, ordering the groups by increasing evidence of a varenicline effect. We expected this comparison would provide insight into the most suitable summary statistic in the present setting.

Results

Study characteristics

The Medline search yielded 133 citations; of these, 21 met the inclusion criteria, which were all published in 2006 or later (fig 1⇓). We identified one additional unpublished study from the online clinical trials registries. Searching in the Cochrane Central Register of Controlled Trials yielded 83 results which, along with manual searches of the bibliographies of relevant research and review articles, did not yield any additional studies.

In total, we identified 22 randomised controlled trials of varenicline use for tobacco cessation; all were double blind and placebo controlled, and collectively included 9232 participants (5431 randomised to varenicline, 3801 to placebo; table 1⇓). The median overall sample size was 404 (range 31-1210). Seven trials had notable imbalances in sample size by condition—four allocated participants to varenicline and placebo groups in a 2:1 ratio, and three studied multiple varenicline doses or regimens. In 21 trials, the varenicline dose was 1 mg twice daily; three of these trials also studied lower doses of varenicline (table 1). The median duration of study treatment was 12 weeks, the median duration of follow-up for treatment emergent, cardiovascular serious adverse events was 16 weeks, and the median duration of the study period after randomisation was 25 weeks.

Table 1

Characteristics of randomised controlled trials of varenicline included in the meta-analysis

The samples tended to have a majority of male and white participants. Two trials studied smokeless tobacco users2434 and 20 studied cigarette smokers. Among the trials of cigarette smokers, participants averaged 21.5 cigarettes (standard deviation 1.9) per day at study screening and 25.1 years (6.3) of tobacco use. Thirteen trials included patients with current or past cardiovascular disease. Of these trials, one was conducted among smokers admitted to hospital, of whom 57% had an admitting diagnosis that was cardiovascular25; another was conducted specifically among people with stable cardiovascular disease11; and 11 trials included people with a past cardiovascular event (table 1). Nine trials excluded people with any history of cardiovascular disease or the timing for exclusion was not specified. Study quality was strong overall, with all 22 trials being of double blind design and providing adequate descriptions of randomisation, loss to follow-up, and cardiovascular serious adverse events. However, only one trial adjudicated the serious adverse events.11

Risk of treatment emergent, cardiovascular serious adverse events

Across the 22 studies, the crude rates of treatment emergent, cardiovascular serious adverse events were 0.63% (34/5431) for the varenicline group and 0.47% (18/3801) for the placebo group. No events occurred in eight trials, including three trials with more than 100 participants per arm. The summary risk difference was 0.27% (−0.10% to 0.63%, P=0.15, I2=0%; fig 2⇓), with no indication of publication bias in the funnel plot. For comparison, based on 14 studies with at least one event, the relative risk was 1.40 (0.82 to 2.39, P=0.22, I2=0%; table 2⇓), the Mantel-Haenszel odds ratio was 1.41 (0.82 to 2.42, P=0.22, I2=0%), and the Peto odds ratio was 1.58 (0.90 to 2.76, P=0.11, I2=0%).

Neither the individual trials nor the summary estimates showed a significant treatment effect; consequently, we did not calculate the number needed to harm.35 We found no evidence of heterogeneity according to the I2 statistic or the cumulative estimated effect of varenicline on cardiovascular serious adverse events (fig 3⇓).

We conducted four sensitivity analyses excluding trials of participants with active cardiovascular disease1125 (risk difference 0.29% (95% confidence interval −0.04 to 0.62), I2=0%); trials of smokeless tobacco users2434 (0.31% (−0.07 to 0.69), I2=0%); one trial in which all participants were initially exposed to varenicline and then randomised to placebo or to a 12 week maintenance phase of varenicline36 (0.26% (−0.16 to 0.67, I2=0%)); and one unpublished trial40 (0.27% (−0.10 to 0.64, I2=0%)). All four sensitivity analyses differed minimally from the full analysis.

After grouping the trials by presence (v absence) of events and equal (v unequal) numbers of events, we obtained five groups (table 2, fig 2). Group 1 included the one trial with more cardiovascular serious adverse events in the placebo arm than in the varenicline group. Group 2 had no events in either arm. Group 3 had an equal number of events per arm. Group 4 had one or more events in the varenicline arm but none on the placebo arm. Finally, group 5 had events occurring in both arms, with more occurring in the varenicline arm.

For group 2 (no event on either arm; eight trials), “no effect” was estimated by a risk difference of 0%, but not by relative effects equal to 1, because the relative summary statistics exclude trials with no events. For groups 3 and 5 (at least one event on each arm; six trials), the evidence for and against the null hypothesis of no treatment effect was similar across the four statistics in group 3, but the Peto odds ratio seemed to underestimate treatment effects in group 5.

For every trial in groups 1 and 4 (eight trials), in terms of both mean effects and confidence intervals, the Peto odds ratio was far stronger than the relative risk, but the Mantel-Haenszel odds ratio was nearly identical to the relative risk. In all eight trials, the Peto odds ratio exceeded 3.4 (or its inverse) despite very few events, large sample sizes, and differences of only one to two events by arm. For example, the trial by Bolliger and colleagues37 saw one event among 394 participants receiving varenicline (0.25%) compared with no events among 199 participants receiving placebo (0%) (risk difference 0.25%, relative risk of 1.52, Mantel-Haenszel odds ratio of 1.52, Peto odds ratio of 4.50).

Discussion

This meta-analysis included all published randomised, placebo controlled trials of varenicline for tobacco cessation, examined events occurring during drug exposure or within 30 days of discontinuation, and analysed findings via four summary measures. None of these measures identified a significantly elevated risk of treatment emergent, cardiovascular serious adverse events with varenicline use, and the 95% confidence interval of the risk difference excluded an increase larger than 0.63%. In response to the FDA’s call for analysis of cardiovascular serious adverse events attributed to varenicline use, this meta-analysis of 22 independent trials and more than 9000 individuals had high power to detect a significant treatment effect and found negligible variation in the evidence across the trials.

Study participants tended to be chronic, heavy smokers, averaging more than one pack of cigarettes a day for more than two decades, placing them at elevated risk of serious adverse events related to cardiovascular disease. Most trials included individuals with current (two trials) or past (11 trials) cardiovascular disease. More than a third of studies did not observe a cardiovascular serious adverse event; among these trials, five of eight included participants with past cardiovascular disease.3440414446

Comparison with other studies

In their meta-analysis of the safety of varenicline, Singh and colleagues reported a Peto odds ratio of 1.72 (95% confidence interval 1.09 to 2.71) and concluded that “The use of varenicline among tobacco users was associated with a 72% increased risk of serious adverse cardiovascular events.” They questioned the safety of this medication,13 and in subsequent press interviews called for withdrawal of varenicline from the market.38 The researchers’ analysis of cardiovascular serious adverse events at any time during the trial duration was, on average, twice the duration of study drug exposure and did not account for longer follow-up in the varenicline group than in the placebo group.

The discrepancy between the conclusions of our meta-analysis and those of Singh and colleagues’ study is explained not only by differing periods over which events were collected but also by the choice of statistics used to summarise the results, which affected the trials included in the meta-analyses. For direct comparison in our study, we analysed data from all 22 trials using the full study follow-up, because the observation period of interest and the calculated risk difference was 0.47% (95% confidence interval 0.04 to 0.91), far less inflammatory than the previous meta-analysis’s reported risk based on the Peto odds ratio, yet still biased in favour of finding an effect due to the differential inclusion of events related to disease and not just treatment. By contrast, our calculated risk difference of treatment emergent, cardiovascular serious events was 0.27% (−0.10 to 0.63).

Choice of summary statistics

When study participants are selected by outcome status (for example, case or control), an odds ratio must be used to summarise an association with exposure status. However, when participants are selected by exposure status (for example, active or placebo treatment), a risk difference or relative risk can be used. These statistics are more natural choices for randomised trials because they explicitly estimate and contrast effects of interest—namely, event rates in the active and placebo arms. Treatment effects based on relative risks always are as or less extreme than those based on odds ratios.30 The (absolute) risk difference has a further advantage because it can be calculated for trials in which zero events occur, whereas relative statistics cannot be calculated for these trials and therefore can bias summaries against the null hypothesis of “no effect.”

Furthermore, relative statistics are unitless, which hides the fact that a low response rate remains very low even when scaled up by a seemingly large effect; by contrast, the risk difference retains the units of the measurement scale, showing that a difference between low response rates is itself very small. Vandermeer and colleagues’ comprehensive reanalysis of findings from 1613 meta-analyses of safety data indicated that the Peto odds ratio statistic was particularly biased.22 We further demonstrate that, regardless of sample size allocation, when all events are in one study arm, Mantel-Haenszel odds ratios match relative risks well whereas Peto odds ratios are far more extreme. For clinical considerations and in the setting of rare events, the risk difference most clearly conveys the relevant effect.222339

Conclusions and clinical implications

Meta-analysis is an important analytical technique for synthesising treatment effects across trials for maximum power and is particularly useful for analysis of serious adverse events, which can occur with low frequency. Bias in methods, however, is a real concern. Our comparison of four summary statistics identified conditions under which the Peto odds ratio produced extreme estimates that did not reflect the underlying event rates, and identified cases in which it produced smaller estimates than it theoretically should have done. Our results accord with other reports that the Peto statistic can lead to incorrect conclusions.2229 The consequence of inflated risk estimates, such as those from Singh and colleagues’ meta-analysis concerning the effect of varenicline on serious adverse events related to cardiovascular disease,13 can be unnecessary public alarm and real harm, since patients may discontinue their drug treatment out of fear of adverse effects and clinicians may recommend cessation treatments of reduced efficacy or discourage use of the drug treatment altogether.

Smoking is the leading preventable cause of death worldwide. Half of long term smokers die from their tobacco use, and smokers die from cardiovascular disease more than from any other cause.123 Varenicline is a first line treatment for quitting smoking, and quitting smoking is central to the prevention of cardiovascular disease. Our meta-analysis of treatment emergent, cardiovascular serious adverse events, with attention to bias and critical design issues, indicates that the risk of these events associated with varenicline use is small, and statistically and clinically insignificant.

What is already known on this topic

There have been drug safety concerns about the use of varenicline for tobacco cessation and the emergence of cardiovascular serious adverse events

However, this association has since been called into question, owing to less than optimal methodology used, and the US FDA has called for further analysis

What this study adds

Our meta-analysis of all published, randomised controlled trials of varenicline use for tobacco cessation included 50% more studies than a previous meta-analysis by Singh and colleagues; used an unbiased summary estimate and compared findings with three other estimates; and examined events that occurred during drug treatment, which is more biologically relevant and obviates problems with differential drop out

All four summary estimates indicated no significant increase in the risk of treatment emergent, cardiovascular serious adverse events attributed to varenicline use

Notes

Cite this as:BMJ 2012;344:e2856

Footnotes

We thank Romina Kim for assistance with data abstraction and data entry with the systematic review.

Contributors: JJP conceived of the study, conducted the literature search and systematic review, assisted with the analyses, and led writing of the manuscript. JFH led the data analyses and assisted with writing the manuscript. Both authors are study guarantors, and had full access to all of the data (including statistical reports and tables) in the study and can take responsibility for the integrity of the data and the accuracy of the data analysis.

Funding: JJP is funded by the National Institute on Drug Abuse (P50 DA09253) and the State of California Tobacco-Related Disease Research Program (17RT-0077) for the submitted work. The funding agencies had no role in the conduct of the research or preparation of the manuscript.

Competing interests: All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: this study received support from the National Institute on Drug Abuse and the State of California Tobacco-Related Disease Research Program; JJP is principal investigator on R01 MH083684 from the National Institute of Mental Health and an Investigator Initiated Research award from Pfizer (WS981308), and is a collaborator on R34 DA030538 from the National Institute on Drug Abuse and a Cahan Award from the Flight Attendant Medical Research Institute, all of which are tobacco control trials; JFH is coinvestigator on five randomised controlled trials funded by the National Institutes of Health and the Agency for Healthcare Research and Quality, none of which are relevant to the submitted work; JFH has had no relationship with any company that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

Ethical approval: Not required.

Data sharing: Statistical code and dataset available from the corresponding author.