Menu

Sections

Did you know? You can easily access AAP Policy Statements, Clinical Practice Guidelines, and Clinical and Technical Reports with the AAP Policy & Collections link in the black header bar above (or the 3-line menu button on a mobile device), or by clicking here.

Abstract

OBJECTIVE: To describe the Vaccine Safety Datalink (VSD) project's experience with population-based, active surveillance for vaccine safety and draw lessons that may be useful for similar efforts.

PATIENTS AND METHODS: The VSD comprises a population of 9.2 million people annually in 8 geographically diverse US health care organizations. Data on vaccinations and diagnoses are updated and extracted weekly. The safety of 5 vaccines was monitored, each with 5 to 7 prespecified outcomes. With sequential analytic methods, the number of cases of each outcome was compared with the number of cases observed in a comparison group or the number expected on the basis of background rates. If the test statistic exceeded a threshold, it was a signal of a possible vaccine-safety problem. Signals were investigated by using temporal scan statistics and analyses such as logistic regression.

RESULTS: Ten signals appeared over 3 years of surveillance: 1 signal was reported to external stakeholders and ultimately led to a change in national vaccination policy, and 9 signals were found to be spurious after rigorous internal investigation. Causes of spurious signals included imprecision in estimated background rates, changes in true incidence or coding over time, other confounding, inappropriate comparison groups, miscoding of outcomes in electronic medical records, and chance. In the absence of signals, estimates of adverse-event rates, relative risks, and attributable risks from up-to-date VSD data have provided rapid assessment of vaccine safety to policy-makers when concerns about a specific vaccine have arisen elsewhere.

CONCLUSIONS: Care with data quality, outcome definitions, comparison groups, and length of surveillance are required to enable detection of true safety problems while minimizing false signals. Some causes of false signals in the VSD system were preventable and have been corrected, whereas others will be unavoidable in any active surveillance system. Temporal scan statistics, analyses to control for confounding, and chart review are indispensable tools in signal investigation. The VSD's experience may inform new systems for active safety surveillance.

The importance of active postmarketing surveillance for vaccine and drug safety is broadly recognized,1,–,3 and the Food and Drug Administration Amendments Act of 2007 mandated the creation of a national electronic system for actively monitoring medical-product safety. The Vaccine Safety Datalink (VSD) project, sponsored by the Centers for Disease Control and Prevention, conducts near–real-time, population-based, active surveillance for vaccine safety.4,5 Beginning in 2006, by using weekly updated longitudinal data on millions of patients, the VSD has performed weekly surveillance for selected outcomes after all recently introduced routinely administered vaccines. One safety problem has been detected. In this article, we describe the experience with this surveillance system and synthesize lessons that may be useful for future surveillance efforts. One of the most important lessons is that when potential associations between exposures and adverse events are identified, the surveillance team needs to investigate further to differentiate between those that represent true increases in risk and those that do not.

METHODS

Population and Data

The VSD comprises a total population of 9.2 million people annually and has an annual birth cohort of ∼95 000.6 The 8 health care systems participating in the VSD are Group Health Cooperative (Washington State), Harvard Vanguard Medical Associates and Harvard Pilgrim Health Care (Massachusetts), HealthPartners (Minnesota), Kaiser Permanente of Colorado, Kaiser Permanente of Northern California, Kaiser Permanente of Southern California, Marshfield Clinic (Wisconsin), and Northwest Kaiser Permanente (Oregon and Washington).

Every week each VSD site extracts electronic data on vaccinations and outpatient and inpatient diagnoses. Programs are then run to aggregate the relevant data for each vaccine study into counts of each of the various prespecified outcomes that occur during the prespecified postvisit observation windows according to vaccination status, visit week, patient age, patient gender, and site. This aggregation step protects data confidentiality, because individual-level data do not need to be transferred beyond the participating health care systems. The aggregate files are transferred to the data-coordinating center and analyzed weekly. Additional information is available through medical-record abstraction and in historical electronic data.

The VSD also conducts influenza vaccine-safety surveillance, which entails special approaches to deal with the challenges presented by the short time frame for vaccination, the seasonality of vaccine administration, and likely confounding by indication. This work is beyond the scope of this article but has been discussed in depth by Greene et al.10

Comparison Groups and Sequential Statistical Methods

When performing weekly analysis, sequential statistical methods are used to adjust for the multiple testing inherent in the repeated examinations of the data. Depending on the comparison group, we use slightly different methods.

Historical Controls

One approach is to use historical background rates, based on either the number of adverse events after a comparison vaccine, which adjusts for the healthy-vaccinee effect, or population-based incidence rate estimates. For each week, the number of observed adverse events after vaccination is compared with the expected number of events based on the number of doses administered times the probability of an adverse event as calculated from the comparison group and is typically adjusted for VSD site, age, and/or gender. We then use a Poisson-based maximized sequential probability ratio test (maxSPRT),11 which is a modified version of the classical sequential probability ratio test proposed by Wald.12 Each week, the cumulative data are used to construct a likelihood ratio test statistic, which contrasts the likelihood of increased risk (relative risk > 1) with the likelihood of no excess risk (relative risk = 1). If the likelihood ratio test statistic exceeds a threshold (“critical value”), we reject the null hypothesis and declare that a signal has occurred. Such a signal can be the result of either a real problem with the vaccine or another cause (discussed below). When a prespecified upper limit on the length of surveillance is reached without a signal, we stop surveillance.

When large amounts of historical data on a population comparable to our study population are available and the background incidence is constant across sites and years, then the expected numbers can be calculated with great precision, and we ignore the variability in the estimate. However, when the number of cases in the historical data is small (specifically, <5 times the number of cases expected in the surveillance period), then such an assumption leads to a nonnegligible bias toward signaling. We developed a conditional Poisson-based maxSPRT to take the error in the estimated expected number of events in this situation into account.13 This method was not available when VSD active surveillance began.

Self-controls

We have also used a self-control period (eg, the 2–8 weeks before vaccination). We then compare the number of adverse events in the postvaccination window with the number in the comparison window. The number of events of each type is tabulated each week, after which we apply a binomial-based maxSPRT11 with a 1:1 ratio when the windows are of equal length and a different ratio proportional to the lengths when the windows are of unequal length. This method is used in the VSD's influenza vaccine-safety studies10 (not described in this article) to adjust for between-person biases.

Concurrent Matched Controls

In some vaccine studies, we use concurrent matched control visits, which are typically patient visits when other vaccines are received or simply preventive care visits by patients of the same age during the same week. If we specify the number of controls per vaccine recipient in advance, we can use the binomial-based maxSPRT, but that leads to a loss of information because not all potential controls are used. Instead, we use exact sequential analysis, which allows for a variable number of matched controls per person each week.14 This method had not yet been developed when VSD active surveillance began.

For most studies, real-time weekly surveillance began well after the start of vaccine delivery. The period between the data start date (the beginning of vaccine uptake in the VSD) and the surveillance start date was not uniform. At the start of surveillance, a first set of sequential analyses was performed retrospectively, which showed what would have happened if the surveillance had been performed on a weekly basis from the beginning.

Signal Investigation

Before launching the surveillance system, we developed a sequence of steps to be followed in the event of a signal, which has guided our investigations to the present. In studying the signals described here, we first considered the possibility of data-quality problems or errors in the background incidences. If there were enough cases, we used temporal scan statistics available with SaTScan software15 to determine if outcomes clustered during the postvaccination observation window, which would have given credence to a vaccine-related physiologic process. If suspicion remained and there were enough cases, we conducted additional analyses, such as logistic regression with concurrent and/or historical controls, to adjust not only for age and site but also for additional potential confounders such as seasonal trends or concomitant vaccines. In some instances, cases were confirmed or ruled out by chart review; with rare outcomes (few cases), chart review was possible to do early in the investigation. A more complete list of signal-evaluation techniques is shown in Table 2.16

RESULTS

Between 2006 and 2009, we monitored the safety of 5 vaccines, not counting influenza, each with 5 to 7 outcomes (Table 1). Of the 30 vaccine-outcome pairs, 21 generated no signal, which provides some assurance of vaccine safety. One signal was reported to external stakeholders and ultimately led to a change in national vaccination policy. Nine signals (2 of which were in the same vaccine-outcome pair) were investigated internally, and routine data-checking and other investigation determined them to be spurious. The 10 signals, all of which appeared in Poisson-based maxSPRT analyses, are described below.

MMRV Vaccine and Seizure

The seizure signal has been intensively studied and was reported separately.8,17 In brief, a temporal scan statistic showed highly significant clustering of seizure cases after MMRV vaccination. Regression analysis of seizures risk in the 7 to 10 days after vaccination revealed a twofold increased risk of seizure after MMRV vaccination compared with same-day administration of separate measles-mumps-rubella (MMR) and varicella vaccines. The findings were reported to the Centers for Disease Control and Prevention and the Advisory Committee on Immunization Practices (ACIP) before publication.

MMRV Vaccine and Meningitis/Encephalitis

Two apparent cases of the prespecified outcome meningitis/encephalitis occurred after MMRV vaccination within the seizure risk window. Chart review by 2 independent experts, using the Brighton Collaboration case definition of encephalitis,18 determined that 1 case fit the least stringent level of diagnostic certainty and 1 was not a case. The 1 confirmed case could have been a result of chance.

MMRV Vaccine and Ataxia

One site had ∼3 times as many cases of ataxia after MMRV vaccination as were expected on the basis of its 2005–2006 post-MMR vaccine incidence. After chart review at that site, only 1 of the 21 cases was found to be a true ataxia episode. Miscoding of disparate gait problems as ataxia evidently began at this site in mid-2006, after MMRV vaccination had come into use. Because the miscoding could not be corrected, the site was excluded from analyses of ataxia. The signal did not recur.

MMRV Vaccine and Thrombocytopenia

The background rate for thrombocytopenia was calculated from historical VSD person-time of 1-year-olds in general instead of person-time after MMR vaccination because of its low incidence rate. A signal was generated with 5 cases. By the time of the signal, a VSD article had been published in which an increased risk of idiopathic thrombocytopenia purpura after MMR vaccination was noted,19 and an incidence in the 42 days after MMR vaccination in 1-year-olds was available using a definition with the same platelet cutoff value as ours. When we substituted the more appropriate post-MMR vaccine rate, which was approximately twice the original one, there was no signal, a finding consistent with a similar risk of thrombocytopenia after MMRV as after MMR vaccination.

RotaTeq and Gastrointestinal Bleeding, First Signal

A gastrointestinal bleeding signal appeared in the first RotaTeq analysis. The original historical baseline incidence of gastrointestinal bleeding had been calculated by using data from infants aged 4 to 52 weeks during 2000–2004 without adjusting for age. Because the risk of gastrointestinal bleeding decreases with age in the first year of life and RotaTeq vaccination is given at ages 2, 4, and 6 months, the signal had resulted from confounding by age. When the analysis was adjusted according to week of age, the signal disappeared.

RotaTeq and Gastrointestinal Bleeding, Second Signal

Three months after the first gastrointestinal signal and after the age adjustment had been implemented, a second signal arose. Logistic regression analysis in which infants with visits for other vaccines were used as a concurrent comparison group and adjusting for age, site, and week of vaccination revealed no difference in risk between the RotaTeq and other vaccine groups (odds ratio: 1.1 [95% confidence interval: 0.87–1.42]). The source of error or bias in the background rates was not definitively determined, although later analyses of rates in data through 2007 showed an increasing secular trend according to year.

RotaTeq and Meningitis/Encephalitis

Two case records of meningitis/encephalitis from 1 site entered the data set late, ∼1 year after the cases had occurred. Their appearance produced a retrospective, transient signal that included these and 2 additional cases. By the time of the first analysis that included the 2 late cases and revealed the retrospective signal, the relative risk had decreased from 22.5 on the date of the signal to 1.2 because of the accrual of additional RotaTeq doses without a proportional accrual of more post-RotaTeq meningitis/encephalitis cases (just 1 additional case). As a result, the test statistic had decreased to well below the threshold. The signal was ascribed to chance.

HPV Vaccine and Appendicitis

This retrospective, transient signal, ascertained in the first HPV vaccine analysis, was the result of a single case that happened in week 2 of the data, at which time only 26 HPV vaccines had been administered to the study population and the expected number of cases was only 0.009 (Fig 1). The test statistic and relative risk decreased to null values during the next 16 weeks. We concluded that the evidence was insufficient to support an association between HPV vaccine and appendicitis.

Progression of test statistic (log-likelihood ratio) and relative risk of appendicitis after HPV vaccine in 18- to 26-year-old women. The signal appeared because of the very early occurrence of the first case and the practice of simulating weekly surveillance in already elapsed time.

HPV Vaccine and Allergic Reactions

A signal for allergic reactions also appeared in the first HPV analysis. The historical incidence was based on only 14 cases, and the prespecified upper limit of surveillance was 40 expected cases, whereas the ratio of the first to the second, ideally, should be at least 5. Thus, the regular Poisson-based maxSPRT analysis was biased toward signaling because of uncertainty in the background incidence. When the conditional Poisson-based maxSPRT was implemented to adjust for this uncertainty, no signal appeared.

MCV4 and GBS

On the basis of computerized International Classification of Diseases, Ninth Revision codes, 5 apparent cases of GBS generated a signal after vaccination with MCV4. Each of the GBS cases had been reviewed after its appearance, and none represented a true instance of new onset of GBS symptoms during the observation window of days 1 through 42. Two were instances of follow-up for preexisting GBS; 1 had not been diagnosed with GBS but rather with a different neurologic syndrome that did not meet our case definition; 1 was an instance of rule-out of a differential diagnosis of GBS; and 1 had symptom onset on day 0 and, thus, was not considered plausibly related to vaccination.

In summary, the reasons for the 10 signals fell into 1 of 7 categories: (1) a confirmed vaccine-safety problem (MMRV-seizure); (2) temporal changes in incidence or coding (possibly the second RotaTeq gastrointestinal bleeding signal); (3) other confounding (by age: the first RotaTeq-gastrointestinal bleeding signal); (4) inappropriate comparison group (MMRV-thrombocytopenia); (5) miscoding of outcomes in the electronic medical record (MMRV-ataxia, MCV4-GBS); (6) bias attributable to uncertainty in the estimated background rates (HPV-allergic reactions); or (7) chance (RotaTeq-meningitis/encephalitis, HPV-appendicitis, possibly MMRV-encephalitis). Some of the dismissed signals could have been avoided had we known from the beginning what we know now (categories 3, 4, and 6), whereas others would have been hard to avoid (categories 2, 5, and 7).

The system also allows immediate estimation of adverse-event rates, relative risks, and attributable risks from up-to-date data. When concerns about the safety of a specific vaccine have arisen from another source, the absence of signals in our system has provided rapid reassurance about vaccine safety. For example, when an association between MCV4 and GBS was suggested by the Vaccine Adverse Event Reporting System spontaneous reporting system, VSD data on number of cases and number of doses administered, which did not support an association, were useful.20

DISCUSSION

The VSD project has created a real-time active surveillance system that can detect real safety problems and in which spurious signals can be investigated and ruled out internally without generating false alarms to the public. Our experience offers lessons that may be useful to others who are establishing similar systems.

Data Quality

Even with semiautomated procedures, there are multiple opportunities for glitches in data-processing and other errors, so data quality must be checked and maintained weekly. We check the quality of the patient-level data extracts and aggregate files and exclude faulty aggregate files from the analyses. When feedback from data-quality checks is supplied to central and site-based data-managers in a timely way, problems can often be addressed before the next weekly data feed.

At some sites, several weeks are required for data to “settle” (eg, for exposure and diagnostic data to enter the record). Therefore, to avoid complications entailed in reanalyzing already analyzed data and interpreting discrepant results of multiple analyses, we routinely exclude the most recent 14 weeks of data (except in influenza vaccine-safety surveillance), which corresponds to the usual 42-day follow-up period plus 8 weeks of settling. The loss in timeliness of signal detection is considered a necessary trade-off for the greater validity of the results. Sites that do not own their own hospitals rely on claims data for hospital-based diagnoses, and these data often take more than 8 weeks to arrive. A set of procedures for dealing with late corrections to the data has been developed.

Exposures

Some vaccines are typically given with other vaccines, which in theory could complicate the identification of the cause of a signal. In our weekly analyses we do not exclude visits with concomitant vaccines or attempt to adjust for these additional vaccines. Such adjustment can be handled by logistic regression or other analyses.

Some vaccines involve multiple doses, whereas others are given as a single dose. For most multidose vaccines, we treat all doses as equal. For RotaTeq and intussusception, analyses were also stratified according to dose, because the risk from the previously licensed rotavirus vaccine was greatest after the first dose.9

Outcomes

Diagnostic codes in electronic data cannot confidently be interpreted as reflecting actual outcomes until validated. The degree to which electronic codes diverge from truth can differ because of local coding conventions and can change over time, as demonstrated by the MMRV-ataxia signal. As Brown et al21 have observed, using validated outcome definitions would be ideal. Validated definitions are somewhat scarce, although the VSD has produced studies on the positive predictive value of some individual codes and outcome definitions.22,23 In the absence of published studies and chart review, we typically finalize outcome definitions only after examining historical data and checking code frequencies in various settings. One objective in defining outcomes is to separate rare, specific conditions from common conditions (eg, anaphylaxis and allergic reactions are generally treated as separate outcomes).

Consideration of the dismissed MCV4-GBS signal and preparation for influenza A (H1N1) 2009 monovalent vaccine-safety surveillance led us to seek rates of chart-confirmed GBS in our historical data against which to compare any chart-confirmed GBS after a vaccine of interest. Having these rates in hand will aid in the evaluation of any GBS signal that might arise in the sequential analysis of electronic data. However, chart review of historical data is too labor-intensive and time-consuming to feasibly perform for all outcomes.

The length of observation time after a vaccination or comparison visit, specified for each vaccine-outcome pair, must be chosen with care and based on considerations of clinical trials data and biological plausibility. The window must be long enough to include the plausible period of increased risk, but selecting a window that is too long could cause a true increase in risk to be obscured by random noise.

Comparison Groups

Historical and concurrent comparison groups have different sets of advantages and disadvantages. One advantage of using a historical comparison group is that it allows earlier signal detection after only a handful of adverse events. However, background rates may be uncertain (imprecise), vary over time, or introduce confounding. The conditional maxSPRT can adjust for uncertainty at the sacrifice of some power. Regarding confounding, we typically stratify background rates according to VSD site and sometimes age and calendar time, if warranted, but it is impossible to identify and adjust for all possible confounders in a surveillance setting. If a signal is generated and an association is suspected, an analysis such as logistic regression, adjusted for additional potential confounders, is conducted.

Using concurrent matched controls instead of historical controls avoids false signaling or missed signals caused by error in historical background rates or by secular trends. However, it is not always simple to define an appropriate control group. For instance, there can be fundamental differences between those who receive a particular vaccine and those who do not receive it, which may lead to confounding. In addition, vaccines are often adopted rapidly within VSD sites, which leaves few concurrent controls.

To benefit from the advantages and mitigate the respective disadvantages of historical and concurrent controls, sometimes we use both and conduct a primary and secondary analysis for the same vaccine-outcome pair. For example, in the HPV study, 6 comparison groups were used (see Table 1).

Combination vaccines are most appropriately compared with 1 or more of their older component vaccines (as demonstrated by the MMRV-thrombocytopenia signal), because the practical question being posed is whether the new combination vaccine is any riskier than the vaccines it is intended to replace.

Length of Surveillance

It is best to plan the duration of surveillance on the basis of statistical power. The ideal would be to continue surveillance until 90% power is achieved to detect the minimum absolute excess risk that is important for public health. Because of differences in frequencies, it is reasonable for the length of planned surveillance to differ according to outcome within the same vaccine study.

Ideally, surveillance continues until any important risk can be ruled out. However, in some of our early studies, the prespecified length of surveillance was set to be inappropriately short and the power may have been insufficient to detect a potential association between a vaccine and an adverse event of high public health importance. In these cases, such as GBS after MCV4, the VSD project elected to continue surveillance via weekly looks at the accumulating data while recognizing that an α expenditure of 5% would be exceeded (ie, the probability of a false signal would be >5%).

Interpretation of Signals

Spurious signals can arise for a variety of reasons. On no account should a signal be interpreted as indicating an association or causal relationship between vaccine and adverse event until confirmatory studies are conducted (see Table 2). With Poisson maxSPRT signals, an important early step is to scrutinize the incidences used in the calculation of the expected counts. If there are enough cases, the use of a temporal scan statistic on the frequency distribution of cases according to day of diagnosis after vaccination to check for clustering during the observation window can be performed quickly and is useful for determining biological plausibility. Ultimately, if enough cases and a suitable comparison group exist, we conduct additional analyses (such as logistic regression), adjusting for additional confounders. Chart review is important to confirm or rule out cases. If there are many charts to review, a sampling scheme can legitimately be used, particularly when the purpose of the chart review is to validate the findings from electronic data. If the outcome is quite rare, chart review as cases crop up can be useful in improving the timeliness of signal confirmation or rule-out.

CONCLUSIONS

The VSD's active surveillance system improves as we learn from experience and has already proven valuable. In 3 years of operation, we detected 1 vaccine-safety problem that led to a revised Advisory Committee on Immunization Practices recommendation for MMRV vaccine. Nine other signals were fully investigated and ruled out. Causes of these spurious signals were uncertainty in the estimated background rates, changes in true incidence or coding over time, other confounding, inappropriate comparison groups, miscoding of outcomes, and chance. Some of these causes are preventable and have been corrected in subsequent VSD analyses, whereas some will be problematic for any active surveillance system.

New systems for real-time active safety surveillance are being developed nationally,24 and similar issues seem likely to arise. The VSD's ongoing surveillance system, which continues to incorporate newly licensed routinely administered vaccines, provides an important and unique resource for vaccine and drug-safety efforts, policy, and public health.

ACKNOWLEDGMENTS

This work was supported by funding from the Centers for Disease Control and Prevention via America's Health Insurance Plans (contract 200-2002-00732).

FINANCIAL DISCLOSURE: Mr Lewis and Dr Klein have received research support from Merck, Wyeth (now Pfizer), Novartis, Sanofi Pasteur, GlaxoSmithKline, and MedImmune (now AstraZeneca), and all the authors receive funding from the Centers for Disease Control and Prevention as investigators or other staff of the Vaccine Safety Datalink project.

REFERENCES

Institute of Medicine, Committee on the Assessment of the US Drug Safety System. The Future of Drug Safety: Promoting and Protecting the Health of the Public. Washington, DC: Institute of Medicine; 2006

. An assessment of the safety of adolescent and adult tetanus-diphtheria-acellular pertussis (TdaP) vaccine, using active surveillance for adverse events in the Vaccine Safety Datalink. Vaccine. 2009;27(32):4257–4262