Abstract

Objective To describe the process of developing and validating the Canadian Association of Paediatric Health Centres Trigger Tool (CPTT).

Methods Five existing trigger tools were consolidated with duplicate triggers eliminated. After a risk analysis and modified Delphi
process, the tool was reduced from 94 to 47 triggers. Feasibility of use was tested, reviewing 40 charts in three hospitals.
For validation, charts were randomly selected across four age groups, half medical/half surgical diagnoses, from six paediatric
academic health sciences centres. 591 charts were reviewed by six nurses (for triggers and adverse events (AEs)) and three
physicians (for AEs only). The incidence of trigger- and AE-positive charts was documented, and the sensitivity and specificity
of the tool to identify charts with AEs were determined. Identification of AEs by nurses and physicians was compared. The
positive predictive value (PPV) of each trigger was calculated and the ratio of false- to true-positive AE predictors analysed
for each trigger.

Results Nurses rated the CPTT easy to use and identified triggers in 61.1% (361/591; 95% CI 57.2 to 65.0) of patient charts; physicians
identified AEs in 15.1% (89/ 591, 95% CI 0.23 to 0.43). Over a third of patients with AEs were neonates. The sensitivity and
specificity were 0.88 and 0.44, respectively. Nurse and physician AE assessments correlated poorly. The PPV for each trigger
ranged from 0 to 88.3%. Triggers with a false/true-positive ratio of >0.7 were eliminated, resulting in the final 35-trigger
CPTT.

Conclusions The CPTT is the first validated, comprehensive trigger tool available to detect AEs in children hospitalised in acute care
facilities.

Introduction

International data demonstrate that the incidence of adverse events (AEs) in hospitalised adults ranges from 2.9% to 16.6%.1–12 Physiological and developmental differences in children could make them more vulnerable to harm, yet no data from methodologically
similar studies in full paediatric populations exist. Published AE rates in children range from 0.2 to 154 per 10 000 discharge
records,13 and 1 to 2.96 per 100 discharges.1415 Chart reviews targeting specific paediatric populations and specific types of AEs reveal much higher rates.16–18 Identifying a composite paediatric AE rate has been hampered by the absence of a single tool suitable for examining AEs across
paediatric populations and events.

No single method can identify all harm associated with patient care. Concurrent use of complementary methods in real time
and retrospectively for AE detection has been recommended.19–24 Trigger tool (TT) methodology, however, is considered the best single tool, given its tiered approach and sensitivity to
AE detection. It can be customised to diverse settings and, used consistently, can accurately measure harm over time.25 The use of screening criteria to identify charts with possible harm followed by an in-depth review for actual harm has been
used for decades, including the landmark Harvard Medical Practice Study.1–11 More recently, the Institute for Healthcare Improvement (IHI) has recommended use of their Global Trigger Tool (IHI-GTT)
for local quality-improvement activities.26 Although focused paediatric TTs have been described, there remains no TT validated to detect AEs across all age groups of
hospitalised children and youth.

There are growing efforts to improve patient safety in paediatrics, and quantification of the burden of iatrogenic harm in
children could catalyse awareness and stimulate changes in paediatric practice and healthcare policy, as occurred following
publication of adult AE rates. The Patient Safety Collaborative of the Canadian Association of Paediatric Health Centres established
a working group to develop a paediatric TT (CPTT) to detect AEs in children hospitalised in Canada.27 The goal was to develop a reliable and robust tool that could be used for both local quality-improvement activities and research
into the rate, incidence, and factors contributing to AEs across a large population. The rigorous processes used to develop,
test the feasibility of use of and validate the CPTT in detecting AEs are outlined in this paper.

Patients and methods

The early development stages of the CPTT have been previously reported.27 Briefly, five TTs identified through a detailed literature review and personal communications were adapted to the modular
format of the IHI-GTT. The Care, Medication, Surgical and Intensive care modules were retained, and two modules added: Laboratory
and Other. Screening criteria from all other TTs were mapped against this list, duplicate triggers were removed, and additional
triggers recommended and approved by consensus were added.

Each trigger in the resulting 94-trigger tool was evaluated for risk priority, frequency and detectability. Triggers above
a pre-established cut-off or mean risk score were retained. Using a modified Delphi process, the remaining triggers were evaluated
by a team of clinical paediatric patient safety experts who eliminated half, resulting in the preliminary 47-trigger CPTT.

Pilot testing of the CPTT

The CPTT was piloted in three acute care paediatric settings in Alberta: a stand-alone paediatric unit in a community hospital,
a paediatric hospital-within-a-hospital and a regional paediatric centre. At each site, five medical and five surgical charts
were selected for audit from fiscal year 2004 from each of three age groups: 29–365 days, 366 days to 5 years and over 5 years;
10 charts were randomly selected from the fourth age group of 0–28 days.

Two nurses visited the hospitals between June and August 2005, and independently reviewed all 40 charts at each hospital using
the draft trigger tool. Each nurse had undergone detailed chart abstraction training as carried out for the Canadian Adverse
Event Study,1 and reliability was tested using a set of standard charts. Reviewers were asked to record any difficulties with interpretation
or application of trigger definitions with particular emphasis on potential duplication or impracticality. Nurses were also
asked to identify any triggers they thought should be considered for exclusion by the Research Team. Their experience was
that the tool could be used to assess the charts, and no triggers were identified that could be eliminated. The κ statistic
for the measurement of inter-rater reliability (IRR) for triggers in 120 charts reviewed was 0.81, and the validation study
was commenced.

Definitions

Based on the definition used in previous national studies of AEs, an AE was defined as an unintended injury or complication
that results in disability at the time of discharge, death, prolonged hospital stay or subsequent hospitalisation, and is
caused by healthcare management.14671011 The AE had to occur within a 3-month period before or during the index admission and be detected during or within the 3-month
period before or after the admission. ‘Healthcare management’ included the actions of individual hospital staff, as well as
the broader systems and care processes, and included acts of omission and commission. To qualify as an AE, healthcare management
must have received a causation rating of at least 4 on a six-point scale, that is, greater than 50% likelihood of being caused
by healthcare management (box 1).

The tool was validated by separate nurse and physician reviews of the same charts of children hospitalised in six academic
paediatric health sciences centres (IWK Hospital Halifax; Children's Hospital of Winnipeg; Alberta Children's Hospital Calgary;
Stollery Children's Hospital Edmonton; Children's Hospital of Eastern Ontario Ottawa; SickKids Hospital Toronto) during the
fiscal year 2005/2006. Validation included the use of the TT on a sample of paediatric patient charts and the assessment of
those charts to determine if AEs were present.

Training for nurses and physicians

Two nurses from each of the six participating institutions, and three physicians from the validation study team, with at least
10 years' academic paediatric centre experience, attended a 2-day training session at the University of Toronto. A customised
study training manual and standard set of blinded paediatric hospital charts were used. Following training, nurses and physicians
each reviewed 20 standardised charts, and the IRR was calculated.

Data collection and IRR calculation

At each hospital, two nurses reviewed a total of 100 selected charts for evidence of triggers. Nurses determined the presence
of an AE based on the presence of an injury and harm associated with healthcare management and then assigned a National Coordinating
Council for Medication Error Reporting and Prevention (NCC-MERP) classification of harm for each event (table 1). Blinded to the nurses' results, one of the three physicians reviewed the same 100 charts at two sites and classified all
injuries as AEs or not, using the standard definition of AE used in other studies (table 1). Ten per cent of charts were reabstracted, and the IRR for the six nurse teams calculated.

Distribution of nurse and physician assessment of adverse events (AEs) as compared with the nurses' classification of harm
using the National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) scale

Validation study

Biostatistical consultation determined that 600 charts from six paediatric healthcare centres would enable the reliability
and validity of the tool to be established with a high degree of confidence (α 0.05; β 0.10). A standardised chart selection
algorithm was applied to the Discharge Abstract Database for all paediatric separations in participating hospitals for fiscal
2006. One hundred charts plus a 20% oversampling (n=120) to accommodate for missing charts, stratified by age group as described
above, were selected for review from those identified by the algorithm at each site.

Charts of children receiving all levels of inpatient care were eligible for inclusion. However, patients were excluded if
they were 18 years of age and older or their most responsible diagnosis at discharge was obstetrical or psychiatric. For patients
with more than one hospitalisation during fiscal year 2006, one stay was randomly selected to be reviewed for the study. No
more than 100 eligible charts were reviewed at any single centre.

Analysis

Descriptive statistics were used for the trigger frequency and the number of AEs identified. The validity of the CPTT was
judged by its sensitivity, that is, the percentage of trigger-positive charts that physicians judged to have an AE; and specificity,
that is, the percentage trigger-negative charts that had no AE according to physician review. The positive predictive value
(PPV) of each trigger, and the sensitivity and specificity of the overall tool were also determined.

Ethics

Research Ethics Board approval was obtained at each participating centre as well as the University of Toronto, and provincial
legislation regarding privacy and protection of evidence was respected.

Results of validation testing

After training for the validation study, the κ statistic for IRR for nurse-identified triggers and physician-identified AEs
was 0.62 and 0.57, respectively, and 0.67 for the IRR for nurses at each participating site. Five hundred and ninety-one of
the 600 charts reviewed by nurses were available for physician review; the others were misfiled or on the clinical wards where
the patient was currently hospitalised. Of the 591 charts reviewed by both nurses and physicians, 361 (61.1%; 95% CI 57.2
to 65.0) had at least one trigger: 148 (41%) had one, 145 (40.2%) had two, three or four, and 59 (16.3%) had five or more
triggers.

Physicians identified 340 injuries in 180 of the 591 charts. In 89 charts, the injuries met the criteria for AEs, for an AE
rate of 15.1% (95% CI 12.2% to 17.1%). The distribution of AEs by age and gender is displayed in table 2. Of the 89 patients with an AE, 37 (41.6%) were designated as resulting from acts of commission, 34 (38.2%) from acts of
omission and six events (6.7%) from both; and 12 (13.5%) were not categorised.

Sensitivity and specificity of the 47-Trigger Canadian Paediatric Trigger Tool

Nurses' assessment of harm and AEs varied by the classification method used, and also differed from physicians' assessment.
MDs found AEs in 34 charts that nurses assigned ‘No Harm’; otherwise, nurses were more likely to identify AEs in charts at
all levels of harm severity (table 1). Nurses and physicians agreed on the presence of an AE in 489 (82.7%) of the charts for a κ of 0.34 (95% CI 0.23 to 0.43).

The PPV (table 4) and ratio of false-positive to true positive predictors of an AE for each trigger were calculated. The PPV ranged from 0
to 83.3. Based on these results, and in order to improve the efficiency of the CPTT, 11 triggers were removed: eight with
a low predictive value and ratio of >0.7, and three that were never selected. A fourth unselected trigger, ‘wrong site surgery,’
was retained, and the two readmission triggers merged (C03 and C16). These revisions resulted in the final 35-trigger CPTT
(table 5).

Discussion

Several agencies, including the Agency for Healthcare Research and Quality, have argued that indicators needed to be developed
for specific populations, including paediatrics.28 We concluded that a robust TT, validated for paediatric use, was necessary for evaluating AEs and conducting a national paediatric
study in Canada. Unlike the IHI-GTT where individual modules were validated separately, the CPTT has been validated for use
as a single tool for all paediatric populations.

Similar to the other national studies of AEs, use of our tool calls for a two-stage review process: first, nurses use the
trigger tool to identify charts likely to have an AE, followed by physicians who review the triggered charts for AEs. However,
in order to establish the sensitivity of the nurse review, all charts in this study were reviewed independently for triggers
(by nurses) and for AEs (by physicians and nurses). This process is the first of its kind to establish a false-negative rate
for triggered charts, by identifying patients with AEs whose charts were not triggered. The number of false-negative charts
may be reduced through education highlighting use of the ‘other’ trigger by the nurses. Nurse and physician review of all
charts for AEs also demonstrated differences between these groups in designating AEs. Nurses and physicians assessed the charts
using somewhat different methods. Nurses reviewed charts using the CPTT and assessed all charts in terms of the NCC-MERP Harm
Scale and the presence of AEs,29 whereas physicians reviewed all charts for AEs only. These different methods mean their results are not strictly comparable.
However, the physicians read each case from admission to discharge, and based on these findings, it appears that physicians
may be better qualified to assess the presence of AEs. This difference between nurse and physician assessment of AEs is consistent
with previous findings,3031 but these findings should be explored further.

There are inherent limitations in TT methodology. The utility of any chart review is subject to the quality of the documentation.32 Like other screening tests, TTs have a high sensitivity and relatively low specificity. However, their purpose is to cull
charts unlikely to have an AE, leaving behind for physician review only those charts with a high probability of having an
AE. The IRR of identifying AEs through chart review is variable; previous studies reported κ scores ranging from −0.077 to
0.66.1–9161932–35 Combining ratings from more than one physician1936 and monitoring reviewer performance during the study with personal feedback may improve IRR.30 However, a recent report suggests that a review process involving two physicians per chart is no more reliable than only
one in detecting AEs.36 Additional training that includes a consensus approach on the number of AEs in the training charts has been used successfully.35 Finally, previous national studies of AEs have focused on those characterised by significant harm, that is NCC-MERP categories
F–I, underestimating the total burden of harm by disregarding events with temporary harm requiring intervention only (category
E events). Although achieving high IRR for category E can be challenging,35 capturing these events in future studies will result in a more comprehensive view of the nature of iatrogenic injury, and
permit comparisons with recent studies of AEs in children and youth that have used the expanded definition.1718

Despite these limitations, the use of a broad paediatric population sample to validate the CPTT permits its use in other acute
care paediatric institutions. With its final refinements, the validated CPTT offers the first comprehensive evidence-based
tool for assessing harm in hospitalised children and youth.

Conclusion

The CPTT is the first valid and reliable TT for detecting harm in children and youth of all ages hospitalised in acute care.
This 35-trigger tool is reliable and robust, and can be used in quality-improvement initiatives and for more rigorous research
agendas.

Future research should focus on improving the efficiency of the CPTT and investigating the differences between nurse and physician
assessments of AEs. The CPTT is likely to be of interest to researchers wishing to comprehensively study the epidemiology
and burden of iatrogenic harm in hospitalised children and youth, using a methodology similar to the previous national studies
of AEs in hospitalised adults. Such studies will enhance current efforts to raise the profile of paediatric patient safety
issues.

. Hospital staff should use more than one method to detect adverse events and potential adverse events: incident reporting,
pharmacist surveillance and local real-time record review may all have a place. Qual Saf Health Care2007;16:40–4.