Description of the development and validation of the Canadian Paediatric Trigger Tool
- Anne G Matlow1,2,3,
- Catherine M G Cronin4,5,
- Virginia Flintoft6,
- Cheri Nijssen-Jordan7,
- Mark Fleming8,
- Barbara Brady-Fryer9,
- Mary-Ann Hiltz10,
- Elaine Orrbine11,
- G Ross Baker3,6
- 1Department of Paediatrics, Hospital for Sick Children, Toronto, Canada
- 2Department of Paediatrics, University of Toronto, Toronto, Canada
- 3University of Toronto Centre for Patient Safety, Toronto, Canada
- 4Department of Paediatrics and Child Health, University of Manitoba, Winnipeg, Canada
- 5Child Health Program, Winnipeg Regional Health Authority, Winnipeg, Canada
- 6Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Canada
- 7Department of Paediatrics, Alberta Children's Hospital, University of Calgary, Calgary, Canada
- 8Department of Psychology, Saint Mary's University, Halifax, Canada
- 9Grant MacEwan University, Edmonton, Canada
- 10IWK Health Centre, Halifax, Canada
- 11Canadian Association of Paediatric Health Centres, Ottawa, Canada
- Correspondence to Dr Anne Matlow, Infection Prevention & Control Programme, The Hospital for Sick Children, 555 University Avenue, Toronto, ON, Canada;
Contributors All authors contributed substantially to the study and write-up of the manuscript.
- Accepted 6 December 2010
- Published Online First 17 January 2011
Objective To describe the process of developing and validating the Canadian Association of Paediatric Health Centres Trigger Tool (CPTT).
Methods Five existing trigger tools were consolidated with duplicate triggers eliminated. After a risk analysis and modified Delphi process, the tool was reduced from 94 to 47 triggers. Feasibility of use was tested, reviewing 40 charts in three hospitals. For validation, charts were randomly selected across four age groups, half medical/half surgical diagnoses, from six paediatric academic health sciences centres. 591 charts were reviewed by six nurses (for triggers and adverse events (AEs)) and three physicians (for AEs only). The incidence of trigger- and AE-positive charts was documented, and the sensitivity and specificity of the tool to identify charts with AEs were determined. Identification of AEs by nurses and physicians was compared. The positive predictive value (PPV) of each trigger was calculated and the ratio of false- to true-positive AE predictors analysed for each trigger.
Results Nurses rated the CPTT easy to use and identified triggers in 61.1% (361/591; 95% CI 57.2 to 65.0) of patient charts; physicians identified AEs in 15.1% (89/ 591, 95% CI 0.23 to 0.43). Over a third of patients with AEs were neonates. The sensitivity and specificity were 0.88 and 0.44, respectively. Nurse and physician AE assessments correlated poorly. The PPV for each trigger ranged from 0 to 88.3%. Triggers with a false/true-positive ratio of >0.7 were eliminated, resulting in the final 35-trigger CPTT.
Conclusions The CPTT is the first validated, comprehensive trigger tool available to detect AEs in children hospitalised in acute care facilities.
International data demonstrate that the incidence of adverse events (AEs) in hospitalised adults ranges from 2.9% to 16.6%.1–12 Physiological and developmental differences in children could make them more vulnerable to harm, yet no data from methodologically similar studies in full paediatric populations exist. Published AE rates in children range from 0.2 to 154 per 10 000 discharge records,13 and 1 to 2.96 per 100 discharges.14 15 Chart reviews targeting specific paediatric populations and specific types of AEs reveal much higher rates.16–18 Identifying a composite paediatric AE rate has been hampered by the absence of a single tool suitable for examining AEs across paediatric populations and events.
No single method can identify all harm associated with patient care. Concurrent use of complementary methods in real time and retrospectively for AE detection has been recommended.19–24 Trigger tool (TT) methodology, however, is considered the best single tool, given its tiered approach and sensitivity to AE detection. It can be customised to diverse settings and, used consistently, can accurately measure harm over time.25 The use of screening criteria to identify charts with possible harm followed by an in-depth review for actual harm has been used for decades, including the landmark Harvard Medical Practice Study.1–11 More recently, the Institute for Healthcare Improvement (IHI) has recommended use of their Global Trigger Tool (IHI-GTT) for local quality-improvement activities.26 Although focused paediatric TTs have been described, there remains no TT validated to detect AEs across all age groups of hospitalised children and youth.
There are growing efforts to improve patient safety in paediatrics, and quantification of the burden of iatrogenic harm in children could catalyse awareness and stimulate changes in paediatric practice and healthcare policy, as occurred following publication of adult AE rates. The Patient Safety Collaborative of the Canadian Association of Paediatric Health Centres established a working group to develop a paediatric TT (CPTT) to detect AEs in children hospitalised in Canada.27 The goal was to develop a reliable and robust tool that could be used for both local quality-improvement activities and research into the rate, incidence, and factors contributing to AEs across a large population. The rigorous processes used to develop, test the feasibility of use of and validate the CPTT in detecting AEs are outlined in this paper.
Patients and methods
The early development stages of the CPTT have been previously reported.27 Briefly, five TTs identified through a detailed literature review and personal communications were adapted to the modular format of the IHI-GTT. The Care, Medication, Surgical and Intensive care modules were retained, and two modules added: Laboratory and Other. Screening criteria from all other TTs were mapped against this list, duplicate triggers were removed, and additional triggers recommended and approved by consensus were added.
Each trigger in the resulting 94-trigger tool was evaluated for risk priority, frequency and detectability. Triggers above a pre-established cut-off or mean risk score were retained. Using a modified Delphi process, the remaining triggers were evaluated by a team of clinical paediatric patient safety experts who eliminated half, resulting in the preliminary 47-trigger CPTT.
Pilot testing of the CPTT
The CPTT was piloted in three acute care paediatric settings in Alberta: a stand-alone paediatric unit in a community hospital, a paediatric hospital-within-a-hospital and a regional paediatric centre. At each site, five medical and five surgical charts were selected for audit from fiscal year 2004 from each of three age groups: 29–365 days, 366 days to 5 years and over 5 years; 10 charts were randomly selected from the fourth age group of 0–28 days.
Two nurses visited the hospitals between June and August 2005, and independently reviewed all 40 charts at each hospital using the draft trigger tool. Each nurse had undergone detailed chart abstraction training as carried out for the Canadian Adverse Event Study,1 and reliability was tested using a set of standard charts. Reviewers were asked to record any difficulties with interpretation or application of trigger definitions with particular emphasis on potential duplication or impracticality. Nurses were also asked to identify any triggers they thought should be considered for exclusion by the Research Team. Their experience was that the tool could be used to assess the charts, and no triggers were identified that could be eliminated. The κ statistic for the measurement of inter-rater reliability (IRR) for triggers in 120 charts reviewed was 0.81, and the validation study was commenced.
Based on the definition used in previous national studies of AEs, an AE was defined as an unintended injury or complication that results in disability at the time of discharge, death, prolonged hospital stay or subsequent hospitalisation, and is caused by healthcare management.1 4 6 7 10 11 The AE had to occur within a 3-month period before or during the index admission and be detected during or within the 3-month period before or after the admission. ‘Healthcare management’ included the actions of individual hospital staff, as well as the broader systems and care processes, and included acts of omission and commission. To qualify as an AE, healthcare management must have received a causation rating of at least 4 on a six-point scale, that is, greater than 50% likelihood of being caused by healthcare management (box 1).
Six-point scale for physician determination of causation2
Virtually no evidence for management causation.
Slight to modest evidence for management causation.
Management causation not likely; less than 50/50 but close call.
Management causation more likely; more than 50/50 but close call.
Moderate to strong evidence for management causation.
Virtually certain evidence for management causation.
The tool was validated by separate nurse and physician reviews of the same charts of children hospitalised in six academic paediatric health sciences centres (IWK Hospital Halifax; Children's Hospital of Winnipeg; Alberta Children's Hospital Calgary; Stollery Children's Hospital Edmonton; Children's Hospital of Eastern Ontario Ottawa; SickKids Hospital Toronto) during the fiscal year 2005/2006. Validation included the use of the TT on a sample of paediatric patient charts and the assessment of those charts to determine if AEs were present.
Training for nurses and physicians
Two nurses from each of the six participating institutions, and three physicians from the validation study team, with at least 10 years' academic paediatric centre experience, attended a 2-day training session at the University of Toronto. A customised study training manual and standard set of blinded paediatric hospital charts were used. Following training, nurses and physicians each reviewed 20 standardised charts, and the IRR was calculated.
Data collection and IRR calculation
At each hospital, two nurses reviewed a total of 100 selected charts for evidence of triggers. Nurses determined the presence of an AE based on the presence of an injury and harm associated with healthcare management and then assigned a National Coordinating Council for Medication Error Reporting and Prevention (NCC-MERP) classification of harm for each event (table 1). Blinded to the nurses' results, one of the three physicians reviewed the same 100 charts at two sites and classified all injuries as AEs or not, using the standard definition of AE used in other studies (table 1). Ten per cent of charts were reabstracted, and the IRR for the six nurse teams calculated.
Biostatistical consultation determined that 600 charts from six paediatric healthcare centres would enable the reliability and validity of the tool to be established with a high degree of confidence (α 0.05; β 0.10). A standardised chart selection algorithm was applied to the Discharge Abstract Database for all paediatric separations in participating hospitals for fiscal 2006. One hundred charts plus a 20% oversampling (n=120) to accommodate for missing charts, stratified by age group as described above, were selected for review from those identified by the algorithm at each site.
Charts of children receiving all levels of inpatient care were eligible for inclusion. However, patients were excluded if they were 18 years of age and older or their most responsible diagnosis at discharge was obstetrical or psychiatric. For patients with more than one hospitalisation during fiscal year 2006, one stay was randomly selected to be reviewed for the study. No more than 100 eligible charts were reviewed at any single centre.
Descriptive statistics were used for the trigger frequency and the number of AEs identified. The validity of the CPTT was judged by its sensitivity, that is, the percentage of trigger-positive charts that physicians judged to have an AE; and specificity, that is, the percentage trigger-negative charts that had no AE according to physician review. The positive predictive value (PPV) of each trigger, and the sensitivity and specificity of the overall tool were also determined.
Research Ethics Board approval was obtained at each participating centre as well as the University of Toronto, and provincial legislation regarding privacy and protection of evidence was respected.
Results of validation testing
After training for the validation study, the κ statistic for IRR for nurse-identified triggers and physician-identified AEs was 0.62 and 0.57, respectively, and 0.67 for the IRR for nurses at each participating site. Five hundred and ninety-one of the 600 charts reviewed by nurses were available for physician review; the others were misfiled or on the clinical wards where the patient was currently hospitalised. Of the 591 charts reviewed by both nurses and physicians, 361 (61.1%; 95% CI 57.2 to 65.0) had at least one trigger: 148 (41%) had one, 145 (40.2%) had two, three or four, and 59 (16.3%) had five or more triggers.
Physicians identified 340 injuries in 180 of the 591 charts. In 89 charts, the injuries met the criteria for AEs, for an AE rate of 15.1% (95% CI 12.2% to 17.1%). The distribution of AEs by age and gender is displayed in table 2. Of the 89 patients with an AE, 37 (41.6%) were designated as resulting from acts of commission, 34 (38.2%) from acts of omission and six events (6.7%) from both; and 12 (13.5%) were not categorised.
Nurses' assessment of harm and AEs varied by the classification method used, and also differed from physicians' assessment. MDs found AEs in 34 charts that nurses assigned ‘No Harm’; otherwise, nurses were more likely to identify AEs in charts at all levels of harm severity (table 1). Nurses and physicians agreed on the presence of an AE in 489 (82.7%) of the charts for a κ of 0.34 (95% CI 0.23 to 0.43).
The PPV (table 4) and ratio of false-positive to true positive predictors of an AE for each trigger were calculated. The PPV ranged from 0 to 83.3. Based on these results, and in order to improve the efficiency of the CPTT, 11 triggers were removed: eight with a low predictive value and ratio of >0.7, and three that were never selected. A fourth unselected trigger, ‘wrong site surgery,’ was retained, and the two readmission triggers merged (C03 and C16). These revisions resulted in the final 35-trigger CPTT (table 5).
Several agencies, including the Agency for Healthcare Research and Quality, have argued that indicators needed to be developed for specific populations, including paediatrics.28 We concluded that a robust TT, validated for paediatric use, was necessary for evaluating AEs and conducting a national paediatric study in Canada. Unlike the IHI-GTT where individual modules were validated separately, the CPTT has been validated for use as a single tool for all paediatric populations.
Similar to the other national studies of AEs, use of our tool calls for a two-stage review process: first, nurses use the trigger tool to identify charts likely to have an AE, followed by physicians who review the triggered charts for AEs. However, in order to establish the sensitivity of the nurse review, all charts in this study were reviewed independently for triggers (by nurses) and for AEs (by physicians and nurses). This process is the first of its kind to establish a false-negative rate for triggered charts, by identifying patients with AEs whose charts were not triggered. The number of false-negative charts may be reduced through education highlighting use of the ‘other’ trigger by the nurses. Nurse and physician review of all charts for AEs also demonstrated differences between these groups in designating AEs. Nurses and physicians assessed the charts using somewhat different methods. Nurses reviewed charts using the CPTT and assessed all charts in terms of the NCC-MERP Harm Scale and the presence of AEs,29 whereas physicians reviewed all charts for AEs only. These different methods mean their results are not strictly comparable. However, the physicians read each case from admission to discharge, and based on these findings, it appears that physicians may be better qualified to assess the presence of AEs. This difference between nurse and physician assessment of AEs is consistent with previous findings,30 31 but these findings should be explored further.
There are inherent limitations in TT methodology. The utility of any chart review is subject to the quality of the documentation.32 Like other screening tests, TTs have a high sensitivity and relatively low specificity. However, their purpose is to cull charts unlikely to have an AE, leaving behind for physician review only those charts with a high probability of having an AE. The IRR of identifying AEs through chart review is variable; previous studies reported κ scores ranging from −0.077 to 0.66.1–9 16 19 32–35 Combining ratings from more than one physician19 36 and monitoring reviewer performance during the study with personal feedback may improve IRR.30 However, a recent report suggests that a review process involving two physicians per chart is no more reliable than only one in detecting AEs.36 Additional training that includes a consensus approach on the number of AEs in the training charts has been used successfully.35 Finally, previous national studies of AEs have focused on those characterised by significant harm, that is NCC-MERP categories F–I, underestimating the total burden of harm by disregarding events with temporary harm requiring intervention only (category E events). Although achieving high IRR for category E can be challenging,35 capturing these events in future studies will result in a more comprehensive view of the nature of iatrogenic injury, and permit comparisons with recent studies of AEs in children and youth that have used the expanded definition.17 18
Despite these limitations, the use of a broad paediatric population sample to validate the CPTT permits its use in other acute care paediatric institutions. With its final refinements, the validated CPTT offers the first comprehensive evidence-based tool for assessing harm in hospitalised children and youth.
The CPTT is the first valid and reliable TT for detecting harm in children and youth of all ages hospitalised in acute care. This 35-trigger tool is reliable and robust, and can be used in quality-improvement initiatives and for more rigorous research agendas.
Future research should focus on improving the efficiency of the CPTT and investigating the differences between nurse and physician assessments of AEs. The CPTT is likely to be of interest to researchers wishing to comprehensively study the epidemiology and burden of iatrogenic harm in hospitalised children and youth, using a methodology similar to the previous national studies of AEs in hospitalised adults. Such studies will enhance current efforts to raise the profile of paediatric patient safety issues.
We would like to thank the Alberta Children's Hospital, Calgary Alberta; Canadian Association of Paediatric Health Centres (CAPHC); Children's Hospital of Eastern Ontario, Ottawa, Ontario; Hospital for Sick Children Toronto, Ontario, IWK Health Centre, Halifax, Nova Scotia; Stollery Children's Hospital, Edmonton, Alberta; and the Child Health Quality Team, Winnipeg Regional Health Authority, Winnipeg, Manitoba for their support.
Funding This study was supported by the Canadian Medical Protection Association; Canada's Research Based Pharmaceutical Companies (Rx&D); Department of Paediatrics University of Calgary; Health Canada; Manitoba Institute of Patient Safety (MIPS); Winnipeg Regional Health Authority (WRHA); and Spelman Cronin Consulting (SCC).
Competing interests None.
Ethics approval Ethics approval was provided by the Hospital for Sick Children, Stollery Children's Hospital, Calgary Children's Hospital, Children's Hospital of Eastern Ontario, Winnipeg Children's Hospital, and IWK Hospital.
Provenance and peer review Not commissioned; externally peer reviewed.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.