Comparison of traditional trigger tool to data warehouse based screening for identifying hospital adverse events
- Kevin J O'Leary1,
- Vikram K Devisetty1,
- Amitkumar R Patel1,
- David Malkenson2,
- Pradeep Sama2,
- William K Thompson3,
- Matthew P Landler1,
- Cynthia Barnard2,
- Mark V Williams1
- 1Division of Hospital Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
- 2Northwestern Memorial Hospital, Chicago, Illinois, USA
- 3Center for Genetic Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
- Correspondence to Dr Kevin J O'Leary, Division of Hospital Medicine, Northwestern University Feinberg School of Medicine, 211 E. Ontario Street, Suite 211, Chicago, IL 60611, USA;
- Received 23 April 2012
- Revised 21 August 2012
- Accepted 2 September 2012
- Published Online First 4 October 2012
Background Research supports medical record review using screening triggers as the optimal method to detect hospital adverse events (AE), yet the method is labour-intensive.
Method This study compared a traditional trigger tool with an enterprise data warehouse (EDW) based screening method to detect AEs. We created 51 automated queries based on 33 traditional triggers from prior research, and then applied them to 250 randomly selected medical patients hospitalised between 1 September 2009 and 31 August 2010. Two physicians each abstracted records from half the patients using a traditional trigger tool and then performed targeted abstractions for patients with positive EDW queries in the complementary half of the sample. A third physician confirmed presence of AEs and assessed preventability and severity.
Results Traditional trigger tool and EDW based screening identified 54 (22%) and 53 (21%) patients with one or more AE. Overall, 140 (56%) patients had one or more positive EDW screens (total 366 positive screens). Of the 137 AEs detected by at least one method, 86 (63%) were detected by a traditional trigger tool, 97 (71%) by EDW based screening and 46 (34%) by both methods. Of the 11 total preventable AEs, 6 (55%) were detected by traditional trigger tool, 7 (64%) by EDW based screening and 2 (18%) by both methods. Of the 43 total serious AEs, 28 (65%) were detected by traditional trigger tool, 29 (67%) by EDW based screening and 14 (33%) by both.
Conclusions We found relatively poor agreement between traditional trigger tool and EDW based screening with only approximately a third of all AEs detected by both methods. A combination of complementary methods is the optimal approach to detecting AEs among hospitalised patients.
Despite a number of national initiatives since the publication of the Institute of Medicine's sentinel report, o Err Is Human: Building a Safer Health System. Washington D.C.: Institute of Medicine 1999,1 the existing evidence suggests we have made little progress in reducing the incidence of preventable harm.2 ,3 A major barrier to improving patient safety is the difficulty we face in detecting adverse events (AEs), defined as injuries due to medical care rather than as a result of underlying condition(s).4 ,5 Certain methods, such as incident reporting systems and review of patient complaints, reveal only a fraction of the total events occurring within an organisation.6–8 Other methods, such as using administrative data, have also been shown to miss a significant portion of events.9 ,10 Recent studies suggest a two stage medical record review is the most sensitive method to detect AEs.9 ,11 In this method, a clinician (eg, nurse) screens medical records in the first stage to identify potential AEs using criteria like the Institute for Healthcare Improvement Global Trigger Tool.12 In the second stage, a physician reviews narrative summaries of potential AEs identified in the first stage to confirm presence of AEs and assess preventability and severity of injury. Though the method is rigorous, it is also is time consuming and labour-intensive.
With the increasing use of electronic health records and computerised provider order entry, screening criteria for potential AEs may be automated. Several teams of researchers have tested automated screens to identify adverse drug events (ADEs) using sudden medication stop orders, antidote ordering and certain abnormal laboratory values.13–17 However, very little research has evaluated the potential for automated screens to identify non-drug related AEs18 or the use of a combination of information systems to facilitate the identification of AEs.19 Many healthcare organisations are implementing business intelligence tools, such as data warehouses, to identify, extract and analyse data to assist operational, clinical and research efforts.20–24 Data warehouses integrate data from disparate information systems into a single common data model from which analyses and reports may be performed without impacting systems in operation.
The goal of this study was to create automated queries for a wide range of hospital AEs and to compare the performance of this data warehouse based screening approach with a traditional trigger tool method.
Setting and study design
The study was conducted at Northwestern Memorial Hospital, an 897-bed tertiary care teaching hospital in Chicago, Illinois, and was approved by the Institutional Review Board of Northwestern University. This retrospective medical record review compared a traditional trigger tool with a data warehouse based screening method to detect AEs among 250 general medical patients. All patients underwent screening for AEs using both a traditional trigger tool and the data warehouse based screening method.
The Northwestern Medicine Enterprise Data Warehouse (EDW) is a single, integrated database of all clinical and research data from all patients receiving treatment through Northwestern University healthcare affiliates. Data from the hospital's fully integrated electronic health records/computerised provider order entry system (PowerChart Millennium from Cerner Corporation), hospital and physician billing systems, incident reporting system, and admission/discharge/transfer system populate the EDW with nightly updates from activities occurring in the preceding 24 h.
Development of EDW queries
We created 50 automated queries based on 33 screening criteria from the Harvard Medical Practice Study, a study on AEs in Utah and Colorado, and the Institute for Healthcare Improvement Global Trigger Tool (ie, the Care and Medication Module Triggers).12 ,25 ,26 We leveraged various information systems in the EDW and wrote Structured Query Language (SQL) queries with the goal to mimic what a clinician would do to identify potential AEs in the medical record using the traditional trigger tool. We created 1–7 EDW queries for each traditional trigger. We did not create automated queries that were not based on traditional triggers. The principal investigator (KJO) reviewed a sample of 20 screen negative patients and five screen positive patients for each initial query. Further adaptations resulted in a final group of 51 automated queries (ie, abnormal glucose was divided into two categories corresponding with high and low values). Examples of queries include discrete laboratory values beyond a specified threshold occurring during the hospitalisation (eg, international normalised ratio (INR)>6 after hospital day 2 and excluding patients with INR>4 on day 1), administration of certain medications (eg, naloxone), incident reports in certain categories (eg, pressure ulcer), specific documentation fields (eg, pressure ulcer noted in nursing documentation form), international classification of diseases - ninth revision (ICD-9) codes suggesting hospital acquired conditions (eg, venous thromboembolism), and text searches of progress notes and discharge summaries using natural language processing (NLP) to identify potential AEs. NLP extracts information from unstructured narrative text and maps it to structured discrete values from a specific code set. NLP helps overcome the problem of synonyms and abbreviations commonly used in medical language, and takes into account contextual factors such as temporal status and negation (eg, ‘perforation’ not preceded by ‘no’ within the same sentence). The full set of automated screening queries is listed in the online supplementary table 1.
We randomly selected 250 patients admitted to general medical services at Northwestern Memorial Hospital between 1 September 2009 and 31 August 2010. We excluded patients admitted under observation status and those cared for by either of the two medical record abstractors (VKD and ARP). We employed a two stage medical record review similar to that used in prior research.2 ,11 ,25 ,26 In this method, a clinician searches the medical record for potential AEs in the first stage. For each potential AE, the clinician creates a brief narrative summary. In the second stage, a physician–researcher reviews narrative summaries to confirm whether or not an AE occurred. The hypothesis tested in this study was whether EDW based screening would perform as well as a traditional trigger tool in the first stage of the method described above.
We assigned half of the 250 patient samples to one medical record abstractor and the other half to a second for traditional trigger tool medical record abstraction (see figure 1). Medical record abstractors identified potential AEs using traditional triggers and created a narrative summary for each potential AE identified. After completion of medical record abstraction using traditional trigger tool, we assigned patients with positive EDW screens to undergo facilitated abstraction by the complementary abstractor. Specifically, ARP conducted facilitated abstractions for EDW screen positive patients who had undergone traditional trigger tool abstraction by VKD (and vice versa). EDW screen positive patient reviews were facilitated in that abstractors were given the specific positive screening criteria, along with a date and time stamp, and value when available (eg, INR=7.2 on 10 February 2009 at 06:46). We defined an AE as an injury due to medical management rather than the natural history of the illness.
In the second stage, a hospitalist–researcher (MPL) reviewed narrative summaries of potential AEs to determine the presence of AEs and their preventability. The physician reviewer was blind as to whether narrative summaries originated from the traditional trigger tool or EDW based screening method. The physician reviewer used a 6-point confidence scale, similar to that used in prior research studies,26–29 to rate the presence of AEs: (1) no evidence that outcome was due to treatment, (2) little evidence that outcome was due to treatment, (3) outcome was possibly due to treatment but was more likely due to disease, (4) outcome was more likely due to treatment than to disease, (5) outcome was probably due to treatment and (6) outcome was definitely due to treatment. Because the same AE could be detected by more than one screening criteria, the physician reviewer indicated duplicate screens mapping to the same AE. We used a 6-point confidence scale to assess the preventability of AEs: (1) virtually no evidence of preventability, (2) slight to modest evidence of preventability, (3) preventability not quite likely, (4) preventability more likely than not, (5) strong evidence of preventability and (6) virtually certain evidence of preventability.26–29 We required a confidence score ≥4 for determination of presence and preventability of an AE. The physician reviewer also classified AEs according to four levels of severity (life-threatening, serious, clinically significant or trivial).30 An example of a life-threatening event is a nosocomial urinary tract infection that leads to septic shock. Serious events include those that lead to interventions or prolonged hospitalisations (eg, a pressure ulcer or a wound infection that requires debridement). Clinically significant events are more transient (eg, an ADE causing transient laboratory abnormalities). Trivial or insignificant events include minor injuries, such as pain at a venipuncture site. Finally, AEs were assigned to 1 of 10 prespecified categories.
The physicians performing medical record abstractions were experienced hospitalists and received specific training for this study, including an overview of the study, definitions of terms, practice with data collection tools and discussion of examples of AEs. The physician performing ratings of potential AEs had prior experience with the research method31 and received additional specific training for the study.
We assessed the performance of the first stage of the traditional trigger tool medical record review method by conducting duplicate abstractions and ratings for presence, preventability and severity of AEs for a randomly selected sample of 25 patients. The inter-rater reliability for identification of patients with potential AEs was excellent (κ=0.78; 95% CI 0.36 to 1.00). The inter-rater reliability for identifying patients with confirmed AEs was good (κ=0.68; 95% CI 0.40 to 0.97) and moderate for confirmed preventable (κ=0.50; 95% CI 0.02 to 0.99) and severe (κ=0.52; 95% CI 0.07 to 0.97) AEs.
Patient demographic data were obtained from the EDW and complemented the information from medical record review. Primary discharge diagnosis ICD-9 codes were grouped into diagnosis clusters using the Clinical Classification Software developed by the Healthcare Cost and Utilisation Project.32 We calculated the number and percentage of patients experiencing one or more AE as detected by each method and by both methods. We also calculated the number and percentage of AEs detected by each method and by both methods, stratified by preventability, severity and category of event. We also report the number of positive screens for each specific EDW query and the percentage of EDW positive screens with confirmed AEs. All analyses were conducted using Stata V.11.0 (College Station, Texas, USA).
Patient demographic characteristics are shown in table 1. Patients were a mean 60.3±18.5 years of age, 54% female, 40% non-white and hospitalised for a mean 5.7±8.4 days. Traditional trigger tool and EDW based screening identified 54 (22%) and 53 (21%) patients, respectively, with one or more AE. Overall, 140 (56%) patients had one or more positive EDW screen (total 366 positive screens).
Comparison of methods in detecting AEs
Of the 137 AEs detected by at least one method, 86 (63%) were detected by a traditional trigger tool, 97 (71%) by EDW based screening and 46 (34%) by both methods (see table 2). Of the 11 total preventable AEs, 6 (55%) were detected by a traditional trigger tool, 7 (64%) by EDW based screening and 2 (18%) by both methods. Of the 43 total serious AEs, 28 (65%) were detected by a traditional trigger tool, 29 (67%) by EDW based screening and 14 (33%) by both.
Identification of specific categories of AEs by each method is shown in table 2. The most common category of AE was ADE. Although traditional trigger tool generally detected a similar percentage of AEs as EDW based screening, there was relatively little agreement between methods with a range of 0 pressure ulcer events detected by both methods to 83% (5/6) of venous thromboembolism events detected by both methods.
Representative examples of discrepancies between traditional trigger tool and EDW based screening are shown in table 3. Examples reveal missed screens on the part of each method. Additionally, variation in descriptions of potential AEs for related screens resulted in different ratings by the physician–researcher for presence of AEs.
Performance of EDW queries
The number of positive screens per EDW query ranged from 0 (eg, new dialysis, falls identified by note type or incident report, digoxin immune Fab) to 72 (subsequent hospitalisation within 30 days). The overall percentage of positive screens with confirmed AEs was 31% (see table 4). Specifically, 113 total AEs were confirmed from 366 total positive screens (AEs>97 because each screen associated with an AE was counted; ie, different queries may identify the same AE). The percentage of positive screens with confirmed AEs for individual EDW queries was highly variable. Among queries with more than two positive screens in the sample, CT brain, death, use of fresh frozen plasma and use of haloperidol had 0 confirmed AEs.
We found relatively poor agreement between traditional trigger tool and EDW based screening with only approximately a third of all AEs detected by both methods. Our results were consistent across designations of preventability and severity of AEs. Prior studies similarly revealed poor agreement between computerised and trigger tool based strategies to detect AEs. In a study focusing on ADEs, Jha and colleagues reported that a computer based method detected 45% of events, trigger tool based screening detected 65% and only 12% were detected by both methods.14 More recently, Tinoco and colleagues reported that a computerised surveillance system detected more hospital acquired infections than trigger tool based screening, but a similar number of ADEs.19 Importantly, the study found that only 26% of hospital acquired infections and 3% of ADEs were detected by both methods.19 Our study provides additional support that computer facilitated screening may complement the traditional trigger tool approach and suggests that the rate of hospital AEs may be even higher than previously reported.
Examples of discrepancies between methods in our study illustrate how each method missed some AEs. Notably, differences in the description of events from related screens contributed to discrepancies in ratings on the part of the physician–researcher. Our assessment of inter-rater reliability between abstractors using a traditional trigger tool medical record abstraction also supports the conclusion that narrative summaries can be a potential source of unwanted variation in measuring AEs. Specifically, inter-rater reliability was lower for confirmation of presence of AEs than for identification of potential AEs. Experienced clinicians, abstracting the same positive screen from the same medical record may include different clinical information in the narrative summary, resulting in discrepancies regarding presence, preventability or severity of AEs.
Performance of EDW queries was highly variable with percentage of positive screens with confirmed AEs ranging from 0% to 100%. We designed EDW queries with the goal of identifying all potential AEs. We believe some EDW queries may be eliminated without adversely affecting the method's overall ability to detect AEs. Specifically, CT brain, death, use of fresh frozen plasma and use of haloperidol were positive for multiple patients, but identified no confirmed AEs. Subsequent hospitalisation, defined as any emergency department visit or rehospitalisation within 30 days of discharge from the index admission, was the most frequently positive screen. In a post hoc analysis, we evaluated the effect of reducing the timeframe for subsequent hospitalisation from 30 to 14 days. The percentage of positive screens with confirmed AEs improved slightly from 22% to 26%, but the 14-day timeframe missed five of the initial 16 AEs detected with the 30-day timeframe.
Our study has several limitations. First, we assessed data warehouse based screening to detect AEs during hospitalisation of general medical patients at a single site. Further research should address this method's ability to detect AEs occurring in patients cared for by other clinical services and in other hospitals. Second, we were unable to calculate sensitivity or specificity of the EDW facilitated method in light of the relatively poor overlap between the methods. That is, some AEs were missed by the traditional trigger tool abstraction while others were missed by the EDW facilitated abstraction. Therefore, a gold standard by which to assess true positive results could not be described. Importantly, our study confirms the need to use complementary approaches as a single gold standard for measuring patient safety does not exist.4 ,5
Our study is unique in that we aimed to leverage multiple information systems and approaches (eg, discrete laboratory values, incident reports, ICD-9 codes and NLP) through the use of a data warehouse to detect AEs. Though researchers have used automated searches for discrete data elements to identify ADEs and text searches of discharge summaries to detect AEs,33–35 few researchers have used a suite of health information systems, such as that found in a data warehouse.36 The development of a data warehouse facilitated strategy is important because the traditional trigger tool method is time consuming and labour-intensive. We hoped a data warehouse facilitated approach would allow for greater efficiency and result in a refined number of medical records for targeted abstraction. We plan to further modify the EDW queries used in this study in an effort to improve their performance.
In conclusion, there was relatively poor agreement between a traditional trigger tool and data warehouse based screening approach in detecting hospital AEs. A combination of complementary methods is the optimal approach for detecting AEs among hospitalised patients.
The authors express their gratitude to Taylor Pomeranz for assistance in data preparation.
Contributors KJO, CB, MVW: designed the study protocol. VKD, ARP, DM, PS, WKT, MPL: conducted data collection. KJO: conducted data analysis and developed the first draft and revised drafts of the manuscript after review by all coauthors. All coauthors approved the final manuscript.
Funding KJO, MPL, MVW received salary support from the Agency for Healthcare Research and Quality, grant #R18 HS019630.
Competing interests None.
Ethics approval Northwestern University Institutional Review Board.
Provenance and peer review Not commissioned; externally peer reviewed.