You are here

Article Text

Article menu
Original research
Comparison of traditional trigger tool to data warehouse based screening for identifying hospital adverse events
  1. Kevin J O'Leary1,
  2. Vikram K Devisetty1,
  3. Amitkumar R Patel1,
  4. David Malkenson2,
  5. Pradeep Sama2,
  6. William K Thompson3,
  7. Matthew P Landler1,
  8. Cynthia Barnard2,
  9. Mark V Williams1
  1. 1Division of Hospital Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
  2. 2Northwestern Memorial Hospital, Chicago, Illinois, USA
  3. 3Center for Genetic Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
  1. Correspondence to Dr Kevin J O'Leary, Division of Hospital Medicine, Northwestern University Feinberg School of Medicine, 211 E. Ontario Street, Suite 211, Chicago, IL 60611, USA; keoleary{at}nmh.org

Abstract

Background Research supports medical record review using screening triggers as the optimal method to detect hospital adverse events (AE), yet the method is labour-intensive.

Method This study compared a traditional trigger tool with an enterprise data warehouse (EDW) based screening method to detect AEs. We created 51 automated queries based on 33 traditional triggers from prior research, and then applied them to 250 randomly selected medical patients hospitalised between 1 September 2009 and 31 August 2010. Two physicians each abstracted records from half the patients using a traditional trigger tool and then performed targeted abstractions for patients with positive EDW queries in the complementary half of the sample. A third physician confirmed presence of AEs and assessed preventability and severity.

Results Traditional trigger tool and EDW based screening identified 54 (22%) and 53 (21%) patients with one or more AE. Overall, 140 (56%) patients had one or more positive EDW screens (total 366 positive screens). Of the 137 AEs detected by at least one method, 86 (63%) were detected by a traditional trigger tool, 97 (71%) by EDW based screening and 46 (34%) by both methods. Of the 11 total preventable AEs, 6 (55%) were detected by traditional trigger tool, 7 (64%) by EDW based screening and 2 (18%) by both methods. Of the 43 total serious AEs, 28 (65%) were detected by traditional trigger tool, 29 (67%) by EDW based screening and 14 (33%) by both.

Conclusions We found relatively poor agreement between traditional trigger tool and EDW based screening with only approximately a third of all AEs detected by both methods. A combination of complementary methods is the optimal approach to detecting AEs among hospitalised patients.

  • Patient safety
  • Adverse events, epidemiology and detection
  • Chart review methodologies

https://doi.org/10.1136/bmjqs-2012-001102

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Despite a number of national initiatives since the publication of the Institute of Medicine's sentinel report, o Err Is Human: Building a Safer Health System. Washington D.C.: Institute of Medicine 1999,1 the existing evidence suggests we have made little progress in reducing the incidence of preventable harm.2 ,3 A major barrier to improving patient safety is the difficulty we face in detecting adverse events (AEs), defined as injuries due to medical care rather than as a result of underlying condition(s).4 ,5 Certain methods, such as incident reporting systems and review of patient complaints, reveal only a fraction of the total events occurring within an organisation.6–8 Other methods, such as using administrative data, have also been shown to miss a significant portion of events.9 ,10 Recent studies suggest a two stage medical record review is the most sensitive method to detect AEs.9 ,11 In this method, a clinician (eg, nurse) screens medical records in the first stage to identify potential AEs using criteria like the Institute for Healthcare Improvement Global Trigger Tool.12 In the second stage, a physician reviews narrative summaries of potential AEs identified in the first stage to confirm presence of AEs and assess preventability and severity of injury. Though the method is rigorous, it is also is time consuming and labour-intensive.

    With the increasing use of electronic health records and computerised provider order entry, screening criteria for potential AEs may be automated. Several teams of researchers have tested automated screens to identify adverse drug events (ADEs) using sudden medication stop orders, antidote ordering and certain abnormal laboratory values.13–17 However, very little research has evaluated the potential for automated screens to identify non-drug related AEs18 or the use of a combination of information systems to facilitate the identification of AEs.19 Many healthcare organisations are implementing business intelligence tools, such as data warehouses, to identify, extract and analyse data to assist operational, clinical and research efforts.20–24 Data warehouses integrate data from disparate information systems into a single common data model from which analyses and reports may be performed without impacting systems in operation.

    The goal of this study was to create automated queries for a wide range of hospital AEs and to compare the performance of this data warehouse based screening approach with a traditional trigger tool method.

    Methods

    Setting and study design

    The study was conducted at Northwestern Memorial Hospital, an 897-bed tertiary care teaching hospital in Chicago, Illinois, and was approved by the Institutional Review Board of Northwestern University. This retrospective medical record review compared a traditional trigger tool with a data warehouse based screening method to detect AEs among 250 general medical patients. All patients underwent screening for AEs using both a traditional trigger tool and the data warehouse based screening method.

    The Northwestern Medicine Enterprise Data Warehouse (EDW) is a single, integrated database of all clinical and research data from all patients receiving treatment through Northwestern University healthcare affiliates. Data from the hospital's fully integrated electronic health records/computerised provider order entry system (PowerChart Millennium from Cerner Corporation), hospital and physician billing systems, incident reporting system, and admission/discharge/transfer system populate the EDW with nightly updates from activities occurring in the preceding 24 h.

    Development of EDW queries

    We created 50 automated queries based on 33 screening criteria from the Harvard Medical Practice Study, a study on AEs in Utah and Colorado, and the Institute for Healthcare Improvement Global Trigger Tool (ie, the Care and Medication Module Triggers).12 ,25 ,26 We leveraged various information systems in the EDW and wrote Structured Query Language (SQL) queries with the goal to mimic what a clinician would do to identify potential AEs in the medical record using the traditional trigger tool. We created 1–7 EDW queries for each traditional trigger. We did not create automated queries that were not based on traditional triggers. The principal investigator (KJO) reviewed a sample of 20 screen negative patients and five screen positive patients for each initial query. Further adaptations resulted in a final group of 51 automated queries (ie, abnormal glucose was divided into two categories corresponding with high and low values). Examples of queries include discrete laboratory values beyond a specified threshold occurring during the hospitalisation (eg, international normalised ratio (INR)>6 after hospital day 2 and excluding patients with INR>4 on day 1), administration of certain medications (eg, naloxone), incident reports in certain categories (eg, pressure ulcer), specific documentation fields (eg, pressure ulcer noted in nursing documentation form), international classification of diseases - ninth revision (ICD-9) codes suggesting hospital acquired conditions (eg, venous thromboembolism), and text searches of progress notes and discharge summaries using natural language processing (NLP) to identify potential AEs. NLP extracts information from unstructured narrative text and maps it to structured discrete values from a specific code set. NLP helps overcome the problem of synonyms and abbreviations commonly used in medical language, and takes into account contextual factors such as temporal status and negation (eg, ‘perforation’ not preceded by ‘no’ within the same sentence). The full set of automated screening queries is listed in the online supplementary table 1.

    View this table:
    Table 1

    Patient characteristics

    AE identification

    We randomly selected 250 patients admitted to general medical services at Northwestern Memorial Hospital between 1 September 2009 and 31 August 2010. We excluded patients admitted under observation status and those cared for by either of the two medical record abstractors (VKD and ARP). We employed a two stage medical record review similar to that used in prior research.2 ,11 ,25 ,26 In this method, a clinician searches the medical record for potential AEs in the first stage. For each potential AE, the clinician creates a brief narrative summary. In the second stage, a physician–researcher reviews narrative summaries to confirm whether or not an AE occurred. The hypothesis tested in this study was whether EDW based screening would perform as well as a traditional trigger tool in the first stage of the method described above.

    We assigned half of the 250 patient samples to one medical record abstractor and the other half to a second for traditional trigger tool medical record abstraction (see figure 1). Medical record abstractors identified potential AEs using traditional triggers and created a narrative summary for each potential AE identified. After completion of medical record abstraction using traditional trigger tool, we assigned patients with positive EDW screens to undergo facilitated abstraction by the complementary abstractor. Specifically, ARP conducted facilitated abstractions for EDW screen positive patients who had undergone traditional trigger tool abstraction by VKD (and vice versa). EDW screen positive patient reviews were facilitated in that abstractors were given the specific positive screening criteria, along with a date and time stamp, and value when available (eg, INR=7.2 on 10 February 2009 at 06:46). We defined an AE as an injury due to medical management rather than the natural history of the illness.

    Figure 1
    Figure 1

    Overview of study design. AE, adverse event; EDW, enterprise data warehouse.

    In the second stage, a hospitalist–researcher (MPL) reviewed narrative summaries of potential AEs to determine the presence of AEs and their preventability. The physician reviewer was blind as to whether narrative summaries originated from the traditional trigger tool or EDW based screening method. The physician reviewer used a 6-point confidence scale, similar to that used in prior research studies,26–29 to rate the presence of AEs: (1) no evidence that outcome was due to treatment, (2) little evidence that outcome was due to treatment, (3) outcome was possibly due to treatment but was more likely due to disease, (4) outcome was more likely due to treatment than to disease, (5) outcome was probably due to treatment and (6) outcome was definitely due to treatment. Because the same AE could be detected by more than one screening criteria, the physician reviewer indicated duplicate screens mapping to the same AE. We used a 6-point confidence scale to assess the preventability of AEs: (1) virtually no evidence of preventability, (2) slight to modest evidence of preventability, (3) preventability not quite likely, (4) preventability more likely than not, (5) strong evidence of preventability and (6) virtually certain evidence of preventability.26–29 We required a confidence score ≥4 for determination of presence and preventability of an AE. The physician reviewer also classified AEs according to four levels of severity (life-threatening, serious, clinically significant or trivial).30 An example of a life-threatening event is a nosocomial urinary tract infection that leads to septic shock. Serious events include those that lead to interventions or prolonged hospitalisations (eg, a pressure ulcer or a wound infection that requires debridement). Clinically significant events are more transient (eg, an ADE causing transient laboratory abnormalities). Trivial or insignificant events include minor injuries, such as pain at a venipuncture site. Finally, AEs were assigned to 1 of 10 prespecified categories.

    The physicians performing medical record abstractions were experienced hospitalists and received specific training for this study, including an overview of the study, definitions of terms, practice with data collection tools and discussion of examples of AEs. The physician performing ratings of potential AEs had prior experience with the research method31 and received additional specific training for the study.

    We assessed the performance of the first stage of the traditional trigger tool medical record review method by conducting duplicate abstractions and ratings for presence, preventability and severity of AEs for a randomly selected sample of 25 patients. The inter-rater reliability for identification of patients with potential AEs was excellent (κ=0.78; 95% CI 0.36 to 1.00). The inter-rater reliability for identifying patients with confirmed AEs was good (κ=0.68; 95% CI 0.40 to 0.97) and moderate for confirmed preventable (κ=0.50; 95% CI 0.02 to 0.99) and severe (κ=0.52; 95% CI 0.07 to 0.97) AEs.

    Data analysis

    Patient demographic data were obtained from the EDW and complemented the information from medical record review. Primary discharge diagnosis ICD-9 codes were grouped into diagnosis clusters using the Clinical Classification Software developed by the Healthcare Cost and Utilisation Project.32 We calculated the number and percentage of patients experiencing one or more AE as detected by each method and by both methods. We also calculated the number and percentage of AEs detected by each method and by both methods, stratified by preventability, severity and category of event. We also report the number of positive screens for each specific EDW query and the percentage of EDW positive screens with confirmed AEs. All analyses were conducted using Stata V.11.0 (College Station, Texas, USA).

    Results

    Patient characteristics

    Patient demographic characteristics are shown in table 1. Patients were a mean 60.3±18.5 years of age, 54% female, 40% non-white and hospitalised for a mean 5.7±8.4 days. Traditional trigger tool and EDW based screening identified 54 (22%) and 53 (21%) patients, respectively, with one or more AE. Overall, 140 (56%) patients had one or more positive EDW screen (total 366 positive screens).

    Comparison of methods in detecting AEs

    Of the 137 AEs detected by at least one method, 86 (63%) were detected by a traditional trigger tool, 97 (71%) by EDW based screening and 46 (34%) by both methods (see table 2). Of the 11 total preventable AEs, 6 (55%) were detected by a traditional trigger tool, 7 (64%) by EDW based screening and 2 (18%) by both methods. Of the 43 total serious AEs, 28 (65%) were detected by a traditional trigger tool, 29 (67%) by EDW based screening and 14 (33%) by both.

    View this table:
    Table 2

    Detection of AEs by method stratified by preventability, severity and category

    Identification of specific categories of AEs by each method is shown in table 2. The most common category of AE was ADE. Although traditional trigger tool generally detected a similar percentage of AEs as EDW based screening, there was relatively little agreement between methods with a range of 0 pressure ulcer events detected by both methods to 83% (5/6) of venous thromboembolism events detected by both methods.

    Representative examples of discrepancies between traditional trigger tool and EDW based screening are shown in table 3. Examples reveal missed screens on the part of each method. Additionally, variation in descriptions of potential AEs for related screens resulted in different ratings by the physician–researcher for presence of AEs.

    View this table:
    Table 3

    Examples of discrepancies between traditional trigger tool and EDW based screening methods

    Performance of EDW queries

    The number of positive screens per EDW query ranged from 0 (eg, new dialysis, falls identified by note type or incident report, digoxin immune Fab) to 72 (subsequent hospitalisation within 30 days). The overall percentage of positive screens with confirmed AEs was 31% (see table 4). Specifically, 113 total AEs were confirmed from 366 total positive screens (AEs>97 because each screen associated with an AE was counted; ie, different queries may identify the same AE). The percentage of positive screens with confirmed AEs for individual EDW queries was highly variable. Among queries with more than two positive screens in the sample, CT brain, death, use of fresh frozen plasma and use of haloperidol had 0 confirmed AEs.

    View this table:
    Table 4

    Performance of individual EDW queries

    Discussion

    We found relatively poor agreement between traditional trigger tool and EDW based screening with only approximately a third of all AEs detected by both methods. Our results were consistent across designations of preventability and severity of AEs. Prior studies similarly revealed poor agreement between computerised and trigger tool based strategies to detect AEs. In a study focusing on ADEs, Jha and colleagues reported that a computer based method detected 45% of events, trigger tool based screening detected 65% and only 12% were detected by both methods.14 More recently, Tinoco and colleagues reported that a computerised surveillance system detected more hospital acquired infections than trigger tool based screening, but a similar number of ADEs.19 Importantly, the study found that only 26% of hospital acquired infections and 3% of ADEs were detected by both methods.19 Our study provides additional support that computer facilitated screening may complement the traditional trigger tool approach and suggests that the rate of hospital AEs may be even higher than previously reported.

    Examples of discrepancies between methods in our study illustrate how each method missed some AEs. Notably, differences in the description of events from related screens contributed to discrepancies in ratings on the part of the physician–researcher. Our assessment of inter-rater reliability between abstractors using a traditional trigger tool medical record abstraction also supports the conclusion that narrative summaries can be a potential source of unwanted variation in measuring AEs. Specifically, inter-rater reliability was lower for confirmation of presence of AEs than for identification of potential AEs. Experienced clinicians, abstracting the same positive screen from the same medical record may include different clinical information in the narrative summary, resulting in discrepancies regarding presence, preventability or severity of AEs.

    Performance of EDW queries was highly variable with percentage of positive screens with confirmed AEs ranging from 0% to 100%. We designed EDW queries with the goal of identifying all potential AEs. We believe some EDW queries may be eliminated without adversely affecting the method's overall ability to detect AEs. Specifically, CT brain, death, use of fresh frozen plasma and use of haloperidol were positive for multiple patients, but identified no confirmed AEs. Subsequent hospitalisation, defined as any emergency department visit or rehospitalisation within 30 days of discharge from the index admission, was the most frequently positive screen. In a post hoc analysis, we evaluated the effect of reducing the timeframe for subsequent hospitalisation from 30 to 14 days. The percentage of positive screens with confirmed AEs improved slightly from 22% to 26%, but the 14-day timeframe missed five of the initial 16 AEs detected with the 30-day timeframe.

    Our study has several limitations. First, we assessed data warehouse based screening to detect AEs during hospitalisation of general medical patients at a single site. Further research should address this method's ability to detect AEs occurring in patients cared for by other clinical services and in other hospitals. Second, we were unable to calculate sensitivity or specificity of the EDW facilitated method in light of the relatively poor overlap between the methods. That is, some AEs were missed by the traditional trigger tool abstraction while others were missed by the EDW facilitated abstraction. Therefore, a gold standard by which to assess true positive results could not be described. Importantly, our study confirms the need to use complementary approaches as a single gold standard for measuring patient safety does not exist.4 ,5

    Our study is unique in that we aimed to leverage multiple information systems and approaches (eg, discrete laboratory values, incident reports, ICD-9 codes and NLP) through the use of a data warehouse to detect AEs. Though researchers have used automated searches for discrete data elements to identify ADEs and text searches of discharge summaries to detect AEs,33–35 few researchers have used a suite of health information systems, such as that found in a data warehouse.36 The development of a data warehouse facilitated strategy is important because the traditional trigger tool method is time consuming and labour-intensive. We hoped a data warehouse facilitated approach would allow for greater efficiency and result in a refined number of medical records for targeted abstraction. We plan to further modify the EDW queries used in this study in an effort to improve their performance.

    In conclusion, there was relatively poor agreement between a traditional trigger tool and data warehouse based screening approach in detecting hospital AEs. A combination of complementary methods is the optimal approach for detecting AEs among hospitalised patients.

    Acknowledgments

    The authors express their gratitude to Taylor Pomeranz for assistance in data preparation.

    References

    1. ↵
      1. Wachter RM
      . Patient safety at ten: unmistakable progress, troubling gaps. Health Aff (Millwood) 2010;29:165–73.
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Levinson DR
      . Adverse events in hospitals: national incidence among medical beneficiaries. Washington, DC, USA: U.S. Department of Health and Human Services, Office of the Inspector General, 2010:OEI-06-09-00090.
    3. ↵
      1. Landrigan CP,
      2. Parry GJ,
      3. Bones CB,
      4. et al
      . Temporal trends in rates of patient harm resulting from medical care. N Engl J Med 2010;363:2124–34.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Shojania KG
      . The elephant of patient safety: what you see depends on how you look. Jt Comm J Qual Patient Saf 2010;36:399–401.
      OpenUrlPubMed
    5. ↵
      1. Thomas EJ,
      2. Petersen LA
      . Measuring errors and adverse events in health care. J Gen Intern Med 2003;18:61–7.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Christiaans-Dingelhoff I,
      2. Smits M,
      3. Zwaan L,
      4. et al
      . To what extent are adverse events found in patient records reported by patients and healthcare professionals via complaints, claims and incident reports? BMC Health Serv Res 2011;11:49.
      OpenUrlCrossRefPubMed
    7. ↵
      1. Farley DO,
      2. Haviland A,
      3. Champagne S,
      4. et al
      . Adverse-event-reporting practices by US hospitals: results of a national survey. Qual Saf Health Care 2008;17:416–23.
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Levtzion-Korach O,
      2. Frankel A,
      3. Alcalai H,
      4. et al
      . Integrating incident data from five reporting systems to assess patient safety: making sense of the elephant. Jt Comm J Qual Patient Saf 2010;36:402–10.
      OpenUrlPubMed
    9. ↵
      1. Levinson DR
      . Adverse events in hospitals: methods for identifying events: Department of Health and Human Services. Washington, DC, USA: Office of the Inspector General; 2010:OEI-06-08-00221.
    10. ↵
      1. Romano PS,
      2. Mull HJ,
      3. Rivard PE,
      4. et al
      . Validity of selected AHRQ patient safety indicators based on VA National Surgical Quality Improvement Program data. Health Serv Res 2009;44:182–204.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Classen DC,
      2. Resar R,
      3. Griffin F,
      4. et al
      . ‘Global trigger tool’ shows that adverse events in hospitals may be ten times greater than previously measured. Health Aff (Millwood) 2011;30:581–9.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Griffin FA,
      2. Resar RK
      . IHI global trigger sool for measuring adverse events (second edition). IHI Innovation Series white paper. Cambridge, MA, USA: Institute for Healthcare Improvement, 2009.
    13. ↵
      1. Classen DC,
      2. Pestotnik SL,
      3. Evans RS,
      4. et al
      . Computerized surveillance of adverse drug events in hospital patients. JAMA 1991;266:2847–51.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Jha AK,
      2. Kuperman GJ,
      3. Teich JM,
      4. et al
      . Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. J Am Med Inform Assoc 1998;5:305–14.
      OpenUrlAbstract/FREE Full Text
    15. ↵
      1. Kilbridge PM,
      2. Alexander L,
      3. Ahmad A
      . Implementation of a system for computerized adverse drug event surveillance and intervention at an academic medical center. J Clin Outcomes Manag 2006;13:94–100.
      OpenUrl
    16. ↵
      1. Kilbridge PM,
      2. Campbell UC,
      3. Cozart HB,
      4. et al
      . Automated surveillance for adverse drug events at a community hospital and an academic medical center. J Am Med Inform Assoc 2006;13:372–7.
      OpenUrlAbstract/FREE Full Text
    17. ↵
      1. Kilbridge PM,
      2. Noirot LA,
      3. Reichley RM,
      4. et al
      . Computerized surveillance for adverse drug events in a pediatric hospital. J Am Med Inform Assoc 2009;16:607–12.
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Szekendi MK,
      2. Sullivan C,
      3. Bobb A,
      4. et al
      . Active surveillance using electronic triggers to detect adverse events in hospitalized patients. Qual Saf Health Care 2006;15:184–90.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Tinoco A,
      2. Evans RS,
      3. Staes CJ,
      4. et al
      . Comparison of computerized surveillance and manual chart review for adverse events. J Am Med Inform Assoc 2011;18:491–7.
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Chute CG,
      2. Beck SA,
      3. Fisk TB,
      4. et al
      . The Enterprise Data Trust at Mayo Clinic: a semantically integrated warehouse of biomedical data. J Am Med Inform Assoc 2010;17:131–15.
      OpenUrlAbstract/FREE Full Text
    21. ↵
      1. Dewitt JG,
      2. Hampton PM
      . Development of a data warehouse at an academic health system: knowing a place for the first time. Acad Med 2005;80:1019–25.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Kamal J,
      2. Liu J,
      3. Ostrander M,
      4. et al
      . Information warehouse—a comprehensive informatics platform for business, clinical, and research applications. AMIA Annu Symp Proc 2010;452–6.
      OpenUrl
    23. ↵
      1. Murphy SN,
      2. Morgan MM,
      3. Barnett GO,
      4. et al
      . Optimizing healthcare research data warehouse design through past COSTAR query analysis. Proc AMIA Symp 1999;892–6.
      OpenUrl
    24. ↵
      1. Zapletal E,
      2. Rodon N,
      3. Grabar N,
      4. et al
      . Methodology of integration of a clinical data warehouse with a clinical information system: the HEGP case. Stud Health Technol Inform 2010;160:193–7.
      OpenUrlPubMed
    25. ↵
      1. Hiatt HH,
      2. Barnes BA,
      3. Brennan TA,
      4. et al
      . A study of medical injury and medical malpractice. N Engl J Med 1989;321:480–4.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Thomas EJ,
      2. Studdert DM,
      3. Burstin HR,
      4. et al
      . Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000;38:261–71.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Baker GR,
      2. Norton PG,
      3. Flintoft V,
      4. et al
      . The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 2004;170:1678–86.
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Forster AJ,
      2. Asmis TR,
      3. Clark HD,
      4. et al
      . Ottawa Hospital Patient Safety Study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. CMAJ 2004;170:1235–40.
      OpenUrlAbstract/FREE Full Text
    29. ↵
      1. Leape LL,
      2. Brennan TA,
      3. Laird N,
      4. et al
      . The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991;324:377–84.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Weissman JS,
      2. Schneider EC,
      3. Weingart SN,
      4. et al
      . Comparing patient-reported hospital adverse events with medical record review: do patients know something that hospitals do not? Ann Intern Med 2008;149:100–8.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. O'Leary KJ,
      2. Buck R,
      3. Fligiel HM,
      4. et al
      . Structured interdisciplinary rounds in a medical teaching unit: improving patient safety. Arch Intern Med 2011;171:678–84.
      OpenUrlCrossRefPubMedWeb of Science
    32. ↵
      1. Studdert DM,
      2. Mello MM,
      3. Gawande AA,
      4. et al
      . Disclosure of medical injury to patients: an improbable risk management strategy. Health Aff (Millwood) 2007;26:215–26.
      OpenUrlAbstract/FREE Full Text
    33. ↵
      1. Forster AJ,
      2. Andrade J,
      3. van Walraven C
      . Validation of a discharge summary term search method to detect adverse events. J Am Med Inform Assoc 2005;12:200–6.
      OpenUrlAbstract/FREE Full Text
    34. ↵
      1. Melton GB,
      2. Hripcsak G
      . Automated detection of adverse events using natural language processing of discharge summaries. J Am Med Inform Assoc 2005;12:448–57.
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Murff HJ,
      2. Forster AJ,
      3. Peterson JF,
      4. et al
      . Electronically screening discharge summaries for adverse medical events. J Am Med Inform Assoc 2003;10:339–50.
      OpenUrlAbstract/FREE Full Text
    36. ↵
      1. Horvath MM,
      2. Cozart H,
      3. Ahmad A,
      4. et al
      . Sharing adverse drug event data using business intelligence technology. J Patient Saf 2009;5:35–41.
      OpenUrlCrossRefPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors KJO, CB, MVW: designed the study protocol. VKD, ARP, DM, PS, WKT, MPL: conducted data collection. KJO: conducted data analysis and developed the first draft and revised drafts of the manuscript after review by all coauthors. All coauthors approved the final manuscript.

    • Funding KJO, MPL, MVW received salary support from the Agency for Healthcare Research and Quality, grant #R18 HS019630.

    • Competing interests None.

    • Ethics approval Northwestern University Institutional Review Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.