You are here

Article Text

Article menu
Original research
Using prospective clinical surveillance to identify adverse events in hospital
  1. Alan J Forster1,2,3,4,
  2. Jim R Worthington1,
  3. Steven Hawken3,4,
  4. Michael Bourke5,
  5. Fraser Rubens5,
  6. Kaveh Shojania1,2,3,6,
  7. Carl van Walraven1,2,3,4
  1. 1The Ottawa Hospital, Ottawa, Canada
  2. 2Department of Medicine, University of Ottawa, Ottawa, Canada
  3. 3Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
  4. 4Institute for Clinical Evaluative Sciences, Ottawa, Canada
  5. 5University of Ottawa Heart Institute, Ottawa, Canada
  6. 6Department of Medicine, Sunnybrook Health Sciences Centre and University of Toronto Centre for Patient Safety, Toronto, Canada
  1. Correspondence to Dr Alan J Forster, Ottawa Hospital, Civic Campus, 1053 Carling Avenue, Administrative Services Building 1, Ottawa, ON K1Y 4E9, Canada; aforster{at}ohri.ca

Abstract

Background To improve patient safety, organisations must systematically measure avoidable harms. Clinical surveillance—consisting of prospective case finding and peer review—could improve identification of adverse events (AEs), preventable AEs and potential AEs. The authors sought to describe and compare findings of clinical surveillance on four clinical services in an academic hospital.

Methods Clinical surveillance was performed by a nurse observer who monitored patients for prespecified clinical events and collected standard information about each event. A multidisciplinary, peer-review committee rated causation for each event. Events were subsequently classified in terms of severity and type.

Results The authors monitored 1406 patients during their admission to four hospital services: Cardiac Surgery Intensive Care (n=226), Intensive Care (n=211), General Internal Medicine (n=453) and Obstetrics (n=516). The authors detected 245 AEs during 9300 patient days of observation (2.6 AEs per 100 patient days). 88 AEs (33%) were preventable. The proportion of patients experiencing at least one AE, preventable AE or potential AE was 13.7%, 6.1% and 5.3%, respectively. AE risk varied between services, ranging from 1.4% of Obstetrics to 11% of Internal Medicine and Intensive Care patients experiencing at least one preventable AE. The proportion of patients experiencing AEs resulting in permanent disability or death varied between services: ranging from 0.2% on Obstetrics to 4.9% on Cardiac Surgery Intensive Care. No services shared the most frequent AE type.

Conclusions Using clinical surveillance, the authors identified a high risk of AE and significant variation in AE risks and subtypes between services. These findings suggest that institutions will need to evaluate service-specific safety problems to set priorities and design improvement strategies.

  • Medical errors
  • adverse events
  • prospective surveillance
  • patient safety
  • near miss

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.

https://doi.org/10.1136/bmjqs.2010.048694

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Improving patient safety requires the minimisation of treatment-related harm. This is typically measured as adverse events (AEs) (harms caused by medical care), preventable AEs (harm caused by errors) and potential AEs (errors with the potential for harm). Numerous studies have demonstrated a high incidence of AEs and preventable AEs in hospitalised patients.1–8 These studies have prompted significant investments to improve patient safety.

    Detection of AEs and preventable AEs is a primary step to achieving a safe healthcare system. By systematically measuring these outcomes and analysing their causes, healthcare planners may design system changes to prevent them.9–14 Unfortunately, most hospitals and health systems have rudimentary approaches to identify AEs. These approaches include the use of voluntary reporting, chart reviews and scanning of administrative data.15 Each of these methods has important limitations including a failure to report, inconsistent peer review and poor specificity for patient safety problems.13 15–23

    Detecting AEs is especially challenging because of the invisible nature of treatment-related harm. When a patient experiences a poor outcome, it is often assumed the outcome was caused by the underlying disease process. While this is often true, careful peer review will often determine that medical care was, at least, partially responsible. Furthermore, adverse outcomes and the circumstances leading to them are often not systematically documented. This will often lead to disagreement between peers when chart review is used to identify AEs.24 25

    Clinical surveillance is a promising method of AE detection. In this method, patients and providers are directly and indirectly monitored by a trained observer prospectively.26–29 As soon as clinical events are identified, the observer records prespecified information to facilitate subsequent peer review. This avoids missing information that occurs from incomplete documentation in the patient record.

    Clinical surveillance has several advantages over existing AE detection methods. It consists of active surveillance rather than relying solely on voluntary reporting; it ensures timely collection of information relevant to case classification rather than retrospectively relying on what is documented in charts, and it enhances peer review through timely review of cases when reviewers still remember the case. On the other hand, although it has been used successfully for years in tracking surgical complications and healthcare-associated infections, it has not been widely adopted for all AE types in multiple, diverse settings.

    In this study, we describe our findings from implementing clinical surveillance in our healthcare facility. We describe a general process and the results obtained during its application in four diverse services at an academic hospital. The comparison of results between services was of particular interest, as it would validate our ability to learn useful information from the programme and would direct strategies for improvement.

    Methods

    This study took place at a 1000-bed multicampus urban-based academic hospital. The hospital provides tertiary-level patient care, including organ transplants, and is a Level 1 trauma centre. We performed our clinical surveillance activities as part of a research study, which was approved by the Ottawa Hospital Research Ethics Board.

    Overview

    We conducted surveillance sequentially on the following four services: General Internal Medicine, Obstetrics, Intensive Care, and Cardiac Surgery Intensive Care. For each service, we monitored care for 12 weeks using a common method, which was adapted for the service to ensure relevancy to the patients treated. The general method included establishment of surveillance parameters, case finding, peer review and event classification (figure 1).

    Figure 1
    Figure 1

    Prospective clinical surveillance overview.

    Establishment of surveillance parameters

    Before starting surveillance, we identified several service-specific surveillance parameters describing surveillance location, baseline patient information and triggers.

    We prespecified the surveillance location to ensure consistent and representative case identification. On the Obstetrics service, for example, we monitored patients in the case-room, the obstetrical operating room, and the postpartum ward. For Medicine, however, we monitored patients on the ward only. We also prespecified whether we would include care preceding the observation period. For example, for Intensive Care patients we included events leading to unit admission.

    We prespecified variables describing baseline patient information to enable consistent case reviews and permit future risk adjustments. For example, when we implemented on the Medicine service, we collected information regarding the presence of diagnoses associated with poor outcomes.30

    Most importantly, we prespecified triggers which represented the factors we used to prompt case reviews. Trigger tool methodology has been used successfully in other institutions to address specific AE types such as adverse drug events,18 19 31–37 hospital-acquired infections38 and surgical complications,39–41 so we adapted the concept to create a more generalised model for the identification of all types of AEs in various settings. For each service, we reviewed the literature to identify previously published risk factors for adverse outcomes. A multidisciplinary team specific to each service then determined the face validity of these triggers and discussed additional triggers. Consensus was used to define the final set of triggers. The triggers represent clinical phenomenon, pharmacy orders and system events relevant to the patients treated. For example, ‘stat Caesarean section’ was a system event on the Obstetrics service, while ‘unplanned OR return’ was used on the Cardiac Surgery Intensive Care unit. For all services, we had an ‘Other’ category to account for events that were not predicted a priori. (See online appendix 1) contains the triggers we used for the four services.

    Case finding

    Case finding was performed daily Monday to Friday between 08:00 and 16:00 by a trained nurse (observer). The observer training consisted of a patient safety presentation, relevant literature, practice case report generation and reviews, familiarisation with triggers and service-specific integration. We integrated the observer into the daily activities of the specific service to ensure maximal interaction with the providers and opportunities to observe care yet with minimal disruption to the providers' routine. This meant there were slight differences in how the observations were performed. For example, in the Intensive Care Unit, observations occurred at the bedside during morning team rounds, while on the Medicine service they occurred at the nursing station. The observer was encouraged to ask providers for details if they were not volunteered or documented in the chart.

    All patients in the surveillance locations were eligible for case finding. The observer captured baseline information on all patients admitted to the service. Once data were entered, the observer monitored the patient until discharge. The observer identified triggers using a combination of three common AE detection methods: daily interactions with clinicians and administrators to enquire about or discuss trigger events (prompted and voluntary reporting), daily chart reviews and direct observation. If triggers occurred when our observer was not present, we identified them based on information volunteered by providers or from the chart.

    Once a trigger was identified, we captured standard information describing the case. The observer generated a case summary using a standard format describing the patient and event in detail, the response to the event, the impact of the response and the documented cause of the response if there was one. This information was compiled by the observer through discussions with the relevant staff on the service.

    Peer review

    Peer review took place weekly. For all reviews, we used the same definitions and classification systems as those used in the Harvard Medical Practice Study and other patient safety studies (See online appendix 2).1–7 We used a majority opinion to consolidate the ratings of the multiple reviewers.42 Cases were initially classified as outcomes (in which a patient experienced some form of harm, including pain, other distressing symptoms, such as dyspnoea, or reasonable markers for such symptoms such as increases in respiratory rate) or process problems (in which a task did not occur as planned). For example, if a patient experienced a sudden drop in blood pressure, the event was classified as an outcome. On the other hand if a patient missed a dose of an antibiotic but had no discernible change in status, then the event was classified as a process problem. For outcomes, we first assessed whether they were caused by medical care (ie, an AE). If so, reviewers determined whether it was preventable (ie, it was caused by error or was avoidable in some other way). We used six-point Likert scales with a cut-point of 3 for both of these assessments. For processes, reviewers determined whether the procedural or process problem could have caused harm (ie, a potential AE). Again, we used a six-point Likert scale with a cut-point of 3 for this assessment (ie, a score of 4, 5 or 6 indicated a process problem).

    Event classification

    Event classification occurred following categorisation of cases as AEs, preventable AEs or potential AEs. For these cases, the observer defined the event's severity and type. For severity, events caused one of the following levels of increasing harm: none; laboratory abnormalities only; symptoms only; temporary disability; permanent disability; or death. Events were also classified according to one or more of the following types: hospital-acquired infection; adverse drug event; procedural complication; operative complication; diagnostic error; management error; anaesthetic complication; obstetrical injury; and system design flaw. As almost all errors could be argued to be system errors, only those events that could not be assigned to another type were deemed to be system design flaws.

    Analysis

    We described patient baseline characteristics using median and IQR for continuous variables and frequency distributions for categorical variables. We described events in terms of preventability, severity and type for each service overall. We calculated the risk of experiencing at least one event per hospital encounter. We also calculated the event rate in terms of events per patient day of observation. We used SAS version 9.0 for all data management and analyses.

    Results

    Table 1 describes the patient populations in terms of their baseline characteristics. There were important differences between services. We observed many more patients in Medicine (n=453) and Obstetrics (n=516) than Cardiac Surgery (n=226) or Intensive Care (n=211). Individual patients spent more days on Medicine than Intensive Care, Obstetrics and Cardiac Surgery, with median lengths of stay on the monitored unit of 8 days, 4.8 days, 2.4 days and 2.1 days, respectively. The indications for admission varied extensively between the four services. Medicine patients were older (median age on Medicine, Cardiac Surgery, Intensive Care and Obstetrics was 74, 70, 66 and 30 years, respectively) and frailer (proportion of patients with a Charlson Index of 3 on Medicine, Cardiac Surgery and Intensive Care was 24%, 3% and 7%, respectively).

    View this table:
    Table 1

    Patients participating in the surveillance development cohort study

    Table 2 demonstrates the overall and service-specific AE risk and rate. 192 patients (13.7%) encountered 245 AEs (2.6 events per 100 patient days), 86 patients (6.1%) encountered 88 preventable AEs (0.9 events per 100 patient days), and 75 patients (5.3%) encountered 81 potential AEs (0.9 events per 100 patient days). The greatest number of events was observed on the Medicine service, which accounted for 63% (75/119) of patients with AEs, 56% of (48/86) patients with preventable AEs and 56% (40/75) of patients with potential AEs.

    View this table:
    Table 2

    Adverse event (AE) risk and rate

    The risk and rate of all AEs and preventable AEs were highest for the Intensive Care unit. Of the 211 patients observed on Intensive Care, 52 (24.6%) experienced at least one AE, and 23 (10.9%) experienced a preventable AE. The corresponding event rates were 4.5 and 1.4 per 100 patient days for AEs and preventable AEs, respectively. The lowest risk and rate of events occurred on the Obstetrics service. Of the 516 patients observed on Obstetrics, 20 (3.9%) experienced at least one AE, and seven (1.4%) experienced a preventable AE. The corresponding event rates were 1.5 and 0.5 per 100 patient days for AEs and preventable AEs, respectively.

    Table 3 describes the severity of all AEs detected. Of all 245 events detected, six (2.5%) resulted in/were associated with/potentially led to death, and 16 (6.6%) led to permanent disability. The vast majority of events resulted in symptoms only (n=75, 30.7%) or led to temporary disability (n=112, 45.9%). The majority of events that led to death or permanent disability occurred in the two critical care areas (Cardiac Surgery Intensive Care and Intensive Care). The risk of AEs causing permanent disability or death varied 24-fold across services: Cardiac Surgery Intensive Care (11/226=4.9%), Intensive Care (8/211=3.8%), Medicine (2/453=0.4%), Obstetrics (1/516=0.2%).

    View this table:
    Table 3

    Adverse event severity and type

    Table 3 also describes the types of AEs identified. There were important proportions of all AE types, but overall the most common types were therapeutic errors, procedural complications, and adverse drug events. Within each service, the top three AE types differed, and no service shared the most frequent AE type. For Cardiac Surgery, the top three types were surgical complications (47%), hospital-acquired infections (19%) and procedural complications (11%). For Intensive Care, the top three types were hospital-acquired infections (25%), procedural complications (24%) and therapeutic errors (19%). For Medicine, the top three types were adverse drug events (30%), therapeutic errors (24%) and procedural complications (12%). For Obstetrics, the top three types were therapeutic errors (36%), procedural complications (32%) and adverse drug events (27%).

    Discussion

    We used clinical surveillance to detect AEs on four hospital services. AE risk per encounter was 13% with a range of 4% to 25% depending on service. Overall, 36% of AEs were preventable, though this was highly variable between services, ranging from 13% to 56%. General Internal Medicine had the greatest number of AEs but also had the highest number of patients and comorbidities. After controlling for length of stay and the number of patients observed, the risk of AEs, including those with the most severe consequences, was greatest in the two critical care areas. There were differences in the types of AEs identified; however, therapeutic errors were common in all subgroups studied. Finally, clinical surveillance identified an important number of potential AEs.

    These findings are important because they identify ample opportunities for quality improvement. We found a greater AE risk than most previous studies of AEs which have tended to rely solely on chart review.1–7 We used a clinical observer to identify events within 24 h of their occurrence, and reviewed them within 7 days of their identification. The timely identification of problems would ultimately mean a more rapid response to rectify them. The reviews were performed by clinicians who knew the clinical setting and were well respected by their local peers. We believe that these factors resulted in classifications with a much greater degree of face validity than existing methods to identify AEs (such as incident reports and chart reviews). These factors also led us to identify a significantly greater proportion of events attributed to diagnostic and therapeutic errors, than prior studies. Furthermore, the systematic nature of our data collection ensured our methods were less biased than morbidity and mortality reviews or closed-claims analysis.15 43 44

    Our study supports the concept that improvement strategies must be locally developed and must target specific problems. We found that AE risk and type varied by service. The underlying explanation for this finding likely relates to the frequency of underlying processes occurring within these areas that in turn lead to more opportunities to cause harm. Regardless of the cause, it follows that solutions will also necessarily vary. For example, in the Cardiac Surgery Intensive Care unit, interventions should be directed to preventing certain types of surgical complications. On the Medicine service, interventions that focus specifically on reducing errors related to the medication administration process would be most beneficial. The conclusion that errors and their solutions vary by patient service is intuitively obvious but is often not incorporated in the ‘top down’ approaches to improve quality and safety taken by accreditation organisations or other regulators.

    Our study also supports the conclusion that priority setting, as it pertains to safety improvement, must occur in a highly strategic manner. The greatest number of AEs occurred on the service with the frailest patients. Effective interventions to improve safety in these patients would have the largest reduction on the institutional number of events. However, after controlling for the number of patients and length of observation, these patients had a risk of events similar to the least frail patients. The highest event risk and the most severe events occurred in services that had very invasive therapies and where patients were most acutely ill. Therefore, efforts to prevent injuries in these groups might be more efficient and associated with a greater overall impact in terms of reducing the economic burden of AEs. For all these reasons, organisations will have to be very clear about their goals when targeting the safety problems they hope to solve.

    Finally, this research has highlighted the feasibility of clinical surveillance. We used a comprehensive strategy involving the use of direct observation, voluntary and prompted reporting, as well as daily chart reviews. This combination of AE detection methods has been recommended by others15 and serves to increase the overall performance of the monitoring system. Our programme was well accepted in the four services studied with very little financial support (the clinical observer was a funded position). However, all other functions required to maintain the process were voluntary, and there was very little infrastructure support required. There was no evidence that providers who were being observed were overtly or covertly averse to the programme or changed their behaviours. In fact, the methodology was so well received that our institution wanted to and is currently taking steps to implement it on other services.

    Our findings are important in that we used a reproducible method of case detection and case review applying a standard method of classification to facilitate comparison. However, there are three important limitations which we would like to highlight. First, we are uncertain of the programme's reliability. Threats to reliability include the observer and peer-review processes. We attempted to mitigate these concerns through standardisation of the process of observation using triggers and case report forms; and, for the peer review, the use of multiple reviewers. These approaches have been successful in improving reliability in prior settings. However, we recommend further research before our method be adopted by health systems to compare institutions. Second, the generalisability of our surveillance feasibility is unknown, particularly because it was performed using a single observer. Even though we successfully implemented our programme within multiple settings in different facilities, it is possible that other settings may create challenges. Further work is recommended to establish whether the programme can be successfully adapted within non-teaching hospitals and in different health systems. Third, the observer cannot be everywhere at once and may miss some events as they are happening. This means the programme cannot be used to intervene systematically and directly in patient care when it does not meet the standard. However, this is not the stated purpose of the programme. Furthermore, if cases are detected in which an immediate intervention is warranted, then this appropriate response should be easily performed. For example, we identified a few instances when a critical laboratory abnormality had not been recognised and simply brought it to the attention of the treating team.

    In summary, our study supports the conclusion that clinical surveillance appears to be an effective means of detecting patient safety issues. Before making a general recommendation for widely adopting this method, we recommend further evaluations to address four issues. First, we recommend comparing the findings of clinical surveillance with other methods of identifying patient safety events. Second, we recommend an economic analysis to determine the most efficient method of adverse event detection. Third, we would recommend evaluations of this method in different hospitals. Finally, and most importantly, we recommend studies to determine if adverse event detection leads to improvement in patient outcomes.

    Acknowledgments

    We would like to thank A Jennings for her assistance throughout the study and in the preparation of the manuscript. This research received financial support from the Canadian Patient Safety Institute, the Canadian Institute for Health Research, the Healthcare Insurance Reciprocal of Canada, the University of Ottawa Heart Institute, and the Ottawa Hospital Centre for Patient Safety.

    References

    1. ↵
      1. Baker GR,
      2. Norton PG,
      3. Flintoft V,
      4. et al
      . The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 2004;170:1678–86.
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Forster AJ,
      2. Asmis TR,
      3. Clark HD,
      4. et al
      . Ottawa Hospital Patient Safety Study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital. CMAJ 2004;170:1235–40.
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Brennan TA,
      2. Leape LL,
      3. Laird NM,
      4. et al
      . Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med 1991;324:370–6.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Thomas EJ,
      2. Studdert DM,
      3. Burstin HR,
      4. et al
      . Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000;38:261–71.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Vincent C,
      2. Neale G,
      3. Woloshynowych M
      . Adverse events in British hospitals: preliminary retrospective record review. BMJ 2001;322:517–19.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Wilson RM,
      2. Runciman WB,
      3. Gibberd RW,
      4. et al
      . The Quality in Australian Health Care Study. Med J Aust 1995;163:458–71.
      OpenUrlPubMedWeb of Science
    7. ↵
      1. Davis P,
      2. Lay-Yee R,
      3. Briant R,
      4. et al
      . Adverse events in New Zealand public hospitals I: occurrence and impact. New Zeal Med J 2002;115:U271.
      OpenUrl
    8. ↵
      1. O'Neil AC,
      2. Petersen LA,
      3. Cook EF,
      4. et al
      . Physician reporting compared with medical-record review to identify adverse medical events. Ann Intern Med 1993;119:370–6.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Cullen DJ,
      2. Bates DW,
      3. Small SD,
      4. et al
      . The incident reporting system does not detect adverse drug events: a problem for quality improvement. Jt Comm J Qual Improv 1995;21:541–8.
      OpenUrlPubMed
    10. ↵
      1. Beckmann U,
      2. Bohringer C,
      3. Carless R,
      4. et al
      . Evaluation of two methods for quality improvement in intensive care: facilitated incident monitoring and retrospective medical chart review. Crit Care Med 2003;31:1006–11.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Flynn EA,
      2. Barker KN,
      3. Pepper GA,
      4. et al
      . Comparison of methods for detecting medication errors in 36 hospitals and skilled-nursing facilities. Am J Health Syst Pharm 2002;59:436–46.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Berwick DM,
      2. Leape LL
      . Reducing errors in medicine. BMJ 1999;319:136–7.
      OpenUrlFREE Full Text
    13. ↵
      1. Leape LL
      . Reporting of adverse events. N Engl J Med 2002;347:1633–8.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Leape LL
      . A systems analysis approach to medical error. J Eval Clin Pract 1997;3:213–22.
      OpenUrlCrossRefPubMed
    15. ↵
      1. Thomas EJ,
      2. Petersen LA
      . Measuring errors and adverse events in health care. J Gen Intern Med 2003;18:61–7.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Kellogg VA,
      2. Havens DS
      . Adverse events in acute care: an integrative literature review. Res Nurs Health 2003;26:398–408.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Walshe K
      . Adverse events in health care: issues in measurement. Qual Health Care 2000;9:47–52.
      OpenUrlFREE Full Text
    18. ↵
      1. Murff HJ,
      2. Patel VL,
      3. Hripcsak G,
      4. et al
      . Detecting adverse events for patient safety research: a review of current methodologies. J Biomed Inform 2003;36:131–43.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Karson AS,
      2. Bates DW
      . Screening for adverse events. J Eval Clin Pract 1999;5:23–32.
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Allan EL,
      2. Barker KN
      . Fundamentals of medication error research. Am J Hosp Pharm 1990;47:555–71.
      OpenUrlAbstract
    21. ↵
      1. Iezzoni LI
      . Assessing quality using administrative data. Ann Intern Med 1997;127:666–74.
      OpenUrlPubMedWeb of Science
    22. ↵
      1. Bates DW,
      2. Evans RS,
      3. Murff H,
      4. et al
      . Detecting adverse events using information technology. J Am Med Inform Assoc 2003;10:115–28.
      OpenUrlAbstract/FREE Full Text
    23. ↵
      1. Weinger MB,
      2. Slagle J,
      3. Jain S,
      4. et al
      . Retrospective data collection and analytical techniques for patient safety studies. J Biomed Inform 2003;36:106–19.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Thomas EJ,
      2. Lipsitz SR,
      3. Studdert DM,
      4. et al
      . The reliability of medical record review for estimating adverse event rates. Ann Intern Med 2002;136:812–16.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Hayward RA,
      2. Hofer TP
      . Estimating hospital deaths due to medical errors: preventability is in the eye of the reviewer. JAMA 2001;286:415–20.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Forster AJ,
      2. Halil RB,
      3. Tierney MG
      . Pharmacist surveillance of adverse drug events. Am J Health Syst Pharm 2004;61:1466–72.
      OpenUrlAbstract/FREE Full Text
    27. ↵
      1. Bates DW,
      2. Cullen DJ,
      3. Laird N,
      4. et al
      . Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA 1995;274:29–34.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Andrews LB,
      2. Stocking C,
      3. Krizek T,
      4. et al
      . An alternative strategy for studying adverse events in medical care. Lancet 1997;349:309–13.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Donchin Y,
      2. Gopher D,
      3. Olin M,
      4. et al
      . A look into the nature and causes of human errors in the intensive care unit. 1995. Qual Saf Health Care 2003;12:143–8.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. Charlson ME,
      2. Pompei P,
      3. Ales KL,
      4. et al
      . A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40:373–83.
      OpenUrlCrossRefPubMedWeb of Science
    31. ↵
      1. Classen DC,
      2. Pestotnik SL,
      3. Evans RS,
      4. et al
      . Description of a computerized adverse drug event monitor using a hospital information system. Hosp Pharm 1992;27:774–9.
      OpenUrlPubMed
    32. ↵
      1. Rozich JD,
      2. Haraden CR,
      3. Resar RK
      . Adverse drug event trigger tool: a practical methodology for measuring medication related harm. Qual Saf Health Care 2003;12:194–200.
      OpenUrlAbstract/FREE Full Text
    33. ↵
      1. Honigman B,
      2. Lee J,
      3. Rothschild J,
      4. et al
      . Using computerized data to identify adverse drug events in outpatients. JAMIA 2001;8:254–66.
      OpenUrlAbstract/FREE Full Text
    34. ↵
      1. Hope C,
      2. Overhage JM,
      3. Seger A,
      4. et al
      . A tiered approach is more cost effective than traditional pharmacist-based review for classifying computer-detected signals as adverse drug events. J Biomed Inform 2003;36:92–8.
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      1. Jha AK,
      2. Kuperman GJ,
      3. Teich JM,
      4. et al
      . Identifying adverse drug events: development of a computer-based monitor and comparison with chart review and stimulated voluntary report. JAMIA 1998;5:305–14.
      OpenUrlAbstract/FREE Full Text
    36. ↵
      1. Jha AK,
      2. Kuperman GJ,
      3. Rittenberg E,
      4. et al
      . Identifying hospital admissions due to adverse drug events using a computer-based monitor. Pharmacoepidemiol Drug Saf 2001;10:113–19.
      OpenUrlCrossRefPubMedWeb of Science
    37. ↵
      1. Rothschild JM,
      2. Landrigan CP,
      3. Cronin JW,
      4. et al
      . The Critical Care Safety Study: The incidence and nature of adverse events and serious medical errors in intensive care. Crit Care Med 2005;33:1694–700.
      OpenUrlCrossRefPubMedWeb of Science
    38. ↵
      1. Haley RW,
      2. Culver DH,
      3. White JW,
      4. et al
      . The efficacy of infection surveillance and control programs in preventing nosocomial infections in US hospitals. Am J Epidemiol 1985;121:182–205.
      OpenUrlAbstract/FREE Full Text
    39. ↵
      1. Neumayer L,
      2. Mastin M,
      3. Vanderhoof L,
      4. et al
      . Using the Veterans Administration National Surgical Quality Improvement Program to improve patient outcomes. J Surg Res 2000;88:58–61.
      OpenUrlCrossRefPubMedWeb of Science
    40. ↵
      1. Khuri SF,
      2. Daley J,
      3. Henderson WG
      . The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs. Arch Surg 2002;137:20–7.
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Rowell KS,
      2. Turrentine FE,
      3. Hutter MM,
      4. et al
      . Use of national surgical quality improvement program data as a catalyst for quality improvement. J Am Coll Surgeons 2007;204:1293–300.
      OpenUrlCrossRefPubMedWeb of Science
    42. ↵
      1. Forster AJ,
      2. O'Rourke K,
      3. Shojania KG,
      4. et al
      . Combining ratings from multiple physician reviewers helped to overcome the uncertainty associated with adverse event classification. J Clin Epidemiol 2007;60:892–901.
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      1. Vincent C,
      2. Taylor-Adams S,
      3. Chapman EJ,
      4. et al
      . How to investigate and analyse clinical incidents: clinical risk unit and association of litigation and risk management protocol. BMJ 2000;320:777–81.
      OpenUrlFREE Full Text
    44. ↵
      1. Vincent C,
      2. Stanhope N,
      3. Crowley-Murphy M
      . Reasons for not reporting adverse incidents: an empirical study. J Eval Clin Pract 1999;5:13–21.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    Footnotes

    • Funding This research received financial support from the Canadian Patient Safety Institute, the Canadian Institute for Health Research, the Healthcare Insurance Reciprocal of Canada, the University of Ottawa Heart Institute and the Ottawa Hospital Centre for Patient Safety.

    • Competing interests None.

    • Ethics approval Ethics approval was provided by the Ottawa Hospital Research Ethics Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.