Table 1

Summary of studies comparing automated harm-detection methods with gold standard chart review

ReferencePatient sample and time frameSampling strategy*SpecialtyEvents identifiedAutomated event dataset sample sizeComparison event dataset sample size
 Nebeker et al18Adults
2001 and 2003
RandomUnknownAdverse drug events3987 admissions3987 admissions
 Zhan et al17Medicare benefic.
2002 to 2004
RandomGeneral SurgeryAdverse event: specifically postoperative deep venous thrombosis and/or pulmonary embolism20 868 hospital discharges identified as surgical patients20 868 hospital discharges identified as surgical patients
 Brossette et al44Unknown
1–3 Dec 2003 and
26–29 Apr 2004
SequentialUnknownInfection907 admissions907 admissions
 Hougland et al30Adults
1 Jan 2001 to
31 Dec 2001
Random, Flagged sample (from records with at least one flagged adverse drug event code)UnknownAdverse drug events3103 inpatients:
1961 random,
1142 flagged
 Polancich et al15UnknownUnknownUnknownHospital acquired decubitus ulcersUnknown123 charts from patients with PSI-identified decubitus ulcers
 Dormann et al26Adults
1 Sept 2000 to
28 Feb 2001
SequentialGastroenterologyAdverse drug events474 admissions of 377 patients; 109 ADEs474 admissions of 377 patients; 109 adverse drug events
 Trick et al20Adults
1 Sept 2001 to
28 Feb 2002
SequentialUnknownInfection135 positive blood cultures144 positive blood cultures
 Levy et al23All age groups
1 Apr 1997 to
31 May 1997
SequentialGeneral MedicalAdverse drug events199 admissions (192 patients)199 admissions
 Azaz-Livshits et al22All age groups
1 Apr 1995 to
31 May 1995
SequentialGeneral MedicalAdverse drug events153 admissions153 admissions
 Jha et al32Adults
1 Oct 1994 to
31 May 1995
SequentialMICU, SICU,
General Medical, General Surgical
Adverse drug events21 964 patient-days21 964 patient-days
 Penz et al47Adults
1 Jun 1999 to
31 Dec 2004
SequentialMICU, SICU and other (placement of CVC)Adverse events related to central venous catheter placement316 patient records40 patients records (10 very low probability records, 30 high probability)
 Forster et al34Adults
FY 2002
RandomGeneral Medical, General SurgicalAdverse event245 patients245 patients
 Melton and Hripcsak48Unknown
Random (charts),
Sequential (electronic discharge summaries)
UnknownAdverse events: specifically 45 NYPORTS event types1000 charts, 57 422 electronic discharge summaries1000 charts
 Murff et al33Adults
1 Jan 2000 to
30 Jun 2000
Random (Cohort 1), Sequential (Cohort 2)General Medical, Medicine subspecialtiesAdverse drug events, adverse events, diagnostic errors, operative complications, fallsCohort 1: 424 admissions.
Cohort 2: 2826 admissions
Cohort 1: 295
Cohort 2: 145 Complex sampling/subsampling and manual review process
  • * Sampling strategy refers to the method by which charts were chosen to be screened by the automated tool. Unless specifically noted, the same sampling strategy also applies to the gold standard method.

  • A scoring system was developed by the authors to reflect the probability of the adverse event in question relating to the central venous catheter placement. This system is described in the text and in table 1 of the paper.

  • CVC, central venous catheter; MICU, Medical Intensive Care Unit; NYPORTS, New York Patient Occurrence Reporting and Tracking System; PSI, patient safety indicators; SICU, Surgical Intensive Care Unit.