Accuracy of automated methods for event identification*
| Reference | Events identified by automated harm-detection method | Sensitivity (95% CI) | Specificity (95% CI) | Positive predictive value (95% CI) | Negative predictive value (95% CI) |
| Field-defined | |||||
| Nebeker et al18 | Calculated separately for bleeding/anticoagulation ADEs and delirium ADEs | Bleeding: 0.86† Delirium: 0.94† | Bleeding: 0.89† Delirium: 0.71† | Bleeding: 0.12† Delirium: 0.03† | NA |
| Zhan et al17 | DVT Cases | 0.67 (0.58 to 0.76) | NA† | 0.31 (0.25 to 0.37) | NA† |
| PE cases | 0.74 (0.59 to 0.90) | NA† | 0.24 (0.16 to 0.33) | NA† | |
| DVT/PE Cases | 0.68 (0.60 to 0.76) | NA† | 0.29 (0.24 to 0.34) | NA† | |
| Brossette et al44 | Hospital-wide nosocomial infection | 0.88†¶ | NA†¶ | 0.78 | NA |
| Hougland et al30 | Codes for inpatient ADE | 0.10 (0.63 to 0.14) | 0.97 (0.96 to 0.98) | 0.32 (0.22 to 0.43) | 0.89 (0.88 to 0.91) |
| Polancich et al15 | Patients with decubitus ulcers | NA† | NA† | 0.50 (0.42 to 0.59) | NA† |
| Dormann et al26 | ADR positive admissions using NEW ALS | 0.91 | 0.23 | 0.18‡ | 0.93 |
| ADR positive admissions using DELTA ALS | 0.41 | 0.76 | 0.25‡ | 0.87 | |
| Trick et al20 | Hospital-acquired episodes of primary CVC associated bloodstream infections | 0.81† | 0.72† | 0.62† | 0.87† |
| Levy et al23 | Admissions | 0.63 (0.51 to 0.74) | 0.42 (0.34 to 0.51) | 0.34 (0.25 to 0.42) | 0.70 (0.60 to 0.80) |
| Azaz-Livshits et al22 | Admissions | 0.66 (0.51 to 0.81) | 0.51 (0.42 to 0.60) | 0.31 (0.21 to 0.41) | 0.82 (0.73 to 0.91) |
| Jha et al32 | ADE | NA† | NA† | 0.16§ (0.16 to 0.19) | NA† |
| NLP | |||||
| Penz et al47 | Cases | PMA: 0.70 † NLP: 0.50 † Combination: 0.72† | PMA: 0.55 † NLP: 0.91 † Combination: 0.80† | PMA: 0.41† NLP: 0.71† Combination: 0.64† | PMA: 0.8† NLP: 0.8† Combination: 0.85† |
| Forster et al34 | Patients | 0.23 (0.11 to 0.35) | 0.92 (0.88 to 0.96) | 0.41 (0.22 to 0.59) | 0.83 (0.78 to 0.88) |
| Melton et al48 | Cases | 0.28 (0.16 to 0.40) | 0.98 (0.97 to 0.99) | 0.47 (0.30 to 0.64) | 0.96 (0.95 to 0.97) |
| Murff et al33 | AE | Fully automated: 0.69 (0.62 to 0.75) Partially automated: 0.64 (0.56 to 0.70) | Fully automated 0.48 (0.42 to 0.55) Partially automated: 0.85 (0.80 to 0.90) | Fully automated: 0.52 (0.46–0.58) Partially automated: 0.78 (0.72–0.85) (cohort 1), 0.84 † (cohort 2) | Fully automated: 0.65 (0.58–0.72) Partially automated: 0.74 (0.69–0.79) |
↵* 95% CIs for independently verified values reported in parentheses.
↵† Denotes figures that we could not independently verify.
↵‡ Dormann et al26 defined the positive predictive value (PPV) as the number of alerts associated with adverse drug reactions (ADRs) out of the total number of alerts. Using this criteria, they found the following PPVs: New automatic laboratory signals (ALS) (574/2328) 25%; Delta ALS (189/580) 32%.
↵§ Jha et al32 report a range of PPVs based on the first and final 8 weeks of data collection (0.16 and 0.23, respectively). We were able to independently verify the PPV for the first 8 weeks of the study only.
↵¶ Brossette et al44 reported a sensitivity of 0.86 and a specificity of 0.98. It is unclear how they identified true negative screens.
ADE, adverse drug event; AE, adverse event; CVC, central venous catheter; DVT, deep venous thrombosis; NLP, natural language processing; PE, pulmonary embolism; PMA, phrase matching algorithm.