You are here

Article Text

Article menu

Download PDFPDF

Original research
Prospective evaluation of medication-related clinical decision support over-rides in the intensive care unit
Free
  1. Adrian Wong1,
  2. Mary G Amato1,2,
  3. Diane L Seger3,
  4. Christine Rehr1,
  5. Adam Wright4,5,
  6. Sarah P Slight6,7,
  7. Patrick E Beeler8,
  8. E. John Orav9,
  9. David W Bates1
  1. 1 Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, Massachusetts, USA
  2. 2 Department of Pharmacy and Therapeutics, MCPHS University, Boston, Massachusetts, USA
  3. 3 Clinical Quality Analysis, Partners Healthcare, Somerville, Massachusetts, USA
  4. 4 Department of Medicine, Brigham and Women’s Hospital, Boston, Massachusetts, USA
  5. 5 Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA
  6. 6 Division of General Internal Medicine, The Center for Patient Safety Research and Practice, Brigham and Women’s Hospital, Boston, Massachusetts, USA
  7. 7 School of Pharmacy, Newcastle University, Newcastle upon Tyne, UK
  8. 8 Research Center for Medical Informatics, University Hospital Zurich, Zurich, Switzerland
  9. 9 Department of Biostatistics, Harvard School of Public Health, Boston, Massachusetts, USA
  1. Correspondence to Adrian Wong, Division of General Internal Medicine and Primary Care, Brigham and Women’s Hospital, Boston, MA 02115, USA; awong22{at}bwh.harvard.edu

Abstract

Background Clinical decision support (CDS) displayed in electronic health records has been found to reduce the incidence of medication errors and adverse drug events (ADE). Recent data suggested that medication-related CDS alerts were frequently over-ridden, often inappropriately. Patients in the intensive care unit (ICU) are at an increased risk of ADEs; however, limited data exist on the benefits of CDS in the ICU. This study aims to evaluate potential harm associated with medication-related CDS over-rides in the ICU.

Methods This was a prospective observational study of adults admitted to any of six ICUs between July 2016 and April 2017 at our institution. Patients with provider-overridden CDS for dose (orders for scheduled frequency and not pro re nata), drug allergy, drug–drug interaction, geriatric and renal alerts (contraindicated medications for renal function or renal dosing) were included. The primary outcome was the appropriateness of over-rides, which were evaluated by two independent reviewers. Secondary outcomes included incidence of ADEs following alert over-ride and risk of ADEs based on over-ride appropriateness.

Results A total of 2448 over-ridden alerts from 712 unique patient encounters met inclusion criteria. The overall appropriateness rate for over-rides was 81.6% and varied by alert type. More ADEs (potential and definite) were identified following inappropriate over-rides compared with appropriate over-rides (16.5 vs 2.74 per 100 over-ridden alerts, Fisher’s exact test P<0.001). An adjusted logistic regression model showed that inappropriate over-rides were associated with an increased risk of ADEs (OR 6.14, 95% CI 4.63 to 7.71, P<0.001).

Conclusions Approximately four of five identified CDS over-rides were appropriately over-ridden, with the rate varying by alert type. However, inappropriate over-rides were six times as likely to be associated with potential and definite ADEs, compared with appropriate over-rides. Further efforts should be targeted at improving the positive predictive value of CDS such as by suppressing alerts that are appropriately over-ridden.

  • adverse drug event
  • clinical decision support
  • critical care
  • informatics
  • patient safety
  • quality of care

https://doi.org/10.1136/bmjqs-2017-007531

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Adverse drug events (ADE) are injuries resulting from a medication, which may result from medication errors. Given their association with increased hospital length of stay, costs, and morbidity and mortality, efforts have been made to reduce these often preventable events.1–3 Medication-related clinical decision support (CDS) has been identified as an effective way to reduce medication errors, along with the introduction of computerised provider order entry.4 5 However, literature regarding the proportion of CDS alerts that are over-ridden including those that are over-ridden inappropriately is increasing.6 7 Therefore, studies evaluating the association between appropriateness of CDS over-rides and ADEs are needed, though relatively few have been done.8

    One patient population that is particularly susceptible to ADEs is intensive care unit (ICU) patients, who are at increased risk compared with other hospitalised patients because they receive a large number of medications, including many high-risk medications; have altered pharmacokinetics and organ systems; and have a relatively increased length of stay.9–11 Two prospective cohort studies identifying ADEs in the ICU found that they were relatively common (30.6–96.5 per 1000 patient-days), and associated with increased morbidity.12 13 A study performed at our institution using our legacy, home-grown electronic health record (EHR) system found that inappropriately over-ridden CDS alerts were associated with an increased risk of ADEs.8 However, this study had limitations including that it was retrospective, had a small sample size, and there were concerns about generalisability as it was done within a home-grown system. Given the lack of published data regarding harms associated with medication-related CDS over-rides and the potential increased risk in ICU patients, our objective was to characterise these issues in a commercial EHR. By identifying the extent of harm associated with these CDS alert over-rides, we will identify ways to improve current CDS alert systems, allowing providers to focus their attention on clinically pertinent alerts.

    Materials and methods

    We performed a prospective, observational study evaluating medication-related CDS alert over-rides by providers (anaesthesiologist, fellow, nurse practitioner, physician, physician assistant, resident). Alert over-rides were generated between July 2016 and April 2017 from patients admitted to one of the following ICUs at Brigham and Women’s Hospital: medical (n=2), neurology (n=2) or surgical (n=2). Alert logic was sourced from First Databank (First Databank, South San Francisco, CA, USA). CDS also included order sentences and default doses, although these were inconsistent across medications in regard to their function and clinical utility.

    Five computerised alert types that are frequent and of clinical significance in the ICU patient population were included: dose, drug allergy, drug–drug interaction (DDI), geriatric and renal. Specific alerts targeted within each alert type were based on a 6-month pilot study (October 2015 to May 2016) evaluating alert over-rides. Factors considered in choosing the alerts to evaluate included clinical experience, frequency and severity of harm. Further details on the specific alerts chosen may be found in the online supplementary appendix A. All alerts that were evaluated in this study were presented to providers at the time of order signing (ie, not informational) and required an action by the provider to continue with the order (ie, over-ride). Therefore, the inclusion criteria for this study were: (1) patient was admitted to one of six ICUs within our institution; (2) the alert was one of five alert types and was in one of the subcategories found in online supplementary appendix A; (3) the alert fired on a patient within our study time period (July 2016 to April 2017); and (4) the alert was presented to the provider at the time of order signing. For geriatric alerts, only ‘contraindicated’ alerts were included as only these were provided to multiple providers (ie, not only the initial ordering provider). Geriatric ‘precaution’ alerts were only presented to the ordering provider and were therefore excluded. Exclusion criteria included patients with a hospital length of stay of <24 hours after the over-ride to allow for adequate time to evaluate potential harm.

    Supplemental material

    Data collection included the patient’s age and gender, Sequential Organ Failure Assessment score at the time of the patient’s first over-ride included in this study, the type of hospital and ICU admission, type of ICU, hospital and ICU length of stay, and documented rationale for over-ride.14 Acute kidney injury (AKI) was defined using guidelines.15 The primary outcome was the appropriateness of the over-ride, assessed by two independent reviewers with a set of predetermined criteria specific for each type of alert. Secondary outcomes included the documented reason for over-ride (which was only required for drug allergy alerts), the incidence of ADEs following alert over-ride and association of over-ride appropriateness with ADE.

    Appropriateness evaluation

    Criteria for appropriateness were created using previously published data, including guidelines as well as clinical experience of a multidisciplinary group.13 16 Criteria were specific for alert categories and modified until a consensus was reached. Parameters included: if the medication was being used for comfort measures only (ie, hospice), gender and baseline QTc (defined as value prior to initiation of medications from alert), past receipt of medication and documented reactions (if present), and if the medication was a home medication and no documented adverse reactions had occurred from its use. For patients administered medications for comfort measures, these over-rides were identified as appropriate if they were used at reasonable doses. An example of our appropriateness criteria for geriatric alerts may be found in the online supplementary appendix B. Appropriateness was independently evaluated for all over-ridden alerts by two clinical pharmacists (one with significant experience in critical care and medication safety, one with significant experience in medication safety). The inter-rater agreement for appropriateness was determined with a Cohen’s κ statistic. Disagreements were resolved by discussions between the two independent reviewers. If consensus was not achieved, a third experienced reviewer (physician with significant experience in medication and patient safety) was consulted.

    ADE evaluation

    To evaluate for ADEs, we performed patient chart reviews on over-rides in which the patient received the over-ridden alert’s specific medication(s). An ADE was defined as an injury occurring from use of a medication. An ADE included instances where the patient’s corrected QT was greater than 500 ms. The period of evaluation started after the over-ride and continued for the time that the medication(s) remained active in the patient’s medication orders, which could have persisted to hospital discharge in some cases. ADEs that were included were specific to the over-ridden alert. Data relevant to an ADE, such as laboratory reports, medication orders and patient notes documented by nurses or providers, were abstracted and summarised by one reviewer. These data were blinded (ie, appropriateness of over-ride was not provided) and forwarded to two independent reviewers to determine the likelihood (no ADE, possible ADE, definite ADE) and severity of the ADE, regardless of likelihood (significant, serious, life-threatening). A definite ADE was defined as harm that only could have occurred due to use of the medication, while a possible ADE was an ADE which could have resulted from other causes (eg, delirium from ICU illness or from medication use). Definitions for severity of the ADE were based on previous work by members of our study team.11 An example of a life-threatening ADE was the requirement for cardiopulmonary resuscitation to sustain life, while a serious ADE was excessive sedation. If consensus was not achieved, a third experienced reviewer (physician) was consulted. Study personnel had undergone training based on guidance developed by the Center for Excellence for Patient Safety Research at Brigham and Women’s Hospital, which has been used in previous studies and previously described.17

    Statistical analysis

    Descriptive statistics were used to summarise patient characteristics. Fisher’s exact test was used to compare categorical variables (rate of over-ride by alert type, appropriateness of over-rides by alert type and rate of ADEs by appropriateness). Multivariable logistic regression was performed to assess the association between the appropriateness of an over-ride and the risk of ADEs (possible and definite). The model was adjusted for the following predefined patient baseline characteristics: age, gender, Sequential Organ Function Assessment score, which may be potential confounders for ADEs. These confounders were chosen based on our expert knowledge. A post hoc multivariable linear regression was performed to assess the association between the appropriateness of an over-ride and the ICU length of stay. This model was also adjusted for potential confounders, as above. A P value of <0.05 was considered significant. Statistical analysis was completed using R V.3.3.3 (R Core Team, Vienna, Austria).

    Results

    A total of 24 231 alerts in the parent alert types (dose, drug allergy, DDI, geriatric, renal) were presented to staff from 3312 unique patient encounters, with an overall over-ride rate of 88.5%. The over-ride rate varied by alert type (dose: 96.8%, drug allergy: 83.6%, DDI: 91.9%, geriatric: 2.3%, renal: 97.1%; P<0.001). For this study, 2448 over-ridden alerts met inclusion criteria (10.1% of total alerts). Patient demographics of unique patient encounters with alerts that met inclusion criteria (n=712) are detailed in table 1.

    View this table:
    Table 1

    Patient demographics

    Characteristics of alert types

    For drug allergy over-rides, most were due to a definite match between the ordered medication and documented allergen (n=248, 89.5%). The most common over-ride was due to acetaminophen for the definite match over-rides (n=99, 39.9%). Of these over-rides, only 10 (10.1%) were due to an acetaminophen-only allergy, while the remainder was due to documented allergies of acetaminophen in combination with other medications (eg, oxycodone). Of the 29 potentially life-threatening drug allergy over-rides, anaphylaxis was the most common documented reaction to the allergen (n=13, 44.8%). Acetaminophen was again the most common medication ordered (n=10, 34.5%), with these over-rides due to a documented allergy of a combination of acetaminophen with other medications. The most commonly documented over-ride reason was ‘Will monitor’ (n=115, 46.4%) and ‘Patient tolerated before’ (n=11, 37.9%) for the definite allergy and life-threatening allergy over-rides, respectively.

    The evaluation of DDI over-rides showed that most alerts were triggered by medication combinations that increase the risk of QTc prolongation (n=1569, 86.9%), with haloperidol the most common medication alerted on (n=716, 39.6%). Subcutaneous heparin (n=18, 50.0%), simvastatin (n=22, 34.4%) and sulfamethoxazole-trimethoprim (n=44, 55.7%) were the most common medications for the anticoagulant, ‘contraindicated’ and ‘other’ DDI alert types, respectively. A total of 38 patients (2.1%) were continued on home medications that resulted in a medication alert. Over-ride reasons were not required and infrequently used (n=504, 27.9%), with ‘Will monitor’ the most common among all DDI over-rides (n=342, 67.9%).

    For the dose alerts, benzodiazepines accounted for most of the over-rides (n=31, 75.6%), with lorazepam the most common (n=23, 74.2%). Hold parameters, which are input by providers (eg, medication should not be administered if a parameter such as low blood pressure exists) for the medication order, were used in only 18 of the alerts (43.9%). ‘Will monitor’ was the most common over-ride reason entered (n=8, 66.7%).

    Regarding the geriatric alerts, chlordiazepoxide and nifedipine were the most common medications alerted on (n=6 each, 28.6%). ‘Patient tolerated before’ and ‘Will monitor’ were the most common over-ride reasons provided (n=3 each, 42.9%).

    For the renal alerts, electrolytes accounted for most over-rides (total: n=154, 50.8%; magnesium sulfate: n=24, potassium chloride: n=130). Most alerts were due to AKI (n=170, 56.1%), with 88 of these alerts (51.8%) indicative of improving AKI and three alerts in patients undergoing continuous renal replacement therapy. A total of 43 alerts (16.3%) were due to continuation of a home medication. Of these alerts, seven (14.0%) had an increasing trend to the patient’s serum creatinine suggestive of developing kidney injury, while 13 (30.2%) were in haemodialysis-dependent patients. Of those over-rides that had an over-ride reason, ‘Will monitor’ was the most commonly documented rationale (n=47, 71.2%).

    Appropriateness of over-rides

    The overall appropriateness rate was 81.6% (table 2). The κ for the criteria agreement of appropriateness was 0.89 (95% CI 0.85 to 0.93) indicating almost perfect agreement, with a percent agreement of 92.1%.

    View this table:
    Table 2

    Rate of appropriateness of over-rides by alert type

    The appropriateness rate differed significantly by alert type (P<0.001). Evaluation of appropriateness rates for the drug allergy and DDI alert subtypes (table 3) showed that over-rides of potentially life-threatening drug allergy alerts were appropriate approximately 70% of the time. Appropriateness of DDI  over - rides differed significantly between subtypes, with amiodarone-digoxin over - rides always being appropriately over - ridden.

    View this table:
    Table 3

    Rate of appropriateness of over-rides by alert subtype

    Adverse drug events

    A total of 1636 over-ridden alerts resulted in medication administration(s) to the patient (66.8% of study sample) and 56 resulted in an ADE (potential or definite) (see online supplementary appendix C). The κ for ADE determination was 0.93 (95% CI 0.90 to 0.96) indicating excellent agreement, with a percent agreement of 95.1%. Most ADEs were considered to be ‘potential’ (n=52, 92.9%). Three out of four ‘definite ADEs’ were a result of a ‘definite’ drug allergy match alert such as vancomycin resulting in red man syndrome. Most ADEs were serious (n=50, 89.3%), with altered mental status (n=10, 20.0%) and QTc prolongation (n=36, 72.0%) the most common ADEs encountered. No ADEs resulted from the anticoagulant DDIs. Only one ADE resulted from continuation of a home medication (geriatric alert, amitriptyline).

    There was one life-threatening event, that was a potential ADE and possibly related to an over-ride. This occurred in a patient with a baseline QTc of 535 ms on hospital admission. The over-ride was for trazodone (new medication) in conjunction with ritonavir (home medication), while the patient was being administered quetiapine (new medication) for agitation. The patient suffered a cardiac arrest with ventricular fibrillation and ultimately died a few days later.

    As only medications that were administered to the patient could have been evaluated for ADEs, table 4 details the number of alerts that resulted in medication administration and the rate of ADEs by appropriateness of over-ride.

    View this table:
    Table 4

    Rate of adverse drug events by appropriateness of over-ride

    There was a significant increase in the rate of ADEs with inappropriate over-rides, compared with appropriate over-rides. The unadjusted logistic regression found that inappropriate over-rides were associated with an increased risk of ADEs (OR 6.13, 95% CI 4.64 to 7.69, P<0.001), with similar findings from the adjusted logistic regression (OR 6.14, 95% CI 4.63 to 7.71, P<0.001). Inappropriate over-rides were associated with an increased ICU length of stay by an additional 2.25 days (95% CI 0.52 to 3.98, P=0.011).

    Discussion

    We evaluated the appropriateness of medication-related CDS over-rides in the ICU and ADEs associated with these over-rides. A significant proportion of medication-related CDS alerts are over-ridden in the ICU (88.5%). Based on the over-rides that were studied, appropriateness was also high (81.6%) and varied significantly by the type of alert, with drug allergy over-rides being commonly appropriately over-ridden, whereas geriatric alerts were frequently inappropriately over-ridden. Inappropriate over-rides were associated with a six-fold increased risk of ADEs (potential and definite), compared with appropriately over-ridden alerts. Appropriateness rates were in line with the published literature, likely due to the alert types studied and the close monitoring that occurs in the ICU.6 13

    Our institution transitioned from our legacy EHR to a commercial EHR in May 2015. Efforts had been continually made to our legacy system to improve available clinical CDS, including increasing its positive predictive value (PPV) to limit potential alert fatigue.18 19 With the transition to the commercial EHR, we found a significant increase in alerts by approximately five-fold, along with a significant increase in over-ride rates of certain alert types.20 Potential reasons for this significant increase were the removal of our tailored CDS as well as the presentation of CDS alerts at the time of order signing, instead of at the time of ordering as in the legacy system. Evaluation of appropriate over-rides to modify available EHR systems and knowledge bases is of importance to increase the clinical relevance of presented CDS alerts. The removal of this tailoring due to our EHR system transition reduced this PPV of our CDS alerts.

    Discussion of specific alert types

    Based on previous experience in evaluating drug allergy over-rides, we focused on two specific subtypes that were of particular clinical interest. The appropriateness of the over-rides differed by the subtype, with definite matches commonly appropriately over-ridden (89.1%), while life-threatening alerts were often appropriately over-ridden (69.0%) but to a decreased extent compared with definite matches. A match between the ordered medication and the documented allergen can increase the PPV of these alerts. However, the premise of combination products (eg, oxycodone-acetaminophen) adds complexity to provider documentation of these allergies.

    DDIs accounted for the majority of studied CDS over-rides due to a large number of QTc-prolonging medication combinations. In evaluating the subtypes of the DDI alerts, the amiodarone-digoxin and anticoagulant subtypes were of particular interest. None of the amiodarone-digoxin over-rides were inappropriate, and no anticoagulant DDIs resulted in ADEs. The lack of ADEs associated with anticoagulants, which are considered to be high-risk medications, is surprising but was often due to discontinuation of one anticoagulant at the same time as ordering another.21 Our CDS system believed that the to-be-discontinued anticoagulant was still an active order, when in fact it was in the process of being discontinued. In evaluating the ‘contraindicated’ DDIs, over-rides were frequently appropriate as simvastatin and a metabolism inhibitor were the most common DDI combinations, which were often not reflective of the dose threshold for simvastatin that is recommended in the guidelines.22 The low rate of ADEs associated with this DDI subtype was due to the large number of simvastatin DDIs. It would be expected to require more time than a typical ICU stay to occur (ie, myopathy/rhabdomyolysis).

    In evaluating the dose alerts, it was concerning that only 43.9% of these orders had any hold parameters associated with the medication. Of the four ADEs that potentially resulted from the over-rides, only one had a hold parameter associated with the order.

    For geriatric alerts, an interesting finding was that the acceptance of these alerts (97.5%) was much more common than reported in the literature.8 This acceptance rate was also significantly greater than the rest of the studied alert types. Potentially, these alerts are the most specific to a patient, resulting in the highest provider acceptance. This is supported by the high rate of inappropriate over-rides found in this study (85.7%). The small number of over-ridden alerts was due to only ‘contraindicated’ alerts that were included in this study because they were presented to all providers. ‘Precaution’ alerts were only presented to the original ordering provider and not to subsequent providers in cases of reordering the same medication or changing the dose. The renal alerts contrasted previous findings within our institution, when they had been tailored to be as specific as possible.18 In the commercial database, electrolyte alerts accounted for most of the over-rides (50.8%), whereas no such alerts were active in our legacy system. These were clinically insignificant alerts, resulting in a high rate of appropriate over-rides.

    Recommendations to improve CDS

    One finding that is noted in the results of this study is the evaluation of why CDS alerts exist within an institution. Malfunctions of CDS have been studied in the literature, which has identified that they may be widespread and may exist for long time periods due to lack of investigation into this matter.23 A few of the malfunctions that were encountered during this study period were focused on the actual medication ordered compared with what the CDS believed was ordered (eg, simvastatin as previously mentioned). This illustrates the need for evaluation of CDS, especially in the time period immediately following EHR implementation, as we did in our institution.

    A general improvement would be linking the over-ride reason to an appropriate intervention. For example, when ‘Will monitor’ is chosen as the over-ride reason for an amiodarone-digoxin DDI, an order for a serum digoxin level would be made. This could become more specific by evaluating additional parameters such as the dose and renal function in determining the date and time of the serum digoxin level order.

    Although it may be difficult to determine which component of a combination product may lead to an allergy, the use of previous tolerance to a component would likely be able to reduce the alert burden. This could be accomplished through the use of machine learning and natural language processing that could more accurately determine culprit medications.24 25 Additionally, our EHR and CDS system can differentiate between a true medication allergy and that of intolerance. By differentiating how CDS presents a true medication allergy (eg, anaphylaxis) to an intolerance (eg, nausea), providers may value CDS more (ie, more correctly respond), which in turn may reduce the incidence of inappropriate over-rides. For dose alerts, incorporation of a hold parameter could potentially prevent ADEs from occurring. Removal of renal alerts for one-time orders of electrolytes as long as the serum lab value (obtained recently within a reasonable amount of time) was within a certain range would help alleviate a significant portion of these alerts.26

    Limitations

    Our study has several limitations. First, this study was completed at a single centre based on only one commercial EHR/database, and therefore may not necessarily be applicable to other institutions. However, data suggest that this commercial EHR is implemented in a significant portion of the market.27 Second, we may not have exhaustively evaluated factors that the provider may have considered in determining the decision to over-ride a CDS alert. Nevertheless, we made considerable efforts to evaluate the appropriateness of over-rides and subsequent ADEs on a case-by-case basis, formulation of criteria using a multidisciplinary expert team and the use of independent adjudicators. Third, we are unable to determine causality of our findings, only association, given the nature of our study design. Fourth, we were time limited as we only evaluated for ADEs that occurred during the hospital stay. Patients may have remained on some medications that were ordered on hospital discharge, which would be important for ADEs that are expected to take a longer time to manifest. Fifth, we were dependent on clinical documentation for the determination of ADEs, which might have limited our findings. Finally, we included in our definition of an ADE some events (eg, QTc greater than 500 ms) that other studies may not have considered to be an ADE.

    Conclusion

    Approximately four of five identified CDS over-rides were appropriately over-ridden, though the rate varied substantially by alert type. Inappropriate over-rides were six times as likely to be associated with an ADE (potential and definite) compared with appropriate over-rides, confirming that decision support can identify clinically important situations. Further efforts should be targeted at improving the PPV of CDS in a number of ways, including by following human factor principles in alert presentation, by suppressing alerts that are appropriately over-ridden, by using clinical information about individual patients to improve the likelihood that the alerts will be appropriate and by using techniques such as artificial intelligence to help determine which alerts to present.

    References

    1. ↵
      2. Leape LL ,
      3. Brennan TA ,
      4. Laird N , et al
      . The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991;324:377–84.doi:10.1056/NEJM199102073240605
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      2. Classen DC ,
      3. Pestotnik SL ,
      4. Evans RS , et al
      . Adverse drug events in hospitalized patients. Excess length of stay, extra costs, and attributable mortality. JAMA 1997;277:301–6.doi:10.1001/jama.1997.03540280039031
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      2. Bates DW , et al
      . The costs of adverse drug events in hospitalized patients. JAMA 1997;277:307–11.doi:10.1001/jama.1997.03540280045032
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      2. Bates DW ,
      3. Leape LL ,
      4. Cullen DJ , et al
      . Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998;280:1311–6.doi:10.1001/jama.280.15.1311
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      2. Bates DW ,
      3. Teich JM ,
      4. Lee J , et al
      . The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc 1999;6:313–21.doi:10.1136/jamia.1999.00660313
      OpenUrlCrossRefPubMed
    6. ↵
      2. Lin CP ,
      3. Payne TH ,
      4. Nichol WP , et al
      . Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs’ Computerized Patient Record System. J Am Med Inform Assoc 2008;15:620–6.doi:10.1197/jamia.M2453
      OpenUrlCrossRefPubMed
    7. ↵
      2. Nanji KC ,
      3. Slight SP ,
      4. Seger DL , et al
      . Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014;21:487–91.doi:10.1136/amiajnl-2013-001813
      OpenUrlCrossRefPubMed
    8. ↵
      2. Wong A ,
      3. Amato MG ,
      4. Seger DL , et al
      . Evaluation of medication-related clinical decision support alert overrides in the intensive care unit. J Crit Care 2017;39:156–61.doi:10.1016/j.jcrc.2017.02.027
      OpenUrl
    9. ↵
      2. Cullen DJ ,
      3. Sweitzer BJ ,
      4. Bates DW , et al
      . Preventable adverse drug events in hospitalized patients: a comparative study of intensive care and general care units. Crit Care Med 1997;25:1289–97.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      2. Rothschild JM ,
      3. Landrigan CP ,
      4. Cronin JW , et al
      . The Critical Care Safety Study: the incidence and nature of adverse events and serious medical errors in intensive care. Crit Care Med 2005;33:1694–700.doi:10.1097/01.CCM.0000171609.91035.BD
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      2. Morimoto T ,
      3. Sakuma M ,
      4. Matsui K , et al
      . Incidence of adverse drug events and medication errors in Japan: the JADE study. J Gen Intern Med 2011;26:148–53.doi:10.1007/s11606-010-1518-3
      OpenUrlCrossRefPubMed
    12. ↵
      2. Nazer LH ,
      3. Hawari F ,
      4. Al-Najjar T
      . Adverse drug events in critically ill patients with cancer: incidence, characteristics, and outcomes. J Pharm Pract 2014;27:208–13.doi:10.1177/0897190013513302
      OpenUrlCrossRefPubMed
    13. ↵
      2. Ohta Y ,
      3. Sakuma M ,
      4. Koike K , et al
      . Influence of adverse drug events on morbidity and mortality in intensive care units: the JADE study. Int J Qual Health Care 2014;26:573–8.doi:10.1093/intqhc/mzu081
      OpenUrlCrossRefPubMed
    14. ↵
      2. Vincent JL ,
      3. Moreno R ,
      4. Takala J , et al
      . The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med 1996;22:707–10.
      OpenUrlCrossRefPubMedWeb of Science
    15. ↵
      Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group: KDIGO clinical practice guideline for acute kidney injury. Kidney Int 2012;2:1–138.
      OpenUrlCrossRef
    16. ↵
      American Geriatrics Society 2015 Beers Criteria Update Expert Panel: American Geriatrics Society 2015. Updated beers criteria for potentially inappropriate medication use in older adults. J Am Geriatr Soc 2015;63:2227–46.
      OpenUrlCrossRefPubMed
    17. ↵
      2. Morimoto T ,
      3. Gandhi TK ,
      4. Seger AC , et al
      . Adverse drug events and medication errors: detection and classification methods. Qual Saf Health Care 2004;13:306–14.doi:10.1136/qshc.2004.010611
      OpenUrlAbstract/FREE Full Text
    18. ↵
      2. Chertow GM ,
      3. Lee J ,
      4. Kuperman GJ , et al
      . Guided medication dosing for inpatients with renal insufficiency. JAMA 2001;286:2839–44.doi:10.1001/jama.286.22.2839
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      2. Peterson JF ,
      3. Kuperman GJ ,
      4. Shek C , et al
      . Guided prescription of psychotropic medications for geriatric inpatients. Arch Intern Med 2005;165:802–7.doi:10.1001/archinte.165.7.802
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      2. Wong A ,
      3. Wright A ,
      4. Seger DL , et al
      . Comparison of overridden medication-related clinical decision support in the intensive care unit between a commercial system and a legacy system. Appl Clin Inform 2017;8:866–79.doi:10.4338/ACI-2017-04-RA-0059
      OpenUrl
    21. ↵
      ISMP List of High-Alert Medications in Acute Care Settings. Institute for safe medication practices website.Updated 2014 https://www.ismp.org/tools/institutionalhighAlert.asp (acessed 31 Jul 2016).
    22. ↵
      Food and Drug Administration. FDA drug safety communication: important safety label changes to cholesterol-lowering statin drugs. Food and drug administration website.Updated February 28, 2012 https://www.fda.gov/Drugs/DrugSafety/ucm293101.htm (accessed 20 Aug 2017).
    23. ↵
      2. Wright A ,
      3. Hickman TT ,
      4. McEvoy D , et al
      . Analysis of clinical decision support system malfunctions: a case series and survey. J Am Med Inform Assoc 2016;23:1068–76.doi:10.1093/jamia/ocw005
      OpenUrlCrossRefPubMed
    24. ↵
      2. Goss FR ,
      3. Plasek JM ,
      4. Lau JJ , et al
      . An evaluation of a natural language processing tool for identifying and encoding allergy information in emergency department clinical notes. AMIA Annu Symp Proc 2014;2014:580–8.
      OpenUrlPubMed
    25. ↵
      2. Hammann F ,
      3. Gutmann H ,
      4. Vogt N , et al
      . Prediction of adverse drug reactions using decision tree modeling. Clin Pharmacol Ther 2010;88:52–9.doi:10.1038/clpt.2009.248
      OpenUrlCrossRefPubMed
    26. ↵
      2. Eschmann E ,
      3. Beeler PE ,
      4. Schneemann M , et al
      . Developing strategies for predicting hyperkalemia in potassium-increasing drug-drug interactions. J Am Med Inform Assoc 2017;24:60–6.doi:10.1093/jamia/ocw050
      OpenUrlCrossRefPubMed
    27. ↵
      Office of the National Coordinator for Health Information Technology. Hospital EHR vendors.Updated September 2016 https://dashboard.healthit.gov/quickstats/pages/FIG-Vendors-of-EHRs-to-Participating-Hospitals.php (accessed 1 May 2017).

    Footnotes

    • Funding This study was funded by CRICO.

    • Competing interests DWB reported receiving equity from Intensix, which makes software to support clinical decision-making in intensive care; being named as coinventor on Patent Number 6029138 held by Brigham and Women’s Hospital on the use of decision support software for medical management, licensed to the Medicalis, and holding a minority equity position in Medicalis, which develops web-based decision support for radiology test ordering; consulting for Early Sense, which makes patient safety monitoring systems; receiving equity and cash compensation from QPID, a company focused on intelligence systems for electronic health records; receiving cash compensation from CDI (Negev), which is a not-for-profit incubator for health IT start-ups; receiving equity from Enelgy, which makes software to support evidence-based clinical decisions, from Ethosmart, which makes software to help patients with chronic diseases, and from MDClone, which takes clinical data and produces deidentified versions of it. The remaining authors have disclosed that they do not have any conflicts of interest.

    • Ethics approval Partners Institutional Review Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.