You are here

  • Home
  • Archive
  • Volume 23, Issue 9
  • Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review

Article Text

Article menu

Download PDFPDF

Narrative review
Computerised provider order entry combined with clinical decision support systems to improve medication safety: a narrative review
  1. Sumant R Ranji,
  2. Stephanie Rennke,
  3. Robert M Wachter
  1. Division of Hospital Medicine, Department of Medicine, University of California San Francisco, San Francisco, California, USA
  1. Correspondence to Dr Sumant R Ranji, Division of Hospital Medicine, Department of Medicine, UCSF, 533 Parnassus Avenue, Box 0131, San Francisco, CA 94143-0131, USA; sumantr{at}medicine.ucsf.edu

Abstract

Background Adverse drug events (ADEs) are a major cause of morbidity in hospitalised and ambulatory patients. Computerised provider order entry (CPOE) combined with clinical decision support systems (CDSS) are being widely implemented with the goal of preventing ADEs, but the effectiveness of these systems remains unclear.

Methods We searched the specialised database Agency for Healthcare Research and Quality (AHRQ) Patient Safety Net to identify reviews of the effect of CPOE combined with CDSS on ADE rates in inpatient and outpatient settings. We included systematic and narrative reviews published since 2008 and controlled clinical trials published since 2012.

Results We included five systematic reviews, one narrative review and two controlled trials. The existing literature consists mostly of studies of homegrown systems conducted in the inpatient setting. CPOE+CDSS was consistently reported to reduce prescribing errors, but does not appear to prevent clinical ADEs in either the inpatient or outpatient setting. Implementation of CPOE+CDSS profoundly changes staff workflow, and often leads to unintended consequences and new safety issues (such as alert fatigue) which limit the system's safety effects.

Conclusions CPOE+CDSS does not appear to reliably prevent clinical ADEs. Despite more widespread implementation over the past decade, it remains a work in progress.

Keywords
  • computerized provider order entry
  • clinical decision support systems
  • medication errors
  • adverse drug events
  • patient safety
  • health information technology

https://doi.org/10.1136/bmjqs-2013-002165

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Keywords

    Introduction

    Adverse drug events (ADEs) are a major source of preventable harm in both hospitalised and ambulatory patients. Preventable ADEs—which are primarily a result of prescribing errors1 ,2—occur in 7–10 of every 100 hospital admissions,3–5 and may be even more common in the ambulatory setting.6

    Computerised provider order entry (CPOE) systems require clinicians to directly place orders for medications, tests or studies into an electronic system, which then transmits the order directly to the recipient responsible for carrying out the order (the pharmacy, laboratory or radiology department). These systems have the potential to greatly reduce prescribing errors, since at a minimum they ensure standardised, legible and complete orders. Initially implemented in the inpatient setting with the goal of preventing prescribing errors and ADEs, the use of CPOE systems has increasingly been broadened to include entry of all types of orders in both the inpatient and outpatient settings.

    Clinical decision support systems (CDSS) are often integrated with CPOE, and aim to optimise the safety and quality of clinical decisions by providing clinicians with reminders or recommendations at the point of care. For example, a medication CDSS may offer default values for doses, routes of administration and frequency for commonly used drugs. In more advanced forms, CDSS can also check for drug allergies or drug–drug interactions, provide reminders for appropriate laboratory monitoring (eg, checking renal function if a patient is prescribed a potentially nephrotoxic medication) or even suggest appropriate orders based on patient factors (eg, ordering broad-spectrum antibiotics in a patient admitted with sepsis). At the highest level of sophistication, the combination of CPOE and CDSS—hereafter referred to as CPOE+CDSS—has the potential to prevent ADEs by preventing errors of commission and errors of omission.

    The pace of uptake of CPOE and CDSS has been relatively slow in both the inpatient and outpatient environments,7–9 but is increasing in the USA due to the federal Health Information Technology for Economic and Clinical Health (HITECH) Act. HITECH stipulates that healthcare providers must demonstrate the ‘meaningful use’ of electronic health records (EHRs) by 2015, and has provided incentive payments for implementing such systems, switching to penalties for failing to achieve the meaningful use standard by 2016. The meaningful use criteria require in part that EHRs must include a CDSS system. In the UK, universal implementation of CPOE in both inpatient and ambulatory settings is an area of priority for the National Health Service. Despite this increased usage, questions about the effectiveness of CPOE+CDSS at preventing clinical ADEs remain. A recent systematic review10 estimated that CPOE can reduce medication prescribing errors in hospitalised patients by 48% on average, but acknowledged that ‘it is unclear whether this translates into reduced harm for patients’.

    The 2001 Making Health Care Safer report, published by the US Agency for Healthcare Research and Quality, reviewed evidence on the effectiveness of CPOE+CDSS, as well as isolated CDSS, at improving medication safety.11 The review defined level 1 outcomes as ADEs, and level 2 and 3 outcomes as medication errors and change in prescribing practices, respectively. These definitions were used in order to distinguish the effects of CPOE and CDSS on clinical outcomes (eg, preventable ADEs) and surrogate outcomes that may not have caused patient harm (eg, medication errors). The review identified four studies, all of which found improvement in level 2 and 3 outcomes, but did not document a reduction in preventable ADEs. The review also identified significant problems with generalisability of these results, as all of the studies evaluated ‘homegrown’, institution-specific systems (as opposed to commercial systems purchased from vendors) and often focused on safety of a specific medication or medication class. The review assigned CPOE+CDSS a ‘medium strength of evidence’ recommendation, noting the limited effect of CPOE+CDSS on preventing clinical patient harm. This conclusion proved to be somewhat controversial, with follow-up commentaries arguing that as clinical ADEs are rare, conducting randomised trials of CPOE+CDSS would be costly and impractical, and that proof of clinical benefit should not be required before wider adoption given the face validity of such systems.12 The evidence report's authors responded that rigorous research to evaluate the effectiveness and generalisability of CPOE+CDSS—and all patient safety practices—was necessary in order to appropriately assess and prioritise methods of improving safety.13

    Since this debate, CPOE+CDSS usage has indeed been more widely adopted, and more high-quality research has been undertaken and disseminated. In view of the increasing use of these systems, we performed a narrative review to assess the state of the evidence regarding the effectiveness of CPOE+CDSS at preventing clinically significant ADEs, and evaluate factors such as implementation issues and unintended consequences that may play a role in the success or failure of CPOE+CDSS systems.

    Methods

    This review was performed as part of Making Healthcare Safer II, a series of comparative effectiveness reviews of patient safety strategies commissioned by the US Agency for Healthcare Research and Quality (AHRQ).14 We sought to narratively synthesise the results of recent systematic reviews of the effect of CPOE+CDSS on medication safety. We identified eligible articles by searching the specialised database AHRQ Patient Safety Net (http://psnet.ahrq.gov), which indexes published literature and grey literature on patient safety, using the keywords ‘CPOE’, ‘clinician decision support system’, ‘health information technology (HIT)’, ‘electronic health record’ and ‘electronic medical record’. For the AHRQ technical report on which this article is based, we included systematic and narrative reviews specifically assessing the effect of CPOE+CDSS on clinical ADEs, as well as articles that measured costs, unintended consequences or specific implementation issues relating to CPOE+CDSS implementation. For this article, we updated the systematic review search, and also included controlled studies (randomised or non-randomised) published after the most recent inclusion date of the systematic reviews.

    Results

    Effect of CPOE+CDSS on ADEs

    We identified five systematic reviews and one narrative review published since 2008 that met our inclusion criteria (table 1). Wolfstadt et al15 identified 10 trials of CPOE+CDSS (none of which were randomised controlled trials (RCTs)), and concluded that CPOE+CDSS may be effective at reducing ADEs, as five of the 10 studies found a statistically significant reduction in ADEs and four others reporting a non-significant improvement. However, most of these studies utilised homegrown systems, and nine of the 10 included studies were conducted in the inpatient setting.

    View this table:
    Table 1

    Evidence for the effect of CPOE+CDSS on adverse drug event rates

    Schedlbauer et al16 identified 20 studies (including four RCTs) that evaluated a total of 27 forms of CDSS. The authors classified the CDSS alerts as ‘basic’ (including only information about allergies, drug–drug interactions and default dosing), ‘advanced’ (including alerts targeting errors of omission and patient-specific dosing and safety guidelines) and ‘complex’ (including features of both basic and advanced systems). Only four of these studies—only one of which used a ‘complex’ alert—evaluated the effect of CPOE+CDSS on clinical ADEs; three of them did find statistically significant reductions in preventable ADEs. Ammenwerth et al’s review17 also identified seven observational studies of the effect of CPOE+CDSS on ADE rates; the four studies reporting a significant reduction in ADEs all utilised a CDSS with patient-specific alerts.

    Van Rosse et al’s review18 specifically focused on the effectiveness of CPOE+CDSS in adult and paediatric intensive care units. The 12 observational studies they identified collectively demonstrated reductions in medication prescribing errors; however, no overall effect was found on ADEs or mortality rates.

    As part of a larger systematic review of the effectiveness of HIT on medication management, McKibbon et al19 identified 78 RCTs of various HIT-based interventions on medication safety, including 10 trials specifically evaluating the effect of CPOE+CDSS on ADE rates. Although the studies yielded no conclusive evidence that any form of HIT can improve clinical outcomes, they did find that CDSS that targeted specific clinical problems (eg, ordering blood tests to monitor the safety of anticoagulants) appeared to be more effective than those offering general decision support (eg, providing non-specific recommendations for cardiovascular risk reduction).

    Paediatric inpatients are particularly vulnerable to prescribing errors. A narrative review by Stultz and Nahata20 identified 11 studies evaluating the effectiveness of CPOE+CDSS in this population. They concluded that CPOE+CDSS may reduce ADEs, but were unable to identify specific CDSS features that were associated with ADE prevention.

    These reviews identified several common weaknesses in the CPOE+CDSS literature base. The vast majority of studies have been conducted in the inpatient setting, often within a single hospital, and consisted of relatively small patient populations evaluated for short intervention periods. Many studies also evaluated homegrown systems, with comparatively less evaluation of commercial CPOE+CDSS systems. We did identify two recent controlled trials that studied the effectiveness of systems developed by commercial vendors. Westbrook et al21 conducted a controlled study of the implementation of two different commercial CPOE+CDSS systems in two teaching hospitals, and found that both systems reduced the incidence of serious medication prescribing errors (those considered to have a high likelihood of causing patient harm). The study did not formally assess clinical outcomes, however. Leung et al22 evaluated the effect of a commercial CPOE+CDSS system in five community hospitals over a 5-year period. The overall rate of ADEs actually increased during the study period, although this was primarily due to non-preventable events; the rate of preventable ADEs fell (from 10.7 to 7.0 ADEs per 100 admissions, p<0.001).

    The use of CPOE+CDSS in the ambulatory care setting is less extensively studied. Two recent studies23 ,24 conducted in large, community-based practice settings found that mandatory use of CPOE+CDSS achieved reductions in prescribing errors, but not clinical ADEs—mirroring the evidence from the inpatient setting.

    Taken together, these data indicate that while CPOE+CDSS clearly reduce medication prescribing errors, their effect on prevention of clinical ADEs is inconsistent. Two recent systematic reviews that evaluated the effect of CDSS on a broad range of outcomes provide insight into why CDSS, with or without CPOE, may be less effective than originally thought. A meta-analysis and meta-regression by Roshanov et al25 reviewed 162 RCTs of various CDSS with the goal of identifying the features of effective CDSS. Somewhat surprisingly, systems that integrated CDSS within CPOE were actually less effective, compared with those employing other ways of delivering reminders. Bright et al26 reviewed 148 RCTs of CDSS that provided decision support for a variety of clinical processes. Although CDSS improved the ordering of preventive services, clinical studies and appropriateness of prescribing, the magnitude of these effects was small to moderate, and did not appear to improve clinical outcomes. These disappointing findings are likely explained by the complexity of integrating CPOE+CDSS into the busy clinical environment, and the unintended consequences that can result from flawed implementation of these systems.

    Unintended consequences and adverse effects associated with CPOE+CDSS

    The growth in use of CPOE+CDSS has yielded a more nuanced appreciation of the unintended consequences of the technology. These unintended consequences, which include unfavourable workflow changes, generation of new types of errors and overdependence on the technology, were classified in a seminal 2006 article27 and are summarised in table 2. Clinicians perceive these unintended consequences to be common, and to adversely affect care.28

    View this table:
    Table 2

    Unintended consequences associated with implementation of CPOE+CDSS

    A detailed discussion of the various potential adverse effects of CPOE is beyond the scope of this review, but one particular problem deserves comment, as it likely explains in part why CPOE+CDSS has not yet delivered its promised safety benefits. CPOE+CDSS systems generate alerts for clinicians during the ordering process in order to warn them of the possibility of harmful prescribing, and approximately 3–6%29 ,30 of orders generate alerts in a typical system. Since a busy clinician could enter hundreds of orders in a day, he or she will likely receive multiple, perhaps dozens, of alerts every day. ‘Alert fatigue’ refers to the phenomenon whereby users of a CDSS that generates an excessive number of warning messages tend to ignore many of these alerts—even the ones that warn of potentially serious errors. Alert fatigue is well documented in both the inpatient and ambulatory settings,29 as the CPOE+CDSS literature consistently shows that clinicians override the majority of alerts, even those ‘critical’ alerts warning of potentially severe drug–drug interactions. In one study of an outpatient CPOE+CDSS system, clinicians overrode nearly 90% of ‘high-severity’ alerts,30 and another hospital-based study found that clinicians ignored 75% of even ‘critical’ drug–drug interaction alerts.31 This problem arises in part because most existing CPOE+CDSS systems favour providing comprehensive alerts for all potential drug safety problems rather than focusing alerts on the most clinically significant problems. Studies have shown that as many as 40% of drug interaction alerts may represent false positives,32 and when surveyed, clinicians often cite the questionable clinical significance of alerts as a major reason for overriding them.33

    There is consensus that alert fatigue diminishes the potential safety effects of CPOE+CDSS, but no standardised approach exists to avert this problem. Some studies have successfully ‘tailored’ alerts by incorporating patient-specific characteristics into algorithms for displaying drug warnings. For example, Seidling et al34 achieved a reduction in prescribing errors by tailoring drug alerts at a German hospital. In this study, providers accepted nearly 25% of warnings, much higher than rates generally reported in the literature. However, efforts to tailor drug warnings are currently limited by the lack of standardised consensus definitions for drug–drug interactions that are likely to lead to ADEs, and unclear malpractice implications for users and manufacturers of CDSS systems35 should patients be harmed if an alert is not provided. Recent commentaries35 ,36 have called for better guidance and legal protection to allow greater tailoring of alerts, and a recent consensus conference37 identified the key issues in developing more effective alert mechanisms. Initial recommendations have also been published that list specific drug–drug interactions that pose minimal clinical risk and therefore should not require interruptive alerts.38

    Implementation and costs

    Healthcare organisations must pay very close attention to how CPOE+CDSS is configured and implemented, as failure to effectively implement CPOE+CDSS can lead to substantial frustration on the part of clinicians,39 decreased efficiency,40 and even clinical harm. Information technology implementation is much more than a technical intervention, as it profoundly affects workflow for both clinical and non-clinical staff.41 The workflow changes that result—which include the need to continuously interact with the system, shifting of roles among providers, and creation of workarounds to avert problems with the system—have been implicated in the high-profile abandonment of some systems after implementation,42 and have been associated with patient harm in others.43 Even when systems are implemented successfully, specific aspects of system configuration can lead to unintended consequences. For example, one study44 evaluated the effect of a ‘hard-stop’ warning that essentially prevented co-prescribing of the anticoagulant warfarin and the antibiotic trimethoprim–sulfamethoxazole—a combination associated with serious bleeding risks. The warning was effective at its intended aim, but was abandoned after 6 months because four patients experienced delays in needed treatment with one of the drugs. The technical aspect of how alerts are configured is thus critical to ensuring their effectiveness, as overly rigid rules could lead to delays in therapy, whereas more frequent but less consequential warnings run the risk of inducing alert fatigue.

    Unfortunately, no clear consensus exists on the optimal implementation methods in either the hospital or ambulatory setting. It has become clear that EHR implementation in general must take into account the principles of human factors engineering, tailoring the introduction of the systems so as to minimise disruptions to existing clinician workflow and to avoid problems such as alert fatigue.45 Ongoing support and follow-up with frontline users is also required in order to respond to clinicians’ concerns and make system improvements, since user perceptions of information technology invariably change over time.46 The AHRQ has published the online ‘Guide to reducing unintended consequences of electronic health records’ (http://www.ucguide.org), and several case studies of implementation of commercial CPOE+CDSS systems have also been published.47–51

    The cost–effectiveness of CPOE+CDSS is also unclear; we did not identify any formal cost–effectiveness analyses of CPOE+CDSS published in the past 5 years. Individual institutions with homegrown CPOE+CDSS systems have estimated considerable cost savings52 due to ADE prevention and optimising medication use, but these data may not be generalisable to other settings and systems. In the USA, the HITECH Act's considerable financial incentives have shifted the cost–benefit equation markedly despite the absence of formal cost–effectiveness data.

    Conclusions

    CPOE+CDSS are effective at reducing medication prescribing errors, but there is no clear evidence that these systems reduce clinical ADEs in either the inpatient or outpatient setting. Implementation issues, including failure to adequately tailor warnings and the resultant alert fatigue, may explain this lack of success. Unfortunately, a decade of wider CPOE+CDSS implementation and intensive research does not appear to change the conclusion reached more than a decade ago when the effectiveness of these systems was first systematically reviewed11: despite considerable promise and face validity, these systems do not clearly prevent patient harm.

    The use of CPOE+CDSS will certainly continue to increase worldwide as part of the move towards EHR. Despite their mixed record thus far, it is conceivable that overall safety performance may improve with more widespread use. Multiple studies have demonstrated that increased user experience with EHR is required before safety benefits are achieved—for example, one recent inpatient study50 found that prescribing errors consistently decreased as users became more familiar with the system, and another primary care-based study53 found that clinicians’ perceptions of the system's patient safety effects markedly improved after 1 year of experience. The adverse effects currently associated with CPOE+CDSS implementation may therefore decrease as organisations gain more experience with tailoring these systems to better integrate with clinician workflow and minimise problems such as alert fatigue. However, it is clear that further research and dissemination of best practices in EHR implementation is sorely needed, especially for commercial applications. Until such approaches have been defined, the actual process of implementing CPOE+CDSS will remain exercises in trial and error.

    Although CPOE has specifically been recommended as a patient safety strategy by a wide range of influential organisations, it may not be capable of preventing ADEs on its own. CPOE+CDSS systems are designed to primarily target prescribing errors, and in specific clinical circumstances may also target appropriate monitoring of therapies. However, they have no effect on medication administration errors, which account for a large proportion of medication errors in inpatients. ‘Closed loop’ medication management systems, which combine CPOE+CDSS with other proven technologies to prevent errors at each step of the medication pathway, show great promise for preventing clinical ADEs.54 Fundamentally, though, improving medication safety cannot be viewed as an isolated safety issue. Analyses of serious medication errors55 invariably reveal other underlying system flaws, such as human factors engineering issues and impaired safety culture, which allow individual prescribing or administration errors to reach the patient and cause serious harm. Despite the promise of technological approaches to medication safety, the potential for error will remain unless these systems are carefully implemented and greater attention is paid to developing safer systems of care.

    Acknowledgments

    The authors gratefully acknowledge Yimdriuska Magan, BS for assistance with article preparation, and Lisha Lo, BA for assistance with article abstraction.

    References

    1. ↵
      1. Gurwitz JH,
      2. Field TS,
      3. Harrold LR,
      4. et al
      . Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA 2003;289:1107–16.
      OpenUrlCrossRefPubMedWeb of Science
    2. ↵
      1. Nebeker JR,
      2. Hoffman JM,
      3. Weir CR,
      4. et al
      . High rates of adverse drug events in a highly computerized hospital. Arch Intern Med 2005;165:1111–6.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Abramson EL,
      2. Bates DW,
      3. Jenter C,
      4. et al
      . Ambulatory prescribing errors among community-based providers in two states. J Am Med Inform Assoc 2012;19:644–8.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Bates DW,
      2. Leape LL,
      3. Petrycki S
      . Incidence and preventability of adverse drug events in hospitalized adults. J Gen Intern Med 1993;8:289–94.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Lazarou J,
      2. Pomeranz BH,
      3. Corey PN
      . Incidence of adverse drug reactions in hospitalized patients: a meta-analysis of prospective studies. JAMA 1998;279:1200–5.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Gandhi TK,
      2. Weingart SN,
      3. Borus J,
      4. et al
      . Adverse drug events in ambulatory care. N Engl J Med 2003;348:1556–64.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. DesRoches CM,
      2. Rosenbaum SJ
      . Meaningful use of health information technology in U.S. hospitals. N Engl J Med 2008;362:1153–5.
      OpenUrlWeb of Science
    8. ↵
      1. Jha AK,
      2. DesRoches CM,
      3. Campbell EG,
      4. et al
      . Use of electronic health records in U.S. hospitals. N Engl J Med 2009;360:1628–38.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Aarts J,
      2. Koppel R
      . Implementation of computerized physician order entry in seven countries. Health Aff (Millwood) 2009;28:404–14.
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Radley DC,
      2. Wasserman MR,
      3. Olsho LE,
      4. et al
      . Reduction in medication errors in hospitals due to adoption of computerized provider order entry systems. J Am Med Inform Assoc 2013;20:470–6.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Kaushal RBD
      . Computerized Physician Order Entry (CPOE) with Clinical Decision Support Systems (CDSSs). In: Shojania K, Duncan B, McDonald K, Wachter RM, eds. Making health care safer: a critical analysis of patient safety practices. Rockville, MD: Agency for Healthcare Research and Quality, 2001; Chapter 6, pp 59–70.
    12. ↵
      1. Leape LL,
      2. Berwick DM,
      3. Bates DW
      . What practices will most improve safety? Evidence-based medicine meets patient safety. JAMA 2002;288:501–7.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Shojania KG,
      2. Duncan BW,
      3. McDonald KM,
      4. et al
      . Safe but sound: patient safety meets evidence-based medicine. JAMA 2002;288:508–13.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Shekelle PG,
      2. Pronovost PJ,
      3. Schoelles K,
      4. et al
      . Making Health Care Safer II: An Updated Critical Analysis of the Evidence for Patient Safety Practices. Comparative Effectiveness Review No 211 (Prepared by the Southern California-RAND Evidence-based Practice Center under Contract No 290-2007-10062-I) AHRQ Publication No 13-E001-EF Rockville, MD: Agency for Healthcare Research and Quality March 2013. http://wwwahrqgov/research/findings/evidence-based-reports/ptsafetyuptphtml
    15. ↵
      1. Wolfstadt JI,
      2. Gurwitz JH,
      3. Field TS,
      4. et al
      . The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 2008;23:451–8.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Schedlbauer A,
      2. Prasad V,
      3. Mulvaney C,
      4. et al
      . What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc 2009;16:531–8.
      OpenUrlAbstract/FREE Full Text
    17. ↵
      1. Ammenwerth E,
      2. Schnell-Inderst P,
      3. Machan C,
      4. et al
      . The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 2008;15:585–600.
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. van Rosse F,
      2. Maat B,
      3. Rademaker CM,
      4. et al
      . The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics 2009;123:1184–90.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. McKibbon KA,
      2. Lokker C,
      3. Handler SM,
      4. et al
      . The effectiveness of integrated health information technologies across the phases of medication management: a systematic review of randomized controlled trials. J Am Med Inform Assoc 2012;19:22–30.
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Stultz JS,
      2. Nahata MC
      . Computerized clinical decision support for medication prescribing and utilization in pediatrics. J Am Med Inform Assoc 2012;19:942–53.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Westbrook JI,
      2. Reckmann M,
      3. Li L,
      4. et al
      . Effects of two commercial electronic prescribing systems on prescribing error rates in hospital in-patients: a before and after study. PLoS Med 2012;9:e1001164.
      OpenUrlCrossRefPubMed
    22. ↵
      1. Leung AA,
      2. Keohane C,
      3. Amato M,
      4. et al
      . Impact of vendor computerized physician order entry in community hospitals. J Gen Intern Med 2012;27:801–7.
      OpenUrlCrossRefPubMed
    23. ↵
      1. Devine EB,
      2. Hansen RN,
      3. Wilson-Norton JL,
      4. et al
      . The impact of computerized provider order entry on medication errors in a multispecialty group practice. J Am Med Inform Assoc 2010;17:78–84.
      OpenUrlAbstract/FREE Full Text
    24. ↵
      1. Kaushal R,
      2. Kern LM,
      3. Barron Y,
      4. et al
      . Electronic prescribing improves medication safety in community-based office practices. J Gen Intern Med 2010;25:530–6.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Roshanov PS,
      2. Fernandes N,
      3. Wilczynski JM,
      4. et al
      . Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ (Clinical Research Ed) 2013;346:f657.
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Bright TJ,
      2. Wong A,
      3. Dhurjati R,
      4. et al
      . Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012;157:29–43.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Campbell EM,
      2. Sittig DF,
      3. Ash JS,
      4. et al
      . Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2006;13:547–56.
      OpenUrlAbstract/FREE Full Text
    28. ↵
      1. Ash JS,
      2. Sittig DF,
      3. Poon EG,
      4. et al
      . The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14:415–23.
      OpenUrlAbstract/FREE Full Text
    29. ↵
      1. Isaac T,
      2. Weissman JS,
      3. Davis RB,
      4. et al
      . Overrides of medication alerts in ambulatory care. Arch Intern Med 2009;169:305–11.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Lin CP,
      2. Payne TH,
      3. Nichol WP,
      4. et al
      . Evaluating clinical decision support systems: monitoring CPOE order check override rates in the Department of Veterans Affairs’ Computerized Patient Record System. J Am Med Inform Assoc 2008;15:620–6.
      OpenUrlAbstract/FREE Full Text
    31. ↵
      1. Payne TH,
      2. Nichol WP,
      3. Hoey P,
      4. et al
      . Characteristics and override rates of order checks in a practitioner order entry system. Proc AMIA Symp 2002:602–6.
    32. ↵
      1. Weingart SN,
      2. Toth M,
      3. Sands DZ,
      4. et al
      . Physicians’ decisions to override computerized drug alerts in primary care. Arch Intern Med 2003;163:2625–31.
      OpenUrlCrossRefPubMedWeb of Science
    33. ↵
      1. van der Sijs H,
      2. Aarts J,
      3. Vulto A,
      4. et al
      . Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13:138–47.
      OpenUrlAbstract/FREE Full Text
    34. ↵
      1. Seidling HM,
      2. Schmitt SP,
      3. Bruckner T,
      4. et al
      . Patient-specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care 2010;19:e15.
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Greenberg M,
      2. Ridgely MS
      . Clinical decision support and malpractice risk. JAMA 2011;306:90–1.
      OpenUrlCrossRefPubMedWeb of Science
    36. ↵
      1. Kesselheim AS,
      2. Cresswell K,
      3. Phansalkar S,
      4. et al
      . Clinical decision support systems could be modified to reduce ‘alert fatigue’ while still minimizing the risk of litigation. Health Aff (Millwood) 2011;30:2310–17.
      OpenUrlAbstract/FREE Full Text
    37. ↵
      1. Riedmann D,
      2. Jung M,
      3. Hackl WO,
      4. et al
      . How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study. J Am Med Inform Assoc 2011;18:760–6.
      OpenUrlCrossRefPubMedWeb of Science
    38. ↵
      1. Phansalkar S,
      2. van der Sijs H,
      3. Tucker AD,
      4. et al
      . Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc 2013;20:489–93.
      OpenUrlCrossRefPubMedWeb of Science
    39. ↵
      1. Ghahramani N,
      2. Lendel I,
      3. Haque R,
      4. et al
      . User satisfaction with computerized order entry system and its effect on workplace level of stress. J Med Syst 2009;33:199–205.
      OpenUrlCrossRefPubMed
    40. ↵
      1. Poissant L,
      2. Pereira J,
      3. Tamblyn R,
      4. et al
      . The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005;12:505–16.
      OpenUrlAbstract/FREE Full Text
    41. ↵
      1. Niazkhani Z,
      2. Pirnejad H,
      3. Berg M,
      4. et al
      . The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review. J Am Med Inform Assoc 2009;16:539–49.
      OpenUrlAbstract/FREE Full Text
    42. ↵
      1. Aarts J,
      2. Doorewaard H,
      3. Berg M
      . Understanding implementation: the case of a computerized physician order entry system in a large Dutch university medical center. J Am Med Inform Assoc 2004;11:207–16.
      OpenUrlAbstract/FREE Full Text
    43. ↵
      1. Han YY,
      2. Carcillo JA,
      3. Venkataraman ST,
      4. et al
      . Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 2005;116:1506–12.
      OpenUrlAbstract/FREE Full Text
    44. ↵
      1. Strom BL,
      2. Schinnar R,
      3. Aberra F,
      4. et al
      . Unintended effects of a computerized physician order entry nearly hard-stop alert to prevent a drug interaction: a randomized controlled trial. Arch Intern Med 2010;170:1578–83.
      OpenUrlCrossRefPubMedWeb of Science
    45. ↵
      IOM. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: The National Academies Press, 2012.
    46. ↵
      1. Hoonakker PL,
      2. Carayon P,
      3. Brown RL,
      4. et al
      . Changes in end-user satisfaction with Computerized Provider Order Entry over time among nurses and providers in intensive care units. J Am Med Inform Assoc 2013;20:252–9.
      OpenUrlCrossRefPubMedWeb of Science
    47. ↵
      1. Longhurst CA,
      2. Parast L,
      3. Sandborg CI,
      4. et al
      . Decrease in hospital-wide mortality rate after implementation of a commercially sold computerized physician order entry system. Pediatrics 2010;126:14–21.
      OpenUrlAbstract/FREE Full Text
    48. ↵
      1. Weir CR,
      2. McCarthy CA
      . Using implementation safety indicators for CPOE implementation. Jt Comm J Qual Patient Saf 2009;35:21–8.
      OpenUrlPubMed
    49. ↵
      1. Wetterneck TB,
      2. Walker JM,
      3. Blosky MA,
      4. et al
      . Factors contributing to an increase in duplicate medication order errors after CPOE implementation. J Am Med Inform Assoc 2011;18:774–82.
      OpenUrlCrossRefPubMedWeb of Science
    50. ↵
      1. Abramson EL,
      2. Malhotra S,
      3. Osorio SN,
      4. et al
      . A long-term follow-up evaluation of electronic health record prescribing safety. J Am Med Inform Assoc 2013;20:e52–8.
      OpenUrlCrossRefPubMedWeb of Science
    51. ↵
      1. Cooley TW,
      2. May D,
      3. Alwan M,
      4. et al
      . Implementation of computerized prescriber order entry in four academic medical centers. Am J Health Syst Pharm 2012;69:2166–73.
      OpenUrlAbstract/FREE Full Text
    52. ↵
      1. Kuperman GJ,
      2. Gibson RF
      . Computer physician order entry: benefits, costs, and issues. Ann Intern Med 2003;139:31–9.
      OpenUrlCrossRefPubMedWeb of Science
    53. ↵
      1. El-Kareh R,
      2. Gandhi TK,
      3. Poon EG,
      4. et al
      . Trends in primary care clinician perceptions of a new electronic health record. J Gen Intern Med 2009;24:464–8.
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      1. Poon EG,
      2. Keohane CA,
      3. Yoon CS,
      4. et al
      . Effect of bar-code technology on the safety of medication administration. N Engl J Med 2010;362:1698–707.
      OpenUrlCrossRefPubMedWeb of Science
    55. ↵
      1. Smetzer J,
      2. Baker C,
      3. Byrne FD,
      4. et al
      . Shaping systems for better behavioral choices: lessons learned from a fatal medication error. Jt Comm J Qual Patient Saf 2010;36:152–63.
      OpenUrlPubMed
      1. Pirnejad H,
      2. Niazkhani Z,
      3. Aarts J,
      4. et al
      . What makes an information system more preferable for clinicians? a qualitative comparison of two systems. Stud Health Technol Inform 2011;169:392–6.
      OpenUrlPubMed
      1. Koppel R,
      2. Metlay JP,
      3. Cohen A,
      4. et al
      . Role of computerized physician order entry systems in facilitating medication errors. JAMA 2005;293:1197–203.
      OpenUrlCrossRefPubMedWeb of Science
      1. Van Der Sijs H,
      2. Rootjes I,
      3. Aarts J
      . The shift in workarounds upon implementation of computerized physician order entry. Stud Health Technol Inform 2011;169:290–4.
      OpenUrlPubMed
      1. Abramson EL,
      2. Malhotra S,
      3. Fischer K,
      4. et al
      . Transitioning between electronic health records: effects on ambulatory prescribing safety. J Gen Intern Med 2011;26:868–74.
      OpenUrlCrossRefPubMedWeb of Science

    Footnotes

    • Contributors SRR was responsible for acquisition of data, synthesis of data, drafting and revision of the manuscript. SR was responsible for acquisition and synthesis of data. RMW was responsible for drafting and revision of the manuscript.

    • Funding This study was funded by the Agency for Healthcare Research and Quality, US Department of Health and Human Services (contract no. HHSA-290-2007-100621). The Agency for Healthcare Research and Quality reviewed contract deliverables to ensure adherence to contract requirements and quality, and a copyright release was obtained from the Agency for Healthcare Research and Quality before submission of this manuscript. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.

    • Competing interests None.

    • Provenance and peer review Not commissioned; internally peer reviewed.