Article Text

The effectiveness of interruptive prescribing alerts in ambulatory CPOE to change prescriber behaviour & improve safety
  1. Oliver Cerqueira1,
  2. Mohsain Gill2,
  3. Bishr Swar2,
  4. Katherine Ann Prentice3,
  5. Shannon Gwin1,
  6. Brent W Beasley1
  1. 1 Department of Internal Medicine, The University of Oklahoma School of Community Medicine, Tulsa, Oklahoma, USA
  2. 2 Internal Medicine, University of Oklahoma College of Medicine, Oklahoma City, Oklahoma, USA
  3. 3 Houston Academy of Medicine–Texas Medical Center Library, Houston, Texas, USA
  1. Correspondence to Dr Brent W Beasley, Department of Internal Medicine, The University of Oklahoma Health Sciences Center, Tulsa, OK 73104, USA; bbeasle3{at}


Background Prescribing alerts of an electronic health record are meant to be protective, but often are disruptive to providers. Our goal was to assess the effectiveness of interruptive medication-prescriber alerts in changing prescriber behaviour and improving patient outcomes in ambulatory care settings via computerised provider order entry (CPOE) systems.

Methods A standardised search strategy was developed and applied to the following key bibliographical databases: PubMed, Embase, CINAHL and The Cochrane Library. Non-comparison studies and studies on non-interrupted alerts were eliminated. We developed a standardised data collection form and abstracted data that included setting, study design, category of intervention alert and outcomes measured. The search was completed in August 2018 and repeated in November of 2019 and of 2020 to identify any new publications during the time lapse.

Results Ultimately, nine comparison studies of triggered alerts were identified. Each studied at least one outcome measure illustrating how the alert affected prescriber decision-making. Provider behaviour was influenced in the majority, with most noting a positive change. Alerts decreased pharmaceutical costs, moved medications toward preferred medications tiers and steered treatments toward evidence-based choices. They also decreased prescribing errors. Clinician feedback, rarely solicited, expressed frustration with alerts creating a time delay.

Conclusion The current evidence shows a clear indication that many categories of alerts are effective in changing prescriber behaviour. However, it is unclear whether these behavioural changes lead to improved patient outcomes. Despite the rapid transition to CPOE use for patient care, there are few rigorous studies of triggered alerts and how workflow interruptions impact patient outcomes and provider acceptance.

  • ambulatory care
  • healthcare quality improvement
  • information technology
  • interruptions
  • medication safety

Data availability statement

All data relevant to the study are included in the article or uploaded as supplemental information.

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


Computerised provider order entry (CPOE) systems built into most electronic medical records provide the capacity to execute a number of clinical decision support (CDS) tools to present medication-related alerts to providers. Some of the commonly available categories of interruptive medication prescribing alerts include drug–drug interactions (DDIs) and dosage adjustments.1 For example, DDI alerts may be programmed to respond to drug interaction severity, limiting minor reactions such as rash, while presenting more severe ones, such as anaphylaxis.2 The enabling of selective categories of interruptive alerts (eg, serious DDI or inadvertent overdosing) may have the potential to reduce prescribing error rates further. This is especially important when (1) medication errors harm an estimated 1.5 million people every year, costing at least $3.5 billion annually and (2) spending in the USA for prescription drugs continues to escalate.3 4

However, alert fatigue and high rates of alert over-ride are well-recognised consequences of the excessive generation of interruptive alerts. One systematic review reported that 49%–96% of medication alerts were over-ridden by prescribers.5 Implementers must be judicious in their decision to include or exclude available interruptive prescribing alerts in their CPOE systems. While it seems prudent to employ an automated system for monitoring drug interactions, its utility is diminished when half of the alerts are ignored. Previous studies have shown placing a small number of well-designed alerts, those critical to health and safety, may maximise clinician buy-in and achieve the desired safety benefits, without the risk of alert fatigue.6–8

A number of previous studies and reviews have been conducted on CDS for CPOE systems. In 2012, Bright et al included a broad range of decision support services in a systematic review, beyond the interruptive alerts that are our focus. However, it was apparent even then that among ‘…the 12 studies that provided statistical data about provider use, 8 documented low use … or that less than 50% of clinicians used the CDS or received alerts to guide therapeutic action.’9

Other previous systemic reviews focused on the inpatient platform.10 11 These inpatient settings often have existing systems in place to limit potential complications and missed interactions (eg, on-site pharmacists, cardiac and oxygen saturation monitors) and healthcare workers ready to intervene day and night if/when an adverse event or interaction takes place. Outpatient and ambulatory care centres lack this constant surveillance.12 13 While both settings may benefit, theoretically alerts could be more critical to the outpatient arena.


Our specific research objective was to determine the strength of the evidence of the effectiveness of CDS alerts to change prescriber behaviour and/or improve patient outcomes in the outpatient and ambulatory care setting.


Identification of studies

This review included CDS alerts defined as ‘knowledge and person-specific information, intelligently filtered or presented at appropriate times, to enhance health and healthcare.’14 We conducted a systematic review of the literature according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. A standardised search strategy was developed and applied to the following databases: PubMed, Scopus, Web of Science, Embase (Ovid) and CINAHL, and The Cochrane Library. The search was conducted to pinpoint studies describing order entry systems, clinical pharmacy information systems, and the patient safety or medication alerts generated during order entry for ambulatory care settings. A health science librarian performed the searches in each database and used subject headings and keyword groupings as appropriate for each search platform (see online supplemental appendix). The search was completed in August 2018, repeated in July 2019, and again in December 2020 to determine if any new papers meeting our criteria were published during the time lapsed (see figure 1: Search strategies chart).

Supplemental material

Figure 1

Search Strategy flow diagram.

Eligibility criteria

Two independent reviewers then evaluated all records for eligibility. The inclusion criterion were:

  1. Articles must be published from a peer-reviewed journal written in English.

  2. Interventions must have been in an outpatient/ambulatory setting.

  3. Interventions must have used a CPOE system with a CDS tool.

  4. Studies must have evaluated the impact of automatic interruptive alerts that provide immediate notification of potential errors or risks at time of order entry.

  5. Studies must have had a comparison component (eg, comparison/control group, pre/post-test) that examined effectiveness.

  6. Outcome measures included changes in prescribers’ behaviour (eg, cancelling an order or prescribing new/appropriate order and patient outcomes (eg, signs or symptoms suggesting potential or actual harm).

The review was further narrowed by excluding papers meeting the following criteria:

  1. Studies in other practice settings other than outpatient/ambulatory (eg, inpatient).

  2. Clinical information systems other than ambulatory electronic health records (EHRs) (eg, surgery or anaesthesia document systems, intensive care unit systems).

  3. Non-medication intervention (eg, pathology, radiology) order entries.

  4. Other CDS entries (eg, passive decision support or non-interruptive alerts).

  5. Studies with no comparators (eg, no pre-test or comparison or control group).

The PRISMA flow diagram (figure 1) outlines the derivation of the total number of articles identified in table 1.

Table 1

Summary of the studies and their characteristics; of these, only three of the included studies were randomised controlled trials, two were cohort studies, three were pre/post-intervention studies and one was cross-sectional

Data collection process

We developed a standardised data collection form, pilot tested it on 10 studies and refined it further based on group discussion. Data abstracted included the study design, the focus and category of the study’s alert, the publication year, the duration of the study, the setting of the study, the primary outcomes measured, the sample size, whether a power analysis was described in the methodology, whether the authors performed a process evaluation of the intervention, the measurements obtained and whether any validated tools were used for measurement. For all the included studies, we used the Evidence Project’s risk of bias tool, designed to be used across a variety of studies (table 2).15 We also employed the PEDro score for the three randomised trials as a measure of the strength of the study design (table 1).16

Table 2

Risk of bias

We differentiated the primary measures used in each study as ‘outcome measures’, denoted by ‘(O)’, for important clinical or financial outcomes, or ‘process measures’, denoted by ‘(P)’, for measures that are steps in a process or lead to a particular outcome metric.

Data synthesis

The lack of homogeneity between the studies did not allow for data synthesis for a meta-analysis.


Study selection

Figure 1 encapsulates the results of our literature search and selection process. With this methodology, our search identified 796 potential records meeting our criteria. In total, we ultimately screened 527 articles for this review, and found 9 studies meeting our inclusion and exclusion criteria. Table 2 demonstrates the risk of bias with each included study, the randomised trials meeting five to seven of the criteria, Pevnik and Ruhland having the highest risk of bias.


Three studies took place within an academic medical centre, two within a health maintenance organisation/managed care organisation, one within a Veteran’s Administration health centre, two within multiple primary care sites and one within a single standalone primary care site. Two of the studies used standalone electronic prescribing software with decision support that lacked clinical documentation function (history and examination) but technically met our search criteria. Two of the studies were from countries outside the USA.

Study design

For convention, we refer to the first author’s name when discussing each article. The study designs of the three randomised controlled trials were heterogeneous in methodology and focus.

The alert in the Fortuna study focused on appropriate prescribing of sedative hypnotics. This was a cluster randomised trial of clinics and individual clinicians were not tracked through the study period.16 Tamblyn randomised patient encounters to receive computer-triggered versus on-demand CDS alert to decrease prescribing errors.17 Awdishu investigated the impact of a CDS for dosing medications in patients with impaired renal function. The majority of patients in this study were ambulatory, but 38% of the patients were hospitalised. They used ‘prospective’ medication adjustment alerts for active prescribing and ‘look-back’ alerts which fired when the prescribing function was initiated for medications already prescribed.18

For the two cohort studies, the designs (size, numbers of sites) were also disparate. McMullin, the earliest paper (2005) in our review, studied a standalone electronic prescribing system with CDS. The CDS alert presented prescribers with low-cost, evidence-based medication options when they entered a diagnosis, facilitating the order but providing limited choices.19 Andersson compared the impact of drug alerts in two cohorts of clinics when a DDI database was brought online in their EHR.20

The three pre–post studies were varied in focus and outcomes. Rattinger21 reported on types of CDS alerts governing medications prescribed for acute respiratory illness, Ruhland22 did so for diabetes in the elderly. Abramson23 used a standalone prescribing software with CDS and performed a pre–post analysis after installation in 11 primary care practices.

Pevnick studied the adherence to CDS recommendations when using a standalone electronic prescribing system.24 This last article used a cross-sectional methodology.


Summary of outcomes of studies investigating specific medication alert categories

The units of measure differed among the studies. A few used provider adherence as the unit of measurement (eg, provider compliance with alert), some used prescriptions per patient (eg, glyburide orders placed per elderly patient) and a few focused on cost (eg, pharmacy costs after implementation). Several concentrated on alert over-rides by providers, some on the change in the number of alerts after an intervention, some on decreasing prescribing errors and others on reducing the prescribing of targeted medications. The heterogeneity of the studies did not lend themselves to any meta-analysis.

Targets of the alerts

Seven of the nine studies demonstrated significant provider behaviour change. For our included articles, only one reported on patient outcomes—it found a decrease in drug utilisation costs.20 It is noteworthy that the majority of articles omit or do not discuss provider feedback in reference to the impact of alerts. Only one reported provider feedback that was overall ‘positive’. The other provider-provided commentary indicated the alerts were inappropriate and intrusive. Providers valued alerts less when they already addressed risks and benefits with the patients in the clinical encounter.

Results by alert category

Alert to reduce cost and increase preferred tier adherence

Fortuna’s interruptive alert was designed to decrease pharmaceutical cost, triggered by a provider prescribing a new sedative-hypnotic medication rather than a generic.16 The CDS alert delivered recommendations for alternative medications, including ‘specific co-payment information, prescribing information, and patient educational materials…also offered the ability to assign an appropriate diagnostic code and level of service to the encounter.’ Prescribers not receiving the alert prescribed these medications significantly more often than those not receiving them (relative risk (RR)=1.31, 95% CI 1.08 to 1.60).

McMullin’s19 cohort study similarly showed using a standalone prescribing software with CDS lowered average costs per prescription by $1.00 (2.4%) but costs increased by $3.75 (9.0%) in the control group.

Pevnick24 found in their cross-sectional study on a standalone prescribing software (non-EHR), that high users of the formulary decision support more commonly prescribed preferred medication tiers (p<0.04).

Alert to reduce prescribing errors

Prescribing errors occur when a prescription has a problem that reduces likelihood of successfully filling the prescription or increases potential harm. Tamblyn17 studied computer triggered versus on-demand decision support to decrease the number of prescribing errors, showing little difference. Common reasons providers gave for over-riding the CDS were ‘problem already known,’ ‘benefit greater than the risk’ and ‘not clinically important.’

Abramson23 studied the impact of CDS on ‘prescribing errors.’ It compared adopters versus non-adopters of e-prescribing. They found a 1.5-fold (p<0.001) decrease in prescription errors. Non-adopters had no change in their error rate.

Alert to identify DDIs

Andersson used a cohort study design to study the prevalence of potential (not actual) DDIs before and after implementing a DDI warning system.20 The study identified potential DDI’s by linking the EHR’s prescription file to a DDI database.

They found the study group had lower potential drug pair interactions after implementation (RR=0.81; 95% CI: 0.60 to 0.99) than in the control group (RR=0.91, 95% CI: 0.32 to 1.29).

Alert to improve drug and drug dosing appropriateness

Awdishu18 found a ‘prospective’ alert used to drive providers to adjust medication dosages based on the patient’s creatinine clearance was acted on 36% of the time. It significantly increased appropriate prescriber dosages (OR=1.89, p<0.0001). Providers seldom acted on ‘look-back’ alerts that fired when the EHR’s prescription application was opened.

The alert used in Rattinger21 decreased the proportion of unwarranted prescriptions for acute respiratory illness from 22% to 3% (p=0.0001). Similarly, Ruhland22 decreased prescribing of glyburide in older patients from 3.3% to 1.2%.


Summary of main results

In our systematic review, we found CDS alerts drive significant change in clinician prescribing behaviour. Alerts decreased pharmaceutical costs, moved providers toward preferred medications tiers and steered them toward more evidence-based choices to treat disease. They also decreased prescribing errors increasing potential patient safety and decreasing probable downstream pharmacist time and costs. But on the rare times provider feedback was solicited, they felt frustrated when receiving alerts about medication interactions, creating a time delay.

A Veteran’s Administration study found primary care providers already receive on average 56 alerts per day with new information in their EHR inbox, and spend on average 49 min of that day processing and responding to them.25 These are above and beyond the additional time required to digest and react to the CPOE alerts studied in our systematic review. Providers need assurance each interruption is necessary and of high yield, as these ancillary duties required in EHR systems are a known source of burnout.26 A review of eHealth technologies from 2011 suggested a large gap between their postulated and empirical benefits.27

CPOE pop-up alerts lose their credibility with a provider when they are not value added, when the provider believes the alerts are unhelpful and when alerts call attention to known risk–benefit issues she already considered. As their frequency increases, frustration mounts.

A systematic review of outpatient CPOE systems 13 years ago did not demonstrate improved safety or reduced cost of care over paper prescriptions. CPOE did increase adherence to guidelines. But, it increased total prescribing time, and providers often ignored alerts.8 Clinicians’ disdain might be mitigated if alerts portended some benefit to their patients. But, in past reviews and in inpatient settings, medication decision support leads to clinician ‘non-compliance’ because of the ‘irrelevance’ of the alerts.5 6

There are also potential downstream costs when alerts are over-ridden. Like the boy who cried wolf, the providers do not respond to important alerts. Slight et al extrapolated from their institution’s experience and projected providers nationally may be over-riding ~5.5 million medication alerts yearly, leading to associated treatment costs of adverse drug events (ADEs) from these over-rides of over $871 million.28

Providers want to preserve their protected time with patients and to ensure its value. Why must a medication alert system be interruptive, when artificial intelligence in other life sectors and certain healthcare applications is facilitative?8 Smarter medication alert systems offer the provider limited choices, but improve the flow and speed. Medication alerts should protect the patient, empirically lead to decreased costs AND streamline the care.

In the past 13 years, technology has changed significantly. But many CPOE systems have not changed tactics. Some of the innovative inertia may come from national and government initiatives focused on ADEs, such as Leapfrog and the US Department of Health and Human Services.29 30 These external forces focused on having CPOE in use, and while the bar to achieve compliance was relatively low, the cost for organisations was high. So, it is not surprising health systems may not have selected the most elaborate EHRs that facilitate safe workflow, rather than an interruptive system that places the burden on the providers.

More recently in 2018, Marcilly et al defined six overarching design principles for medication alert systems based on their literature review.30 Alerts should:

  1. Improve the system’s signal-to-noise ratio, in order to decrease the frequency of overalerting.

  2. Support collaborative work, advocate a team approach and make the alert system a ‘team player’.

  3. Fit with clinicians’ workflow and their mental model.

  4. Display relevant data within the alert.

  5. Make the system transparent for the user.

  6. Include actionable tools within the alert.

Clinicians have a fast and a slow pathway in their decision-making process. The slow pathway is the ruminative, deliberate thinking. The intuitive, tacit reflex system is the conduit used in prescribing based on clinicians’ historic patterns and well-known algorithms.31 If an alert ‘stops’ that quick thinking, it elicits frustration. Instead, a CDS alert should facilitate the speed at the same time it shuttles care into safe, evidence-based decisions.

Drilling down on item six, the authors recommend alerts that ‘propose formulary drug lists,’ ‘allow ordering a drug suggested’ and ‘open a prepopulated ordering screen for the new drug.’ Recommendations like these go beyond simple interruptive alerts and facilitate the provider’s workflow.

Our review found the quality of the studies is no better than they were 13 years ago, and more needs to be explored. We have three recommendations:

  1. The randomised controlled trials are few in number, and more robust studies are needed. We expect pharmaceutical companies to prove their medications through well-powered clinical trials. CDS and alert trials should do no less.

  2. Developers of medication alert systems should collaborate with clinically active providers to determine what will provide value, defined as eliminating waste (ie, extra provider time, unnecessary medications and lab studies), ensuring safety (ie, decreasing DDI and allergic reactions) and reducing costs (ie, high-cost medications, costs of provider burnout). We need to hear more about the clinicians’ experiences with the alerts.

  3. Data collected in the daily operations of EHR systems should be available for formulating high-specificity CDS for optimal chronic disease management prescribing. Our current laws and rules impair our quick use of EHR data.32


Limitations in this review are largely due to the limited number of high-quality studies existing in this topic. We focused on outpatient ambulatory comparative studies. We wanted to tease out what is specific to outpatient providers and avoid a false sense of precision by pooling outcomes (ie, including inpatient experiences and other professions such as pharmacists). The strength of a systematic review could be potentially hampered by not observing PRISMA guidelines. We attempted to adhere tightly to our review protocols, including two-person review with cross-checks at each step. We repeated our literature searches as the review progressed in an attempt to ensure that all qualified interval publications were included, as well.


Despite the rapid transition to EHRs, there has not been a concomitant acceleration of high-quality research in clinical decision support and medication alerts. Summarising our systematic review, we found limited evidence demonstrating a variety of interruptions in the flow of routine practice that contributes significantly to the safety of patients and improves patient outcomes. Where there was evidence, it was not strong evidence. What we hope to see in the future are high-quality randomised controlled trials powered appropriately to demonstrate what types of medication alerts are high yield for the safety and quality of ambulatory patient care and the sanity of the providers.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplemental information.

Ethics statements

Patient consent for publication


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.


  • MG and BS contributed equally.

  • Contributors All authors above have contributed to this paper and have provided the final version to review.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Linked Articles