Background Computerised provider order entry (CPOE) systems are widely used in clinical settings for the electronic ordering of medications, laboratory tests and radiological therapies. However, evidence regarding effects of CPOE-based medication ordering on clinical and safety outcomes is mixed. We conducted an overview of systematic reviews (SRs) to characterise the cumulative effects of CPOE use for medication ordering in clinical settings.
Methods MEDLINE, EMBASE, CINAHL and the Cochrane Library were searched to identify published SRs from inception to 12 February 2018. SRs investigating the effects of the use of CPOE for medication ordering were included. Two reviewers independently extracted data and assessed the methodological quality of included SRs.
Results Seven SRs covering 118 primary studies were included for review. Pooled studies from the SRs in inpatient settings showed that CPOE use resulted in statistically significant decreases in medication errors and adverse drug events (ADEs); however, there was considerable variation in the magnitude of their relative risk reduction (54%–92% for errors, 35%–53% for ADEs). There was no significant relative risk reduction on hospital mortality or length of stay. Bibliographic analysis showed limited overlap (24%) among studies included across all SRs.
Conclusion SRs on CPOEs included predominantly non-randomised controlled trials and observational studies with varying foci. SRs predominantly focused on inpatient settings and often lacked comparison groups; SRs used inconsistent definitions of outcomes, lacked descriptions regarding the effects on patient harm and did not differentiate among the levels of available decision support. With five of the seven SRs having low to moderate quality, findings from the SRs must be interpreted with caution. We discuss potential directions for future primary studies and SRs of CPOE.
- medication safety
- patient safety
- information technology
Statistics from Altmetric.com
Computerised provider order entry (CPOE) systems support the electronic ordering of medications, laboratory tests and radiology therapies.1 2 By sheer volume, use of CPOEs in clinical environments is staggering: rough estimates suggest that on average 25–30 orders are placed per admission with nearly four to six orders per patient per day.3 4 CPOE systems have led to marked improvements in coordinating care delivery and treatment, streamlining ordering workflow and improving patient safety.5 6 These improvements in CPOE effectiveness have also been spurred by the development of advanced clinical decision support (CDS) functionalities that aid in real-time diagnostic and therapeutic decisions.7
However, the use of CPOE systems has also resulted in undesirable outcomes1 8 including socio-technical issues9 10 such as care coordination difficulties,11 user interface challenges,1 increased workload and fatigue12 and the creation of new types of medication errors.13 14 These unintended consequences have been associated with increased length of hospital stays, adverse drug events (ADEs), higher costs of care and mortality.15–20 Most importantly, the direct impact of CPOEs in reducing medication errors and ADEs has also been questioned.21 The published research literature on CPOE use for medication ordering is considerably vast, disparate, fragmented and heterogeneous (~60 systematic reviews (SRs)) with discordant evidence on the effects of CPOE-based medication ordering on clinical and safety outcomes.22 For example, a recent SR has described CPOEsystems-related medication errors, relative to all medication ordering errors, to range between 6% and 77%12; whereas other SRs have reported that CPOE use has resulted in significant reductions in medication errors.18 23 24
We conducted an overview of SRs (or umbrella review)25–27 to systematically identify, appraise and summarise all SRs on effects of CPOE-based medication ordering on safety and clinical outcomes. Towards this end, we applied rigorous methods from the Cochrane Collaboration26 to minimise any potential bias arising from different sources within and across SRs, and the overview of SR process itself. The goals of this overview of SRs were to: (1) describe the characteristics of SRs on CPOE-based medication ordering and its impact on outcomes; (2) identify gaps and limitations of current CPOE-based SRs and potentially their associated primary research studies; and (3) provide recommendations and directions for future CPOE-based research studies and considerations for performing future SRs on this topic.
Data sources and searches
A systematic search of MEDLINE, EMBASE, CINAHL and the Cochrane Library (Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects and the Health Technology Assessment Database) was developed and conducted by the second author (SK) to identify SRs of CPOE systems published prior to 12 February 2018. Searches were supplemented with manual searching on Google Scholar and screening of relevant SRs’ bibliographies. The search strategy combined multiple keywords (eg, electronic order entry, CDS systems, computerised medication order) with subject heading terms (eg, Medical Order Entry Systems, Decision Support Systems, Clinical) and specialised queries for SRs (eg, systematic[sb] in MEDLINE). No language or publication type restrictions were applied. The Cochrane Collaboration methodology28 and other relevant guidelines29 30 for overviews of SRs were rigorously applied. Extended details of the full search strategy are provided in online supplementary appendix S1.
Selection of SRs
During the preliminary stage of searching, all titles and abstracts were independently screened by two reviewers (JA, SB) according to prespecified inclusion criteria (table 1). Articles eligible for inclusion were SRs investigating the effects of CPOE systems for medication ordering (with or without a CDS) in any clinical setting and evaluating any outcome(s). Based on published literature, we defined CPOE as ‘computer-based systems used for entering orders’18 for medication prescriptions. We excluded SRs that did not meet the inclusion criteria: SRs that reported findings from CPOE systems and isolated CDS systems in a single (ie, same) SR; SRs that were unrelated to CPOE-based medication ordering (eg, laboratory tests); SRs that did not have a comparison group; SRs that specifically focused on evaluation of CDS features (eg, alerts) and scoping, qualitative, or narrative SRs.
Full-text articles of SRs that appeared to be relevant were retrieved and independently assessed for inclusion by two reviewers (JA, SB). Disagreements in the selection of reviews were resolved through discussion in team meetings with three reviewers (JA, SK, TK).
Data extraction and quality assessment
Two reviewers (JA, HV) independently extracted data from each SR using an electronic data extraction form developed for this study. All extracted data were checked for consistency by the third author (AM), and discrepancies were resolved through discussion. Data extraction elements included key characteristics of the reviews including information on study objectives, participants, intervention features and implementation, outcomes assessed and comparisons performed.
Two reviewers (TK, HV) independently evaluated the methodological quality of included SRs using the Assessment of Multiple Systematic Reviews (AMSTAR) tool.31 AMSTAR is a validated instrument with 11 items to assess whether the methods used in a SR are unbiased and methodologically sound. Each of these items are categorised into a standardised set of four possible responses: ‘yes’, ‘no’, ‘can’t answer’, or ‘not applicable’. In order to maintain consistency across assessments for each SR,32 we used decision support rules for scoring each criterion developed by the second author (SK) (online supplementary appendix S2).33
All AMSTAR responses by the two reviewers (TK, HV) were reviewed by the first author (JA), and any identified discrepancies were resolved through discussion among them. Based on the results of this critical appraisal, SRs were categorised into three categories: ‘low’ (score 0 to 3); ‘middle’ (score 4 to 7); and ‘upper’ (score 8 to 11). These groups reflect the existence of ‘major’, ‘moderate’ and ‘minor or no methodological limitations’ in a SR.32 Similar categorisation of methodological quality have been used in other overviews of SRs (eg, studies include Kitsiou et al, Jaspers et al and Sequeira-Byron et al 33–35).
Data synthesis and analysis
Extracted data from each included SR was aggregated into summary figures and tables. The data was summarised in narrative form(online supplementary appendix S3). Pooled summary estimates were reported for SRs reporting on similar outcome measures. We also categorised studies from included SRs into four types of evidence (based on Kaushal et al’s evidence framework36 : randomised controlled trials (RCTs), non-RCTs, observational studies with control groups and observational studies without control groups.
In order to evaluate the representativeness of the included SRs, we conducted a citation and a bibliographic analysis. For the citation analysis, we created citation matrices, similar to the methods described by Martel et al,37 where we cross-linked included SRs with previously published SRs. Using this, we computed the citation ratio, that is, a ratio of cited SRs to available published SRs. A 1-year lag time was used for the calculation of citation counts and citation ratio. This was done to evaluate whether newer SRs included empirical insights derived from SRs published more than a year prior. In addition, we also computed the Pearson correlation of citation counts (ie, of cited SRs) with AMSTAR scores. For the bibliographic analysis, we organised individual studies published across the included SRs, and examined for the presence and type of overlap. All analyses were performed using Microsoft Excel 2016.
Our initial search yielded 5603 citations after removal of 727 duplicates (figure 1). We excluded 5551 references which, based on titles and abstracts, did not meet the inclusion criteria. Next, we retrieved and reviewed the full text of the remaining 52 SRs, and manually screened their references to identify any relevant SRs that were not originally captured by our electronic search strategy. This process yielded five additional SRs. After full-text review, we excluded 50 SRs that did not meet our eligibility criteria. A final list of seven SRs was included.18 23 24 38–41 The excluded articles along with the reasons for their exclusion are provided in online supplementary appendix S4.
Characteristics of included SRs
The number of primary studies included in each SR varied (mean=23.4, median=16, range 10–67, 95% CI: (8.28 to 38.58)), due to differences in their search strategies and inclusion criteria.
Study settings and population
With regards to the type of study settings, three SRs included only inpatient settings,18 24 39 two combined inpatient and outpatient settings (all)23 40 and two included inpatient along with emergency department (ED) settings (ie, inpatient/ED).38 41
Patient populations varied across the SRs: one SR focused exclusively on the adult population38 whereas other SRs included combinations of adult, paediatric and/or neonatal populations.18 23 24 39–41 Only one SR explicitly evaluated physicians as the primary CPOE user.23 Others included combinations of physicians, nurses, nurse practitioners, pharmacists and physician assistants. Two SRs did not specify target users.18 38
Interventions (CPOE functionalities)
Because of the relatively long time period of the primary studies in included SRs (earliest primary study was published in 1989,42 CPOE functionalities varied considerably: CPOE only; CPOE with basic CDS (assist in tasks such as drug selection, dosing and management, checks for interactions with drug–drug, drug–allergy36; and CPOE with advanced CDS (integrates patient-specific diagnoses, risk factors and prior treatments to make a drug recommendation.43 Two SRs did not provide specific details regarding the CPOE functionalities that were considered in the studies39 41; three SRs provided summary analysis based on the CPOE functionalities.18 21 38 Six SRs included both homegrown and commercial CPOEs; one SR focused exclusively on commercial systems18 (table 3).
Study designs and comparisons
The included SRs identified 118 unique primary studies. Informed by Kaushal and colleagues’ evidence framework36, the primary studies were categorised into 8 RCTs (7 in inpatient settings, 1 in an outpatient setting); 110 non-RCTs (88 in inpatient settings, 6 in outpatient settings, 4 in an ED and 12 in both inpatient and outpatient settings).
Comparison groups varied across studies: handwritten orders were compared with CPOEs,18 23 24 38 CPOE without CDS compared with CPOE with basic or advanced CDS18 23 and homegrown CPOE or handwritten orders compared with commercial CPOE.18 One SR did not include details of the comparison or control group41; and another only described the control group as ‘usual care’, without providing additional details40 (online supplementary appendix S5).
Methodological quality of SRs
The methodological quality among the SRs varied, with several SRs having considerable methodological limitations (online supplementary appendix S6). Two of the seven SRs18 38 were classified as ‘high quality’, with scores of 8 or more on the 11-point AMSTAR scale. These SRs closely adhered to most of the methodological criteria, indicating thoroughness in design and execution along with minimal risk of bias. Three SRs were of ‘moderate quality’, with AMSTAR scores between 4 and 723 40 41 and two SRs were ranked as ‘low quality’, with an AMSTAR score of 3.24 39
Only one SR (14%) provided evidence of a previously published protocol or design (Question 1, AMSTAR instrument).31 Only two SRs (29%) mentioned searching the grey literature for eligible studies or restricted their search to articles published in English, limiting their searches considerably and increasing risk for publication and language bias (Question 4). Only one SR (14%) included a list of included and excluded studies, with the remaining six SRs failing to provide one or both lists (Question 5). With regards to scientific quality of primary studies, only four SRs (57%) provided documentation of scientific quality assessment (Question 7). Four SRs (57%) reported on the scientific quality of included studies appropriately and discussed their overall findings (Question 8). Finally, two SRs (29%) assessed publication bias (Question 10), which may have an impact on the interpretation of results in cases where publication bias exists in a body of literature. None of the SRs (0%) assessed conflicts of interest (COI) or sources of support thoroughly of the included studies (Question 11), although three mentioned COIs associated with their SRs. In conclusion, it is plausible that biassed studies may have impacted the findings reported in the included SRs.
Effects of CPOE on outcomes
All SRs assessed the impact of CPOE use on medication safety: six included medication errors as the primary outcome of interest18 23 24 38 39 41 and five used ADEs.23 24 38 40 41 Other outcomes of interest included intensive care unit (ICU) and hospital mortality,18 24 41 length of stay (LOS)18 and implementation outcomes such as cost, usability and user satisfaction.24 41 Two SRs reported on time-related factors such as turnaround time and medication ordering time.24 41 One SR had additional outcomes including impact of alerts and adherence to guidelines as evaluation measures of CPOE effectiveness.41 Three SRs synthesised their findings on effect of CPOE use on outcomes.39–41
Four SRs reported meta-analyses on the effects of CPOE on medication errors and related clinical outcomes. However, the clinical settings and considered outcomes of the included studies differed, making direct comparisons across these difficult: two of these18 24 included all ICUs (neonatal, paediatric and adult); one included all clinical settings,23 while the other included only studies conducted in hospital settings.38 These SRs also did not provide separate analyses based on the CPOE functionalities (ie, based on their CDS features).
Of the SRs reporting meta-analysis, two SRs—Ammenwerth et al and van Rosse et al 23 24—were categorised as having moderate methodological limitations; the remaining two—Prgomet et al 18 and Nuckols et al 38—had only minor or no limitations, indicating their high methodological quality. A summary of the findings based on the outcome measures reported in the SRs with meta-analysis is described in the following sections.
All four SRs with meta-analysis reported the effect of CPOE use on medication errors and found that there was significant decrease in medication ordering errors with CPOE use. Pooled studies showed significant reduction in medication errors with CPOE use, with considerable variation in the magnitude of relative risk reduction (from 54% to 92%): Nuckols et al 38 (setting=inpatient/ED) (RR=0.46, 95% CI: (0.35 to 0.60), n=16 studies, all non-RCT, I 2=98.8%); Prgomet et al 18 (I) (RR=0.15, 95% CI: (0.03 to 0.80), n=9 studies, all non-RCT, I 2=99.6%); and van Rosse et al 24 (I) (RR=0.08, 95% CI: (0.01 to 0.76), n=3 studies, all non-RCT, I 2=35%).
Ammenwerth et al 23 (all settings) did not provide a pooled summary or tests of heterogeneity of the effect of medication errors. Of the 25 studies included in their meta-analysis, 23 studies showed a statistically significant reduction in medication errors with a relative risk ratio ranging between 0.01 and 0.87.23
Adverse drug events
Three SRs also reported on the incidence of ADE with CPOE use.23 24 38 Similar to medication errors, pooled studies showed significant reduction in ADEs with CPOE use, with considerable variation in the magnitude of relative risk reduction (from 35% to 53%): Nuckols et al 38 (RR=0.47, 95% CI: (0.31 to 0.71), n=6 studies, all non-RCT, I 2=69.4%); and van Rosse et al 24 (RR=0.65, 95% CI: (0.40 to 1.08), n=3 studies, all non-RCT, I 2=65%). Ammenwerth et al 23 found seven studies that described changes in ADE after CPOE use; of these, four studies (all non-RCTs) showed a statistically significant reduction in ADE after CPOE use with a relative risk ratio ranging from 0.16 to 0.70.
Two SRs reported on the effect of the use of CPOEs on mortality. However, there were differences across these SRs with Prgomet et al 18 reporting on ICU and hospital mortality separately, and van Rosse et al 24 on all mortality. Pooled studies from Prgomet et al 18 showed that ICU mortality decreased after CPOE use (RR=0.88, 95% CI: (0.78 to 0.99), n=6 studies, all non-RCT, I 2=0%), but there was no significant change in hospital mortality (RR=1.17, 95% CI: (0.53 to 2.54), n=4 studies, all non-RCT, I 2=82.5%).18 Similarly, van Rosse et al 24 also found no significant change in overall mortality after CPOE use (RR=1.02, 95% CI: (0.52 to 1.94), n=4 studies, all non-RCT, I 2=0%).
Length of stay
Prgomet et al 18 was the only study that evaluated the effect of CPOE use on LOS, and found that there was no evidence of relative reduction in LOS with CPOE use (mean difference=−0.10, 95% CI: (−0.81 to 0.60), n=7 studies, all non-RCT, I 2=54.4%).
Citation and bibliographic analysis
The pattern of citations of previous SRs is shown in online supplementary appendix S7. One SR, by van Rosse et al,24 did not cite any prior SRs. Two of the most recent SRs by Prgomet et al 18 and Nuckols et al 38 cited five and four of the previous SRs (included for this review), respectively. Mean number of cited SRs was 2.80 (SD=1.9, 95% CI: (0.91 to 4.68), Range 0–5). The mean citation ratio across all SRs was 0.73 (SD=0.43, 95% CI: (0.31 to 1.15), Range 0–1). There was a high correlation between citation counts (ie, number of previous SRs that were cited) and AMSTAR score (r=0.71).
We also characterised the overlap across primary studies included in the SRs (online supplementary appendix S8). Of the 118 studies that were included in the seven SRs, 29 of them were included in only one SR (24%) showing the low degree of overlap of included studies across SRs. Only two studies44 45 were included in six of the seven SRs..
In this overview of SRs, we identified, critically appraised and synthesised evidence from seven SRs evaluating the effects of CPOE-based medication ordering on safety and clinical outcomes. To the best of our knowledge, this is the first overview of SRs on CPOEs; the seven SRs include 118 primary studies published between 1989 and 2014.
Informed by this overview, we highlight several characteristics regarding the current empirical research evidence on CPOE-based medication ordering. A majority of the primary studies included in the SRs were conducted in inpatient settings. This can potentially be explained by the relative complexities of inpatient medication ordering, where the volume of orders is considerably higher.2 Apart from one that focused on only physicians,23 all SRs either included all clinician users or did not describe the target user (potentially including all clinicians). Similarly, comparison groups in the SRs were unclear in nearly half of the SRs,39–41 making it difficult to draw confirmatory conclusions regarding the effects of CPOE on medication ordering.
Of the 118 primary studies in the SRs, RCTs accounted for less than 7% (n=8) and the randomisation was at the clinical unit-level,46–48 individual clinician-level49 50 or care team-level.51–53 The limited number of RCTs is not surprising, given the challenges of conducting randomised studies with heterogeneous, complex and multi-functional health information technologies.
Despite the overall positive outcomes of CPOEs on medication ordering, our concerns are similar to those raised in prior SR39 54 including the nuanced descriptions and inconsistent characterisation of the outcomes, and the varying methods for capturing, reporting and measuring these outcomes. For example, different versions of ADE definitions were used including general ADEs, preventable ADEs38 and potential ADEs.23 The definition of errors, a commonly used outcome for CPOE studies, should be an important consideration for future studies. Although CPOE use increases the potential for new types of medication-related errors such as duplication,55 wrong-dose,12 55–57 and wrong-patient errors,58–60 errors were often grouped into a single generic class of medication or prescription errors. For developing precise predictions and meaningful CPOE-based CDS interventions, it is imperative that future studies consider robust taxonomies of medication errors.1
Similarly, an understanding of the effects of errors—from no effect to patient harm—can help in further delineating the clinical and safety impact of CPOE-based errors based on its severity. However, only one SR39 conducted a narrative synthesis on the severity of medication errors.
Among the included SRs, CPOE–CDS was described as ‘basic’ or ‘advanced’ with limited characterisation of the CDS features for discerning functional differences. Some SRs treated CDS as a ‘black-box’ with no description of the features, making it impossible to ascertain features that resulted in improved safety and clinical outcomes. Given the rapid growth in CPOE–CDS technology over the past two decades, longitudinal comparisons across findings from CPOE research may require additional interpretations for appropriate use in the modern context. For example, drug–drug interactions, drug–allergy, or dosing for renal impairment may have been considered as advanced CDS features a decade ago, but are part of most modern commercial CPOE systems.61
Similarly with paper-based ordering and home-grown systems being largely obsolete and most healthcare settings in the USA adopting vendor-based systems, the basic features across systems are relatively similar. However, this also provides unique opportunities to develop and test novel CDS algorithms targeting specific types of medication ordering errors (eg, Schedlbauer et al 62). Similarly, with vendor-based systems used in multiple large academic and community hospital systems, it is also possible to conduct multi-centre studies on the effects of CPOE-based medication ordering on patient safety.
The bibliographic analysis demonstrated limited overlap across the primary studies (24%), illustrating limited duplication of CPOE primary studies (given the different research objectives, settings, outcomes, or type of CPOE users). In contrast, the citation analysis illustrated a high mean rate of citation of previously published SRs (M=0.73). It is likely that in order to differentiate from previously published SRs, newer SRs relied on modifying the population (eg, adult, paediatric) and/or setting (ICU, inpatient) of the inclusion/exclusion criteria. Publishing reviews by modifying specific population or settings, rather than broader reviews with sub-group analysis of specific settings or populations, has contributed to some of the duplication and a relatively fragmented evidence base for CPOE-related research.63 Poor reporting practices of SRs also led to redundancies in research efforts. Three relatively similar SRs were published in 2008 (with slight modifications to the setting/population),23 40 41 with none of them having a published protocol. The quality of the SRs was skewed towards low to moderate methodological quality (five of the seven studies). A quality concern common across all SRs was the lack of acknowledgement of the sources of their funding and the funding of the primary studies (Question 11, AMSTAR), which was a potential source of bias.
We acknowledge several limitations. First, our analysis of SRs in this overview of SRs was limited by the reporting characteristics of the included SRs, the quality of evidence and the time lag between the publication of original studies and the reviews. Second, the heterogeneity across the SRs limited our ability to make generalisable observations regarding the effectiveness of CPOE use. Finally, although pooled analysis was considered across SRs that assessed the same outcomes, differences in the comparison groups (or in some cases no clarity on the comparison groups) across SRs may have inflated the positive effects of CPOE.
Informed by this overview of CPOE SRs, we have highlighted potential directions and recommendations for future SRs and primary studies, with emphasis on concerted efforts to build cumulative evidence on the effectiveness of CPOEs on outcomes (table 4). Furthermore with the widespread adoption of CPOEs across the world (eg, Mozaffar et al 64), this overview can inform and facilitate evidence-driven decision making on future CPOE–CDS research/practice.
Contributors JA and SK were involved in developing and refining the search strategy; JA and SB duplicated the screening and full-text for inclusion; JA and HV extracted data from the articles and the final extracted forms were reviewed by AM; HV and TK completed the quality assessment. JA, SK, AM and TK were involved in the analysis. All authors were involved in manuscript writing.
Funding This study was funded by Agency for Healthcare Research and Quality.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data extracted from the included systematic reviews are available upon reasonable request.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.