You are here

Article Text

Article menu

Download PDFPDF

Quality improvement report
Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation
  1. Norman Lance Downing1,2,
  2. Joshua Rolnick3,4,
  3. Sarah F Poole5,
  4. Evan Hall6,
  5. Alexander J Wessels7,
  6. Paul Heidenreich8,
  7. Lisa Shieh7
  1. 1 Department of Medicine - Biomedical Informatics Research, Hospital Medicine, and Primary Care and Population Health, Stanford University, Stanford, California, USA
  2. 2 Clinical Excellence Research Center, Stanford University, Stanford, California, USA
  3. 3 Division of General Internal Medicine, Department of Medicine and the National Clinician Scholars Program, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
  4. 4 Corporal Michael J. Crescenz VA Medical Center, Pennsylvania, PA, United States
  5. 5 Biomedical Informatics Training Program, Stanford University, Stanford, California, USA
  6. 6 Medicine, Hematology and Oncology, Stanford University, Stanford, California, USA
  7. 7 Medicine, Stanford School of Medicine, Stanford, California, USA
  8. 8 Department of Medicine, Cardiovascular Medicine, Stanford University School of Medicine, Stanford, California, USA
  1. Correspondence to Dr Norman Lance Downing, Medicine, Biomedical Informatics Research, Hospital Medicine, and Primary Care and Population Health, Stanford University, Stanford, CA 94025, USA; ldowning{at}stanford.edu

Abstract

Background Sepsis remains the top cause of morbidity and mortality of hospitalised patients despite concerted efforts. Clinical decision support for sepsis has shown mixed results reflecting heterogeneous populations, methodologies and interventions.

Objectives To determine whether the addition of a real-time electronic health record (EHR)-based clinical decision support alert improves adherence to treatment guidelines and clinical outcomes in hospitalised patients with suspected severe sepsis.

Design Patient-level randomisation, single blinded.

Setting Medical and surgical inpatient units of an academic, tertiary care medical centre.

Patients 1123 adults over the age of 18 admitted to inpatient wards (intensive care units (ICU) excluded) at an academic teaching hospital between November 2014 and March 2015.

Interventions Patients were randomised to either usual care or the addition of an EHR-generated alert in response to a set of modified severe sepsis criteria that included vital signs, laboratory values and physician orders.

Measurements and main results There was no significant difference between the intervention and control groups in primary outcome of the percentage of patients with new antibiotic orders at 3 hours after the alert (35% vs 37%, p=0.53). There was no difference in secondary outcomes of in-hospital mortality at 30 days, length of stay greater than 72 hours, rate of transfer to ICU within 48 hours of alert, or proportion of patients receiving at least 30 mL/kg of intravenous fluids.

Conclusions An EHR-based severe sepsis alert did not result in a statistically significant improvement in several sepsis treatment performance measures.

  • sepsis
  • electronic health record
  • clinical decision support
  • alert
  • protocol

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0

https://doi.org/10.1136/bmjqs-2018-008765

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Background

    Sepsis affects an estimated 650 000–750 000 Americans annually, costing the American healthcare system an estimated $20 billion.1–3 Early recognition, antibiotics and intravenous fluids are the cornerstones of sepsis management. The Surviving Sepsis Campaign recommends administration of antibiotics within 1 hour, citing evidence that even short delays in antibiotic administration lead to increases in mortality. However, despite this recommendation, delays in treatment still often occur.4

    Given the difficulties in the timely recognition of sepsis, healthcare institutions are increasingly leveraging clinical data captured in electronic health records (EHR) and clinical decision support (CDS) systems that incorporate patient-specific context such as vital signs and laboratory values to alert clinicians to the possible presence of sepsis and other clinical deteriorations.5 Despite a growing body of research, the impact of electronic sepsis alerts remains unclear. Alerts have been developed for emergency departments, intensive care units (ICU), and medical wards, and in a range of software platforms: home-grown EHRs, stand-alone software programs and commercial EHRs. However, published studies have reported a wide range of findings: ranging from no impact on bundle compliance or outcome measures, to impact on process measures like antibiotic administration, and one pre-post study that showed a very large effect on mortality.6–14 Inconsistent findings may reflect methodological weaknesses in the existing literature with only a few studies employing rigorous methodologies such as randomisation or reflect implementation differences such as alert algorithms and associated downstream care.

    Most sepsis alert studies have incorporated either home-grown EHRs or other custom software. While this approach can increase flexibility, it creates significant barriers to dissemination and generalisability–most medical centres cannot invest in building, buying or supporting stand-alone alerting systems. Since few studies have limited alert design to the constraints of a standard EHR functionality, implications for broader dissemination, replication and long-term sustainability have been limited.

    In addition to information technology infrastructure, the clinical action downstream of the alert is critical to positive clinical impact. This entails recognition of the alert, acceptance of the recommendation and correct action by the clinician. Few studies have described the protocolised care, change management and follow-up assessment methodology associated with and necessary for an alert to positively effect clinical care.

    To address these gaps, we studied the implementation of a sepsis alert for hospitalised patients with the following approach: (1) patient-level randomisation to control for confounding; (2) using only the EHR functionality to deliver real-time CDS alerts; and (3) a change management strategy that incorporates user-centred design via clinician feedback and standardised clinical workflow.

    Methods

    Definitions and terms

    Severe Sepsis Alert Criteria: The co-occurrence of one or more criteria of suspected infection, one or more criteria of organ dysfunction and three or more criteria of systemic inflammatory response syndrome (online supplementary file 1). Definition was based on the most current Surviving Sepsis Campaign guidelines published at the time of development.

    Supplemental material

    EHR-based alert: This EHR allows customised alerts via pop-up screen or text-page based on specified criteria. A previously developed alert was modified to increase positive predictive value (PPV) to above 60% to decrease alert fatigue which is comparable to most previously published alerts. This standard functionality was used to implement the Severe Sepsis Alert Criteria into an alert that used criteria collected as a part of usual patient care (vital signs, laboratory values, physician orders). The antibiotics (anti-infectives) included in this definition were determined by the hospital quality committee (online supplementary file 2). The alert was delivered via text-page by the paging system directly to prespecified users rather than a pop-up window within the EHR to ensure users received the alert even when not logged into the EHR.

    Supplemental material

    Alert recipients and workflow: Severe sepsis alerts were sent via text-page to a pool of intensive care-trained nurses (crisis nurse) whose dedicated role is to circulate throughout the hospital and assist providers with patients with deteriorating clinical status. The alert was also delivered to the medicine primary treatment teams if the patient was under their care. These roles were decided by stakeholders in nursing, quality improvement and the medical staff.

    Design

    Screening and enrolment

    From December 2014 to July 2016, we conducted a pragmatic, randomised, single-blind evaluation in the hospital medical and surgical wards and step-down units. This protocol was reviewed by the university institutional review board and determined to be a quality improvement project so consent requirement was waived. This project was registered to ClinicalTrials.gov (NCT02376842). Inclusion criteria: age 18 years or older, admitted to the hospital. Exclusion criteria included patients admitted to the ICU, patient code status is comfort care only. Because we believe that even patients on antibiotics with ongoing signs of sepsis may benefit from a consideration of change in therapy, patients on antibiotics prior to the alert were included.

    Randomisation and blinding

    Each inpatient encounter was randomised 1:1 to either control group (silent alert visible retrospectively to study staff) or intervention group (live alert transmitted to clinical staff). Assignment to control or intervention group was not known by patients but, in the case of the intervention group, known to the alert recipients (figure 1).

    Figure 1
    Figure 1

    Enrolment and randomisation.

    Intervention and change management

    For patients assigned to the intervention group, when the alert criteria were met, the alert automatically generated a text-page sent via the paging system to the crisis nurse and the primary treatment team physician for medical teams. Surgical and other services opted out of receiving the alert directly so alerts generated for their patients were sent to the crisis nurse only.

    The text-page displays the patient’s bed location and the most recent set of vital signs. Before and during implementation of this alert, nursing and physician leadership developed and codified standardised clinical workflow that should occur downstream of the alert. Recipients are instructed to go to the patient bedside to assess and discuss next steps including a clinical assessment of whether or not severe sepsis is present. The physician places appropriate orders and the crisis registered nurse (RN) mobilises other necessary clinical resources and documents the clinical diagnosis and actions taken. The crisis RN also has protocol orders which enable them to order blood cultures and repeat lactate if the primary team is not available. After firing, the alert is silenced for 48 hours. The implementing team educated care teams and the crisis nurses about the alert, the expected response and the rationale for implementation. Educational efforts included how to recognise severe sepsis, treatment guidelines and the associated order set, including early antibiotic administration and fluid resuscitation recommendations.

    Ad hoc feedback was obtained from end users after implementation to iterate on educational efforts and to improve buy-in. We also formally surveyed end users to understand attitudes towards order sets and the impact of educational efforts.

    Physician surveys

    A member of the research team reviewed 98 alerts during a random 4-week period in March 2015, of which 63 alerts (64%) were deemed to be a true positive. The associated clinicians were then contacted by phone or email to understand why treatment decisions often differed from institution recommendations. Since antibiotic compliance was high, the survey focused on fluid resuscitation and the use of an order set designed to improve fluid resuscitation. Forty-two clinicians responded to the survey, answering whether they used the EHR sepsis order set, and what, if any, concerned them about 30 mL/kg intravenous fluid resuscitation.

    Data collection

    Patients were followed until hospital discharge. Demographic and baseline characteristics, vital signs, laboratory results, orders, medication administration, ICU transfer and clinical outcomes were collected within the EHR as part of routine clinical care and extracted for analysis with assistance from an institutional clinical data warehouse (Stanford Translational Research Integrated Database Environment).15 These data were abstracted into a database by blinded personnel. The primary endpoint was the percentage of patients who had a new order for antibiotics within 3 hours of the alert firing (see online supplementary file 2 for definition of antibiotics). Post hoc secondary endpoints included: blood cultures ordered, serum lactate ordered, intravenous fluids, at least 30 mL/kg of intravenous fluids, length of stay (LOS) greater than 72 hours, rate of transfer to ICU and inpatient mortality.

    Statistical analysis

    Based on prior studies, we predicted that the alert might lead to a 10% increase in antibiotic administration.9 To be conservative, we assumed that administration would increase only among true positives. Given an initially estimated true positive rate of 40%, we therefore expected a 4% overall change. With a significant threshold of 0.05, planned enrolment was 3000 encounters for both groups. However, randomisation was terminated early by the hospital quality committee to expand the alert to all eligible patients—only 1123 encounters were randomised before early termination.

    Continuous outcomes were evaluated using a two-sided t-test. Categorical outcomes were assessed with a Pearson’s χ2. A significance threshold of 0.05 was used for all comparisons. Bonferroni correction was employed for secondary analyses. All analyses were performed using R statistical software V.3.1.1, with epitools V.0.5–7 and forestplot V.1.1 packages (RStudio Team 2015). All analyses were performed at the hospitalisation level.

    Results

    Enrolment and baseline characteristics

    Before termination of randomisation, 1123 patients met the alert criteria and were randomised. The 528 patients to the silent alert and the 595 patients randomised to the active alert had similar baseline characteristics and prealert vital signs and relevant laboratory values. There were no significant differences between groups in baseline characteristics (table 1). Patients averaged 63 years old, were equally male or female, largely non-Hispanic and White and largely cared for by the internal medicine teaching service. Approximately 20% had a diagnosis of heart failure (International Classification of Diseases, Ninth Revision, ICD-9 428.0) and 26% sepsis, respectively (ICD-9 038.0, 038.12, 038.2, 038.3, 038.40, 038.42, 038.43, 038.49, 038.8, 038.9, 003.1).

    View this table:
    Table 1

    Baseline characteristics by treatment group

    Primary and secondary outcomes

    There was no significant difference between intervention and control groups in the percentage of patients with new antibiotic orders within 3 hours of the alert (35.0% vs 36.7%, p=0.53), though a large proportion were already on broad spectrum antibiotics at the time of the alert (66.0% vs 66.3%, p=0.93). There was no difference in secondary outcomes of the percentage of patients who received any intravenous fluids (23.70%% vs 19.89%, p=0.86), lactate ordered (14.12% vs 10.8%, p=0.75), at least 30 mL/kg intravenous fluids (1.5% vs 1.5%, p=1.0), in-hospital mortality at 30 days, (6.2% vs 6.8%, p=1.0), LOS greater than 72 hours (3.2 vs 3.2, p=1.0) or rate of transfer to ICU within 48 hours of alert (6.7% vs 5.7%, p=1.0) (figure 2).

    Figure 2
    Figure 2

    Primary and secondary outcomes. ICU, intensive care unit.

    Survey results

    Of the 63 clinician contacts, 42 completed the survey (67%), which was administered either in person or over the phone. Most were first year, male resident physicians (table 2). Twenty-six of the 42 surveys that were completed related to episodes that were chart validated as true positive for severe sepsis (62%). Only two out of the 26 physicians (8%) used the EHR sepsis order set. Physicians reported a variety of reasons, most commonly reporting that they preferred to enter sepsis orders individually (69%) and that the alert did not capture a true change in clinical status (58%). Only two out of the 26 physicians administered the recommended 30 mL/kg of intravenous fluids in cases of chart review confirmed sepsis hypoperfusion (hypotension or hyperlactaemia). The physicians reported that the primary reasons they did not do so were that they were worried about fluid overload (63%) and that the patient was not sufficiently hypotensive to warrant that degree of intravenous fluids (54%) though several also reported disagreement of the diagnosis of sepsis (17%) or were not aware of fluid resuscitation guidelines (13%).

    View this table:
    Table 2

    Demographic characteristics and physician attitudes of sepsis order set and intravenous fluid recommendations

    Discussion

    This pragmatic, randomised evaluation of an EHR-based sepsis CDS alert in an inpatient population demonstrated no significant improvement in either the primary endpoint or any secondary endpoint. While some prior studies have shown an impact of sepsis alerts on antibiotics and other therapeutic endpoints, many of these observational studies have weak designs.5 More robust evaluations have yielded inconsistent results. One randomised evaluation of a modified systemic inflammatory response alert in an ICU found no impact on therapeutic endpoints, while a ward-level comparison of an alert to usual care showed an increase in antibiotic administration.8 9 Given the multifactorial nature of sepsis care, pre-post comparisons are subject to significant confounding and these results should be interpreted with caution.12 16 17 Indeed, in the two randomised trials done in the same academic centre, no difference was seen in outcomes.7 8

    Our study offers a new contribution to the small body of randomised evaluations of sepsis alerts. It is the first patient-level randomised evaluation of severe sepsis alert, one targeting patients with organ dysfunction from sepsis. Our alert is therefore more consistent than prior studies with the trend to conceptualise sepsis as organ dysfunction caused by host dysregulation.18 In addition, most prior studies leveraged expensive, customised software that would be difficult to replicate—akin to using a specialised medication or procedure unavailable to most of the world. Our alert uses only off-the-shelf functionality from a common EHR platform, making it potentially generalisable to many health systems. Finally, we invested considerable time and resources to develop a protocolised response to the alert and train staff appropriately, believing that this investment would increase the likelihood that the alert would impact care. However, despite the protocolised care, we did not observe an effect of the alert.

    There are multiple possible reasons that we did not observe a significant effect of the alert. First, baseline levels of compliance were high with approximately 66% of patients already receiving antibiotics at the time of alert (online supplementary file 3). Our institution had previously taken steps to educate staff and promote awareness of sepsis management. These steps led to a high baseline compliance, leading to a lower marginal return on improved detection through an alert. Second, survey data revealed that clinicians had already diagnosed severe sepsis and initiated treatment in a significant portion of true positive cases—the alert lagged behind their clinical judgement. Third, some of the non-responses may be attributable to false positives. We determined by chart review that the alert’s PPV was 61%. This PPV is equal or superior to that of most prior studies: one review found alerts to have PPVs ranging from 20.5% to 53.8%.5 Nevertheless, nearly 40% of alerts were false positives, and clinicians may have reacted appropriately to these by not initiating treatment. Although we calibrated the alert to a high PPV, we were limited by the detection parameters available from the EHR and the baseline prevalence of severe sepsis among non-ICU patients. Lastly, randomisation was terminated early by the institution’s quality improvement leadership, outside of the authors’ control, so there was insufficient power to detect differences between groups. Early termination occurred because of institutional priorities and reflects the realities of conducting a randomised evaluation in the context of a quality improvement programme. Another notable finding was that the in-hospital mortality rate was 6%, lower than many previously published reports of sepsis mortality.2 3 There are several possible reasons the mortality in this study was lower. First, our study excludes ICU patients, whereas many prior studies included them. Second, changes in identification, treatment and coding of severe sepsis make it difficult to establish a clear reference point for severe sepsis. Past estimates including ICU patients using severe sepsis diagnostic codes may obtain a higher rate. In addition, the alert’s PPV was approximately 60%, and not all subjects had sepsis.

    Supplemental material

    Results of a clinician survey suggest some reasons that the alert did not impact care. In a survey of clinicians’ attitudes towards the alert, respondents’ concern for fluid overload appears to have greatly limited adherence to the recommended amount of intravenous fluids though only 20% of patients carried a diagnosis of heart failure. Though the recommended volume of intravenous fluid is not always indicated, the low achievement in this area represents a potential opportunity to change attitudes towards fluid resuscitation. It highlights the importance of change management and local culture in achieving change in clinician behaviour. A significant barrier in improved care remained clinician adherence to treatment recommendations—a common theme identified in CDS.19–22 Though there are advantages to providing CDS within the EHR at the point of order entry, we opted for text-paging to ensure prompt delivery. This, and the inability to provide an explicit recommendation via the text message due to length constraints, may have decreased the alert’s impact as well.

    Future research should explore how to improve the effectiveness of sepsis alerts. Improved accuracy, and in particular PPV, is greatly needed as low specificity alert contributes to alert fatigue which further reduces the potential efficacy.23 Even if accuracy is improved, achieving clinical outcomes depends on the subsequent clinical actions. Implementation science research is needed to understand why alerts alone do not change provider behaviour. Why do providers still fail to always meet established process outcomes, such as administration of antibiotics and intravenous fluids, even when the alert is accompanied by an educational campaign? What could be done to redesign alerts to be more responsive to provider needs? Human–computer interaction research could contribute to the ergonomics of alerts, finding the best channels and mechanisms to deliver them. Machine learning techniques offer potential to produce more accurate and predictive alerts which in turn may reduce alert fatigue and engender trust by clinicians. However, routine use of more advanced models may not permit explanations for predictions and will be limited by the capabilities of commercial EHR systems until these platforms support more complex algorithms in real time. Collaboration between vendors and researchers will be needed to ensure that advances in sepsis detection have a practical path to implementation and additional randomised evaluations will be critical to evaluating the ultimate clinical impact.

    Conclusions

    Although it is feasible for a health system to develop and randomise the implementation of a severe sepsis alert using only functionality from a commercial EHR, the effect on sepsis care outcomes is unclear. Results of observational studies of sepsis alerts likely overestimate the impact of alerts, due in part to increased identification and documentation of sepsis. More research is needed to understand how to design sepsis alerts to change provider behaviour.

    References

    1. 1.↵
      2. Angus DC ,
      3. Linde-Zwirble WT ,
      4. Lidicker J , et al
      . Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med 2001;29:1303–10.doi:10.1097/00003246-200107000-00002
      OpenUrlCrossRefPubMedWeb of Science
    2. 2.↵
      2. Gaieski DF ,
      3. Edwards JM ,
      4. Kallan MJ , et al
      . Benchmarking the incidence and mortality of severe sepsis in the United States. Crit Care Med 2013;41:1167–74.doi:10.1097/CCM.0b013e31827c09f8
      OpenUrlCrossRefPubMedWeb of Science
    3. 3.↵
      2. Dombrovskiy VY ,
      3. Martin AA ,
      4. Sunderram J , et al
      . Rapid increase in hospitalization and mortality rates for severe sepsis in the United States: a trend analysis from 1993 to 2003. Crit Care Med 2007;35:1244–50.doi:10.1097/01.CCM.0000261890.41311.E9
      OpenUrlCrossRefPubMedWeb of Science
    4. 4.↵
      2. Sterling SA ,
      3. Miller WR ,
      4. Pryor J , et al
      . The impact of timing of antibiotics on outcomes in severe sepsis and septic shock: a systematic review and meta-analysis. Crit Care Med 2015;43:1907–15.doi:10.1097/CCM.0000000000001142
      OpenUrlCrossRefPubMed
    5. 5.↵
      2. Makam AN ,
      3. Nguyen OK ,
      4. Auerbach AD
      . Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J Hosp Med 2015;10:396–402.doi:10.1002/jhm.2347
      OpenUrlCrossRefPubMed
    6. 6.↵
      2. Nelson JL ,
      3. Smith BL ,
      4. Jared JD , et al
      . Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med 2011;57:500–4.doi:10.1016/j.annemergmed.2010.12.008
      OpenUrlCrossRefPubMed
    7. 7.↵
      2. Semler MW ,
      3. Weavind L ,
      4. Hooper MH , et al
      . An electronic tool for the evaluation and treatment of sepsis in the ICU. Crit Care Med 2015;43:1595–602.doi:10.1097/CCM.0000000000001020
      OpenUrlCrossRefPubMed
    8. 8.↵
      2. Hooper MH ,
      3. Weavind L ,
      4. Wheeler AP , et al
      . Randomized trial of automated, electronic monitoring to facilitate early detection of sepsis in the intensive care unit*. Crit Care Med 2012;40:2096–101.doi:10.1097/CCM.0b013e318250a887
      OpenUrlCrossRefPubMedWeb of Science
    9. 9.↵
      2. Sawyer AM ,
      3. Deal EN ,
      4. Labelle AJ , et al
      . Implementation of a real-time computerized sepsis alert in nonintensive care unit patients. Crit Care Med 2011;39:469–73.doi:10.1097/CCM.0b013e318205df85
      OpenUrlCrossRefPubMedWeb of Science
    10. 10.↵
      2. Umscheid CA ,
      3. Betesh J ,
      4. VanZandbergen C , et al
      . Development, implementation, and impact of an automated early warning and response system for sepsis. J Hosp Med 2015;10:26–31.doi:10.1002/jhm.2259
      OpenUrlCrossRefPubMed
    11. 11.↵
      2. Narayanan N ,
      3. Gross AK ,
      4. Pintens M , et al
      . Effect of an electronic medical record alert for severe sepsis among ED patients. Am J Emerg Med 2016;34:185–8.doi:10.1016/j.ajem.2015.10.005
      OpenUrlPubMed
    12. 12.↵
      2. Manaktala S ,
      3. Claypool SR
      . Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inform Assoc 2017;24:88–95.doi:10.1093/jamia/ocw056
      OpenUrlCrossRefPubMed
    13. 13.↵
      2. Nguyen SQ ,
      3. Mwakalindile E ,
      4. Booth JS , et al
      . Automated electronic medical record sepsis detection in the emergency department. PeerJ 2014;2:e343.doi:10.7717/peerj.343
    14. 14.↵
      2. Meurer WJ ,
      3. Smith BL ,
      4. Losman ED , et al
      . Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc 2009;57:40–5.doi:10.1111/j.1532-5415.2008.02094.x
      OpenUrlCrossRefPubMedWeb of Science
    15. 15.↵
      2. Lowe HJ ,
      3. Ferris TA ,
      4. Hernandez PM , et al
      . STRIDE--An integrated standards-based translational research informatics platform. AMIA Annu Symp Proc 2009;2009:391–5.
      OpenUrlPubMed
    16. 16.↵
      2. McRee L ,
      3. Thanavaro JL ,
      4. Moore K , et al
      . The impact of an electronic medical record surveillance program on outcomes for patients with sepsis. Heart Lung 2014;43:546–9.doi:10.1016/j.hrtlng.2014.05.009
      OpenUrl
    17. 17.↵
      2. Evans RS ,
      3. Kuttler KG ,
      4. Simpson KJ , et al
      . Automated detection of physiologic deterioration in hospitalized patients. J Am Med Inform Assoc 2015;22:350–60.doi:10.1136/amiajnl-2014-002816
      OpenUrlCrossRefPubMed
    18. 18.↵
      2. Shankar-Hari M ,
      3. Phillips GS ,
      4. Levy ML , et al
      . Developing a new definition and assessing new clinical criteria for septic shock: for the third International consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:775–87.doi:10.1001/jama.2016.0289
      OpenUrlCrossRefPubMed
    19. 19.↵
      2. Roshanov PS ,
      3. Fernandes N ,
      4. Wilczynski JM , et al
      . Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ 2013;346.doi:10.1136/bmj.f657
    20. 20.↵
      2. Garg AX ,
      3. Adhikari NKJ ,
      4. McDonald H , et al
      . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005;293:1223–38.doi:10.1001/jama.293.10.1223
      OpenUrlCrossRefPubMedWeb of Science
    21. 21.↵
      2. Bright TJ ,
      3. Wong A ,
      4. Dhurjati R , et al
      . Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012;157:29–43.doi:10.7326/0003-4819-157-1-201207030-00450
      OpenUrlCrossRefPubMedWeb of Science
    22. 22.↵
      2. Kawamoto K ,
      3. Houlihan CA ,
      4. Balas EA , et al
      . Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005;330.doi:10.1136/bmj.38398.500764.8F
    23. 23.↵
      2. Bonafide CP ,
      3. Localio AR ,
      4. Holmes JH , et al
      . Video analysis of factors associated with response time to physiologic monitor alarms in a children's Hospital. JAMA Pediatr 2017;171:524–31.doi:10.1001/jamapediatrics.2016.5123
      OpenUrl

    Footnotes

    • NLD and JR are joint first authors.

    • Contributors NLD and JR contributed equally to this manuscript.

    • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

    • Competing interests None declared.

    • Patient consent for publication Not required.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    Linked Articles