To catch a killer: electronic sepsis alert tools reaching a fever pitch?
========================================================================

* Halley Ruppel
* Vincent Liu

*   sepsis
*   alert
*   clinical decision support
*   predictive model
*   machine learning

Sepsis is a global health priority of staggering impact, resulting in at least 6 million deaths worldwide each year and contributing to as many one half of all hospital deaths in the USA.1–4 Sepsis is also tremendously costly, as reflected in total healthcare expenditures,5 6 short-term and long-term morbidity and mortality7–9 and the heavy burden placed on caregivers and society.10 11 Large-scale efforts, including those of the WHO and the Global Sepsis Alliance, have helped to elevate sepsis to a highly prominent concern visible to ‘the public, political leaders and leaders of healthcare systems’.1 12 Emerging public awareness campaigns—for example, the Sepsis Alliance’s ‘It’s About TIME’ motto emphasises Temperature, Infection, Mental decline and Extreme illness as concerning patient symptoms13—further drive home the need for timely and aggressive patient screening, identification and treatment. Together, these clarion calls highlight the need to leverage all available tools and modalities to enhance the earlier identification and treatment of patients to combat sepsis.

Not surprisingly, over the last decade, we have witnessed a rapid expansion in the number of electronic sepsis alert tools in development or use, particularly in locales that have also seen widespread deployment of modern electronic health record (EHR) systems.14 In some cases, simpler rule-based sepsis screening or prognostication tools, built around the systemic inflammatory response syndrome (SIRS) or quick Sepsis-related Organ Failure Assessment (qSOFA) criteria,15–18 have been electronically implemented as sepsis ‘sniffers’ that offload the burden of sepsis detection in patients who already meet relevant clinical criteria.14 In other cases, the role of these simpler ‘predictive’ models has been questioned alongside the rise of machine learning (ML)-driven sepsis predictive models.19–24 Numerous ML-based sepsis predictive models have already demonstrated excellent predictive performance with still many others being designed and tested today.20–25 

ML algorithms are particularly useful in sifting through large, complex and heterogeneous data in order to maximise the signal within the noise of messy EHR data.26 27 By rapidly peering through vast swaths of data, these algorithms can bolster model discrimination—most often quantified as c-statistic values—and improve model calibration.28 29 Although many existing studies trumpet modest increases in c-statistics, improving the positive predictive value (PPV) is also critical because it describes the likelihood that a patient triggering an alert will actually have the outcome of interest. Thus, it can be used to estimate the clinical burden associated with predictive model alerts when embedded in practice. In some cases, partly because of poor PPV performance, reports reveal that sepsis alerting tools have already been shut off.30 

Understanding the impact of predictive model performance in the context of clinical workflow is essential because these tools do not exist in a vacuum. Instead, predictive models must be paired with effective interventions in a prediction-action or afferent-efferent dyad.31 32 The term ‘precision delivery’ was recently coined to reflect the need for risk tools to be embedded within clinical delivery systems to facilitate targeted and personalised care.33 Decisions on what actions should follow a predictive model alert can be highly variable. When faced with a predictive model of given characteristics, operational leaders must make decisions related to the tool’s use in the areas of alert delivery modalities and thresholds, end-user staffing and interfaces, clinical decision support, workflow changes and/or educational programming. For sepsis, multifaceted interventions that coordinate improvements across several clinical domains have become the norm and, in nearly all cases, have already proven highly effective for improving mortality.34–37 

Against this background, what is the evidence showing that electronic sepsis alert tools benefit patients? Several prior studies suggest that their use is associated with incredible benefits in outcomes. For example, when embedded within clinical workflow redesign, a system incorporating electronic surveillance criteria resulted in a stunning 53% reduction in sepsis mortality.38 A more complex ML-driven sepsis predictive model similarly resulted in a 58% reduction in hospital mortality, with no associated increase in adverse events.39 In a condition as deadly as sepsis and with purported effects of this magnitude, it comes as no surprise that many hospitals and health systems are racing towards implementation of electronic alerts and predictive models. However, not all studies have shown such promising results.14 40–42 

In this issue, an important study by Downing and others43 helps to enrich our understanding of the potential utility of electronic sepsis alert tools. The authors modified a previously developed severe sepsis identification algorithm44 based on SIRS, suspected infection and organ dysfunction criteria, to enhance PPV. They implemented this EHR-based alert for patients in medical, surgical and stepdown units, excluding those in intensive care units or on comfort care. In coordination with hospital leadership, they carefully developed a standardised workflow following the alerts and educated the relevant clinical teams. When alert criteria were met, a circulating ‘crisis’ nurse, and in some cases the patient’s treating physician, would receive a pager alert with the intended goal of having clinicians assess the patient at the bedside and implement appropriate orders within an order set. Overall, their work reflects the type of careful approach needed to implement an effective and sustainable prediction-action dyad.

What makes this study particularly valuable is that the authors implemented the sepsis alerts in a randomised fashion. Among patients meeting alert criteria, some of their treating clinicians received a paged alert (intervention, n=595) whereas other patients had a ‘silent’ alert invisible to their treating teams (control, n=528). As a result, the study findings should reflect the causal effect of the sepsis alert and workflow itself, rather than other exogenous or confounding factors that often impact more commonplace before-after studies.45 46 

Their randomisation process succeeded in achieving intervention and control groups that shared similar baseline characteristics. However, their sepsis alert did not significantly affect the primary outcome of a new antibiotic order placed within 3 hours of the alert (35.0% vs 36.7%). Nor were there any significant differences between intervention and control patients in a diverse set of secondary care processes and outcomes including lactate orders, intravenous fluid administration, blood cultures, prolonged length of stay, intensive care transfer or hospital mortality. Even while this iteration of their tool had no significant effect on sepsis care processes or outcomes, ironically, the study was cut short by hospital leadership who requested that the alert be turned on for all users.

Many factors may have contributed to the lack of benefit seen with the intervention. For example, in both groups, two-thirds of patients were already being treated with antibiotics at the time of the alert. However, the study also did not demonstrate significant differences in intravenous fluid administration or lactate orders between groups. The alert was also designed primarily to improve detection of severe sepsis, rather than to predict onset, limiting the tool’s utility for pre-empting severe sepsis with clinical intervention. The authors found that alert-driven workflows were inconsistently applied by clinicians and wisely conducted surveys to better understand their findings. Of physicians surveyed who had cared for a patient with sepsis, the majority felt that the alert did not flag an important change in a patient’s condition that required new action.

Sepsis is a deadly, prevalent and costly healthcare problem that demands urgent attention. Promising electronic alert tools are increasingly being implemented in the hopes that they can drive improved patient outcomes. However, as this study and others show, well-designed tools demonstrating excellent in silico performance are not guaranteed to improve care or outcomes. In real-world practice, some may even result in unintended consequences like alert fatigue,47 distraction48 and wasted resources. Given the tremendous heterogeneity in the electronic tools themselves as well as in the clinical settings in which they are implemented, we expect variability in reported benefits to persist. Rigorous study designs, as well as the confidence to publish ‘negative’ results, are essential for identifying effective and sustainable interventions that benefit our patients with sepsis.

## Footnotes

*   Funding Vincent Liu is funded by NIH R35128672. Halley Ruppel is a Kaiser Permanente Division of Research Delivery Science Fellow.

*   Competing interests None declared.

*   Patient consent for publication Not required.

*   Provenance and peer review Commissioned; internally peer reviewed.

## References

1.   Reinhart K , Daniels R , Kissoon N , et al . Recognizing sepsis as a global health priority — a WHO resolution. N Engl J Med 2017;377:414–7.[doi:10.1056/NEJMp1707170](http://dx.doi.org/10.1056/NEJMp1707170) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMp1707170&link_type=DOI) 

2.   Fleischmann C , Scherag A , Adhikari NKJ , et al . Assessment of global incidence and mortality of Hospital-treated sepsis. Current estimates and limitations. Am J Respir Crit Care Med 2016;193:259–72.[doi:10.1164/rccm.201504-0781OC](http://dx.doi.org/10.1164/rccm.201504-0781OC) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1164/rccm.201504-0781OC&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=26414292&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

3.   Rhee C , Dantes R , Epstein L , et al . Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009-2014. JAMA 2017;318:1241–9.[doi:10.1001/jama.2017.13836](http://dx.doi.org/10.1001/jama.2017.13836) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2017.13836&link_type=DOI) 

4.   Liu V , Escobar GJ , Greene JD , et al . Hospital deaths in patients with sepsis from 2 independent cohorts. JAMA 2014;312:90–2.[doi:10.1001/jama.2014.5804](http://dx.doi.org/10.1001/jama.2014.5804) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2014.5804&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24838355&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000338131100026&link_type=ISI) 

5.   Paoli CJ , Reynolds MA , Sinha M , et al . Epidemiology and costs of sepsis in the United States—An analysis based on timing of diagnosis and severity Level*. Critical Care Medicine 2018;46:1889–97.[doi:10.1097/CCM.0000000000003342](http://dx.doi.org/10.1097/CCM.0000000000003342) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/CCM.0000000000003342&link_type=DOI) 

6.   Angus DC , Linde-Zwirble WT , Lidicker J , et al . Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Critical Care Medicine 2001;29:1303–10.[doi:10.1097/00003246-200107000-00002](http://dx.doi.org/10.1097/00003246-200107000-00002) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/00003246-200107000-00002&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=11445675&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000169944400002&link_type=ISI) 

7.   Prescott HC , Langa KM , Iwashyna TJ . Readmission diagnoses after hospitalization for severe sepsis and other acute medical conditions. JAMA 2015;313:1055–7.[doi:10.1001/jama.2015.1410](http://dx.doi.org/10.1001/jama.2015.1410) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2015.1410&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25756444&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

8.   Prescott HC , Osterholzer JJ , Langa KM , et al . Late mortality after sepsis: propensity matched cohort study. BMJ 2016;353.[doi:10.1136/bmj.i2375](http://dx.doi.org/10.1136/bmj.i2375) 
    
    

9.   Cuthbertson BH , Elders A , Hall S , et al . Mortality and quality of life in the five years after severe sepsis. Crit Care 2013;17.[doi:10.1186/cc12616](http://dx.doi.org/10.1186/cc12616) 
    
    

10.  Prescott HC , Langa KM , Liu V , et al . Increased 1-year healthcare use in survivors of severe sepsis. Am J Respir Crit Care Med 2014;190:62–9.[doi:10.1164/rccm.201403-0471OC](http://dx.doi.org/10.1164/rccm.201403-0471OC) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1164/rccm.201403-0471OC&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=24872085&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

11.  Iwashyna TJ , Ely EW , Smith DM , et al . Long-term cognitive impairment and functional disability among survivors of severe sepsis. JAMA 2010;304:1787–94.[doi:10.1001/jama.2010.1553](http://dx.doi.org/10.1001/jama.2010.1553) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2010.1553&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20978258&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000283501300019&link_type=ISI) 

12. World Health Assembly. Improving the prevention, diagnosis and clinical management of sepsis. Available: [http://apps.who.int/gb/ebwha/pdf\_files/WHA70/A70\_R7-en.pdf?ua=1](http://apps.who.int/gb/ebwha/pdf_files/WHA70/A70_R7-en.pdf?ua=1) [Accessed 31 Mar 2019].
    
    

13. Sepsis alliance. Available: [https://www.sepsis.org/itsabouttime/](https://www.sepsis.org/itsabouttime/) [Accessed 31 Mar 2019].
    
    

14.  Makam AN , Nguyen OK , Auerbach AD . Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review. J. Hosp. Med. 2015;10:396–402.[doi:10.1002/jhm.2347](http://dx.doi.org/10.1002/jhm.2347) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1002/jhm.2347&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25758641&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

15.  Seymour CW , Liu VX , Iwashyna TJ , et al . Assessment of clinical criteria for sepsis: for the third International consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016;315:762–74.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2016.0288&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=26903335&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

16.  Bone RC , Balk RA , Cerra FB , et al . Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. Chest 1992;101:1644–55.[doi:10.1378/chest.101.6.1644](http://dx.doi.org/10.1378/chest.101.6.1644) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1378/chest.101.6.1644&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=1303622&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=A1992HX70800034&link_type=ISI) 

17.  Kaukonen K-M , Bailey M , Pilcher D , et al . Systemic inflammatory response syndrome criteria in defining severe sepsis. N Engl J Med 2015;372:1629–38.[doi:10.1056/NEJMoa1415236](http://dx.doi.org/10.1056/NEJMoa1415236) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMoa1415236&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25776936&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

18.  Levy MM , Fink MP , Marshall JC , et al . SCCM/ESICM/ACCP/ATS/SIS international sepsis definitions conference. Crit Care Med 2001;2003:1250–6.
    
    

19.  Churpek MM , Snyder A , Sokol S , et al . Investigating the impact of different suspicion of infection criteria on the accuracy of quick sepsis-related organ failure assessment, systemic inflammatory response syndrome, and early warning Scores*. Critical Care Medicine 2017;45:1805–12.[doi:10.1097/CCM.0000000000002648](http://dx.doi.org/10.1097/CCM.0000000000002648) 
    
    

20.  Henry KE , Hager DN , Pronovost PJ , et al . A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 2015;7.[doi:10.1126/scitranslmed.aab3719](http://dx.doi.org/10.1126/scitranslmed.aab3719) 
    
    

21.  Horng S , Sontag DA , Halpern Y , et al . Creating an automated trigger for sepsis clinical decision support at emergency department triage using machine learning. PLoS One 2017;12:e0174708.[doi:10.1371/journal.pone.0174708](http://dx.doi.org/10.1371/journal.pone.0174708) 
    
    

22.  Mao Q , Jay M , Hoffman JL , et al . Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open 2018;8:e017833.[doi:10.1136/bmjopen-2017-017833](http://dx.doi.org/10.1136/bmjopen-2017-017833) 
    
    

23.  Taneja I , Reddy B , Damhorst G , et al . Combining biomarkers with EMR data to identify patients in different phases of sepsis. Sci Rep 2017;7.[doi:10.1038/s41598-017-09766-1](http://dx.doi.org/10.1038/s41598-017-09766-1) 
    
    

24.  Taylor RA , Pare JR , Venkatesh AK , et al . Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach. Acad Emerg Med 2016;23:269–78.[doi:10.1111/acem.12876](http://dx.doi.org/10.1111/acem.12876) 
    
    

25.  Calvert JS , Price DA , Chettipally UK , et al . A computational approach to early sepsis detection. Computers in Biology and Medicine 2016;74:69–73.[doi:10.1016/j.compbiomed.2016.05.003](http://dx.doi.org/10.1016/j.compbiomed.2016.05.003) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.compbiomed.2016.05.003&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=27208704&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

26.  Beam AL , Kohane IS . Big data and machine learning in health care. JAMA 2018;319:1317–8.[doi:10.1001/jama.2017.18391](http://dx.doi.org/10.1001/jama.2017.18391) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=29532063&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

27.  Chen JH , Asch SM . Machine learning and prediction in medicine — beyond the peak of inflated expectations. N Engl J Med 2017;376:2507–9.[doi:10.1056/NEJMp1702071](http://dx.doi.org/10.1056/NEJMp1702071) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMp1702071&link_type=DOI) 

28.  Alba AC , Agoritsas T , Walsh M , et al . Discrimination and calibration of clinical prediction models: users' guides to the medical literature. JAMA 2017;318:1377–84.
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=29049590&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

29.  Kipnis P , Liu V , Escobar GJ . Accuracy of hospital standardized mortality rates: effects of model calibration. Med Care 2014;52:378–84.
    
    

30.  Strickland E . Hospitals roll out AI systems to keep patients from dying of sepsis. Available: [https://spectrum.ieee.org/biomedical/diagnostics/hospitals-roll-out-ai-systems-to-keep-patients-from-dying-of-sepsis](https://spectrum.ieee.org/biomedical/diagnostics/hospitals-roll-out-ai-systems-to-keep-patients-from-dying-of-sepsis) [Accessed 31 Mar 2019].
    
    

31.  DeVita MA , Smith GB , Adam SK , et al . “Identifying the hospitalised patient in crisis”—A consensus conference on the afferent limb of Rapid Response Systems. Resuscitation 2010;81:375–82.[doi:10.1016/j.resuscitation.2009.12.008](http://dx.doi.org/10.1016/j.resuscitation.2009.12.008) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1016/j.resuscitation.2009.12.008&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=20149516&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000276707200003&link_type=ISI) 

32.  Verghese A , Shah NH , Harrington RA . What this computer needs is a physician: Humanism and artificial intelligence. JAMA 2018;319:19–20.[doi:10.1001/jama.2017.19198](http://dx.doi.org/10.1001/jama.2017.19198) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2017.19198&link_type=DOI) 

33.  Parikh RB , Kakad M , Bates DW . Integrating predictive analytics into high-value care: the dawn of precision delivery. JAMA 2016;315:651–2.[doi:10.1001/jama.2015.19417](http://dx.doi.org/10.1001/jama.2015.19417) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2015.19417&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=26881365&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

34.  Ferrer R et al . Improvement in process of care and outcome after a multicenter severe sepsis educational program in Spain. JAMA 2008;299:2294–303.[doi:10.1001/jama.299.19.2294](http://dx.doi.org/10.1001/jama.299.19.2294) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.299.19.2294&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=18492971&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000255984500024&link_type=ISI) 

35.  Rhodes A , Evans LE , Alhazzani W , et al . Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Crit Care Med 2017;45:486–552.[doi:10.1097/CCM.0000000000002255](http://dx.doi.org/10.1097/CCM.0000000000002255) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/CCM.0000000000002255&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=28098591&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

36.  Seymour CW , Gesten F , Prescott HC , et al . Time to treatment and mortality during mandated emergency care for sepsis. N Engl J Med Overseas Ed 2017;376:2235–44.[doi:10.1056/NEJMoa1703058](http://dx.doi.org/10.1056/NEJMoa1703058) 
    
    

37.  Liu VX , Morehouse JW , Baker JM , et al . Data that drive: closing the loop in the learning hospital system. J Hosp Med 2016;11 Suppl 1:S11–S17.[doi:10.1002/jhm.2651](http://dx.doi.org/10.1002/jhm.2651) 
    
    

38.  Manaktala S , Claypool SR . Evaluating the impact of a computerized surveillance algorithm and decision support system on sepsis mortality. J Am Med Inform Assoc 2017;24:88–95.[doi:10.1093/jamia/ocw056](http://dx.doi.org/10.1093/jamia/ocw056) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1093/jamia/ocw056&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=27225197&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

39.  Shimabukuro DW , Barton CW , Feldman MD , et al . Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Resp Res 2017;4:e000234.[doi:10.1136/bmjresp-2017-000234](http://dx.doi.org/10.1136/bmjresp-2017-000234) 
    
    

40.  Warttig S , Alderson P , Evans DJW , et al . Automated monitoring compared to standard care for the early detection of sepsis in critically ill patients. Cochrane Database Syst Rev 2018;181.[doi:10.1002/14651858.CD012404.pub2](http://dx.doi.org/10.1002/14651858.CD012404.pub2) 
    
    

41.  Umscheid CA , Betesh J , VanZandbergen C , et al . Development, implementation, and impact of an automated early warning and response system for sepsis. J. Hosp. Med. 2015;10:26–31.[doi:10.1002/jhm.2259](http://dx.doi.org/10.1002/jhm.2259) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1002/jhm.2259&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25263548&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

42.  Semler MW , Weavind L , Hooper MH , et al . An electronic tool for the evaluation and treatment of sepsis in the ICU: a randomized controlled trial. Crit Care Med 2015;43:1595–602.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1097/CCM.0000000000001020&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=25867906&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

43.  Downing NL , Rolnick J , Poole SF , et al . Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ Qual Saf 2019;28:762–8.[doi:10.1136/bmjqs-2018-008765](http://dx.doi.org/10.1136/bmjqs-2018-008765) 
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjI4LzkvNzYyIjtzOjQ6ImF0b20iO3M6MTg6Ii9xaGMvMjgvOS82OTMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

44.  Rolnick J , Downing NL , Shepard J , et al . Validation of test performance and clinical time zero for an electronic health record embedded severe sepsis alert. Appl Clin Inform 2016;7:560–72.[doi:10.4338/ACI-2015-11-RA-0159](http://dx.doi.org/10.4338/ACI-2015-11-RA-0159) 
    
    

45.  Mauri L . Why we still need randomized trials to compare effectiveness. N Engl J Med 2012;366:1538–40.[doi:10.1056/NEJMe1202866](http://dx.doi.org/10.1056/NEJMe1202866) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1056/NEJMe1202866&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22452337&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000302934800013&link_type=ISI) 

46.  Angus DC . Fusing randomized trials with big data: the key to self-learning health care systems? JAMA 2015;314:767–8.[doi:10.1001/jama.2015.7762](http://dx.doi.org/10.1001/jama.2015.7762) 
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1001/jama.2015.7762&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=26305643&link_type=MED&atom=%2Fqhc%2F28%2F9%2F693.atom) 

47.  Kane-Gill SL , O'Connor MF , Rothschild JM , et al . Technologic Distractions (Part 1): summary of approaches to manage alert quantity with intent to reduce alert fatigue and suggestions for alert fatigue metrics. Crit Care Med 2017;45:1481–8.[doi:10.1097/CCM.0000000000002580](http://dx.doi.org/10.1097/CCM.0000000000002580) 
    
    

48.  Volchenboum SL , Mayampurath A , Göksu-Gürsoy G , et al . Association between in-hospital critical illness events and outcomes in patients on the same ward. JAMA 2016;316:2674–5.[doi:10.1001/jama.2016.15505](http://dx.doi.org/10.1001/jama.2016.15505)