• sepsis
• alert
• clinical decision support
• predictive model
• machine learning

Sepsis is a global health priority of staggering impact, resulting in at least 6 million deaths worldwide each year and contributing to as many one half of all hospital deaths in the USA.1–4 Sepsis is also tremendously costly, as reflected in total healthcare expenditures,5 6 short-term and long-term morbidity and mortality7–9 and the heavy burden placed on caregivers and society.10 11 Large-scale efforts, including those of the WHO and the Global Sepsis Alliance, have helped to elevate sepsis to a highly prominent concern visible to ‘the public, political leaders and leaders of healthcare systems’.1 12 Emerging public awareness campaigns—for example, the Sepsis Alliance’s ‘It’s About TIME’ motto emphasises Temperature, Infection, Mental decline and Extreme illness as concerning patient symptoms13—further drive home the need for timely and aggressive patient screening, identification and treatment. Together, these clarion calls highlight the need to leverage all available tools and modalities to enhance the earlier identification and treatment of patients to combat sepsis.

Not surprisingly, over the last decade, we have witnessed a rapid expansion in the number of electronic sepsis alert tools in development or use, particularly in locales that have also seen widespread deployment of modern electronic health record (EHR) systems.14 In some cases, simpler rule-based sepsis screening or prognostication tools, built around the systemic inflammatory response syndrome (SIRS) or quick Sepsis-related Organ Failure Assessment (qSOFA) criteria,15–18 have been electronically implemented as sepsis ‘sniffers’ that offload the burden of sepsis detection in patients who already meet relevant clinical criteria.14 In other cases, the role of these simpler ‘predictive’ models has been questioned alongside the rise of machine learning (ML)-driven sepsis predictive models.19–24 Numerous ML-based sepsis predictive models have already demonstrated excellent predictive performance with still many others being designed and tested today.20–25

ML algorithms are particularly useful in sifting through large, complex and heterogeneous data in order to maximise the signal within the noise of messy EHR data.26 27 By rapidly peering through vast swaths of data, these algorithms can bolster model discrimination—most often quantified as c-statistic values—and improve model calibration.28 29 Although many existing studies trumpet modest increases in c-statistics, improving the positive predictive value (PPV) is also critical because it describes the likelihood that a patient triggering an alert will actually have the outcome of interest. Thus, it can be used to estimate the clinical burden associated with predictive model alerts when embedded in practice. In some cases, partly because of poor PPV performance, reports reveal that sepsis alerting tools have already been shut off.30

Understanding the impact of predictive model performance in the context of clinical workflow is essential because these tools do not exist in a vacuum. Instead, predictive models must be paired with effective interventions in a prediction-action or afferent-efferent dyad.31 32 The term ‘precision delivery’ was recently coined to reflect the need for risk tools to be embedded within clinical delivery systems to facilitate targeted and personalised care.33 Decisions on what actions should follow a predictive model alert can be highly variable. When faced with a predictive model of given characteristics, operational leaders must make decisions related to the tool’s use in the areas of alert delivery modalities and thresholds, end-user staffing and interfaces, clinical decision support, workflow changes and/or educational programming. For sepsis, multifaceted interventions that coordinate improvements across several clinical domains have become the norm and, in nearly all cases, have already proven highly effective for improving mortality.34–37

Against this background, what is the evidence showing that electronic sepsis alert tools benefit patients? Several prior studies suggest that their use is associated with incredible benefits in outcomes. For example, when embedded within clinical workflow redesign, a system incorporating electronic surveillance criteria resulted in a stunning 53% reduction in sepsis mortality.38 A more complex ML-driven sepsis predictive model similarly resulted in a 58% reduction in hospital mortality, with no associated increase in adverse events.39 In a condition as deadly as sepsis and with purported effects of this magnitude, it comes as no surprise that many hospitals and health systems are racing towards implementation of electronic alerts and predictive models. However, not all studies have shown such promising results.14 40–42

In this issue, an important study by Downing and others43 helps to enrich our understanding of the potential utility of electronic sepsis alert tools. The authors modified a previously developed severe sepsis identification algorithm44 based on SIRS, suspected infection and organ dysfunction criteria, to enhance PPV. They implemented this EHR-based alert for patients in medical, surgical and stepdown units, excluding those in intensive care units or on comfort care. In coordination with hospital leadership, they carefully developed a standardised workflow following the alerts and educated the relevant clinical teams. When alert criteria were met, a circulating ‘crisis’ nurse, and in some cases the patient’s treating physician, would receive a pager alert with the intended goal of having clinicians assess the patient at the bedside and implement appropriate orders within an order set. Overall, their work reflects the type of careful approach needed to implement an effective and sustainable prediction-action dyad.

What makes this study particularly valuable is that the authors implemented the sepsis alerts in a randomised fashion. Among patients meeting alert criteria, some of their treating clinicians received a paged alert (intervention, n=595) whereas other patients had a ‘silent’ alert invisible to their treating teams (control, n=528). As a result, the study findings should reflect the causal effect of the sepsis alert and workflow itself, rather than other exogenous or confounding factors that often impact more commonplace before-after studies.45 46

Their randomisation process succeeded in achieving intervention and control groups that shared similar baseline characteristics. However, their sepsis alert did not significantly affect the primary outcome of a new antibiotic order placed within 3 hours of the alert (35.0% vs 36.7%). Nor were there any significant differences between intervention and control patients in a diverse set of secondary care processes and outcomes including lactate orders, intravenous fluid administration, blood cultures, prolonged length of stay, intensive care transfer or hospital mortality. Even while this iteration of their tool had no significant effect on sepsis care processes or outcomes, ironically, the study was cut short by hospital leadership who requested that the alert be turned on for all users.

Many factors may have contributed to the lack of benefit seen with the intervention. For example, in both groups, two-thirds of patients were already being treated with antibiotics at the time of the alert. However, the study also did not demonstrate significant differences in intravenous fluid administration or lactate orders between groups. The alert was also designed primarily to improve detection of severe sepsis, rather than to predict onset, limiting the tool’s utility for pre-empting severe sepsis with clinical intervention. The authors found that alert-driven workflows were inconsistently applied by clinicians and wisely conducted surveys to better understand their findings. Of physicians surveyed who had cared for a patient with sepsis, the majority felt that the alert did not flag an important change in a patient’s condition that required new action.

Sepsis is a deadly, prevalent and costly healthcare problem that demands urgent attention. Promising electronic alert tools are increasingly being implemented in the hopes that they can drive improved patient outcomes. However, as this study and others show, well-designed tools demonstrating excellent in silico performance are not guaranteed to improve care or outcomes. In real-world practice, some may even result in unintended consequences like alert fatigue,47 distraction48 and wasted resources. Given the tremendous heterogeneity in the electronic tools themselves as well as in the clinical settings in which they are implemented, we expect variability in reported benefits to persist. Rigorous study designs, as well as the confidence to publish ‘negative’ results, are essential for identifying effective and sustainable interventions that benefit our patients with sepsis.