Performance and interventions

Performance of predictive models consists of two related components: overall performance and threshold-specific performance. Measures such as discrimination and calibration assess the model as a whole. However, many applications require separation of patients into two (or more) distinct groups (eg, high risk and low risk) by selecting a defining probability, or threshold for intervention. Several factors are important when selecting a threshold.

The application of an intervention to individual patients is heavily dependent on the model’s ‘certainty’ of a patient being high risk, measured by positive predictive value (PPV). Low PPV is common for broad identification applications, such as screening or populating registries, as it casts a wide net (high sensitivity). Similarly, a PEOLC intervention such as a goals-of-care discussion is straightforward, inexpensive, with little downside and, thus, modest false-positive rates are reasonable. The mHOMR authors acknowledge this reality and select an alert threshold corresponding to specificity=90%, sensitivity=59% with a PPV=36%. That is, two of every three alerted patients will not die within 1 year. By contrast, an appropriate threshold for more aggressive PEOLC interventions, such as de-escalation of treatment, would require much higher PPV, likely ≥75%.

For time and resource-intensive interventions, the volume of patients identified can become a limiting factor and should thus be considered. Prospectively, mHOMR identifies 15.8% and 12.2% of all internal medicine patients admitted across the two sites. This one-in-six volume is sustainable for simpler interventions that are distributable to the care teams, for example, goals-of-care discussions, but would likely not be feasible for more time or resource-intensive interventions. Palliative care consults for more than 10% of patients would likely overwhelm any institution’s palliative care team without significant investment in staffing. Regardless of the specific intervention, institutional support is crucial when adding onto the clinical workload.

In addition to workload considerations, any intervention must also be appropriately matched to the population’s expected survival for effective implementation. Many PEOLC systems, including mHOMR, focus on ‘early’ identification of patients at risk of dying in 1 year, often to prompt palliative care, citing the fact that palliative care is often initiated too near a patient’s death (median (IQR) of 59 (13, 200) days11). Alternatively, identification of ‘sentinel hospitalizations,’12 the turning point of a patient’s trajectory, has been proposed as an appropriate moment to prompt initiation of palliative care.13 However, patients follow different trajectories that warrant PEOLC interventions at different times and the inability to distinguish patients with weeks or many months to live complicates recommendation of any single intervention.

The authors of mHOMR describe multiple potential interventions but do not recommend any single one, instead prompting a general palliative approach in their notification. In our opinion, goals-of-care discussion is the only intervention feasible for all patients identified by mHOMR given the time horizon, and predictive performance selected. Naturally, some patients will receive other PEOLC interventions, with or without goals-of-care. The authors report combined rates of either goals-of-care discussion or palliative care consult, allowing clinicians to choose. Concerningly, the authors describe a potential ‘ceiling effect’ related to the high baseline palliative care/goals-of-care rates at site 2, suggesting clinicians are self-moderating possibly due to a lack of ‘certainty’ or that universal goals-of-care is implausible in this broad population. A useful approach to further investigate would be to employ provider agreement to the patient’s risk as a process measure. With a low PPV, a long time horizon and an ambiguous intervention, mHOMR may not be persuasive enough to change a clinician’s opinion, obstructing effective implementation.

Applying mHOMR in practice

In restricting intervention to internal medicine patients, the investigators implement a constrained version of mHOMR. Because the original model was developed on 15 services and included interaction terms using service, many coefficients rely on services other than internal medicine (14 other services × 3 interaction terms=42 of the original 60 coefficients). When implementation is restricted to only include internal medicine, these parameters become irrelevant and are effectively dropped from the model. This may cause model performance to degrade to a point no longer comparable to that initially reported.

Model-identified subcohorts

Existing methods to prompt PEOLC typically estimate long-term mortality in cohorts of older adults using administrative or geriatric assessment data14–17 but few have translated into widespread clinical practice.18 One limitation of methods targeting older adults is the explicit omission of younger adults. Avati et al 19 used deep learning methods for the prediction of 3–12 months’ all-cause mortality in patients of all ages (but received criticism for their experimental design and ill-defined intervention). mHOMR is similarly derived in a general cohort of adult patients. However, the model identifies an older, frail population (mean (SD) age=83 (7.8); 55% frail), similar to other work that explicitly selects an older population.

Prediction-based systems learn patterns in data. Any model can overfit particular parameters, or combinations of data, which can produce counter-intuitive results, for example, surprising high or low risk patients at a given threshold. The advantage of simple models like mHOMR (in contrast to machine learning ‘black boxes’) is the potential to explore the model coefficients to identify outlier cohorts and potentially restrict use of these untrustworthy predictions. mHOMR only uses nine data elements which minimises the number of data combinations and simplifies identification of outlier cohorts that do exist.

We reproduced mHOMR for all possible data combinations (omitting hospital service) in search of outlier cohorts (code available at: github.com/vincentmajor/reproducing-mHOMR). Overall, we find that age is such an important factor that mHOMR automatically identifies patients as high risk with little other data. For instance, a 77-year-old male patient with zero prior emergency department (ED) visits or admissions by ambulance, living independently at home, presenting electively and without an intensive care unit (ICU) stay will be identified as high risk due to his age. At the opposite extreme, acute admissions of high utilisers will be identified as low risk below a certain age. A patient with two prior admissions by ambulance, two prior ED visits, presenting by ambulance to the ED from home with home care services, as a 30-day readmission, and admitted directly to the ICU, will be low risk unless he is at least 58 or she is at least 64. mHOMR’s reliance on age may be caused by a lack of alternative parameters, such as disease state, that may be more influential than age for particular subgroups.

Operational implementation

From a technical perspective, there are several hurdles to generalise this work into other institutions. First, historical data may be limited in regions with fragmented healthcare systems that may lead to systematically lower risk estimates. Second, mHOMR is calculated inside the EHR which necessitates reproduction, including parameter mapping of non-standardised fields such as hospital service and current living situation. Third, the delivery of the notification may not be easily reproduced at other institutions, as described in this study: the electronic sign-out tool was effective at one site but could not be implemented at the second. Moreover, both the sign-out and email methods have limitations and the preferred notification may vary by institution and recipient type. Overall, widespread deployment of systems like mHOMR will require EHR vendors to develop a toolbox of configurable functionalities and willingness of individual institutions to invest technical and clinical resources.

Timing within patient trajectory

Though we recognise the paper’s aim was to assess feasibility, the authors present results that inadvertently challenge mHOMR’s utility and future efforts to evaluate PEOLC notifications. As previously described, the identified population is generally elderly with a mean age of 83, an age that is comparable to the life expectancy for Ontario.20 Moreover, 38% of alerted patients were admitted with a current do not resuscitate order and 11% died during their hospitalisation, making the notification irrelevant to them. A comprehensive evaluation should include rates of (1) hospice care, (2) readmission, and (3) patients not willing to engage in PEOLC. Future effectiveness studies should report both process measures describing provider agreement and adherence to the recommendations as well as patient-centred outcome measures of quality and intensity of care.

mHOMR succeeds in demonstrating implementation and initial evaluation of a mortality prediction model to prompt PEOLC interventions. We anticipate growing application of prediction models driven by increasingly sophisticated algorithms. With model performance, reproduction, implementation and transportability21 to external populations will continue to be key challenges to success, especially for more complex models. Before that, we must consider the broader clinical application: whether the notifications come at an impactful time in a patient’s trajectory; how, when and to whom the notification is delivered; the patient volume necessary for the feasibility of planned intervention; the data driving each high risk prediction; and performance expectations when applied to different populations. Without addressing these challenges, implementation of predictive modelling will continue to be ad hoc, struggling to reproduce in other populations and systems.