Article Text

Download PDFPDF

Competing risks in quality and safety research: a framework to guide choice of analysis and improve reporting
  1. Perla J Marang-van de Mheen1,
  2. Hein Putter2,
  3. Esther Bastiaannet3,
  4. Alex Bottle4
  1. 1 Department of Biomedical Data Sciences, Medical Decision Making, J10-S, Leiden University Medical Center, Leiden, The Netherlands
  2. 2 Department of Biomedical Data Sciences, Medical Statistics, Leiden University Medical Center, Leiden, The Netherlands
  3. 3 Department of Surgery, Leiden University Medical Center, Leiden, Zuid-Holland, The Netherlands
  4. 4 School of Public Health, Faculty of Medicine, Imperial College London, London, UK
  1. Correspondence to Dr Perla J Marang-van de Mheen, Department of Biomedical Data Sciences, Medical Decision Making, J10-S, Leiden University Medical Center, 2300 RC Leiden, The Netherlands; p.j.marang-van_de_mheen{at}lumc.nl

Abstract

When comparing hospitals on their readmission rates as currently done in the Hospital Readmission and Reduction Program (HRRP) in the USA, should we include the competing risk of mortality after discharge, which precludes the readmission, in the analysis? Not including competing risks in current HRRP metrics was raised recently as a limitation with possible unintended consequences, as financial penalties for higher readmission rates are more severe than for higher mortality rates. Incorrectly including or ignoring competing risks can both induce bias. In this paper, we present a framework to clarify situations when competing risks should be taken into account and when they should not. We argue that the research question and the perspective from which it is asked determine whether the competing risk is also of interest and should be included in the analysis, or if only the event of interest should be considered. This information is often not explicitly reported but is needed to interpret whether the results are valid. Using the examples of readmissions and cancer, we show how different research questions fit different perspectives from which these are asked (patient, system, regulatory/insurance). Slightly changing the research question or perspective may thus change the analysis. Even though some may argue that any introduced bias is likely to be small, in the context of the HRRP, even small changes may mean that a hospital will face (higher) financial penalties. The impact of getting it wrong matters.

  • health services research
  • pay for performance
  • performance measures
  • statistics

Data availability statement

There are no data in this work.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Introduction

Competing risks are often encountered in evaluation of patient outcomes, particularly in the analysis of survival or other time-to-event data. A competing risk is defined as an event whose occurrence precludes the occurrence of the primary outcome or the event of primary interest.1 Examples from cardiovascular, oncology and nephrology research show that not taking into account these competing risks tends to overestimate the incidence of the event of interest. This may result in biased estimates of the probability of the event of interest as well as differences between groups.1–5 Florence Nightingale already pointed to this issue when arguing for hospital statistics to take into account not just the proportion of recoveries and the proportion of deaths but also the average time in hospital.6 Together with William Farr, she argued that the number of hospital deaths should be related to the patient time at risk rather than dividing by the size of the patient cohort—a rate rather than a proportion (or risk). When calculating a rate, people experiencing a competing event will no longer be counted in the patient time at risk, the denominator, which is why the rate tends to be higher than the risk.

That this is also relevant for the quality of care field is illustrated by the recent debate on the Hospital Readmission Reduction Program (HRRP) in the USA. When comparing hospitals on readmission rates as currently done in the HRRP, where hospitals are incentivised to reduce readmissions, should we take into account the competing risk of mortality after discharge, which prevents people from being readmitted? Researchers have reported an increase in mortality since the HRRP introduction for patients with heart failure, although not for acute myocardial infarction and pneumonia,7 8 resulting in concerns about unintended consequences. The issue of not including competing risks in HRRP metrics has been specifically mentioned as a limitation with possible unintended consequences, because penalties for readmissions under the HRRP are more severe than penalties for mortality under the Hospital Value-Based Purchasing Program.9

If there are competing risks, competing-risk models are able to deal with this but are not always appropriate and, depending on the outcome of interest, may themselves give biased estimates.10 Whether competing-risk models are appropriate depends on the question at hand, in particular whether the interest is to estimate a risk or a rate. There have been various overviews and tutorials that introduce ways to analyse data in the presence of competing risks,1 2 11–13 but few have offered guidance in a non-statistical way regarding situations in which competing-risk models should and should not be used. In this paper, we aim to clarify situations relevant for the quality and safety field in which competing risks should be taken into account and situations when they should not, by presenting a framework that can be used to guide decision-making on what type of analysis is best used. The research question and the perspective from which this question is asked determine whether the competing risk is also of interest and should be included in the analysis, or whether it is sufficient to consider only the event of interest. This information is often not explicitly reported but is needed to interpret whether the results are valid. Using the examples of readmissions and cancer, we will show how different research questions will fit different perspectives from which these are asked and determine whether the competing risk is of interest and should be included in the analysis.

Is there a competing risk?

If we are interested in all-cause mortality, there generally is no interest in competing risks. However, things may be different when examining the risk of death for patients while on dialysis, where kidney transplantation may be a relevant competing risk.2 The same is true for outcomes like readmissions, where death may preclude the occurrence of readmission. In time-to-event analysis, the time of subjects who do not experience the event of interest during the observation period is censored at the end of follow-up. Such censoring may occur for various reasons, for example, because people are lost to follow-up (for instance, for a particular study site because of moving to another city), but also if they experience another event (eg, death) that makes further follow-up impossible or futile. Analyses that do not take into account competing risks therefore assume that such censoring gives no information on the chance of developing the event of interest (called non-informative censoring). In other words, they assume that censoring occurs independently from the event of interest so that censored subjects are representative of those still at risk. Whether this is a reasonable assumption will depend on the research question and what constitutes the outcome of interest to be estimated.

When considering which analysis to use, important issues to consider are (1) whether competing risks need to be included or can be ignored, and (2) whether rates or risks are needed given the research question (see table 1 for description of frequently used statistical models). Models that include competing risks, such as the Fine-Gray model, take into account both the process of the event of interest and the competing event, also referred to as reflecting what we see in practice.10 14 Methods like the Kaplan-Meier ignore the competing event and thus give an estimate of ‘what would happen if the competing event could be prevented [from occurring], creating an imaginary world in which an individual remains at risk of failure from the event of interest’,11 also known as the net risk.10 14 These analyses give different results if the competing event occurs frequently before the event of interest, as then these patients will be censored in the non-competing-risk models (ie, excluded from the population considered to be at risk for the event of interest), whereas competing-risk models keep these patients included in the at-risk population (figure 1). The cause-specific cumulative incidence function estimated in the competing-risk analysis will generally be lower than the naïve cumulative incidence function estimated from the (complement of the) Kaplan-Meier, as a different quantity is estimated where subjects remain at risk for the event of interest rather than being censored, that is, the denominator in the competing-risk analysis is larger. Because censoring due to the competing event does not occur, the cause-specific cumulative incidence function calculates each patient’s actual risk for the specific event of interest that occurs in practice. Similarly, the risk ratios to indicate differences between patient groups from competing-risk models like Fine-Gray will estimate a different quantity than the rate ratios from the cause-specific Cox models (that censor competing risks), thereby answering different questions.15 It is thus crucial to consider what quantity we need to estimate given the research question.

Figure 1

Difference between competing-risk and non-competing-risk models in calculation of readmission incidence as event of interest with death as competing event. Figure created by the authors, adapted from Noordzij et al.2 Initially, 50 patients are at risk for readmission (depicted in white). For the patients at risk, some may experience a readmission (depicted in blue) or death as the competing event (depicted in grey). Competing-risk models will keep patients experiencing the competing event in the at-risk population (white and grey combined), whereas non-competing-risk models will exclude those patients (leaving only white) from the at-risk population. The bottom rows show the difference in calculation applied to readmission as the event of interest, which will differ more as the frequency of the competing event increases. For example, at t+2, two patients had already been readmitted (the event of interest) and four patients had already died (the competing risk). The competing-risk model has a denominator of 50–2=48 at-risk patients because it retains the deaths; the non-competing-risk model has a denominator of 50−2−4=44 at-risk patients because it censors the deaths (adapted from Noordzij et al).2

Table 1

General description of frequently used time-to-event statistical analyses

The perspective and research question determine the quantity of interest

Any research question is asked from a certain perspective, for example, the patient’s perspective in prognostic research aiming to estimate their readmission risk or death from cancer and the type of information a doctor may need in their communication with patients. Therefore, whether it is of interest to include the competing event will depend on the perspective and question asked. Relevant other perspectives that we consider here are the system’s perspective (eg, a hospital trying to estimate required resources for readmissions) and the regulatory or insurance perspective (eg, comparing different hospitals or insurance plans on readmissions), as these will also cover many research questions in the quality and safety field.

Within each perspective, different types of questions will determine which quantity is of interest and the recommended type of analysis. Taking readmission as the example and death as the competing risk (table 2) or death from cancer as the event of interest and non-cancer death as the competing risk (table 3), the question will differ whether it is asked from the perspective of a patient, a hospital or an insurer. Even within these perspectives, a different question may change the quantity that is of interest and the analysis. Patients, for instance, are largely interested in the risk of an event actually occurring to them, either the risk of being readmitted after discharge (table 2) or the risk of dying from cancer if no treatment is given (table 3). Therefore, the competing risk needs to be included to estimate their real-life risk using the cause-specific cumulative incidence function from competing-risk models. However, if that same patient is comparing two hospitals on their readmission rates, the interest is in comparing only the readmissions, and therefore competing risks can be censored.

Table 2

Readmissions: the research question differs depending on the perspective and subsequently determines the quantity of interest and recommended analysis

Table 3

Cancer: application of the framework to cause-of-death related questions

From the system perspective, for instance, a hospital, a relevant question may be around the extent to which the readmission risk changes over time, where the interest is in both the readmission risk and the competing risk of death, reflecting what will occur in practice (table 2). However, when the question changes slightly to include whether readmissions have changed after implementing a new discharge policy compared with before, then we are interested in drawing inferences on whether this new policy could have worked to change the readmission rate. For such an aetiological research question, we are mostly interested in evaluating the causal effect, and the Cox proportional hazard model can be used to estimate each cause-specific hazard while censoring the competing event.1 2 15 This has the advantage that the assumption of independent competing events is not required to obtain valid estimates regarding the intervention effect.16 The hazard ratios from these analyses can be interpreted for those at risk, that is, those patients who have not experienced a readmission and are still alive. This is aligned with the research question, whether patients being discharged (alive) after implementation of this new policy have a different readmission rate than before this policy was implemented.

From the regulatory/insurance perspective, a relevant prognostic question for an insurance plan may be to assess the amount of future spending or resources needed per person to treat cancer recurrence (table 3). Both cancer and non-cancer deaths will then be of interest as these both determine the need to spend money or use resources, meaning that the competing risk should be included using the cause-specific cumulative incidence function. However, one could think of prognostic questions where competing risks can be ignored. For instance, when the aim is to select the implant with the best longevity, the interest is only in revision (as a failure of the implant) to know how long the implant might function theoretically, so competing risks can be ignored.10 This shows that it is the research question, rather than a rule of thumb on whether it is prognostic or aetiological, that determines whether the competing risk is of interest and the quantity that needs to be estimated. For readmissions in the context of the HRRP, if the aim is to compare readmission rates between hospitals and identify outliers, the interest is only in the readmissions and in drawing inferences whether hospital is a risk factor, so competing risks can be censored (table 2). Cause-specific Cox models can be used when adjustment for other factors is required, or the naïve cumulative incidence function from (the complement of) the Kaplan-Meier for unadjusted estimates, provided that the assumption of independent censoring is met. In this case, that means that the mortality process occurs independently from the readmission risk when comparing hospitals. This seems to be met for a number of patient groups for which previous studies have shown that hospital readmission and mortality rates are not correlated.17–19

The issue being raised with regard to competing risks and the HRRP metrics is that of unintended consequences,9 specifically the increase in mortality observed for heart failure patients since HRRP’s introduction.7 8 This could point to readmissions and mortality being related, for example, because the discharge policy introduced to decrease readmissions is now increasing mortality, which could be relevant when considering the assumption of independent censoring of the competing risk. However, even if there is a competing risk present and it occurs frequently before the event of interest, as could be the case for some patient groups like heart failure, the research question and the perspective determine whether it is of interest and needs to be included in the analysis (or metrics). For future studies and discussion, the above framework may help guide decision-making on whether to use competing-risk analysis, but also highlights the key issues to report, that is, the perspective combined with the (type of) research question, and to align that with the quantity of interest that needs to be estimated to answer the question at hand.

When does it matter and give different results in practice?

Putting aside which analysis to use and the fact that (not) including the competing risk may induce bias, one may wonder about the extent of bias. As noted above, the estimates obtained from the two approaches differ only after the first competing event has occurred, meaning that the difference in their absolute estimates becomes larger when the competing event is more frequent (figure 1). The timing of the competing event is important, though, as the difference will occur particularly when the competing events occur early, that is, before the event of interest, but will not be relevant if they only occur after it.20 Therefore, competing risks may pose problems in the following situations:

  • Longer duration of follow-up, as the cumulative incidence of the competing event will increase.

  • For patient groups with higher competing-risk event rates, such as elderly patients21 or patient groups with higher mortality risks like oncological or cardiovascular patients.

Previous studies have been conducted to show the difference in estimated Cox cause-specific or Fine-Gray subdistribution hazard ratios in various patient groups, across different durations of follow-up and different frequency of the event of interest and competing event, that is, how much it matters to look at a change in risk or a change in the rate of an event (when there are competing risks). For instance, in patients undergoing hip or knee arthroplasty followed for a median duration of 3.1 years, the ratios associated with several variables (eg, older or younger than 70) and revision estimated by these two types of models differed by a maximum of 3.5%, explained by the low frequency of revision as the event of interest (3% for hip and knee) and heavy censoring of death as the competing event (18% for hip, 19% for knee).15 For patients on dialysis, of whom 51% died over the course of 5 year follow-up with 22% receiving transplantation as the competing event, the ratios associated with older age and mortality from these two types of models differed by 35%.2 Among patients with diabetes, of whom 10% died of cardiovascular disease and 6% from cancer during 9-year follow-up, ratios associated with albuminuria status differed by 11% for cardiovascular mortality and 15% for cancer mortality.22 In some cases, however, the two types of models may show qualitatively different effects, for example, among patients with heart failure the presence of cancer was associated with a decrease in the risk of cardiac death (subdistribution hazard ratio 0.82) but had no association with the rate of cardiac death in those still alive (cause-specific hazard ratio 0.96).1 It was shown that this apparent reduction in risk of cardiac death associated with the presence of cancer, was explained by the (indirect) effect of this variable on non-cardiac death, as cardiovascular deaths cannot occur in these individuals but are included in the at-risk population of the Fine-Gray model. Examples like this illustrate how in combination, these models give a better understanding of the relationships with the different competing events and thereby a more comprehensive view on the effect of a variable on both the risk and the rate of occurrence.1 23 Future studies on readmissions may similarly provide further insights by examining the impact of specific changes in discharge policies and both the risk and the rate of readmissions, with mortality as the competing event. It is important to note that if, rather than understanding the relationships, the focus is to estimate the cumulative incidence of both the event of interest and the competing event, that combining estimates from different subdistribution hazard models may result in the total risk exceeding 1 for some subjects and times, as shown recently by Austin et al to be a fundamental problem with the Fine-Gray subdistribution hazard model.24

Follow-up in the above studies is considerably longer than the 30 days after discharge for the HRRP metrics, but the occurrence of death as the competing event may still be significant for some patient groups on which the HRRP is focused. In general, it has been recommended that absolute percentages of competing events of more than 10% requires serious consideration.1 The observed aggregate death rates within 30 days after discharge for acute myocardial infarction, heart failure and pneumonia were all below 10%, both before and after HRRP implementation.7 8 This may seem to point to any introduced bias likely being small, but even small changes may render an increased readmission estimate such that a hospital will face (higher) financial penalties. The issue stops being merely theoretical for such organisations. So, the main issue remains to select the analysis that will fit both the research question and the perspective from which it is asked and thus the quantity of interest. The impact of getting it wrong also matters.

Conclusion

It is not just the presence and frequency of competing risks that matter, but also the perspective from which a particular research question is asked. We offer a framework to guide the analyst and encourage explicit reporting on the perspective taken and the quantity of interest to be aligned with the question at hand, to enable interpretation whether it is appropriate to include competing risks or whether, for this particular question and perspective, they can be ignored.

Data availability statement

There are no data in this work.

Ethics statements

Patient consent for publication

References

Footnotes

  • Twitter @DrAlexBottle

  • Contributors PJM-vdM and AB conceived this study. PJM-vdM wrote the first draft. All the authors contributed to the development of the paper, critically reviewed the manuscript and approved the final version. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.