Article Text
Abstract
Introduction When comparing mortality rates between hospitals to explore hospital performance, there is an important role for adjustment for differences in case-mix. Identifying outcome measures that are less influenced by differences in case-mix may be valuable. The main goal of this study was to explore whether hospital differences in anastomotic leakage (AL) and postoperative mortality are due to differences in case-mix or to differences in treatment factors.
Methods Data of the Dutch Surgical Colorectal Audit were used. Case-mix factors and treatment-related factors were identified from the literature and their association with AL and mortality were analysed with logistic regression. Hospital differences in observed AL and mortality rates, and adjusted rates based on the logistic regression models were shown. The reduction in hospital variance after adjustment was analysed with Levene's test for equality of variances.
Results 17 of 22 case-mix factors and 4 of 11 treatment factors related to AL derived from the literature were available in the database. Variation in observed AL rates between hospitals was large with a maximum rate of 17%. This variation could not be attributed to differences in case-mix but more to differences in treatment factors. Hospital variation in observed mortality rates was significantly reduced after adjustment for differences in case-mix.
Conclusions Hospital variation in AL is relatively independent of differences in case-mix. In contrast to ‘postoperative mortality’ the observed AL rates of hospitals evaluated in our study were only slightly affected after adjustment for case-mix factors. Therefore, AL rates may be suitable as an outcome indicator for measurement of surgical quality of care.
- Audit and feedback
- Mortality (standardized mortality ratios)
- Surgery
- Quality measurement
Statistics from Altmetric.com
Introduction
Nowadays there is a growing public interest in quality of medical and surgical care, with an increasing need for outcome measures that represent hospital performance. The outcome measure postoperative mortality is often used to benchmark surgical performance.1–3 When comparing mortality rates between hospitals, there is an important role for risk adjustment.4 ,5 Observed variations in mortality may be caused by differences in patient and tumour characteristics (case-mix), and high-risk patients may not be evenly distributed between hospitals.6
However, valid case-mix adjustments require a substantial amount of reliable data collected on a patient level. These data are rarely available and require a substantial registration effort. Therefore, it may be valuable to identify outcome measures that are less influenced by differences in case-mix and represent the actual differences in quality of care processes.
Colorectal cancer is a significant source of mortality, with nearly 10 000 new cases diagnosed in the Netherlands each year.7 The cornerstone of this treatment is surgical resection. Patients undergoing surgical resection have a considerable risk for postoperative complications, which can lead to significant morbidity, mortality and large costs. Internationally, several quality improvement programmes have been initiated to reduce postoperative complications after colorectal surgery.
Anastomotic leakage (AL) is one of the most feared complications after colorectal surgery, often causing prolonged hospital stay, morbidity, mortality and possibly worse oncological outcomes.8 The percentage of patients developing AL depends on multiple factors. In the literature, several elements have been identified as risk factors. These can be patient or tumour related, often referred to as case-mix, such as height of the anastomosis, a malnourished status, steroid use and male gender.9–13 Treatment-related factors such as surgeons’ experience, operative duration, blood loss, preoperative radiation and a defunctioning stoma have also been shown to be associated with the occurrence of AL.9–13
The aim of this study was to explore whether hospital differences in AL rates are related to differences in case-mix. We compared the role of case-mix adjustment for AL and postoperative mortality. With this objective, we drew up the following research questions:
-
Which case-mix and treatment-related risk factors are associated with AL and postoperative mortality after colorectal surgery?
-
What are the differences in AL and mortality rates between hospitals and are these due to differences in case-mix or due to differences in treatment patterns?
Methods
Patients
Data were derived from the Dutch Surgical Colorectal Audit (DSCA), a national quality improvement project in which over 200 variables concerning the patient, comorbidity, diagnostics, disease-specific details, treatment and outcomes are collected prospectively. The DSCA contains data of patients registered by 92 hospitals (all hospitals performing colorectal cancer surgery). The dataset is disease specific for colorectal cancer and shows a nearly 100% accordance on most items, including AL on validation against the National Cancer Registry dataset.14
All patients undergoing resection for primary colorectal cancer between 1 January 2009 and 31 December 2011 and registered in the DSCA before 15 March 2012 were evaluated. Minimal data requirements for inclusion in analyses were information on tumour location, date of surgery and mortality. Patients with metastases at the time of primary surgery and resections for multiple synchronous colorectal tumours were excluded because these represent subgroups of patients with other treatment perspectives and subsequent different expected outcomes. Also, patients in whom a primary end colostomy was constructed were excluded from the analysis.
Risk factors
Since part of the dataset of the DSCA was designed with the objective of performing case-mix adjustment, particularly for postoperative mortality, variables were determined as risk factors for postoperative mortality at an early stage of development of the dataset. These factors were based on existing evidence on potential risk factors for mortality and determined by an expert panel using a Delphi method.6
To assess whether there are additional case-mix and treatment-related risk factors that need to be taken into account when adjusting for AL, we performed a systematic search of the literature published between 1990 and 2012 on biomedical bibliographical databases PubMed and the Cochrane Library. The search headings ‘anastomotic leak and colorectal surgery’ were used in combination with the keyword ‘risk factor’. The ‘related articles’ function was used to expand the search. References from the articles were also used when appropriate. Letters, reviews without original data, non-English language papers, overlapping patient populations and animal studies were excluded.
From the articles retrieved from the literature search, different risk factors for AL were selected. A distinction was made between patient and tumour related factors (case-mix factors) and treatment-related factors. We selected risk factors with a statistical significance of p<0.05, which were analysed with multivariate logistic regression.
Outcomes
Various definitions of AL have been previously presented.15 The definition of AL in this study was ‘a clinically relevant anastomotic leak requiring a reintervention’. Both radiological and surgical reinterventions were included. Postoperative mortality was defined as ‘death during postoperative hospital stay or within 30 days after the date of surgery’.
Analyses
The association of case-mix and treatment factors, and AL and mortality were tested with multivariate logistic regression models. Separate models were used for each outcome.
To analyse the differences in AL and mortality between hospitals and investigate whether these were due to differences in case-mix or due to differences in treatment patterns we applied three different models. model 1: unadjusted (observed) variation in outcome; model 2: adjusted for patient (case-mix) characteristics; model 3: ‘adjusted’ for case-mix and treatment characteristics. Adjustment was performed by calculating expected outcomes (E) using case-mix (model 2), and case-mix and treatment (model 3) coefficients from the regression analysis. Next, for each hospital, the observed percentage (O) was divided by the expected value (E) and multiplied by the overall mean (observed/E* mean) to obtain the adjusted percentages.
Hospital differences in AL and mortality rates before and after adjustment were plotted in a graph; a summary measure of the between-hospital variance was given with ranges and SDs. The reduction in between-centre variance after adjustment for (model 2) case-mix and (model 3) case-mix and treatment factors was analysed with Levene's test for equality of variances. A p<0.05 was considered statistically significant.
Furthermore, a mixed logistic regression model with hospitals as random effects was performed. A likelihood ratio test was used to test whether the variance of the random effects was statistically significant after adjustment for case-mix and treatment factors.
Hospitals with more than 15% missing case-mix factors were excluded from multivariate analyses. All statistical analyses were performed in PASW Statistics, Rel. 18.0.2009 and R V.2.14.16
Results
On 15 March 2012, 92 hospitals (8 university, 47 teaching and 37 non-teaching hospitals) registered a total of 25 555 eligible patients with primary colorectal cancer with a date of surgery between 1 January 2009 and 31 December 2011 in the DSCA. Nine hospitals had more than 15% missing case-mix factors in total and were therefore excluded (n=1460). After additional exclusion of patients with multiple synchronous tumours (n=598), distant metastases (n=2032) and without an anastomosis (n=5480), a total of 15 236 patients were included in the analysis. Characteristics of the included patients are shown in table 1.
Of all patients, 1207 (8%) developed AL and 525 (3.4%) died within 30 days or during hospital admission.
Risk factors
The literature search gave a total of 39 studies describing risk factors for AL.8 ,10–13 17–49 In total, 22 case-mix factors and 11 treatment-related factors were identified. Table 1 shows the results.
The case-mix factors described most frequently were gender, American Society of Anesthesiologists (ASA) score and location of the tumour and/or anastomosis. Treatment factors often described were blood loss/transfusion, duration of the operation and the use of a defunctioning stoma.
Of the 22 case-mix factors for AL identified in literature, 17 were available in the DSCA. The database had no information on the factors weight loss, nutrition status, alcohol abuse, smoking and leukocytosis. Treatment factors were less often available; 4 out of 11 were available in the dataset.
The case-mix and treatment-related risk factors that were found for AL in the literature were similar to those that were used for risk adjustment for postoperative mortality in the DSCA dataset.
A multivariate analysis was performed to investigate the association of case-mix and treatment factors with AL and postoperative mortality. The results of the analysis are shown in table 2.
Individual case-mix factors predicting AL were male gender, urgency of the resection, renal disease and tumour location. Treatment-related factors associated with AL were short preoperative radiotherapy, the absence of a defunctioning stoma and postoperative blood transfusion. For postoperative mortality the case-mix factors age, gender, ASA score, pulmonary disease, tumour location sigmoid and urgency of the resection were individual predicting factors. Treatment-related factors were chemo-radiotherapy and blood transfusion.
Hospital variation
Anastomotic leakage
Unadjusted hospital variation in AL rates was large: the hospital with the lowest percentage had an AL rate of 0% (n=0/166); the hospital with the highest percentage had an AL rate of 18% (n=12/70) (SD 0.036, figure 1A). After adjustment for case-mix, there was still a large variation between hospitals: the adjusted AL rates per hospital ranged from 0 to 17% (SD 0.033). The reduction in variation after adjustment for case-mix was not statistically significant (p=0.52).
The variance in AL rates significantly decreased after including treatment factors in the adjustment model (p<0.01). Case-mix and treatment-adjusted AL rates varied from 0 to 12% (SD 0.024).
For 60% of the hospitals (50/83), the unadjusted AL rate was similar to the case-mix adjusted AL rate. In 36% of the hospitals, AL rates slightly increased or decreased by 1%, and in 4% of the hospitals by 2% (figure 2A).
For 75% of the hospitals (63/83), the unadjusted AL rate changed after adjustment for treatment factors by at least 1%; for 32% of the hospitals, the unadjusted rate changed by more than 3%; and for 10% of the hospitals, the rate changed by more than 5%.
Although hospital variance decreased after adjustment for case-mix and treatment factors, there was still variability between hospitals as a likelihood ratio test showed that the variance of the random effects was statistically significant in all models.
Postoperative mortality
Hospitals’ unadjusted mortality rates ranged from 0 to 10% (SD 0.017). The variance in postoperative mortality significantly decreased after case-mix adjustment (p<0.01) (range 0–6%, SD 0.012, figure 1B).
The variance in postoperative mortality rates slightly increased (range 0–6%, SD 0.013) after including treatment factors in the adjustment model, although not statistically significant (p=0.81).
For 84% of the hospitals (70/83), the unadjusted postoperative mortality rate changed after adjustment for treatment factors by at least 1%; for 24% of the hospitals, the unadjusted rate changed by more than 3%; and for 6% of the hospital, the rate changed by more than 5% (figure 2B).
Adjustment for treatment factors had a slight effect on two hospitals compared with the case-mix adjusted mortality rate. In these hospitals, the case-mix adjusted mortality rate changed by 1% after adjustment for treatment factors.
Hospital variability in postoperative mortality was still significant after adjustment for case-mix and treatment factors, as a likelihood ratio test showed that the variance of the random effects was statistically significant in all models.
Discussion
The present study suggests that ‘AL rate’ is an outcome indicator for measurement of surgical quality of care that is relatively independent of differences in case-mix between hospitals. We found a large variation in AL rates between Dutch hospitals. This confirms the ability of this outcome indicator to be discriminative. In contrast to ‘postoperative mortality’, the observed AL rates of hospitals evaluated in our study could not be explained by differences in case-mix. In addition, we found that the influence of treatment factors on the variation in AL rates was substantial. These findings imply that AL rates may be much more related to treatment factors and in-hospital care processes than to characteristics of the patient population treated in a certain hospital. AL rates may therefore be a good reflection of the quality of care provided.
Outcome measures
Optimising surgical outcomes can be seen as ‘the bottom line’ of what surgeons do, and outcome indicators have the advantage that they have ‘face validity’ for surgeons and their patients. Also, measurement in itself may improve surgical outcomes, as suggested by the so-called Hawthorne effect.4 As shown in our study, outcome indicators can present meaningful differences between hospitals. However, there was still significant variability in AL and mortality rates after adjustment for case-mix factors and treatment factors in our study. This suggests that there are other characteristics of the hospital, its staff and the care they deliver that may explain the observed differences. Although outcomes of care are important, process and structure information is essential to identify which area is susceptible for innovation. Therefore, adopting the Donabedian paradigm,50 a balanced indicator set needs to include information on structures, processes and outcomes.
Limitations
The results presented in this study should be interpreted in the light of some important limitations. First, despite the fact that most patient-related risk factors were available in the database of the DSCA, it lacked data on some important host-related factors, such as smoking, alcohol consumption, nutrition status and preoperative leukocytosis. Although unlikely, it is possible that a strong case-mix adjustment model for AL could have been made if those four missing factors were available from the dataset.
Also, high-risk patients according to the surgeons’ preoperative risk judgment or patients with impaired continence at baseline may not have been selected for a primary anastomosis and therefore excluded, which may have caused a potential selection bias. It is not exactly clear how these differences in patient selection might affect the between-hospital comparisons.
Moreover, due to a lack of clear agreements on definitions, the factors we used may not have been identical to the ones found in the literature.
Although we found that case-mix adjustment does not seem to play a large role when comparing hospitals’ AL rates, there are some limitations to using it as an outcome indicator. It may unintentionally lead to the perverse incentive of aiming for the lowest possible AL rate by constructing more end colostomies or defunctioning stomas. This defensive attitude would not immediately contribute to increased quality of care as a surgeon or clinic that has zero AL rates at the cost of constructing defunctioning stomas or end colostomies in all patients would not be regarded as providing the best practice. Obviously, AL rates are only calculated for patients in whom an anastomosis has been created. Therefore hospitals with lower rates of patients with anastomoses could automatically have better scores without providing better quality of care, as the stoma itself may cause morbidity, lead to increased readmissions51 ,52 and may be associated with morbidity at the time of surgical removal of the stoma.53 In reality, there is probably an optimum percentage of defunctioning stomas and end colostomies to be constructed, and AL rates should always be seen in the light of these percentages. However, the exact optimum is unclear and it may vary among different surgeons or clinics. Auditing programmes like the DSCA may help to clarify in what range this optimum should be. A composite quality measure might be a solution; that is, a metric which includes whether or not AL occurred, creation of a defunctioning stoma or end colostomy, readmission or mortality. Patient-reported outcomes are of additive value in this context. The choice between an anastomosis with or without a defunctioning stoma or an end colostomy can and should always be influenced by patient preferences.
Improvement of outcomes
When AL is used in hospital comparisons, it should be under the condition that practices with higher AL rates have the opportunity to improve their performance. Unfortunately, the actual cascade of factors resulting in AL still remains a ‘black box’. Our findings suggest that this black box consists of factors that represent multiple elements of the care processes taking place within a hospital. Peroperative factors, such as blood loss and duration of the operation, have been described as important predictors for AL by several authors.9–13 Longer duration, more blood loss than anticipated, an increased anastomotic strain and limited vascular supply at the anastomotic sites may be a proxy for a more complicated procedure, suggesting that AL rates might be related to surgical technical skill and experience. Additionally, factors more related to perioperative care than to surgical skill, such as oliguria during the operation, are also said to enhance the risk of leakage.54
The ultimate challenge for outcome researchers is to understand the complex clinical mechanisms that lead to success or failure, so that the excellence of best practices can be transferred to all hospitals performing these procedures.
Definition of AL
Comparison of AL between hospitals also requires the use of standard definitions and methods of measurement of AL. However, it has been stated before that the definition of AL varies; a systematic review by Bruce et al15 found 56 separate definitions for AL used in the literature. A valuable feature of an audit registration system is that it applies one definition that is used by all participants. In the DSCA, only clinically apparent leaks requiring reintervention have been registered, and a distinction has been made between radiological and surgical reintervention. Further (international) agreement on a standard definition that is valid and reliable, and can distinguish between clinical minor and major anastomotic leaks are explicitly important when using AL as an outcome indicator.
Conclusions
Hospital variation in AL rates is relatively independent of differences in case-mix. Differences in treatment factors contributed more to the variation in AL rates. Further exploration of in-hospital factors may give insight into further improvement possibilities and understanding the multifactorial process that underlies AL. Audit programmes may provide data for targeted visitation of clinics with bad outcomes, and best practices, aiding in identification of the most important areas for improvement.
Acknowledgments
The authors thank all surgeons, registrars, physician assistants, and administrative nurses who registered all the patients in the Dutch Surgical Colorectal Audit, and the Dutch Surgical Colorectal Audit group (E H Eddes, W A Bemelman, R M van Dam, E van der Harst, J H J M van Krieken, J Meijerink, M Elferink, B Groot-Koerkamp, F Ferenschild, D Boerma, E de Graaf, V E Lemmens, E R Manusama, H J T Rutten, C J H van de Velde), and the methodological board for their advice.
References
Footnotes
-
Contributors Study concept: HS, DH, JWD, KH, TW, TK, MW; study design: HS, DH, MW, MF; data acquisition: TK, KH, NvL, TW, RT; quality control of data and algorithms: DH, NvL, MF; data analysis and interpretation: HS, JWD, MW; manuscript preparation: HS, DH, MW; manuscript editing: DH. NvL, MtB, TK, KH, TW, JWD, RT, MW; manuscript review: DH. NvL, MtB, TK, KH, TW, JWD, RT, MW.
-
Competing interests None.
-
Provenance and peer review Not commissioned; externally peer reviewed.