Background: There is debate about the role of crude mortality rates and case-mix adjusted mortality rates in monitoring the outcomes of treatment. In the context of quality improvement a key purpose of monitoring is to identify special cause variation as this type of variation should be investigated to identify possible causes. This paper investigates agreement between the identification of special cause variation in risk adjusted and observed hospital specific mortality rates after coronary artery bypass grafting in New York hospitals.
Methods: Coronary artery bypass grafting mortality rates between 1994 and 2003 were obtained from the New York State Department of Health's cardiovascular reports for 41 hospitals. Cross-sectional control charts of crude (observed) and risk adjusted mortality rates were produced for each year. Special cause variation was defined as a data point beyond the 99.9% probability limits: hospitals showing special cause variation were identified for each year. Longitudinal control charts of crude (observed) and risk adjusted mortality rates were produced for each hospital with data for all ten years (n = 27). Special cause variation was defined as a data point beyond 99.9% probability limits, two out of three consecutive data points beyond 95% probability limits (two standard deviations from the mean) or a run of five consecutive points on one side of the mean. Years showing special cause variation in mortality were identified for each hospital. Cohen's Kappa was calculated for agreement between special causes identified in crude and risk-adjusted control charts.
Results: In cross sectional analysis the Cohen's Kappa was 0.54 (95% confidence interval: 0.28 to 0.78), indicating moderate agreement between the crude and risk-adjusted control charts with sensitivity 0.4 (95% confidence interval 0.17-0.69) and specificity 0.98 (95% confidence interval: 0.95-0.99). In longitudinal analysis, the Cohen's Kappa was 0.61 (95% confidence interval: 0.39 to 0.83) indicating good agreement between the tests with sensitivity 0.63 (95% confidence interval: 0.39-0.82) and specificity 0.98 (95% confidence interval: 0.96 to 0.99).
Conclusion: There is moderate-good agreement between signals of special cause variation between observed and risk-adjusted mortality. Analysis of observed hospital specific CABG mortality over time and with other hospitals appears to be useful in identifying special causes of variation. Case-mix adjustment may not be essential for longitudinal monitoring of outcomes using control charts.