The statistician's pageCumulative Sum Curves and Their Prediction Limits
Section snippets
Risk Adjustment
Measuring the results of a surgical intervention is essential to determining its effectiveness. Public reporting and comparisons are proceeding, with the ultimate goal of identifying the best providers, whose permissions and reimbursements might be determined by their results. For this task, risk adjustment is imperative; because of differences in patient profiles, no two patients, and thus their intrinsic risks of death and other complications, are exactly the same. Among medical procedures,
Examples
We will begin with a simple example using dummy patients. Suppose you operate on 10 relatively high-risk patients with expected mortality ranging from 10% to 35% (Fig 1). In this example, the sum of the expected mortality risks (in decimal notation) is E = 2.0, meaning that 2.0 deaths are expected. In fact, two patients actually died (O = 2), so the O/E ratio is 1.0, and you are operating as expected. A more detailed analysis of this limited experience can be obtained by plotting the cumulative
CUSUM Prediction Limits
The O-E difference will almost never equal exactly zero, even when performance is as expected. Random variation must be accounted for, before any clinical difference is attributed to performance, by constructing an interval estimate that contains the values that are consistent with the observed (point) estimate. A simple 95% confidence interval can be constructed using the normal (“bell-shaped”) approximation to the binomial distribution (Appendix). To produce prediction intervals for the CUSUM
Comment
Steiner and colleagues [14] note that the limitation of the O-E CUSUM plots is that “… they do not specify how much variation in the plot is expected under good surgical performance, and hence how large a deviation from the expected should be a cause for concern” [14]. Our major purpose was to address this “limitation.” Prediction limits can easily be put around a CUSUM using the familiar normal approximation to the binomial distribution. Also, but technically more difficult, exact prediction
References (25)
- et al.
Blood transfusion reduction in cardiac surgery: multidisciplinary approach at a community hospital
Ann Thorac Surg
(2009) - et al.
Cumulative sum techniques for assessing surgical results
Ann Thorac Surg
(2003) - et al.
Control chart methods for monitoring cardiac surgical performance and their interpretation
J Thorac Cardiovasc Surg
(2004) Monitoring clinical performance: a commentary
J Thorac Cardiovasc Surg
(2004)- et al.
Monitoring cardiac surgical performance: a commentary
J Thorac Cardiovasc Surg
(2004) - et al.
Monitoring the performance of residents during training in off-pump coronary surgery
J Thorac Cardiovasc Surg
(2004) Monitoring surgical performance
J Thorac Cardiovasc Surg
(2004)- et al.
What are the odds?
Ann Thorac Surg
(2007) - et al.
Validation and refinement of mortality risk models for heart valve surgery
Ann Thorac Surg
(2005) - et al.
Real-time monitoring of coronary care mortality: a comparison and combination of two monitoring tools
Int J Cardiol
(2005)
Direct comparison of risk-adjusted and non-risk-adjusted CUSUM analyses of coronary artery bypass surgery outcomes
J Thorac Cardiovasc Surg
Monitoring the results of cardiac surgery by variable life-adjusted display
Lancet
Cited by (23)
Learning Curve for Temporary Hemodialysis Catheter Placement
2023, Transplantation ProceedingsContinuous Monitoring of Risk-Adjusted Outcomes: Excess Deaths vs Lives Saved
2021, Annals of Thoracic SurgeryCitation Excerpt :The answer is provided by computing prediction limits. Two methods have been suggested: pointwise prediction limits for the difference between cumulative expected and cumulative observed deaths (Figure 3)11,12 and the Rocket Tail plot.13 The pointwise prediction limits are symmetric and the Rocket Tails are slightly asymmetric—otherwise, they are very similar.
Analysis of competency in phacoemulsification trainees using the cumulative sum analysis
2017, Journal of Cataract and Refractive SurgeryCitation Excerpt :Cumulative posterior capsule rupture rates require at least 100 cases for meaningful analysis, rendering it difficult to determine which trainees require more attention early on, especially for trainees who are performing neither extremely well nor very poorly. The CUSUM analysis is more sensitive to a change in performance than the cumulative probability plot; hence, the CUSUM would be more useful as a tool for continuous quality improvement in trainees.23,24 This continuous monitoring of performance enables the early detection of trainees with difficulties and allows the training program to be tailored to the individual's capabilities.
Learning curve for EUS in gastric cancer T staging by using cumulative sum analysis
2015, Gastrointestinal EndoscopyCitation Excerpt :Continuous performance monitoring may be needed even after achieving competency. CUSUM analysis would be helpful for self-monitoring and continuous quality improvement in trainees.15,27 In contrast to esophageal cancer, gastric cancer requires relatively complex endoscope control during EUS according to the tumor’s location.
Monitoring methods in the surgical arena: Assumptions and potential pitfalls
2013, European Journal of Surgical OncologyCitation Excerpt :These have been the subject of extensive debates2,5,13. Related parameters, such as the limits set for alarms, how much variation in the plot is expected under “good performance”, and how large a deviation from the expected should be cause of concern have also been discussed by other authors for all three procedures14,13,15. The funnel plot requires definition of the confidence limits so that mortality rates lying outside these intervals are identified and investigated16,8.
Report of a consensus conference on transplant program quality and surveillance
2012, American Journal of Transplantation
For related article, see page 532