ReviewIssues in Quality Measurement: Target Population, Risk Adjustment, and Ratings
Section snippets
Data Source: STS Adult Cardiac Surgery Database
The STS Adult Cardiac Surgery Database (ACD) is an all-payer cardiac surgery registry that has collected granular, clinical data since 1989. Random site audits conducted annually by an external organization have demonstrated high data accuracy. The STS ACD has developed and periodically revised risk models for all major adult cardiac surgical procedures 9, 10, 11, and it has constructed composite performance measures for isolated CABG 12, 13 and isolated AVR [14]. In 2010, it began a
Target Population Heterogeneity
Table 1 compares the overall number of cases identified from the 2009 STS ACD using STS definitions of isolated CABG and isolated AVR, the corresponding number of cases using Leapfrog specifications, the aggregate unadjusted mortality rate in each cohort, and the percentage of high and low observed mortality outliers (two-sided 95% CI). Differences in outliers resulting from these two population definitions are shown graphically in the Figure 1, where outliers for each approach are plotted
Selection of Target Population
Some profiling initiatives use heterogeneous target populations to increase sample size but the resulting provider performance assessments are potentially biased by case mix. If sample size is a concern, data collection over numerous years or a composite of multiple endpoints may be better solutions.
Our analyses illustrate the implications of a heterogeneous target population, which results in a mix of patients, conditions, or procedures having widely variable mortality rates. For example, the
References (51)
- et al.
New York State's cardiac surgery reporting system: four years later
Ann Thorac Surg
(1994) - et al.
The Society of Thoracic Surgeons isolated aortic valve replacement (AVR) composite score: a report of the STS Quality Measurement Task Force
Ann Thorac Surg
(2012) - et al.
Cardiac surgery risk models: a position article
Ann Thorac Surg
(2004) - et al.
Cardiac surgery report cards: comprehensive review and statistical critique
Ann Thorac Surg
(2001) - et al.
Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care: results of the National Veterans Affairs Surgical Risk Study
J Am Coll Surg
(1997) - et al.
Hospital characteristics, clinical severity, and outcomes for surgical oncology patients
Surgery
(2010) - et al.
Ranking hospitals on surgical quality: does risk-adjustment always matter?
J Am Coll Surg
(2008) - et al.
The public health hazards of risk avoidance associated with public reporting of risk-adjusted outcomes in coronary intervention
J Am Coll Cardiol
(2009) - et al.
Making health care quality reports easier to use
Jt Comm J Qual Improv
(2001) What is value in health care?
N Engl J Med
(2010)
ACCF/AHA new insights into the methodology of performance measurement: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures
Circulation
Risk adjustment for measuring health care outcomes
Standards for statistical models used for public reporting of health outcomes
Circulation
The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting surgery
Ann Thorac Surg
The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2—isolated valve surgery
Ann Thorac Surg
The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3—valve plus coronary artery bypass grafting surgery
Ann Thorac Surg
Quality measurement in adult cardiac surgery: part 1—conceptual framework and measure selection
Ann Thorac Surg
Quality measurement in adult cardiac surgery: part 2—Statistical considerations in composite measure scoring and provider rating
Ann Thorac Surg
Public release of clinical outcomes data-online CABG report cards
N Engl J Med
Cited by (25)
Use of artificial intelligence for nonlinear benchmarking of surgical care
2023, Surgery (United States)The Use and Misuse of Indirectly Standardized, Risk-Adjusted Outcomes and Star Ratings
2020, Annals of Thoracic SurgeryDevelopment of a Congenital Heart Surgery Composite Quality Metric: Part 1—Conceptual Framework
2019, Annals of Thoracic SurgeryCitation Excerpt :Their use can be limited, however, when the event rate is low or when sample sizes are small [5]. Other important considerations include the costs associated with detailed data collection and analytics related to the use of outcome metrics [6]. In addition to Donabedian’s triad of structure, process, and outcome, the Institute of Medicine has further delineated six domains of quality [7]:
Risk Aversion and Public Reporting. Part 2: Mitigation Strategies
2017, Annals of Thoracic SurgeryCitation Excerpt :These are typically not published in the peer-reviewed literature nor are they endorsed by the National Quality Forum, currently the best assurance that a measure is suitable for public reporting (all STS measures are submitted to the National Quality Forum). Measures used for health care public reporting must address a number of important statistical and practical considerations that affect their validity and reliability [3, 9, 32, 43–47], at a minimum the important features listed in Table 1. Performance metrics should focus on outcomes, because these are, in Donabedian’s words, the “ultimate validators of the effectiveness and quality of medical care” [48], yet these need not be the only quality metrics.
From Nonclinical Research to Clinical Trials and Patient-registries: Challenges and Opportunities in Biomedical Research
2017, Revista Espanola de Cardiologia