Review
Issues in Quality Measurement: Target Population, Risk Adjustment, and Ratings

https://doi.org/10.1016/j.athoracsur.2013.03.029Get rights and content

This review investigates three fundamental issues in health care performance measurement: selection of a homogeneous target population, risk adjustment, and assignment of quality rating categories. Many but not all organizations involved in quality measurement have adopted similar approaches to these important methodological issues. To illustrate the practical implications of different profiling strategies, we use The Society of Thoracic Surgeons' data to compare profiling results derived using prevailing analytical methodologies with those obtained from alternative approaches, exemplified by those of a well-known health care performance rating organization. We demonstrate the differences in provider classification that may result from these methodologic decisions.

Section snippets

Data Source: STS Adult Cardiac Surgery Database

The STS Adult Cardiac Surgery Database (ACD) is an all-payer cardiac surgery registry that has collected granular, clinical data since 1989. Random site audits conducted annually by an external organization have demonstrated high data accuracy. The STS ACD has developed and periodically revised risk models for all major adult cardiac surgical procedures 9, 10, 11, and it has constructed composite performance measures for isolated CABG 12, 13 and isolated AVR [14]. In 2010, it began a

Target Population Heterogeneity

Table 1 compares the overall number of cases identified from the 2009 STS ACD using STS definitions of isolated CABG and isolated AVR, the corresponding number of cases using Leapfrog specifications, the aggregate unadjusted mortality rate in each cohort, and the percentage of high and low observed mortality outliers (two-sided 95% CI). Differences in outliers resulting from these two population definitions are shown graphically in the Figure 1, where outliers for each approach are plotted

Selection of Target Population

Some profiling initiatives use heterogeneous target populations to increase sample size but the resulting provider performance assessments are potentially biased by case mix. If sample size is a concern, data collection over numerous years or a composite of multiple endpoints may be better solutions.

Our analyses illustrate the implications of a heterogeneous target population, which results in a mix of patients, conditions, or procedures having widely variable mortality rates. For example, the

References (51)

  • NQF Evidence Task Force final report. Available at:...
  • Northern New England Cardiovascular Disease Study Group. Available at: http://www.nnecdsg.org/data_cabgallsites4.htm....
  • Massachusetts Data Analysis Center. Available at: www.massdac.org. Accessed March...
  • J.A. Spertus et al.

    ACCF/AHA new insights into the methodology of performance measurement: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures

    Circulation

    (2010)
  • L.I. Iezzoni

    Risk adjustment for measuring health care outcomes

    (2003)
  • H.M. Krumholz et al.

    Standards for statistical models used for public reporting of health outcomes

    Circulation

    (2006)
  • D.M. Shahian et al.

    The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 1—coronary artery bypass grafting surgery

    Ann Thorac Surg

    (2009)
  • S.M. O'Brien et al.

    The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 2—isolated valve surgery

    Ann Thorac Surg

    (2009)
  • D.M. Shahian et al.

    The Society of Thoracic Surgeons 2008 cardiac surgery risk models: part 3—valve plus coronary artery bypass grafting surgery

    Ann Thorac Surg

    (2009)
  • D.M. Shahian et al.

    Quality measurement in adult cardiac surgery: part 1—conceptual framework and measure selection

    Ann Thorac Surg

    (2007)
  • S.M. O'Brien et al.

    Quality measurement in adult cardiac surgery: part 2—Statistical considerations in composite measure scoring and provider rating

    Ann Thorac Surg

    (2007)
  • T.G. Ferris et al.

    Public release of clinical outcomes data-online CABG report cards

    N Engl J Med

    (2010)
  • STS version 2.61 Procedure Identification Table. Available at:...
  • The Leapfrog Group. Available at: http://www.leapfroggroup.org/. Accessed September...
  • Leapfrog Hospital Survey Reference Book: supporting documentation for the 2010 Leapfrog hospital survey. Available upon...
  • Cited by (25)

    • Development of a Congenital Heart Surgery Composite Quality Metric: Part 1—Conceptual Framework

      2019, Annals of Thoracic Surgery
      Citation Excerpt :

      Their use can be limited, however, when the event rate is low or when sample sizes are small [5]. Other important considerations include the costs associated with detailed data collection and analytics related to the use of outcome metrics [6]. In addition to Donabedian’s triad of structure, process, and outcome, the Institute of Medicine has further delineated six domains of quality [7]:

    • Risk Aversion and Public Reporting. Part 2: Mitigation Strategies

      2017, Annals of Thoracic Surgery
      Citation Excerpt :

      These are typically not published in the peer-reviewed literature nor are they endorsed by the National Quality Forum, currently the best assurance that a measure is suitable for public reporting (all STS measures are submitted to the National Quality Forum). Measures used for health care public reporting must address a number of important statistical and practical considerations that affect their validity and reliability [3, 9, 32, 43–47], at a minimum the important features listed in Table 1. Performance metrics should focus on outcomes, because these are, in Donabedian’s words, the “ultimate validators of the effectiveness and quality of medical care” [48], yet these need not be the only quality metrics.

    View all citing articles on Scopus
    View full text