Elsevier

The Annals of Thoracic Surgery

Volume 72, Issue 6, December 2001, Pages 2155-2168
The Annals of Thoracic Surgery

Review
Cardiac surgery report cards: comprehensive review and statistical critique1

https://doi.org/10.1016/S0003-4975(01)03222-2Get rights and content

Abstract

Public report cards and confidential, collaborative peer education represent distinctly different approaches to cardiac surgery quality assessment and improvement. This review discusses the controversies regarding their methodology and relative effectiveness.

Report cards have been the more commonly used approach, typically as a result of state legislation. They are based on the presumption that publication of outcomes effectively motivates providers, and that market forces will reward higher quality. Numerous studies have challenged the validity of these hypotheses. Furthermore, although states with report cards have reported significant decreases in risk-adjusted mortality, it is unclear whether this improvement resulted from public disclosure or, rather, from the development of internal quality programs by hospitals. An additional confounding factor is the nationwide decline in heart surgery mortality, including states without quality monitoring. Finally, report cards may engender negative behaviors such as high-risk case avoidance and “gaming” of the reporting system, especially if individual surgeon results are published.

The alternative approach, continuous quality improvement, may provide an opportunity to enhance performance and reduce interprovider variability while avoiding the unintended negative consequences of report cards. This collaborative method, which uses exchange visits between programs and determination of best practice, has been highly effective in northern New England and in the Veterans Affairs Administration. However, despite their potential advantages, quality programs based solely on confidential continuous quality improvement do not address the issue of public accountability. For this reason, some states may continue to mandate report cards. In such instances, it is imperative that appropriate statistical techniques and report formats are used, and that professional organizations simultaneously implement continuous quality improvement programs.

The statistical methodology underlying current report cards is flawed, and does not justify the degree of accuracy presented to the public. All existing risk-adjustment methods have substantial inherent imprecision, and this is compounded when the results of such patient-level models are aggregated and used inappropriately to assess provider performance. Specific problems include sample size differences, clustering of observations, multiple comparisons, and failure to account for the random component of interprovider variability. We advocate the use of hierarchical or multilevel statistical models to address these concerns, as well as report formats that emphasize the statistical uncertainty of the results.

Section snippets

Background

Just as Deming and Juran led an industrial quality revolution during the second half of the 20th century, quality improvement has emerged over the last decade as one of the dominant themes in modern health care. Specific areas of interest have included access, appropriateness, geographic variations in care, reduction of medical error and costs, cost-effectiveness, long-term functional status, and quality of the provider–patient relationship 1, 2, 3, 4, 5, 6. Although quality can be assessed on

Market-driven quality improvement (“report cards”)

Report cards have been the more widely used approach to cardiac surgery quality assessment and improvement. They are based on the presumption that publication of outcomes effectively motivates providers, and that market forces will reward higher quality. Numerous studies have challenged the validity of these hypotheses. Furthermore, controversies persist regarding the statistical methodology of report cards, their presentation format, and their actual impact on cardiac surgery quality.

The CQI approach to cardiac surgery quality improvement

Despite the reduction in mortality observed in states with cardiac surgery report cards, this approach has also produced fear, distrust, gaming, and other negative behaviors on the part of providers, which may paradoxically impede continued quality improvement 7, 126. Jencks [127] has stated that it is unreasonable to expect honest self-examination under the threat of media disclosure. In 1989, several years before public report cards were made available in New York or Pennsylvania, Berwick

The balance between public accountability and peer-based quality improvement

A continuous quality improvement approach may offer the best possibility for improving outcomes and reducing variability among hospitals without the unintended negative consequences of report cards. However, it appears inevitable that some states and payers will demand access to cardiac surgery outcome data, despite evidence that such information is imperfect and is rarely used to select providers. As formulated by Epstein [132], the issue is whether to rely on the market impact of such report

Recommendations

Whenever feasible, state and regional cardiac surgical organizations should implement continuous quality improvement programs. These should include exchange site visits, benchmarking, and determination of best practice. This process and systems–oriented approach is the best way to raise the performance level of all cardiac surgery programs; it affirms the commitment of our specialty to excellence, and it should be the cornerstone of any quality improvement effort.

In states where legislation

References (133)

  • E.D. Peterson et al.

    The effects of New York’s bypass surgery provider profiling on access to care and patient outcomes in the elderly

    J Am Coll Cardiol

    (1998)
  • S.W. Dziuban et al.

    How a New York cardiac surgery program uses outcomes data

    Ann Thorac Surg

    (1994)
  • S.W. Dziuban

    Using information from databases to improve clinical practicelessons learned under fire

    Ann Thorac Surg

    (1997)
  • V.A. Ferraris

    The dangers of gathering data

    J Thorac Cardiovasc Surg

    (1992)
  • E.L. Hannan et al.

    Public release of cardiac surgery outcomes data in New Yorkwhat do New York State cardiologists think of it?

    Am Heart J

    (1997)
  • J.M. Bentley et al.

    How Pennsylvania hospitals have responded to publicly released reports on coronary artery bypass graft surgery

    J Comm J Qual Improv

    (1998)
  • K.V. Arom et al.

    Establishing and using a local/regional cardiac surgery database

    Ann Thorac Surg

    (1997)
  • W.L. Holman et al.

    Alabama coronary artery bypass grafting Cooperative Projectbaseline data. Alabama CABG Cooperative Project Study Group

    Ann Thorac Surg

    (1999)
  • J.R. Goss et al.

    Washington State’s model of physician leadership in cardiac outcomes reporting

    Ann Thorac Surg

    (2000)
  • F.L. Grover et al.

    Factors predictive of operative mortality among coronary artery bypass subsets

    Ann Thorac Surg

    (1993)
  • F.L. Grover et al.

    The Veterans Affairs Continuous Improvement in Cardiac Surgery Study

    Ann Thorac Surg

    (1994)
  • K.E. Hammermeister et al.

    Using outcomes data to improve clinical practicewhat we have learned

    Ann Thorac Surg

    (1994)
  • K.E. Hammermeister

    Participatory continuous improvement

    Ann Thorac Surg

    (1994)
  • G. Marshall et al.

    Bayesian-logit model for risk assessment in coronary artery bypass grafting

    Ann Thorac Surg

    (1994)
  • F.L. Grover et al.

    Calculating risk and outcomethe Veterans Affairs Database

    Ann Thorac Surg

    (1996)
  • J. Daley et al.

    Validating risk-adjusted surgical outcomessite visit assessment of process and structure. National VA Surgical Risk Study

    J Am Coll Surg

    (1997)
  • T.B. Ferguson et al.

    The STS National DatabaseCurrent changes and challenges for the new millennium

    Ann Thorac Surg

    (2000)
  • A.L. Shroyer et al.

    Updates to the Data Quality Review ProgramThe Society of Thoracic Surgeons Adult Cardiac National Database

    Ann Thorac Surg

    (1998)
  • A.L. Shroyer et al.

    1995 coronary artery bypass risk modelThe Society of Thoracic Surgeons Adult Cardiac National Database

    Ann Thorac Surg

    (1998)
  • R.E. Clark

    The STS Cardiac Surgery National Databasean update

    Ann Thorac Surg

    (1995)
  • R.E. Clark

    Calculating risk and outcomeThe Society of Thoracic Surgeons Database

    Ann Thorac Surg

    (1996)
  • F.H. Edwards et al.

    The Society of Thoracic Surgeons National Cardiac Surgery Databasecurrent risk assessment

    Ann Thorac Surg

    (1997)
  • F.G. Estafanous et al.

    Increased risk and decreased morbidity of coronary artery bypass grafting between 1986 and 1994

    Ann Thorac Surg

    (1998)
  • E.D. Peterson et al.

    Challenges in comparing risk-adjusted bypass surgery mortality resultsresults from the Cooperative Cardiovascular Project

    J Am Coll Cardiol

    (2000)
  • J.F. Burgess et al.

    Medical profilingimproving standards and risk adjustments using hierarchical models

    J Health Econ

    (2000)
  • L.I. Iezzoni

    Using risk-adjusted outcomes to assess clinical practicean overview of issues pertaining to risk adjustment

    Ann Thorac Surg

    (1994)
  • P.M. Steen

    Approaches to predictive modeling

    Ann Thorac Surg

    (1994)
  • J. Daley

    Criteria by which to evaluate risk-adjusted outcomes programs in cardiac surgery

    Ann Thorac Surg

    (1994)
  • G.A. Diamond

    What price perfection? Calibration and discrimination of clinical prediction models

    J Clin Epidemiol

    (1992)
  • D. Blumenthal

    Part 1. Quality of carewhat is it?

    N Engl J Med

    (1996)
  • R.H. Brook et al.

    Quality of health care.Part 2: measuring quality of care

    N Engl J Med

    (1996)
  • M.R. Chassin

    Quality of health care.Part 3: improving the quality of care

    N Engl J Med

    (1996)
  • D. Blumenthal

    Quality of health care.Part 4: the origins of the quality-of-care debate

    N Engl J Med

    (1996)
  • M.R. Chassin et al.

    The urgent need to improve health care quality. Institute of Medicine National Roundtable on Health Care Quality

    JAMA

    (1998)
  • T. Bodenheimer

    The American health care systemthe movement for improved quality in health care

    N Engl J Med

    (1999)
  • J.S. Carey et al.

    Quality improvement in thoracic surgery

    Bull Am Coll Surg

    (1998)
  • E.J. Topol et al.

    Scorecard cardiovascular medicine. Its impact and future directions

    Ann Intern Med

    (1994)
  • R.H. Jones

    In search of the optimal surgical mortality

    Circulation

    (1989)
  • J. Ivanov et al.

    Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery

    Circulation

    (1999)
  • G. Marshall et al.

    Assessment of predictive models for binary outcomesan empirical approach using operative death from cardiac surgery

    Stat Med

    (1994)
  • Cited by (239)

    View all citing articles on Scopus
    1

    This review is an abridged version of a report submitted by the Massachusetts Cardiac Care Quality Commission to the Massachusetts Legislature, May 2001.

    View full text