Skip to main content
SpringerLink
Log in

Using hierarchical modeling to measure ICU quality

  • Original
  • Published:
Intensive Care Medicine Aims and scope Submit manuscript

Abstract

Objective

To determine whether hierarchical modeling agrees with conventional logistic regression modeling on the identity of ICU quality outliers within a large multi-institutional database.

Design

Retrospective database analysis.

Setting and patients

Subset of the Project IMPACT database consisting of 40,435 adult patients admitted to surgical, medical, and mixed surgical-medical ICUs (n =55) between 1997 and 1999 who met inclusion criteria for SAPS II.

Measurements and results

The SAPS II score was customized to this database using conventional logistic regression and using a hierarchical (random coefficients) model. Both models exhibited excellent discrimination (C statistic) and calibration (Hosmer-Lemeshow statistic). The hierarchical and nonhierarchical models hadC statistics of .870 and .865, and HL statistics of 3.71 (p>.88, df=8) and 8.94 (p>.35, df=8), respectively. Since the random effects component of the hierarchical model accounts for between-hospital variability, only the fixed-effects coefficients were used to calculate the expected mortality rate based on the hierarchical model. The ratio and 95% confidence intervals of the observed to expected mortality rate were calculated using both models for each ICU. ICUs whose observed/expected ratio was either less than 1 or greater than 1, and whose 95% confidence interval did not include 1 were labeled as either high-performance or low-performance outliers, respectively. Analysis using κ statistic revealed almost perfect agreement between the two models (nonhierarchical vs. hierarchical) on the identity of ICU quality outliers.

Conclusions

Models obtained by customizing SAPS II using a nonhierarchical and a hierarchical approach exhibit excellent agreement on the identity of ICU quality outliers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.

Similar content being viewed by others

Reference

  1. Knaus WA, Zimmerman JE, Wagner DP, Draper EA, Lawrence DE (1981) APACHE-acute physiology and chronic health evaluation: a physiologically based classification system. Crit Care Med 9:591–597

    Article  CAS  Google Scholar 

  2. Rivera-Fernandez R, Vazquez-Mata G, Bravo M, Aguayo-Hoyos E, Zimmerman J, Wagner D, Knaus W (1998) The APACHE III prognostic system: customized mortality predictions for Spanish ICU patients. Intensive Care Med 24:574–581

    Article  CAS  Google Scholar 

  3. Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963

    Article  CAS  Google Scholar 

  4. Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J (1993) Mortality probability models (MPM II) based on an international cohort of intensive care unit patients. JAMA 270:2478–2486

    Article  CAS  Google Scholar 

  5. Zimmerman JE, Shortell SM, Rousseau DM, Duffy J, Gillies RR, Knaus WA, Devers K, Wagner DP, Draper EA (1993) Improving intensive care: observations based on organizational case studies in nine intensive care units: a prospective, multicenter study. Crit Care Med 21:1443–1451

    Article  CAS  Google Scholar 

  6. Iezzoni LI (1996) On opening “black boxes” and looking inside. Int J Qual Health Care 8:209–210

    Article  CAS  Google Scholar 

  7. Iezzoni LI (1991) ‘Black box’ medical information systems. A technology needing assessment. JAMA 265:3006–3007

    Article  CAS  Google Scholar 

  8. Iezzoni LI (1997) The risks of risk adjustment. JAMA 278:1600–1607

    Article  CAS  Google Scholar 

  9. Burgess JF Jr, Christiansen CL, Michalak SE, Morris CN (2000) Medical profiling: improving standards and risk adjustments using hierarchical models. J Health Econ 19:291–309

    Article  Google Scholar 

  10. Marshall EC, Spiegelhalter DJ. Institutional Performance. In: Leyland AH, Goldstein H, editors (2001) Multilevel modeling of health statistics. Wiley, New York, pp 127–142

  11. Christiansen CL, Morris CN (1997) Improving the statistical approach to health care provider profiling. Ann Intern Med 127:764–768

    Article  CAS  Google Scholar 

  12. Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI (2001) Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 72:2155–2168

    Article  CAS  Google Scholar 

  13. Moreno R, Matos R (2000) The “new” scores: what problems have been fixed, and what remain? Curr Opin Crit Care 6:158–165

    Article  Google Scholar 

  14. Leyland AH, Goldstein H (eds) (2001) Multilvel modeling of health statistics, ch 1. Wiley, New York

  15. Goldstein H, Spiegelhalter DJ (1996) League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc 159:385–443

    Article  Google Scholar 

  16. Glance LG, Osler T, Dick AW (2002) Identifying quality outliers in a large multi-institutional database using customized versions of SAPS II and MPM II. Crit Care Med 30:1995–2002

    Article  Google Scholar 

  17. Glance LG, Osler T, Dick A (2002) Rating the quality of ICUs—is it a function of the ICU scoring system? Crit Care Med 30:1969–1975

    Article  Google Scholar 

  18. Rapoport J, Teres D, Steingrub J, Higgins T, McGee W, Lemeshow S (2000) Patient characteristics and ICU organizational factors that influence frequency of pulmonary artery catheterization. JAMA 283:2559–2567

    Article  CAS  Google Scholar 

  19. Diringer MN, Edwards PhD DF (2001) Admission to a neurologic/neurosurgical intensive care unit is associated with reduced mortality rate after intracerebral hemorrhage. Crit Care Med 29:635–640

  20. Cook SF, Visscher WA, Hobbs CL, Williams RL (1999) Preliminary results from a pilot validity study of a new observational database (abstract). Crit Care Med 27:36A

    Article  Google Scholar 

  21. Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963

    Article  CAS  Google Scholar 

  22. Leyland AH, Goldstein H (2001) Multilevel modelling of health statistics. Wiley, New York

  23. Rabe-Hesketh S, Skrondal A, Pickles A (2002) Estimation of generalized linear mixed models. Stata J 2:1–21

    Article  Google Scholar 

  24. Hox J (2002) Multilevel analysis: techniques and applications. Erlbaum, Mahwah

  25. Hadorn DD, Keeler E, Rogers WH, Brook RH (1993) Assessing the performance of mortality prediction models. RAND/UCLA/Harvard Center for Health Care Financing Policy Research, Santa Monica

  26. Hosmer DW, Lemeshow S (2000) Applied logistic regression, 2nd edn. Wiley-Interscience, New York

  27. Hosmer DW, Lemeshow S (1995) Confidence interval estimates of an index of quality performance based on logistic regression models. Stat Med 14:2161–2172

    Article  CAS  Google Scholar 

  28. Efron F, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New York

  29. Anonymous (2000) To err is human: building a safer health system. National Academy, Washington

  30. Anonymous (2001) Crossing the quality chasm. National Academy, Washington

  31. Iezzoni LI (1997) The risks of risk adjustment. JAMA 278:1600–1607

    Article  CAS  Google Scholar 

  32. Hurtado MP, Swift EK, Corrigan JM (2001) Envisioning the national health care quality report. National Academy, Washington

  33. Lansky D (2002) Improving quality through public disclosure of performance information. Health Aff (Millwood) 21:52–62

Download references

Author information

Authors and Affiliations

  1. University of Rochester, School of Medicine and Dentistry, 601 Elmwood Avenue, Rochester, NY, 14642, USA

    Laurent G. Glance & Andrew W. Dick

  2. University of Vermont, Medical College, USA

    Turner M. Osler

  3. University of California, Department of Medicine, Irvine 100 Theory, Irvine CA 92697-5800, USA

    Dana Mukamel

Authors
  1. Laurent G. Glance
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew W. Dick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Turner M. Osler
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dana Mukamel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laurent G. Glance.

Additional information

This project was supported by a Research Career Development Award to L.G.G. from the Agency for Healthcare Research and Quality (K08 HS11295). This paper was presented in abstract form as an oral presentation at the Society of Critical Care Medicine 32nd Critical Care Congress in 2003

Electronic Supplementary Material

Rights and permissions

Reprints and permissions

About this article

Cite this article

Glance, L.G., Dick, A.W., Osler, T.M. et al. Using hierarchical modeling to measure ICU quality. Intensive Care Med 29, 2223–2229 (2003). https://doi.org/10.1007/s00134-003-1959-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00134-003-1959-9

Keywords

Search

Navigation