Abstract
Objective
To determine whether hierarchical modeling agrees with conventional logistic regression modeling on the identity of ICU quality outliers within a large multi-institutional database.
Design
Retrospective database analysis.
Setting and patients
Subset of the Project IMPACT database consisting of 40,435 adult patients admitted to surgical, medical, and mixed surgical-medical ICUs (n =55) between 1997 and 1999 who met inclusion criteria for SAPS II.
Measurements and results
The SAPS II score was customized to this database using conventional logistic regression and using a hierarchical (random coefficients) model. Both models exhibited excellent discrimination (C statistic) and calibration (Hosmer-Lemeshow statistic). The hierarchical and nonhierarchical models hadC statistics of .870 and .865, and HL statistics of 3.71 (p>.88, df=8) and 8.94 (p>.35, df=8), respectively. Since the random effects component of the hierarchical model accounts for between-hospital variability, only the fixed-effects coefficients were used to calculate the expected mortality rate based on the hierarchical model. The ratio and 95% confidence intervals of the observed to expected mortality rate were calculated using both models for each ICU. ICUs whose observed/expected ratio was either less than 1 or greater than 1, and whose 95% confidence interval did not include 1 were labeled as either high-performance or low-performance outliers, respectively. Analysis using κ statistic revealed almost perfect agreement between the two models (nonhierarchical vs. hierarchical) on the identity of ICU quality outliers.
Conclusions
Models obtained by customizing SAPS II using a nonhierarchical and a hierarchical approach exhibit excellent agreement on the identity of ICU quality outliers.
Similar content being viewed by others
Reference
Knaus WA, Zimmerman JE, Wagner DP, Draper EA, Lawrence DE (1981) APACHE-acute physiology and chronic health evaluation: a physiologically based classification system. Crit Care Med 9:591–597
Rivera-Fernandez R, Vazquez-Mata G, Bravo M, Aguayo-Hoyos E, Zimmerman J, Wagner D, Knaus W (1998) The APACHE III prognostic system: customized mortality predictions for Spanish ICU patients. Intensive Care Med 24:574–581
Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963
Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J (1993) Mortality probability models (MPM II) based on an international cohort of intensive care unit patients. JAMA 270:2478–2486
Zimmerman JE, Shortell SM, Rousseau DM, Duffy J, Gillies RR, Knaus WA, Devers K, Wagner DP, Draper EA (1993) Improving intensive care: observations based on organizational case studies in nine intensive care units: a prospective, multicenter study. Crit Care Med 21:1443–1451
Iezzoni LI (1996) On opening “black boxes” and looking inside. Int J Qual Health Care 8:209–210
Iezzoni LI (1991) ‘Black box’ medical information systems. A technology needing assessment. JAMA 265:3006–3007
Iezzoni LI (1997) The risks of risk adjustment. JAMA 278:1600–1607
Burgess JF Jr, Christiansen CL, Michalak SE, Morris CN (2000) Medical profiling: improving standards and risk adjustments using hierarchical models. J Health Econ 19:291–309
Marshall EC, Spiegelhalter DJ. Institutional Performance. In: Leyland AH, Goldstein H, editors (2001) Multilevel modeling of health statistics. Wiley, New York, pp 127–142
Christiansen CL, Morris CN (1997) Improving the statistical approach to health care provider profiling. Ann Intern Med 127:764–768
Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, Dreyer PI (2001) Cardiac surgery report cards: comprehensive review and statistical critique. Ann Thorac Surg 72:2155–2168
Moreno R, Matos R (2000) The “new” scores: what problems have been fixed, and what remain? Curr Opin Crit Care 6:158–165
Leyland AH, Goldstein H (eds) (2001) Multilvel modeling of health statistics, ch 1. Wiley, New York
Goldstein H, Spiegelhalter DJ (1996) League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc 159:385–443
Glance LG, Osler T, Dick AW (2002) Identifying quality outliers in a large multi-institutional database using customized versions of SAPS II and MPM II. Crit Care Med 30:1995–2002
Glance LG, Osler T, Dick A (2002) Rating the quality of ICUs—is it a function of the ICU scoring system? Crit Care Med 30:1969–1975
Rapoport J, Teres D, Steingrub J, Higgins T, McGee W, Lemeshow S (2000) Patient characteristics and ICU organizational factors that influence frequency of pulmonary artery catheterization. JAMA 283:2559–2567
Diringer MN, Edwards PhD DF (2001) Admission to a neurologic/neurosurgical intensive care unit is associated with reduced mortality rate after intracerebral hemorrhage. Crit Care Med 29:635–640
Cook SF, Visscher WA, Hobbs CL, Williams RL (1999) Preliminary results from a pilot validity study of a new observational database (abstract). Crit Care Med 27:36A
Le Gall JR, Lemeshow S, Saulnier F (1993) A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 270:2957–2963
Leyland AH, Goldstein H (2001) Multilevel modelling of health statistics. Wiley, New York
Rabe-Hesketh S, Skrondal A, Pickles A (2002) Estimation of generalized linear mixed models. Stata J 2:1–21
Hox J (2002) Multilevel analysis: techniques and applications. Erlbaum, Mahwah
Hadorn DD, Keeler E, Rogers WH, Brook RH (1993) Assessing the performance of mortality prediction models. RAND/UCLA/Harvard Center for Health Care Financing Policy Research, Santa Monica
Hosmer DW, Lemeshow S (2000) Applied logistic regression, 2nd edn. Wiley-Interscience, New York
Hosmer DW, Lemeshow S (1995) Confidence interval estimates of an index of quality performance based on logistic regression models. Stat Med 14:2161–2172
Efron F, Tibshirani RJ (1993) An introduction to the bootstrap. Chapman & Hall, New York
Anonymous (2000) To err is human: building a safer health system. National Academy, Washington
Anonymous (2001) Crossing the quality chasm. National Academy, Washington
Iezzoni LI (1997) The risks of risk adjustment. JAMA 278:1600–1607
Hurtado MP, Swift EK, Corrigan JM (2001) Envisioning the national health care quality report. National Academy, Washington
Lansky D (2002) Improving quality through public disclosure of performance information. Health Aff (Millwood) 21:52–62
Author information
Authors and Affiliations
Corresponding author
Additional information
This project was supported by a Research Career Development Award to L.G.G. from the Agency for Healthcare Research and Quality (K08 HS11295). This paper was presented in abstract form as an oral presentation at the Society of Critical Care Medicine 32nd Critical Care Congress in 2003
Electronic Supplementary Material
Rights and permissions
About this article
Cite this article
Glance, L.G., Dick, A.W., Osler, T.M. et al. Using hierarchical modeling to measure ICU quality. Intensive Care Med 29, 2223–2229 (2003). https://doi.org/10.1007/s00134-003-1959-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00134-003-1959-9