Article Text

other Versions

Download PDFPDF
Random variation and rankability of hospitals using outcome indicators
  1. Anne-Margreet van Dishoeck,
  2. Hester F Lingsma,
  3. Johan P Mackenbach,
  4. Ewout W Steyerberg
  1. Department of Public Health, Center for Medical Decision Making, Erasmus MC, University Medical Center Rotterdam, Rotterdam, The Netherlands
  1. Correspondence to Anne-Margreet van Dishoeck, Room Ae-138, Erasmus MC, PO Box 2040, 3000 CA Rotterdam, The Netherlands; a.m.vandishoeck{at}


Objective There is a growing focus on quality and safety in healthcare. Outcome indicators are increasingly used to compare hospital performance and to rank hospitals, but the reliability of ranking (rankability) is under debate. This study aims to quantify the rankability of several outcome indicators of hospital performance currently used by the Dutch government.

Methods From 52 indicators used by the Netherlands Inspectorate, the authors selected nine outcome indicators presenting a fraction and absolute numbers. Of these indicators, four were combined into two, resulting in seven indicators for analysis. The official data of 97 Dutch hospitals for the year 2007 were used. Uncertainty in the observed outcomes within the hospitals (within hospital variance, σ2) was estimated using fixed effect logistic regression models. Heterogeneity (between hospital variance, τ2) was measured with random effect logistic regression models. Subsequently, the rankability was calculated by relating heterogeneity to uncertainty within and between hospitals (τ2/(τ2 +median σ2)).

Results Sample sizes varied but were typically around 200 per hospital (range of median 90–277) with a median of 2–21 cases, causing a substantial uncertainty in outcomes per hospital. Although fourfold to eightfold differences between hospitals were noted, the uncertainty within hospitals caused a poor (<50%) rankability in three indicators and moderate rankability (50–75%) in the other four indicators.

Conclusion The currently used Dutch outcome indicators are not suitable for ranking hospitals. When judging hospital quality the influence of random variation must be accounted for to avoid overinterpretation of the numbers in the quest for more transparency in healthcare. Adequate sample size is a prerequisite in attempting reliable ranking.

  • Health care quality
  • patient outcomes
  • quality of care
  • reliability

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


  • Funding Internal Erasmus MC grant for healthcare research (Mrace).

  • Competing interests None to declare.

  • Provenance and peer review Not commissioned; externally peer reviewed.