Article Text

PDF
What is the probability of detecting poorly performing hospitals using funnel plots?
  1. Sarah E Seaton1,
  2. Lisa Barker2,
  3. Hester F Lingsma3,
  4. Ewout W Steyerberg3,
  5. Bradley N Manktelow1
  1. 1Department of Health Sciences, University of Leicester, Leicester, UK
  2. 2Neonatal Unit, University Hospitals of Leicester NHS Trust, Leicester, UK
  3. 3Department of Public Health, Erasmus MC, Rotterdam, The Netherlands
  1. Correspondence to Dr B N Manktelow, Department of Health Sciences, University of Leicester, 22–28 Princess Road West, Leicester LE1 6TP, UK; brad.manktelow{at}le.ac.uk

Abstract

Recent high profile cases in the UK have highlighted the impact that the reporting of clinical outcomes can have for healthcare providers and patients. Great importance has been placed on the use of statistical methods to identify healthcare providers with observed poor performance. Such providers are highlighted as potential outliers and the possible causes investigated. It is crucial, therefore, that the methods used for identifying outliers are correctly understood and interpreted. While patients, funders, managers and clinical teams really want to know the true underlying performance of the provider, this true performance generally cannot be known directly and must be estimated using observed outcomes. However, differences between the true and observed performance are likely to arise due to chance variation. Clinical outcomes are often reported through the standardised mortality ratio (SMR), displayed using funnel plots. Providers whose observed SMR falls outside the funnel plot control limits will be identified as potential outliers. However, while it is obviously desirable that a healthcare provider with a true underlying performance different from that expected should be identified, the actual probability that it will be identified from observed SMRs has not previously been described. Here we show that funnel plots for the SMR should be used with caution when the expected number of events is small as the probability of identifying providers with true poor performance is likely to be small. On the other hand, when the expected number of events is large, care must be taken as a provider may be identified as an outlier even when its divergence is of little or no clinical importance.

  • Statistical process control
  • Mortality (standardized mortality ratios)
  • Performance measures
  • Healthcare quality improvement

Statistics from Altmetric.com

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.