Jump to comment:

• What to do with outliers?
  Gareth J Parry
  Published on: 13 December 2005

• Published on: 13 December 2005

  What to do with outliers?

  • Gareth J Parry, Reader
  • Other Contributors:
    • Elizabeth Draper, Patricia McKinney

  Dear Editor,

  The paper by Spiegelhalter is a valuable contribution to the literature on presenting and displaying performance related outcome measures.[1] It provides further methodological guidance on identifying service providers whose performance falls outside control limits using funnel plot methodology. When reporting on performance it is important to have procedures in place which should be followed when...

  Show More

  Dear Editor,

  The paper by Spiegelhalter is a valuable contribution to the literature on presenting and displaying performance related outcome measures.[1] It provides further methodological guidance on identifying service providers whose performance falls outside control limits using funnel plot methodology. When reporting on performance it is important to have procedures in place which should be followed when outliers are identified. These issues have been considered by the Paediatric Intensive Care Audit Network (PICANet) who use the funnel plot methodology for reporting risk-adjusted mortality from all paediatric intensive care units (PICU) in England and Wales. Prior to producing these funnel plots for the latest National Report, we issued a policy statement drawn up in consultation with both our Clinical Advisory and Steering Groups.[2] In summary, the PICANet policy (published in full at http://www.picanet.org.uk) recognises that a PICU whose risk-adjusted mortality lies outside of the control limits will be identified as having returned data that is markedly different to the other PICUs. It is important to note that a PICU lying outside the control limits is not sufficient evidence to suggest that it has either markedly higher or lower mortality than other PICUs, merely that the data they have returned is different to that of other PICUs. To resolve why this data is different, PICANet will work with the units to provide a satisfactory explanation, following the plan below

  i) Review the data to investigate whether there are data driven reasons for a PICU lying outside of the control limits (it is known that risk-adjustment tools can be unreliable when a PICU has a particularly high proportion of patients at either end of the bounds of the tool.)

  ii) Review the quality of data supplied by the PICU. The quality of the data is the PICUs’ responsibility. PICANet will provide feedback from PICU data validation visits and central validation procedures. PICUs will be expected to check the quality of individual data items.

  iii) Plot the data quality indicators over time to identify whether the anomaly can be traced to a certain data collection period.

  iv) Plot the mortality ratio over time to identify whether the anomaly can be traced to a certain data collection period.

  v) Plot the observed mortality over time to identify whether the anomaly can be traced to a certain data collection period.

  vi) Plot the expected mortality over time to identify whether the anomaly can be traced to a certain data collection period.

  vii) Investigate the primary diagnoses for admissions to the PICU. If the PICU has a very different diagnostic casemix when compared with other PICUs this may suggest further refinements to the risk-adjustment method are required.

  viii) Produce a brief summary report of the above for the lead clinician and Chief Executive at the PICU concerned together with an invitation to meet in person to review the data with the PICANet team.

  We believe having such a policy in place, clearly outlining our interpretation and proposed actions before publication of such funnel plots is vital to the chances of such information being accepted by staff at the participating units and thus more likely to result in positive actions being taken.

  Yours sincerely,

  Dr Gareth Parry, co-director of PICANet and reader in Health Services Research, University of Sheffield.

  Dr Elizabeth Draper, co-director of PICANet and senior research fellow. Department of Epidemiology and Public Health, University of Leicester

  Prof. Patricia McKinney, co-director of PICANet and professor of paediatric epidemiology. Paediatric Epidemiology Group, University of Leeds

  References

  1. Spiegelhalter DJ. Handling over-dispersion of performance indicators. Qual Saf Health Care 2005; 14: 347-351.

  2. Paediatric Intensive Care Audit Network. National Report 2003–2004. Universities of Leeds, Leicester and Sheffield, May 2005 (ISBN 0 85316 254 9).

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top