Article Text

Download PDFPDF
Handling missing values in the analysis of between-hospital differences in ordinal and dichotomous outcomes: a simulation study
  1. Reinier C A van Linschoten1,2,3,
  2. Marzyeh Amini1,
  3. Nikki van Leeuwen1,
  4. Frank Eijkenaar4,
  5. Sanne J den Hartog1,5,6,
  6. Paul J Nederkoorn7,
  7. Jeannette Hofmeijer8,9,
  8. Bart J Emmer10,
  9. Alida A Postma11,12,
  10. Wim van Zwam11,
  11. Bob Roozenbeek5,
  12. Diederik Dippel5,
  13. Hester F Lingsma1
  14. MR CLEAN Registry Investigators
  1. 1 Public Health, Erasmus MC, Rotterdam, Netherlands
  2. 2 Gastroenterology and Hepatology, Franciscus Gasthuis en Vlietland, Rotterdam, Netherlands
  3. 3 Department of Gastroenterology & Hepatology, Erasmus MC, Rotterdam, Netherlands
  4. 4 Erasmus School of Health Policy and Management, Erasmus Universiteit Rotterdam, Rotterdam, Netherlands
  5. 5 Neurology, Erasmus MC, Rotterdam, Netherlands
  6. 6 Department of Radiology and Nuclear Medicine, Erasmus MC, Rotterdam, Netherlands
  7. 7 Neurology, Amsterdam UMC, Amsterdam, Netherlands
  8. 8 Neurology, Rijnstate Hospital, Arnhem, Netherlands
  9. 9 Clinical Neurophysiology, University of Twente, Enschede, Netherlands
  10. 10 Radiology and Nuclear Medicine, Amsterdam UMC, Amsterdam, Netherlands
  11. 11 Radiology and Nuclear Medicine, MUMC+, Maastricht, Netherlands
  12. 12 School for Mental Health and Sciences, Maastricht University, Maastricht, Netherlands
  1. Correspondence to Dr Reinier C A van Linschoten, Public Health, Erasmus MC, Rotterdam 3015, Netherlands; r.vanlinschoten{at}erasmusmc.nl

Abstract

Missing data are frequently encountered in registries that are used to compare performance across hospitals. The most appropriate method for handling missing data when analysing differences in outcomes between hospitals with a generalised linear mixed model is unclear. We aimed to compare methods for handling missing data when comparing hospitals on ordinal and dichotomous outcomes. We performed a simulation study using data from the Multicentre Randomised Controlled Trial of Endovascular Treatment for Acute Ischaemic Stroke in the Netherlands (MR CLEAN) Registry, a prospective cohort study in 17 hospitals performing endovascular therapy for ischaemic stroke in the Netherlands. The investigated methods for handling missing data, both case-mix adjustment variables and outcomes, were complete case analysis, single imputation, multiple imputation, single imputation with deletion of imputed outcomes and multiple imputation with deletion of imputed outcomes. Data were generated as missing completely at random (MCAR), missing at random and missing not at random (MNAR) in three scenarios: (1) 10% missing data in case-mix and outcome; (2) 40% missing data in case-mix and outcome; and (3) 40% missing data in case-mix and outcome with varying degree of missing data among hospitals. Bias and reliability of the methods were compared on the mean squared error (MSE, a summary measure combining bias and reliability) relative to the hospital effect estimates from the complete reference data set. For both the ordinal outcome (ie, the modified Rankin Scale) and a common dichotomised version thereof, all methods of handling missing data were biased, likely due to shrinkage of the random effects. The MSE of all methods was on average lowest under MCAR and with fewer missing data, and highest with more missing data and under MNAR. The ‘multiple imputation, then deletion’ method had the lowest MSE for both outcomes under all simulated patterns of missing data. Thus, when estimating hospital effects on ordinal and dichotomous outcomes in the presence of missing data, the least biased and most reliable method to handle these missing data is ‘multiple imputation, then deletion’.

  • Simulation
  • Quality improvement methodologies
  • Quality improvement
  • Performance measures
  • Healthcare quality improvement

Data availability statement

Data may be obtained from a third party and are not publicly available. The data of the study cannot be made available to other researchers, as Dutch law prohibits data sharing when no patient approval was obtained for sharing coded data. However, syntax or output files of the statistical analyses may be made available for academic purposes upon reasonable request.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Data availability statement

Data may be obtained from a third party and are not publicly available. The data of the study cannot be made available to other researchers, as Dutch law prohibits data sharing when no patient approval was obtained for sharing coded data. However, syntax or output files of the statistical analyses may be made available for academic purposes upon reasonable request.

View Full Text

Footnotes

  • Collaborators MR CLEAN Registry Investigators Please see Supplementary File.

  • Contributors RCAvL, MA, NvL and FE designed the study. RCAvL and MA undertook analyses and interpretation of study findings and wrote the first draft of the manuscript. NvL, FE and HFL contributed to the interpretation of study findings, reviews, and revision of the manuscript. All the authors critically reviewed the various versions of the full paper and approved the final manuscript for submission. HFL is the guarantor of the study.

  • Funding This study was funded by Amsterdam University Medical Centers (no award/grant number), Erasmus Medisch Centrum (no award/grant number), TWIN Foundation (no award/grant number), Maastricht Universitair Medisch Centrum (no award/grant number).

  • Competing interests WvZ has received consulting fees from Codman and Stryker. DD has received research grants from Stryker and Bracco Imaging.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.