Article Text

PDF

589
PREDICTING OUTCOME MEASURE PERFORMANCE THROUGH MONTE CARLO SIMULATION
  1. Richard Scoville1,
  2. Manwai Ng2,
  3. Gareth Parry1,
  4. Abiola Faniyan2,
  5. Adam Richman3,
  6. Arthur Evans3
  1. 1Institute for Healthcare Improvement, United States
  2. 2Boston Children's Hospital, United States
  3. 3Dentaquest Institute, United States

Abstract

Background Improvement initiatives rely on measures to assess their impact. When outcomes are expressed as percentages, unanticipated changes to the denominator due to complexities and interactions among processes, population and contextual factors can result in measures that fail to register real improvements. For example a pediatric oral health collaborative focused on the incidence of new dental caries as a key measure. Despite uptake of process changes, a prevalence measure failed to register associated improvement in appointment rates (fig 1). Data exploration suggested the measure was confounded with exposure to process changes and recruitment of new patients.

Objectives To assess whether applying Monte-Carlo simulation can help project designers test the performance of outcome measures under an array of anticipated conditions.

Methods Data were obtained via a database application installed at each practice which calculated monthly collaborative measures using standard definitions, and produced a file that was uploaded to the collaborative team.

A template with a Monte-Carlo simulation platform was constructed to model a typical practice panel. Random variables modeled patient appointment intervals based on hypothetical subpopulations with varying appointment compliance.

Results The model tested two alternative definitions of caries incidence, and suggested that a modified definition would more accurately reflect improvements in caries incidence (fig 2).

Based on predictions, the application was modified to incorporate two alternative operational definitions, which were then applied retrospectively by selected practices.

Data using the revised definitions more closely conformed to simulated predictions (fig 3).

Conclusions Monte Carlo simulation enables advance testing of alternative operational definitions to optimize measure performance.

Figure 2

Simulation results: alternative definitions.

Figure 3

Results: individual team data.

Statistics from Altmetric.com

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.