RT Journal Article
SR Electronic
T1 PREDICTING OUTCOME MEASURE PERFORMANCE THROUGH MONTE CARLO SIMULATION
JF BMJ Quality & Safety
JO BMJ Qual Saf
FD BMJ Publishing Group Ltd
SP 733
OP 734
DO 10.1136/bmjqs-2015-IHIabstracts.19
VO 24
IS 11
A1 Scoville, Richard
A1 Ng, Manwai
A1 Parry, Gareth
A1 Faniyan, Abiola
A1 Richman, Adam
A1 Evans, Arthur
YR 2015
UL http://qualitysafety.bmj.com/content/24/11/733.1.abstract
AB Background Improvement initiatives rely on measures to assess their impact. When outcomes are expressed as percentages, unanticipated changes to the denominator due to complexities and interactions among processes, population and contextual factors can result in measures that fail to register real improvements. For example a pediatric oral health collaborative focused on the incidence of new dental caries as a key measure. Despite uptake of process changes, a prevalence measure failed to register associated improvement in appointment rates (fig 1). Data exploration suggested the measure was confounded with exposure to process changes and recruitment of new patients.Objectives To assess whether applying Monte-Carlo simulation can help project designers test the performance of outcome measures under an array of anticipated conditions.Methods Data were obtained via a database application installed at each practice which calculated monthly collaborative measures using standard definitions, and produced a file that was uploaded to the collaborative team.A template with a Monte-Carlo simulation platform was constructed to model a typical practice panel. Random variables modeled patient appointment intervals based on hypothetical subpopulations with varying appointment compliance.Results The model tested two alternative definitions of caries incidence, and suggested that a modified definition would more accurately reflect improvements in caries incidence (fig 2).Based on predictions, the application was modified to incorporate two alternative operational definitions, which were then applied retrospectively by selected practices.Data using the revised definitions more closely conformed to simulated predictions (fig 3).Conclusions Monte Carlo simulation enables advance testing of alternative operational definitions to optimize measure performance.⇓⇓⇓Figure 1 Original measure.Figure 2 Simulation results: alternative definitions.Figure 3 Results: individual team data.