Article Text
PDF
Abstract
Background Improvement initiatives rely on measures to assess their impact. When outcomes are expressed as percentages, unanticipated changes to the denominator due to complexities and interactions among processes, population and contextual factors can result in measures that fail to register real improvements. For example a pediatric oral health collaborative focused on the incidence of new dental caries as a key measure. Despite uptake of process changes, a prevalence measure failed to register associated improvement in appointment rates (fig 1). Data exploration suggested the measure was confounded with exposure to process changes and recruitment of new patients.
Objectives To assess whether applying Monte-Carlo simulation can help project designers test the performance of outcome measures under an array of anticipated conditions.
Methods Data were obtained via a database application installed at each practice which calculated monthly collaborative measures using standard definitions, and produced a file that was uploaded to the collaborative team.
A template with a Monte-Carlo simulation platform was constructed to model a typical practice panel. Random variables modeled patient appointment intervals based on hypothetical subpopulations with varying appointment compliance.
Results The model tested two alternative definitions of caries incidence, and suggested that a modified definition would more accurately reflect improvements in caries incidence (fig 2).
Based on predictions, the application was modified to incorporate two alternative operational definitions, which were then applied retrospectively by selected practices.
Data using the revised definitions more closely conformed to simulated predictions (fig 3).
