Methods in QI ResearchRecommendations for Evaluation of Health Care Improvement Initiatives
Section snippets
Why New Improvement Ideas Fail So Often
Donald Campbell was a leading figure in program evaluation in the United States.10 Since the 1960s, Campbell’s championing of evaluation in public policy led to the wide establishment of formal evaluation methods. However, in health care, evaluation often entailed only an impact assessment of the overall intervention, with little focus on the processes involved or the context of the participants.11 This narrow focus led to a perception that interventions that work in initial studies lose their
Suggested Approaches to Evaluation of Health Care Improvement
As described above, the guiding evaluative question for health care improvement is, “How and in what contexts does the new model work or can be amended to work?” To answer this question, we propose using theory-driven formative evaluations. The specific approach will be informed by 2 primary considerations. The first is the degree of belief in the new conceptual model and whether it is at the innovation, testing, or scale-up and spread phase. If the improvement work is part of a
Innovation
The innovation phase aims to generate a new model of care or content theory (Table 2). Evaluation here should describe the new content theory, including the underlying concepts that inform it and the context in which the model was developed. In addition, an evaluation should estimate the measured improvement achieved as a result of the new model in this context and indicate the degree of belief that the model is likely to apply in other settings. For the rapid response team example, this
Conclusions
We recommend that the guiding question for those planning to undertake evaluation of health care improvement be, “How and in what contexts does a new model work or can be amended to work?” Evaluators seeking to answer this question will need to understand whether the improvement work is at the innovation, testing, or scale-up and spread phase. We recommend improvement initiatives clarify a program theory that comprises execution and content theories, illustrated by a logic model. Evaluators may
References (39)
- et al.
Penicillin as a chemotherapeutic agent
Lancet
(1940) - et al.
Logic models: a systems tool for performance management
Eval Progr Planning
(2001) - et al.
A 10-year (2000–2010) systematic review of interventions to improve quality of care in hospitals
BMC Health Serv Res
(2012) - et al.
Factors associated with the impact of quality improvement collaboratives in mental healthcare: an exploratory study
Implement Sci
(2012) Creating the evidence base for quality improvement collaboratives
Ann Intern Med
(2004)On the antibacterial action of cultures of a penicillium, with special reference to their use in the isolation of B influenzae
Br J Exp Pathol
(1929)Penicillin: a survey
Br Med J
(1944)- et al.
Methods in health service research. An introduction to Bayesian methods in health technology assessment
BMJ
(1999) Mind and the World Order: Outline of a Theory of Knowledge 1929
(1991)The ongoing process of building a theory of disruption
J Prod Innov Manag
(2006)
Experimental and Quasi-Experimental Designs for Research. Chicago
Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study
Ann Intern Med
The iron law of evaluation and other metallic rules
Res Social Problems Public Policy
Realistic Evaluation
Contradicted and initially stronger effects in highly cited clinical research
JAMA
Care models and associated outcomes in congenital heart surgery
Pediatrics
Replicating cardiac intensive care units: the importance of structure, process, and outcomes
Pediatrics
A physiologically-based early warning score for ward patients: the association between score and outcome
Anaesthesia
Explaining Michigan: developing an ex post theory of a quality improvement program
Milbank Q
Cited by (0)
The views expressed in this report are those of the authors and do not necessarily represent those of the US Department of Health and Human Services, the Agency for Healthcare Research and Quality or the American Board of Pediatrics Foundation.
The authors declare that they have no conflict of interest.
Publication of this article was supported by the Agency for Healthcare Research and Quality and the American Board of Pediatrics Foundation.