Elsevier

Academic Pediatrics

Volume 13, Issue 6, Supplement, November–December 2013, Pages S23-S30
Academic Pediatrics

Methods in QI Research
Recommendations for Evaluation of Health Care Improvement Initiatives

https://doi.org/10.1016/j.acap.2013.04.007Get rights and content

Abstract

Intensive efforts are underway across the world to improve the quality of health care. It is important to use evaluation methods to identify improvement efforts that work well before they are replicated across a broad range of contexts. Evaluation methods need to provide an understanding of why an improvement initiative has or has not worked and how it can be improved in the future. However, improvement initiatives are complex, and evaluation is not always well aligned with the intent and maturity of the intervention, thus limiting the applicability of the results. We describe how initiatives can be grouped into 1 of 3 improvement phases—innovation, testing, and scale-up and spread—depending on the degree of belief in the associated interventions. We describe how many evaluation approaches often lead to a finding of no effect, consistent with what has been termed Rossi’s Iron Law of Evaluation. Alternatively, we recommend that the guiding question of evaluation in health care improvement be, “How and in what contexts does a new model work or can be amended to work?” To answer this, we argue for the adoption of formative, theory-driven evaluation. Specifically, evaluations start by identifying a program theory that comprises execution and content theories. These theories should be revised as the initiative develops by applying a rapid-cycle evaluation approach, in which evaluation findings are fed back to the initiative leaders on a regular basis. We describe such evaluation strategies, accounting for the phase of improvement as well as the context and setting in which the improvement concept is being deployed. Finally, we challenge the improvement and evaluation communities to come together to refine the specific methods required so as to avoid the trap of Rossi’s Iron Law.

Section snippets

Why New Improvement Ideas Fail So Often

Donald Campbell was a leading figure in program evaluation in the United States.10 Since the 1960s, Campbell’s championing of evaluation in public policy led to the wide establishment of formal evaluation methods. However, in health care, evaluation often entailed only an impact assessment of the overall intervention, with little focus on the processes involved or the context of the participants.11 This narrow focus led to a perception that interventions that work in initial studies lose their

Suggested Approaches to Evaluation of Health Care Improvement

As described above, the guiding evaluative question for health care improvement is, “How and in what contexts does the new model work or can be amended to work?” To answer this question, we propose using theory-driven formative evaluations. The specific approach will be informed by 2 primary considerations. The first is the degree of belief in the new conceptual model and whether it is at the innovation, testing, or scale-up and spread phase. If the improvement work is part of a

Innovation

The innovation phase aims to generate a new model of care or content theory (Table 2). Evaluation here should describe the new content theory, including the underlying concepts that inform it and the context in which the model was developed. In addition, an evaluation should estimate the measured improvement achieved as a result of the new model in this context and indicate the degree of belief that the model is likely to apply in other settings. For the rapid response team example, this

Conclusions

We recommend that the guiding question for those planning to undertake evaluation of health care improvement be, “How and in what contexts does a new model work or can be amended to work?” Evaluators seeking to answer this question will need to understand whether the improvement work is at the innovation, testing, or scale-up and spread phase. We recommend improvement initiatives clarify a program theory that comprises execution and content theories, illustrated by a logic model. Evaluators may

References (39)

  • E. Chain et al.

    Penicillin as a chemotherapeutic agent

    Lancet

    (1940)
  • A. Millar et al.

    Logic models: a systems tool for performance management

    Eval Progr Planning

    (2001)
  • M.C. Conry et al.

    A 10-year (2000–2010) systematic review of interventions to improve quality of care in hospitals

    BMC Health Serv Res

    (2012)
  • M.H. Versteeg et al.

    Factors associated with the impact of quality improvement collaboratives in mental healthcare: an exploratory study

    Implement Sci

    (2012)
  • B.S. Mittman

    Creating the evidence base for quality improvement collaboratives

    Ann Intern Med

    (2004)
  • A. Fleming

    On the antibacterial action of cultures of a penicillium, with special reference to their use in the isolation of B influenzae

    Br J Exp Pathol

    (1929)
  • H.W. Florey

    Penicillin: a survey

    Br Med J

    (1944)
  • D.J. Spiegelhalter et al.

    Methods in health service research. An introduction to Bayesian methods in health technology assessment

    BMJ

    (1999)
  • C.I. Lewis

    Mind and the World Order: Outline of a Theory of Knowledge 1929

    (1991)
  • C. Christiansen

    The ongoing process of building a theory of disruption

    J Prod Innov Manag

    (2006)
  • D.T. Campbell et al.

    Experimental and Quasi-Experimental Designs for Research. Chicago

    (1963)
  • B.E. Landon et al.

    Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study

    Ann Intern Med

    (2004)
  • P.H. Rossi

    The iron law of evaluation and other metallic rules

    Res Social Problems Public Policy

    (1987)
  • R. Pawson et al.

    Realistic Evaluation

    (1997)
  • J.P.A. Ioannidis

    Contradicted and initially stronger effects in highly cited clinical research

    JAMA

    (2005)
  • D.S. Burstein et al.

    Care models and associated outcomes in congenital heart surgery

    Pediatrics

    (2011)
  • G.J. Parry

    Replicating cardiac intensive care units: the importance of structure, process, and outcomes

    Pediatrics

    (2011)
  • D.R. Goldhill et al.

    A physiologically-based early warning score for ward patients: the association between score and outcome

    Anaesthesia

    (2005)
  • M. Dixon-Woods et al.

    Explaining Michigan: developing an ex post theory of a quality improvement program

    Milbank Q

    (2011)
  • Cited by (0)

    The views expressed in this report are those of the authors and do not necessarily represent those of the US Department of Health and Human Services, the Agency for Healthcare Research and Quality or the American Board of Pediatrics Foundation.

    The authors declare that they have no conflict of interest.

    Publication of this article was supported by the Agency for Healthcare Research and Quality and the American Board of Pediatrics Foundation.

    View full text