Table 1

Principles, strengths, weaknesses and opportunities for study designs for improvement interventions

Class of studiesPrinciplesStrengths/weaknessesOpportunities for methodological improvementExample
Quality improvement projectsProject is set up primarily as an improvement effort, to learn what works in a local context. It is typically motivated by a well-defined problem and oriented towards a focused aim. PDSA cycles are often applied, allowing for testing incremental, cyclically implemented changes, which are monitored through statistical process controlStrengths: flexibility in testing changes and adapting interventions; incorporation of knowledge generated by local improvement experiences; ability to interactively move from testing the QII locally to applying it more broadly. Weaknesses: generalisability of findings is not straightforward; lack of structured explanation of mechanisms of change; frequent low quality of reportsQuality improvement projects should incorporate theoretical base and qualitative methods more systematically to allow for predicting and explaining the mechanisms of change involved; more scientific vigour is needed in the application and reporting of PDSA cycles and other methods/techniques appliedAn improvement initiative based on social marketing interventions developed to increase access to a psychological therapy service (especially from areas of high deprivation) involved weekly collection of geo-coded referral data and small-scale tests of change57 58
Effectiveness studiesRCTsRCTs may be especially suitable whenever interventions are being considered for widespread use based on their face validity and early or preliminary evidence. Differences in outcomes from delivering two or more interventions to similar groups of people or other entities are attributable to differences between the interventions. Control of confounding factors is an explicit aimStrengths: direct inferences on causality. Weaknesses: neglect the weak boundaries separating context and intervention and the multiple interactions that take place between them; randomisation and blinding may be difficult or even not applicable; risk of contamination between groupsImprovements in the design, conducting, and reporting of RCTs are necessary to limit the high risk of bias observed currently. The awareness of the value of robust design, the need to avoid preconceived judgments about the intervention, and investments in research methods training should be pursuedA study aimed to determine the causal effects of an intervention shown effective in former pre/post studies in reducing central line-associated bloodstream infections in intensive care units.72
Quasiexperimental designsThe intervention is implemented and followed-up over time, ideally with a control. Compared with a RCT, the investigator keeps more control over the intervention, but has less control over confounding factorsStrengths: often more practical to conduct than an RCT. Weaknesses: causality is not inferred directly, and confounding factors’ effects may not be obviousWhether they have controls or not, quasiexperimental studies will be more powerful if they involve multiple measurements before and after the intervention is appliedA before-after study with concurrent controls sought to evaluate an intervention to reduce inpatient length of stay and considered the effect of the reduction on patient safety80
Observational (longitudinal) studiesThe implementation of the intervention is observed over timeStrengths: cases in practice may be the focus of the study; may be especially useful in the evaluation of sustainability of interventions. Weaknesses: inferences about causality may be challengingCan be useful when other studies are not possible. They must be longitudinal and, ideally, prospective. The absence of an explicit control in the study design may be compensated by statistical techniquesA study aimed to examine the sustainability of an in-hospital quality improvement intervention in AMI, including the identification of predictors of physician adherence to AMI-recommended medication87
Systematic reviewsCombining findings/samples from RCTs and quasiexperimental studies on the effectiveness of an intervention allows for more robust and generalisable QII effectiveness resultsStrengths: ability to generate more powerful evidence. Weaknesses: uncritical incorporation and interpretation of studies may lead to inadequate conclusions; low use of meta-analysesThe development of systematic reviews on the effectiveness of QIIs has grown. It needs more critical appraisal of the studies included, more meta-analyses, and to deal with complex interventions in diverse contextsSystematic review with meta-analysis aimed at assessing the effects of QIIs on the management of diabetes88
Process evaluationsUnderstanding what an intervention is in practice important, especially when the aim is to attribute effects to itStrengths: process evaluations make possible an understanding of improvement interventions in practice and the fidelity with which they are implementedProcess evaluations should be embedded in effectiveness studies to capture failures in the QII implementation, and to better understand how QIIs’ components act. They need also to be more oriented towards validating theory-informed strategiesProcess evaluation of a cluster randomised controlled trial aimed to examine which components of two hand hygiene improvement strategies were associated with increased nurses’ hand hygiene compliance70
Qualitative studiesIt is not enough to know that an expected change happened or did not. It is important to understand why and howStrengths: ability to capture, considering different points of view, the extent that interventions are implemented, and to explain mechanisms of change involved, based on theoriesQualitative studies should be included in quality improvement projects and QIIs’ quantitative evaluative studies for better understanding of outcomes and explanation of mechanisms of change involved.Study that developed an ex post theory of the Michigan Intensive Care Unit project to explain how it achieved its effects101
Economic evaluationsIt is important to know that an intervention is effective and also that the investment required is justifiableStrengths: adds information about how justifiable the QII is in face of the investment requiredIn the literature, studies dedicated to economic evaluations of healthcare QIIs are still lacking, and there is recognition that there should be more of them in the fieldCost-effectiveness analysis of a multifaceted intervention to improve the quality of care of children in district hospitals in Kenya104
  • AMI, acute myocardial infarction; PDSA, Plan-Do-Study-Act; QII, quality improvement intervention; RCTs, randomised controlled trials.