Principles, strengths, weaknesses and opportunities for study designs for improvement interventions
| Class of studies | Principles | Strengths/weaknesses | Opportunities for methodological improvement | Example | |
|---|---|---|---|---|---|
| Quality improvement projects | Project is set up primarily as an improvement effort, to learn what works in a local context. It is typically motivated by a well-defined problem and oriented towards a focused aim. PDSA cycles are often applied, allowing for testing incremental, cyclically implemented changes, which are monitored through statistical process control | Strengths: flexibility in testing changes and adapting interventions; incorporation of knowledge generated by local improvement experiences; ability to interactively move from testing the QII locally to applying it more broadly. Weaknesses: generalisability of findings is not straightforward; lack of structured explanation of mechanisms of change; frequent low quality of reports | Quality improvement projects should incorporate theoretical base and qualitative methods more systematically to allow for predicting and explaining the mechanisms of change involved; more scientific vigour is needed in the application and reporting of PDSA cycles and other methods/techniques applied | An improvement initiative based on social marketing interventions developed to increase access to a psychological therapy service (especially from areas of high deprivation) involved weekly collection of geo-coded referral data and small-scale tests of change57 58 | |
| Effectiveness studies | RCTs | RCTs may be especially suitable whenever interventions are being considered for widespread use based on their face validity and early or preliminary evidence. Differences in outcomes from delivering two or more interventions to similar groups of people or other entities are attributable to differences between the interventions. Control of confounding factors is an explicit aim | Strengths: direct inferences on causality. Weaknesses: neglect the weak boundaries separating context and intervention and the multiple interactions that take place between them; randomisation and blinding may be difficult or even not applicable; risk of contamination between groups | Improvements in the design, conducting, and reporting of RCTs are necessary to limit the high risk of bias observed currently. The awareness of the value of robust design, the need to avoid preconceived judgments about the intervention, and investments in research methods training should be pursued | A study aimed to determine the causal effects of an intervention shown effective in former pre/post studies in reducing central line-associated bloodstream infections in intensive care units.72 |
| Quasiexperimental designs | The intervention is implemented and followed-up over time, ideally with a control. Compared with a RCT, the investigator keeps more control over the intervention, but has less control over confounding factors | Strengths: often more practical to conduct than an RCT. Weaknesses: causality is not inferred directly, and confounding factors’ effects may not be obvious | Whether they have controls or not, quasiexperimental studies will be more powerful if they involve multiple measurements before and after the intervention is applied | A before-after study with concurrent controls sought to evaluate an intervention to reduce inpatient length of stay and considered the effect of the reduction on patient safety80 | |
| Observational (longitudinal) studies | The implementation of the intervention is observed over time | Strengths: cases in practice may be the focus of the study; may be especially useful in the evaluation of sustainability of interventions. Weaknesses: inferences about causality may be challenging | Can be useful when other studies are not possible. They must be longitudinal and, ideally, prospective. The absence of an explicit control in the study design may be compensated by statistical techniques | A study aimed to examine the sustainability of an in-hospital quality improvement intervention in AMI, including the identification of predictors of physician adherence to AMI-recommended medication87 | |
| Systematic reviews | Combining findings/samples from RCTs and quasiexperimental studies on the effectiveness of an intervention allows for more robust and generalisable QII effectiveness results | Strengths: ability to generate more powerful evidence. Weaknesses: uncritical incorporation and interpretation of studies may lead to inadequate conclusions; low use of meta-analyses | The development of systematic reviews on the effectiveness of QIIs has grown. It needs more critical appraisal of the studies included, more meta-analyses, and to deal with complex interventions in diverse contexts | Systematic review with meta-analysis aimed at assessing the effects of QIIs on the management of diabetes88 | |
| Process evaluations | Understanding what an intervention is in practice important, especially when the aim is to attribute effects to it | Strengths: process evaluations make possible an understanding of improvement interventions in practice and the fidelity with which they are implemented | Process evaluations should be embedded in effectiveness studies to capture failures in the QII implementation, and to better understand how QIIs’ components act. They need also to be more oriented towards validating theory-informed strategies | Process evaluation of a cluster randomised controlled trial aimed to examine which components of two hand hygiene improvement strategies were associated with increased nurses’ hand hygiene compliance70 | |
| Qualitative studies | It is not enough to know that an expected change happened or did not. It is important to understand why and how | Strengths: ability to capture, considering different points of view, the extent that interventions are implemented, and to explain mechanisms of change involved, based on theories | Qualitative studies should be included in quality improvement projects and QIIs’ quantitative evaluative studies for better understanding of outcomes and explanation of mechanisms of change involved. | Study that developed an ex post theory of the Michigan Intensive Care Unit project to explain how it achieved its effects101 | |
| Economic evaluations | It is important to know that an intervention is effective and also that the investment required is justifiable | Strengths: adds information about how justifiable the QII is in face of the investment required | In the literature, studies dedicated to economic evaluations of healthcare QIIs are still lacking, and there is recognition that there should be more of them in the field | Cost-effectiveness analysis of a multifaceted intervention to improve the quality of care of children in district hospitals in Kenya104 | |
AMI, acute myocardial infarction; PDSA, Plan-Do-Study-Act; QII, quality improvement intervention; RCTs, randomised controlled trials.