• Quality improvement
• Quality measurement
• Quality improvement methodologies

Debates over the degree to which standards of evidence and methods from traditional clinical research can or should apply to quality improvement (QI) have recurred over the past 10 years.1–4 When, if ever, do we need a randomised controlled trial (RCT) demonstrating benefit to decide that an intervention has worked? Can we recommend QI interventions for widespread adoption even without supportive RCTs? On one side of the debate, some have argued that QI and the RCT are like oil and water—never the twain shall mix. Certainly, many have argued, we should not presume that RCTs represent the gold standard for evidence in QI.

On the face of it, the report by Mate et al5 supports this oil and water view of RCTs and QI interventions. The authors report their struggles conducting a pragmatic, multisite RCT of a complex intervention to reduce perinatal transmission of HIV in KwaZulu-Natal Province, South Africa. The intervention included socioadaptive strategies,6 ,7 such as engaging local health system leaders, securing a commitment to the aims of the project, and providing participating health centres with the tools to perform data-driven improvement cycles. It also promoted specific best practices for key steps in the prevention of perinatal transmission of HIV (eg, increasing the proportion of women receiving early antenatal care that includes HIV counselling and testing, increasing the proportion of mothers with low CD4 counts who receive treatment, and so on). The authors initially planned to evaluate this complex intervention using an equally complex study design—a step-wedge, cluster RCT involving 48 clusters of clinics (for a total of 222 individual clinics) in three waves of intervention and control sites; hence, the ‘step-wedge’ label.

It will come as no surprise to most readers that this double dose of complexity—from the intervention itself and the trial design—overwhelmed all parties involved. Citing frustration among participants, tensions between implementation ideas at different levels of the health system and other very legitimate sounding factors, investigators abandoned the planned RCT. They opted instead for a mixed-methods evaluation that included qualitative analysis and time series, with statistical process control charts and interrupted time series methods. (The initially intended intervention itself also underwent simplification.)

The need to abandon this trial seems particularly supportive of the oil and water view of RCTs and QI, since the planned trial contained two pragmatic characteristics generally believed to facilitate the evaluation of QI interventions. Randomisation occurred at the level of clinics (hence, ‘cluster RCT’) rather than individual patients. And, the intervention occurred in waves, accommodating the reality of organisations having differing timelines for implementing new interventions.8 This step-wedge design has found successful applications in other QI settings, including an evaluation of rapid response teams9 and the central line bundle to reduce catheter-associated bacteraemia.10

So, what does this case illustrate: are some interventions simply too complex for RCTs? Do low-resource settings make RCTs impractical? Or, perhaps the problem is context.11 ,12 Maybe important variations in local context prohibit evaluation using a randomised design.

The first two questions are easy to address, since multiple examples of complex RCTs in low-resource settings exist. In one example, the intervention's complexity rivals that seen in any QI study: investigators randomised poor, rural villages to an intervention involving microfinance loans, education about gender equity issues and an HIV training curriculum.13 The trial showed mixed effects, with no reduction in HIV incidence but a 55% reduction in intimate partner violence. Thus, the cluster randomised design did not undermine the intervention even in this explicitly low-resource setting (in fact, in South Africa, like the intervention reported by Mate et al5). Another recent example involved cluster randomisation of 98 healthcare zones in Ghana to evaluate the impact of a complex intervention involving training of community-based surveillance volunteers.14 These volunteers identified pregnant women in their community in order to make two home visits during pregnancy and three in the first week postpartum to promote key newborn care practices, assess babies for concerning signs and make referrals to formal healthcare services as necessary. While the trial achieved only a small, non-significant reduction in neonatal mortality, it achieved multiple significant improvements in the uptake of key practices.

Concerns about context as a barrier to RCTs are also relatively straightforward to address. Despite much emphasis on context as a unique consideration in QI, the challenges of accounting for important variations in trial participants exist routinely confront clinical trialists. As Bond et al15 replied to a critique of the Medical Research Council framework16 for the evaluation of complex interventions: Imagine an intervention whose effects vary within and between individuals and depend on subtle interactions between deliverers and recipients, and in which exposure is uncertain. Given this complexity, who would contemplate conducting a randomised controlled trial? In fact, all these issues must be dealt with in drug or therapeutic trials, as well as in more obviously complex interventions.15

The situation with contextual factors in QI thus resembles that seen for important patient characteristics in conventional clinical trials. Variations in clinical characteristics such as comorbid conditions, different genetic predispositions and socioeconomic factors may greatly impact the effects of a medication. One hopes that randomisation balances out the factors we do not yet know about and we can always choose to stratify the randomisation according to key factors we do know about. In fact, conventional clinical trials must also sometimes deal with what amount to contextual factors, although clinical trialists tend to call them ‘centre effects’. Some centres have greater expertise in delivering the treatment of interest (eg, a new surgical procedure), have relevant infrastructure such as relevant consulting services or attract patient populations with characteristics not easily taken into account with simple risk adjustment.

While clinical trialists do not account for centre effects as often as they should,17 the point remains that conventional clinical trials face the same types of variations—across patients and centres—that we in QI ascribe to context.11 ,12 ,18 ,19 If an intervention might plausibly have different effects in settings with certain leadership styles, levels of patient safety culture, sophistication of clinical informatics infrastructure or any of the numerous other possible elements of context,11 investigators can explicitly balance the study groups for some factors (perhaps the ones they can best measure) and let randomisation take care of the rest.

That said, when we really believe that contextual factors play important roles in an intervention's effectiveness, it may be premature to conduct an RCT. Better to first identify the key contextual factors, so that modifiable ones can be addressed as part of the intervention and unmodifiable ones can be avoided through the trial's exclusion criteria. (‘Organisations that did not have such-and-such features in place were not invited to participate.’)

Once we establish that a given intervention requires a certain safety culture, concrete commitments by leadership, frontline engagement or various other contextual factors, investigators can address these factors in the implementation plan. If we forgo characterising these factors and rush to disseminating the intervention widely (as has often occurred), then we have to expect that the intervention will not work in many places. An RCT would undoubtedly show this. The problem, then, lies not with the RCT, but with the developmental stage of the intervention—dissemination occurred too soon.