Should we use large scale healthcare interventions without clear evidence that benefits outweigh costs and harms? Yes

Imperfect evidence

In the NHS the National Institute for Health and Clinical Excellence (NICE) gives guidance on questions such as “Should the NHS make available gemcitabine for pancreatic cancer?” or “Is there adequate information about the safety and efficacy of laparoscopic repair of abdominal aortic aneurysm?”1 2 using a rigorous approach to integrate the best available evidence on efficacy and where available, cost effectiveness. In these, as in countless other examples, NICE offered guidance about how to proceed in circumstances where benefits are not yet clear. We also know that evidence may change.3

In seeking to clarify the role of primary, secondary, and tertiary care for familial breast cancer,4 a more complex question, NICE made 105 recommendations. For none of them was it able to draw on evidence from randomised controlled trials or even a single controlled or quasi-experimental study. Sixteen of the recommendations rested on non-experimental descriptive studies and the remainder on expert opinion. Should NICE have proposed that no service be offered until there was better evidence (leaving practitioners to face the public without guidance) or make the recommendations that it did?

Interventions designed to improve the quality or safety of care are more complex still. They involve individuals, organisations, and systems. Examples might include steps taken by five primary care trusts to shift care closer to patients’ homes5 or the interventions instituted by an NHS trust to reduce the hospital standardised mortality ratio.6 They present formidable challenges to the experimental evaluator. For example, it is rarely possible to contemplate blinding of investigator or participant. Randomisation, where it can occur, often needs to be at cluster not individual level.

Mechanisms of change

Moreover, learning from other sectors teaches that successful improvement programmes will build in real time feedback on intermediate outcomes and will allow for the adjustment of the intervention as the implementation takes place.7 At the heart of many quality and safety interventions is the need to stimulate and motivate change, and continuous feedback is central to this.

More fundamentally, the nature of the experimental approach to evaluating such programmes is problematic. Experimental evaluation is based on an approach to the establishment of causation that can be described as successionist.8 The changes in outcome that occur in the experimental and control group are all that matters and are observed externally. The context in which these changes occur is relevant only in so far as they can confirm the adequacy of the randomisation process.

This failure to take account of context leads to at least two problems. Where evaluations lead to the conclusion that an intervention works, it is not known why it has worked. They are also prone to lead to the conclusion that an intervention does not work when another perspective is that the impact is place and context specific.9

An alternative approach to studying causation is the generative approach, which requires an understanding of the mechanisms causing change, takes account of the internal factors as well as the external, and requires a deep appreciation of the context. Social scientists have developed models of evaluation based on the generative approach, often referred to as scientific realism.10 Such evaluations are rigorous and exacting, using a combination of quantitative and qualitative approaches with an emphasis on capturing evidence about the context in which interventions take place.

For example, researchers from University College London and the RAND Corporation studied nine healthcare institutions in the US and Europe with a strong reputation for quality improvement.11 Using interviews, narrative accounts, observation, and document analysis the team created a novel framework to describe the organisational context that distinguished these organisations and their most successful teams. The use of such approaches in the study of programmes to improve safety and quality deserves exploration.

Meanwhile, those embarking on an improvement initiative should be clear about how they will tell if a change is an improvement, should monitor the effect of their interventions (including costs) using sound methods for measuring quality, and should capture information about the context in which their intervention is taking place. This is analogous to the approach NICE recommends for new interventions when more evidence is needed.