UNDERSTANDING THE VARIABLE EFFECTIVENESS OF QUALITY IMPROVEMENT

The highly variable effectiveness of quality improvement initiatives presents many challenges for their pioneers and developers when they try to transfer or spread their ideas and experience to others, and for healthcare organisations endeavouring to take approaches they have seen working elsewhere and implement them at home. It is frustrating for policy makers and others whose attempts to mandate quality improvement or to roll out quality programmes across a health system often have disappointingly variable results. For the biomedically inclined, more accustomed to the relative certainties of randomised controlled trials of healthcare interventions like drugs and other therapies, the results of research evaluations of quality improvement interventions can be perplexing and ambiguous.

There are two main reasons for what might be termed the “consistent inconsistency” of quality improvement (table 1). Firstly, quality improvement is a complex and heterogeneous organisational intervention and the names given to different forms or approaches bestow a misleading impression of uniformity on the realities they describe. For example, different total quality management programmes may share little more than the title and basic terminology of total quality management/continuous quality improvement, and can have quite different structures, training, measurement methods, and levels of resource investment.⁸ The variable impact of such programmes found in evaluations in part results from the fact that the programmes themselves are different, and like is not being compared with like. While this heterogeneity may present problems for evaluators, it would be a mistake to imagine that it can simply be eliminated by prescribing the content of the quality improvement intervention in great detail. To do so would be unrealistic since in real life quality improvement interventions are always evolving and changing during their development and implementation—this is an inherent characteristic of the interventions, not a methodological flaw in their design or evaluation.

Secondly, the organisational context for quality improvement initiatives is a crucial determinant of their effectiveness, and differences in context from one organisation to another mean that, even if a quality improvement intervention could be standardised, its effects would still be likely to vary considerably. For example, some organisations come to quality improvement at a time when they really need it because, if they cannot improve performance, their longer term future is threatened. Others embark on quality improvement with less drastic motivations. Some have strong top leadership support for and commitment to quality improvement while, for others, the ideas are given lip service at best. Some organisational cultures seem to provide fertile territory for quality improvement, perhaps because it fits with their existing values, while others seem positively inimical to the development of quality improvement because it cuts across established organisational, professional, or other beliefs. The heterogeneity of organisational context means that transferring quality improvement techniques, even between apparently similar organisations, is not straightforward.

The frustrating inconsistency with which quality improvement interventions work can be used as an argument for not using them at all. The flawed logic goes something like this. Clinical audit (or total quality management, or whatever other form of quality improvement is being criticised) is poorly researched and appears not to work very well. We should stop using it, at least until more research has been done, so that we know better whether or not it works. There are two problems with this argument. Firstly, even if quality improvement programmes do not work very well, we cannot simply do without them and abandon any organised attempt to improve the quality of health care. To do so would be an abrogation of professional and organisational responsibility. So the decision we have to take is not whether to use a quality improvement intervention, but which one to use, and how to use it best. Secondly, the argument assumes that one day we will be able to show definitively which quality improvement interventions work and which do not, and then we can all simply change over to using the effective approaches. Research to date has not identified a “magic bullet” and it seems unlikely that it will do so in the future. Waiting for the definitive study may be like waiting for Godot.⁹

“Which approach to quality improvement is used in an organisation may matter less than how and by whom it is used.”

Of course, for policy makers, practitioners, and especially those who lead healthcare organisations, there is a very important practical message to be taken from the existing research into the effectiveness of quality improvement in health care. It is that, in the words of the Rolling Stones, “the singer, not the song” is important. Which approach to quality improvement is used in an organisation may matter much less than how and by whom it is used. The between-approach variation may be no greater than (or even less than) the within-approach variation. It may be that what matters is that an organisation chooses its approach to quality improvement carefully and then sticks to it in the long term—what Deming termed “consistency of purpose”.¹⁰ In this light, the unceasing hunt for a better way to do quality improvement seen in many healthcare organisations and health systems as they take up, try out, and then discard one method after another looks rather foolish.

FROM RESEARCHING WHETHER QUALITY IMPROVEMENT WORKS TO WHY IT WORKS

So what kind of research into quality improvement do we need in the future? Research which explores the effectiveness of quality improvement and focuses on its impact is likely to be relatively unhelpful because it will probably just reiterate the findings already referred to above—that the impact of these interventions is very variable and their effectiveness is mixed. The research agenda in quality improvement needs to move away from trying to show whether it works and towards understanding how and why it works—the determinants of effectiveness. The heterogeneity in interventions and organisational context referred to in table 1 should then be easier to analyse and take into account when quality improvement programmes are developed.

There is already a developing literature on the factors which help or hinder the progress of quality improvement, and a growing consensus that characteristics like leadership, direction, organisational culture, training, resources, and practical support are all important.^{3, 11, 12} But for those tasked with implementing quality improvement programmes in the real world, the current state of knowledge is frustratingly vague. For example, they are told that organisational culture is important, but not why it matters, what kind of culture they need or, most of all, how they go about changing the culture of their organisation. We now need more specific and practically focused research into these determinants of effectiveness if the findings are to be of real use for healthcare organisations in developing their quality improvement programmes. For example:

• We understand that organisational leaders have an enormously important part to play in the development of quality improvement, and without their serious, long term commitment it is unlikely that such programmes will make much progress. However, what are the leadership characteristics which we should aim to promote, and how can we use this knowledge to make organisational leaders more self-aware of their own contribution to quality improvement?

• We believe that organisational culture is an important determinant of the effectiveness of quality improvement. For this information to be genuinely useful we need a better understanding of what cultural traits either help or hinder the development of quality improvement, whether there is a particular set of cultural attributes essential for quality improvement which we should aim to promote or develop, and what we can do to promote or develop that cultural shift.

• We know that importing quality improvement technologies from outside health care is a perilous business and that such transfers often fail because of differences in the culture, organisation, or environment between the two settings. However, if we had a better understanding of the kinds of differences which impede transfer and how they can be overcome, we might be more able to adopt quality improvement interventions which have been developed and have worked elsewhere.

None of these questions can be answered easily, and they are certainly not amenable to investigation using the traditional tools of the biomedical health services researcher. Experimental methods like randomised controlled trials will not help. These research questions need to be explored through primarily qualitative methods using techniques such as participant observation and in-depth longitudinal organisational case studies. We also need to develop better theoretical frameworks to explain the results of these studies, from which we can then begin to develop more generalisable and transferable findings.

EVERY QUALITY IMPROVEMENT PROGRAMME IS AN EXPERIMENT

Even though our understanding of the effectiveness of quality improvement in health care should improve as more research is done, these will never be “fire and forget” technologies which can simply be deployed and then left to do their job. They will always be uncertain endeavours, liable to produce very variable results, and susceptible to failure if they are not well monitored. For this reason, we should start to incorporate some element of evaluation into every implementation of quality improvement. In a sense, we should view every quality improvement programme as a kind of experiment, and design it to be “autoevaluative” so that the programme itself produces information about its own effectiveness.

“we should start to incorporate some element of evaluation into every implementation of quality improvement”

This may sound ambitious, but the kind of routine evaluation which might be incorporated into every quality improvement programme does not need to be overly expensive or complex, and does not demand a special set of research skills or instruments. Firstly, every quality improvement programme should gather good activity data—how many people are trained in quality improvement methods, how many projects are undertaken, how many groups or teams are set up in departments and services, and what resources are invested in all these activities. Secondly, each quality improvement programme should incorporate some routine analysis of its outcomes or impact—what changes in practice take place, how well they are sustained, and what this means for patient care. Some impacts may only be enumerated or listed and described in qualitative terms; some may be more quantifiable; and, in some cases, it may be both possible and appropriate to attach a financial value to the impacts. Thirdly, every quality improvement programme should incorporate reflective self-evaluation in which those who have participated set some time aside to consider what they have achieved, what problems they have encountered, and how it could have been done better. All this evaluative effort and data would be a waste of time if it were not used to bring about improvements in the systems for improvement themselves.

CONCLUSIONS

Research evaluations of quality improvement interventions will continue to be an important source of insights into how those interventions work, and how they should be used or implemented in healthcare organisations. However, they are unlikely to provide all the answers that clinicians, managers, quality improvement professionals, and others involved in the practice of healthcare quality improvement want or need.¹³ Practitioners need to become active participants in evaluation, seek to incorporate evaluation into their own practice, and contribute towards setting the future research agenda.