Mix of methods

Within the recollection of older researchers and practitioners in medical care, society celebrated medicine as an ‘art,’ and most of its truths were distillations of experience from seasoned practitioners. In the past half century, the standard of proof in medical care has moved strongly towards a scientific model, seeking generalisable truths about the workings of the body and of the interventions proposed. This has led to a useful reification of the randomised controlled trial, with its ability to mask many effects of unmeasured variables, secular trends and random events; and the factual basis of medical practice is much stronger than accretion of experience by practitioners could have achieved. This work has been generally well supported—by government, academic institutions and businesses—and many people make their living in medical research governed by careful statistical proof.

However, the work has had its challenges. Some issues are just too expensive to study rigorously, some are too difficult to measure reliably, and some require so many exclusions before sampling that the scope of generalisability is uncertain or limited. Furthermore, thoroughly inappropriate use of human subjects in an array of medical research projects led to extensive review and regulation aimed to prevent abuse.

Within the last 20 years, a complementary set of process improvement methods has taken root, borrowed from management of manufacturing. Continuous quality improvement uses goal-setting, monitoring, standardisation, testing and strategies for sustainability, and spread in order to tackle potential dysfunctions in service delivery, mostly within one organisation or setting. The methods have often been successful in rapidly and sustainably achieving better outcomes and now form the backbone of major portions of professional and organisational credentialling. But the relationships between the methods of scientific research and of quality improvement have not yet been worked out well. For example, it not clear whether funding appropriated for ‘research’ could be used in ‘quality improvement,’ whether improvement projects deserve to be published or which ‘quality improvement’ projects must undergo review under the regulations governing ‘research on human subjects’. Perhaps most importantly, how to tackle important national priorities using a prudent mix of methods has received scant attention.

Furthermore, as the recent US legislation1 illustrates, sometimes major changes (which may yield improvements) arise from adoption of broad policies, such as coverage of new categories of services by public insurance, coverage of new beneficiaries or imposing performance requirements for practitioners or insurance companies. Those policies, of course, can also affect the funding and other resources available for research or quality improvement, and can dictate the resolution of some points of uncertainty as to methods. But again, how best to use public policy as part of the mix of methods to achieve improvement is often left mostly to power politics, informed only by scant information as to the wise course to follow and constrained to be ‘permanent’ because of the overwhelming practical difficulties in deliberately designing policy interventions as tests that will be monitored and adjusted on the basis of experience. So, when to use policy to achieve reforms currently has no better evidence-based methodology at its disposal (at least in the USA) than deciding how to draw insights from sources like conventional medical research and quality improvement. Other modes of gaining insight and making improvement also play their similarly happenstance roles—for example, epidemiological surveillance, public health interventions outside medical care (eg, housing, nutrition and environment) and programme implementation with evaluation.

Politics as usual

Society has not developed a methodology to guide use of the overall panoply of approaches that yield insights useful to efficient and effective reform. Instead, the developmental process often leaves the setting of goals and the implementation of methods to the political arena. Perhaps enlightened, informed and fair political agenda-setting is the best that complex democratic society can do in setting goals, but evidence and wisdom have strong roles to play in the selection of methods to ensure that political priorities reflect reliable insights.

At present, however, the strong allegiance of many powerful forces to the methods of orthodox medical research serves to denigrate most other methods of gaining insight. This is not merely an annoyance, since it serves to make some kinds of insight unavailable when costs, complexity or time pressures make conventional medical research unavailable. Thus, rather than relying on the experience of multiple sites in improving a clinical process or drawing from the experience of multiple jurisdictions in relieving inequities, leaders tend to discount those insights and thus to abandon the possibility of thoughtful and guided reform, either by not getting reforms under way or leaving them to political forces operating with inadequate information.

Example: the upcoming US investment in comparative effectiveness research

The example of comparative effectiveness could illustrate how to develop a methodology for mixed methods of reform. The US Congress appropriated about $10 billion for start-up and $500 million per year (from a fee assessed on health insurance) for this rather imprecisely defined work. The shortcomings that motivated this commitment centred on the recognition that much of medical research is not readily applicable to the usual situations of patients, for a number of reasons, and that deliberately expanding research to broader populations and to specified subsets could provide more authoritative answers to treatment questions that doctors and patients actually face. In some instances, this only requires repeating randomised clinical trials in broader populations to test whether effects documented in narrowly defined populations still are evident in more typical populations. However, half of the topics in the Institute of Medicine (IOM) list require attention to ‘system’ factors,4 meaning that the way that the delivery system operates is a key component affecting potential effectiveness.

For example, the third topic in the IOM priority list is ‘falls,’ which are a particular plague for older people. Specifically, the IOM Committee called for research to ‘Compare the effectiveness of primary prevention methods, such as exercise and balance training, versus clinical treatments in preventing falls in older adults at varying degrees of risk.’ Some of the issues in reducing the impact of falls are amenable to conventional clinical trials of a manageable size: for example, optimal treatment for reversible postural hypotension or optimal rehabilitation following elective hip replacement. However, reducing the impact of falls in a community will depend heavily upon ‘clinical treatments’ that rely upon local factors such as the ability to make home settings safer, the availability and services of monitoring agencies, and the supportive homemaker and rehabilitation services of the local Medicaid benefit. Indeed, perhaps the most important factor will be the prioritisation of mitigating this problem in many parts of the social system, from housing to transportation as well as healthcare services.

Thus, in devising a strategy to learn what we need to know in order to address the problem of falls, those who allocate funds and shape initiatives will need to develop a balanced strategy that tests medical treatments in broad populations and subsets—and that also tests the merits of various ways of forging the will to address the problem (generating motivating data and organising advocates, for example) and of various ways of managing a process of change (technical assistance, coalition-building and financial incentives, for example). The point would be to generate, within a reasonably short time, a substantially growing expertise in how to prevent and mitigate harm from falls—addressing, for example, individual clinicians and their patients, leaders of institutions providing services, public health officials, contributing social service and public infrastructure entities, and working groups focused upon a geographical locale. The products might include specific interactive models to assist physicians in decision-making about medication combinations and side effects, as well as specific recommendations on lighting, doorways and bathroom fixtures in new housing. In the USA, these opportunities rest not only in the comparative effectiveness initiative but also in the new funding and programmes for Accountable Care Organisations, Medical Homes, Health Homes, Dual Eligibles (beneficiaries with both Medicaid and Medicare) and Community-Based Care Transitions. I am sure that similar opportunities can be found in many other countries.

Organising for responsible advocacy

We also need to learn how to keep, at least those involved in quality improvement, abreast of initiatives arising from healthcare reforms and how to enable us to take action to inform and shape these endeavours. For example, the US Assistant Secretary for Planning and Evaluation has recently asked the public for suggestions as to how to build an inventory of comparative effectiveness research5; quality improvement insights need to show up in such an inventory. The US Internal Revenue Service recently asked for comment as to how non-profit hospitals should have to use their community benefit funds6; quality improvement interests should have put their point of view. The PPACA1 established a Patient-Centred Outcomes Research Board which is required to support a Methodology Committee on comparative effectiveness research; the insights from quality improvement activities need to be part of that work. The opportunities are legion, but people involved in improved management of healthcare services delivery have no mature organisations monitoring and coordinating responses to these opportunities.

Some older issues will take on renewed importance in the months ahead. The persistently ambiguous scope of application of the US regulations concerning protection of human subjects in research will need to be revisited and clarified. The mandates of the National Institutes of Health, which mostly restrict their funding to generalisable research on prevention and cure, may need to be examined and revised to allow their participation in developing methods to draw insights from context-sensitive improvement work. US antitrust law raises new challenges for initiatives that focus upon local populations and require broad cooperation to succeed. Even copyright and trademark issues have a new layer of complications when multiple parties working in the public interest generate an instrument or protocol worthy of marketing to other localities. Having scholars and managers ready to handle these issues would position the field of improvement science well.

Some may believe that these issues have less urgency now that Dr Don Berwick is head of the Centers for Medicare and Medicaid Services in the USA. Dr Berwick has been a motivating force and guiding light for quality improvement, and some will think that he can lead the changes we need. However, Dr Berwick cannot do much on his own. Advocates are going to need to be developing insights, political strategies and action plans that make it possible for him to lead. In any reasonably democratic society, leaders cannot be much ahead of the flock—so people concerned for system performance improvement and other methods are going to need to move ahead briskly in order make it safe enough for Dr Berwick and political leaders in Congress and state legislatures to capitalise on the opportunities, rather than retreating to established patterns that accept delay and resist reform.

People interested in evidence-based improvement can find various strategies to gain the experience to have an effective voice in the current health reform efforts, but we do need to undertake to do them quickly. The Academy for Healthcare Improvement could take on an informing and coordinating function, including attention to a number of countries where its membership is active. The Institute for Healthcare Improvement could expand its scope to help around the world with monitoring, brainstorming and building consensus. Individual leaders and allied professional and trade organisations could invest in skills and contacts needed to participate effectively.

Conclusion

The time is upon us, and it may be a long time returning if the current opportunities end up doing the same old things and replicating the same familiar shortcomings. In policy, as in life generally, the time-honoured aphorism holds: ‘If we keep doing what we've been doing, we'll keep getting what we've been getting.’ Now is a very auspicious time to start doing things differently with regard to developing a methodology sufficient to guide healthcare services delivery reform with confidence and efficiency.