Article Text
PDF
Statistics from Altmetric.com
Targets have assumed a central role in the management of healthcare and public services more generally in the UK over the last 25 years. They emerged from ideas of ‘new public management’ in the 1980s and of a strong performance management approach under prime minister Tony Blair from 1997.1 While targets can be effective2 and are an important part of public accountability, Quinn3 provides more evidence in this issue of the journal that they can also produce unintended or unanticipated consequences, some of which are unhelpful. They investigated the impact that an English NHS target has had on referral practices. The target in question was the ‘18-week referral to treatment standard’ introduced in 2012. The standard states that at least 92% of patients should have been waiting for less than 18 weeks for treatment after their referral. Quinn found strong evidence for a threshold effect over the period 2015–2020, when some patients seem to have been prioritised based on the target rather than clinical need. Specifically, they found evidence of a spike in the number of hospital trusts at the target threshold.
This study joins a long litany of examples of the unintended impact of targets, a number of which are explored in a previous editorial in this journal.4
The theory of the target setters seems to be that hospitals need a clear focus on government-set goals and that, without this, they will direct their attention elsewhere or possibly fail to put in the required effort.5 There are several risks with this approach.
Focusing on one element of a complex system means that important interactions tend to be ignored or oversimplified, which can produce effects that undermine the intent of the target. For example, a rigid focus on hitting the 18-week waiting target for 92% of patients could distort clinical decision making, leading to clinicians being less engaged in the process and also leading to some patients waiting significantly longer once their treatment time exceeds 18 weeks, as their treatment time ceases to influence the reported performance. Hospitals doing better than the target may allow their performance to slip back to the target level.
Hospitals faced with a challenging target may respond in different ways, depending on the position they start from in relation to the target, their local context, and the competences and preferences of their management. Hospitals that have no chance of meeting the target may choose to direct their efforts elsewhere. The most sustainable approach to meeting a target is to redesign processes and realign resources to ensure that the targets are met as a by-product of a well-designed system. However, if there are insufficient resources—for example, in the case of waiting lists where demand exceeds capacity or the organisation lacks the skills and resources to undertake a major review of processes and ways of working—less desirable approaches may be taken.
One option is to demand unsustainable levels of work from staff, focusing on the target to the exclusion of almost everything else. In the case of the NHS Mid-Staffordshire hospital scandal, the pursuit of financial objectives and a short-term focus on meeting the requirements to qualify for greater organisational autonomy (ie, to become a ‘foundation trust’) led to unsafe levels of staffing, a lack of attention to important issues of quality and safety, and a major collapse in elements of a caring culture. This resulted in serious harm to patients, their carers and staff. The hospital had become so fixated on financial targets that it had ceased to notice these consequences.6
Another option is gaming the targets7—for example, finding ways to exclude people from a waiting list or, in the case of emergency departments, move patients to potentially unsuitable accommodation just before the 4-hour waiting time targets might be breached.
At worst, gaming means not only that the target is not met but also that other adverse outcomes can be generated. For example, patients moved to decanting units to meet the target for time spent in the emergency department could wait longer for treatment, and outcomes may be worse when patients are put in the wrong place8 9 or moved.10
This and similar behaviours can lead to an arms race of increasingly complex rules designed to eliminate gaming, followed by even more ingenious methods to meet the target.
Making targets work
Given that targets seem to be here to stay and indeed can sometimes be useful for ensuring accountability or supporting improvement,11 it is important to understand how to minimise their worst effects and maximise their effectiveness.
One of the important purposes of well-designed targets is to provide clarity about the most important goal of the target system. Good targets focus the system’s attention on a desired outcome. Badly designed targets, in contrast, focus attention on easy-to-measure inputs to the system. Too many targets fail to achieve clarity about the objective; this may be about assurance, improvement, relative position in a league table or achieving a specified goal.
It is also possible to have good targets embedded in a performance improvement system that undermines their value by sending the wrong signals about how the system should respond. Sustainable performance improvement depends a great deal on how well those who have to deliver the desired outcomes understand the key factors that matter and have the management capability to make change. Those setting targets need a well-evidenced logic model that links the target to the desired behaviours or to changes in the system that will deliver it.12 Linking targets to high-powered incentives and punishments may increase the probability of gaming and the other dysfunctions detailed previously,13 which can also undermine the usefulness of the data used to monitor improvement.
A useful case study is COMPSTAT, the system used by the New York Police Department in the 1990s to radically reduce serious crime. The designer and leader of the system in its early days, Jack Maple,14 held local police commanders to account for reducing crime but was more likely to fire them for failing to understand their local crime patterns than for not showing headline reductions in their targets. The way the performance system was managed was more important than the use of a target. The target set a clear goal, but the management system around it used the metric to promote better local understanding of how to achieve it.
COMPSTAT also illustrates how systems can fail when focus on outcomes and understanding is lost. In the post-Maple years, the system demanded activity targets (so the mayor could get headlines about how much police activity had increased). This led to perverse gaming of the input statistics and far less actual improvement. The loss of focus on understanding the patterns of crime and the infantilisation of local leaders by targeting inputs was a failure.
While stretching targets can unlock creativity and prompt people to find new solutions, which seemed to be the case with targets for reducing healthcare-acquired infections, this cannot be guaranteed. Dysfunctions seem likely to be endemic in situations where the gap between the aspirations of the target and the capacity and capability of the system is too far apart to be bridged within the time or resources available.
The NHS has often pursued the failed COMPSTAT approach by promoting input targets, not outcomes, and promoting ‘achieving the numbers’ over developing a better understanding of the problem. This means that an understanding of the context and how systems work and interact will be important. Insights from the people who will be working with the target about how it might operate in practice and who can affect the behaviour of different actors in the system can be valuable. Muller suggests involving a diverse range of stakeholders in the design phase of targets, both to identify potential gaming opportunities and to improve the overall design of the instruments to ensure that they reflect measures that are important to those involved.15
Generally, unintended consequences may be less easy to identify in advance because targets represent an intervention in a complex adaptive system where there are multiple dynamic interactions that make behaviours and responses less predictable.16 This argues for ensuring that any use of targets is associated with systems for learning from and evaluating their impact. Employing a portfolio of measures, rather than a single headline metric, will ensure a fuller picture of the outcomes and processes that matter, quickly identifying perverse effects and making gaming more difficult. However, this portfolio approach also carries risk: additional measures should be used as a means to improve understanding rather than adding a further layer of targets.
The experience of the use of targets in the English NHS, as evidenced by Quinn and many other researchers, suggests that over-reliance on a small number of high-profile measures is risky. A richer picture of how the system being measured works and how its staff and managers behave and are motivated is needed for sustainable long-term change.
Effective performance improvement systems cannot be built solely on targets but need a great deal of managerial judgement. As Muller puts it, ‘…measurement is not an alternative to judgement: measurement demands judgement: …about whether to measure, what to measure, how to evaluate the significance of what’s been measured…’.15
The improvement system and the local teams delivering improvement both need the management capacity to make good judgements to avoid the issues of gaming, overpromising and other perverse ways of pursuing the metric while missing the point.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Footnotes
Twitter @nedwards_1
Contributors NE developed the structure. Both authors contributed material and underlying ideas.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Commissioned; internally peer reviewed.