Measuring the performance of health systems

Indicators still fail to take socioeconomic factors into account

Reviews p 248

It seems that the whole world is suddenly talking about measuring the performance of health systems. Last month the World Health Organization published its findings from a comparative study of healthcare systems.1 This time it is the turn of the Department of Health in England, which last week published the results of the second round of performance indicators for 99 health authorities and 275 NHS hospital trusts.2 Unlike last year's figures, these data will inform and shape key aspects of the government's plan for the NHS, which will be published next week.

The main message of the latest performance indicators is that health in England is continuing to improve. However, there is also compelling evidence of variation in health and healthcare performance between areas and between hospitals. For example, the proportion of patients waiting less than two hours to be admitted after attending an accident and emergency unit ranges from 38% to 100%. There is also a twofold variation between the best and worst health authorities in death rates from circulatory diseases.

While the focus of the new performance measures is still on individual indicators, the intention of the government's national plan is to gain an overall view of the performance of hospitals, possibly by aggregating indicators.3 In theory the plan is attractive, and the idea of combining indicators mirrors the approach taken by the WHO. This will provide information for the so called traffic lights system, which will be introduced shortly. Organisations in the NHS will be periodically classed as “green” (excellent), “amber” (having room for improvement), or “red” (poor) depending on their overall performance. Organisations classed as green will be rewarded and can expect to enjoy far greater freedom to manoeuvre in terms of developing local services. The fate of organisations classed as red or amber is less certain, although it seems likely that they will receive help from intervention teams made up of successful NHS managers or from the private sector. Yet one question remains: can the indicators distinguish between good and bad performance? For two fundamental reasons the answer must be no.

Firstly, whether comparing districts or countries, it is not clear how summarised accounts of performance help to identify which parts of a health system contribute most to improved health. The WHO's rankings concealed a range of performance on individual indicators. The United Kingdom, for example, scored second highest on the distribution of health gain, yet dropped to 18th place overall. Similarly, the performance tables published by the Department of Health show that in some health authorities there are low death rates from heart disease and cancer although these same authorities have poor rates of access to NHS dentists or high rates of inappropriate surgery. The WHO weighted different indicators to reflect their perceived importance in the overall indicator of performance. This raises questions about who should provide these weights. The WHO's survey used key informants from around the world, but a case could be made that other people, such as those who use the services, should decide which aspects of the health system matter most.4

Secondly, it is not clear to what extent the indicators identify the contribution of health services to the health of the population in general and to people with poorer health prospects in particular. And further, what light do the indicators shed on the role of different providers—general practitioners, hospitals, community services—in improving health?

In all cases, the answer is little. There is no way of knowing, on the basis of the indicators alone, whether high death rates in some parts of the country are due to poor care. If sanctions are to be applied and incentives offered, these are the things that we need to know. Moreover, if deprived areas are not to be penalised for poor performance then the data must be adjusted to account for socioeconomic factors. The risk, as the Department of Health acknowledges, is that such adjustments mask the true extent of inequalities that the NHS should know about and address.5

The Secretary of State for Health recently announced that the NHS is neither a market nor an administrative hierarchy but a system.3 Systems in general, and health care in particular, are complex, hard to understand, and difficult to manage. Performance indicators alone are not enough; unhappily, unless the national plan for the NHS reveals evidence of new thinking, performance indicators seem to be the only game in town.