Health service planners are increasingly trying to find ways to improve the quality and safety of health care. A wide range of approaches is being used from high level regulatory frameworks, use of clinical guidance and guidelines, to more micro level activity such as audit of care. None of these is easy; all require significant investment of resources, training, time and monitoring. The results are often uneven and result in variations in quality as initiatives diffuse unevenly through the system. It is understandable then that policy makers seek easier ways to deliver these improvements.

Research since the late 1970s seemed to point in the direction of a relatively constant relationship in health care—increased use of a hospital procedure reduces the mortality associated with it. The message emerging from a large number of studies, mainly from the USA, was that patients treated in hospitals which (or by clinicians who) managed high volumes of patients with the same condition had better outcomes than those with lower volumes. This was summed up by Luft and others in an influential report in 1990.¹ Hundreds of studies have been published, many of which are based on analysis of large US administrative databases, and most report an inverse relationship between the volume of activity and mortality or other poor outcomes. These studies have been reviewed and summarised in several publications in the last few years.^2–5 Although there was not full agreement, in general the reviews—especially those published in the USA—support the hypothesis of a volume-outcome relationship and the existence of volume-quality thresholds.

The policy response was predictable. Several national and state regulators and professional associations set volume thresholds for hospital based procedures and for hospital accreditation and pushed for the regionalisation of services. Policy makers like simple messages and interventions more under their control which do not depend heavily for their success on the compliance of clinicians. The idea that volume directly influences outcome has become so accepted that at times volume is used as a proxy for—and even confused with—outcome. The influential Leapfrog Group for Patient Safety recently produced a fact sheet on evidence based hospital referral,⁶ giving advice on the criteria to be used to select high quality hospitals in which they state that:

“Another measure of surgical outcomes is volume – how many procedures of a given type a hospital performs each year. … Choosing the right hospital is not just important in surgery. For example, babies with very low birth weight or major congenital abnormalities are much more likely to survive if they are treated at hospitals with large neonatal care units.”

The consensus around the relationship between volume and outcomes and its potential use as a measure of quality is, however, based on shakier foundations than many leading researchers in the field care to acknowledge. The quality of the studies underpinning the commonsense view has often been poor and their interpretation at times disingenuous; concerns expressed a decade ago were largely ignored.^2,7,8 However, in the last couple of years the pendulum has begun to swing back and a recent methodological critique⁹ and new analyses^10–12 have raised concerns about both the evidence for this relationship and its use for policy making.

One concern with the research in this field is that it consists mainly of cross sectional studies, often using information from administrative databases with relatively little clinical detail. If units with higher volume were admitting patients who, on average, were less seriously ill (indeed, this might be a consequence of having larger capacity), then a straight comparison would automatically show high volume units to have lower mortality than low volume units. The association would, however, be spurious—the result of the confounding effects of case mix. Only if these studies take into account variations in the distribution of prognostic factors in patients across the units—risk adjustment—can the associations be trusted. This was shown, for example, in a meta-analysis of studies of coronary bypass surgery (one of the most studied procedures and strongly regulated by volume thresholds) which showed that studies which poorly adjusted for case mix reported much stronger associations between volume and outcome than those which more adequately took case mix into account.⁸

The quality of research in this area has improved over the last few years with some interesting results.⁴ The largest and most recent study based on a high quality clinical database using very good risk adjustment did not show a strong association between volume and quality.⁹ The authors concluded that the use of hospital procedural volume of coronary artery bypass surgery is of limited value in discriminating between better or worse risk adjusted mortality outcomes.⁹ Another high quality study of the mortality of very low birth weight infants in the US showed that the volume of the neonatal intensive care units “cannot prospectively identify high quality providers”.¹⁰ These two studies strongly suggest that the sort of guidance given and thresholds applied by the Leapfrog Group and others are unjustified and do not warrant the description “evidence based”.

This does not mean that there is no association between volume and quality; for some procedures (such as AIDS treatment, surgery for pancreatic and oesophageal cancer, abdominal aortic aneurysm, and paediatric cardiac procedures) there does appear to be reasonably strong evidence of a relationship between volume and improved patient outcomes.^2–4 In other areas this seems only to be relevant in high risk patients or very low volume providers.

However, even if there is a valid statistical association between volume and outcome, this does not tell us whether it is causal. In other words, and most importantly, it does not provide evidence as to whether a policy of hospitals increasing their volumes will result in an improvement in their clinical outcomes. To establish the impact of changing volume on outcomes needs a prospective and preferably experimental design.¹¹ In their review, Sowden et al⁷ identified only two longitudinal studies which examined changes in outcomes as volumes varied over time. Even though these studies found cross sectional associations between volume and in-hospital mortality, there was no relationship between changes in volume and outcome over time. The absence of reliable prospective data raises serious concerns about our ignorance of the likely impact of policies to concentrate services.

Even if the research showed a causal link, we would need to know more about the mechanism of action so that it could be implemented effectively. For example, if there is an effect, what are the relative contributions from volume at the hospital, unit, team or clinician level? Rarely do studies consider the relative effects of the unit of analysis and their possible interactions. Importantly, Urbach and Baxter, using data from Ontario, Canada, recently reported that the inverse association they found between procedure volume and postoperative mortality risk was not specific to the volume of the procedure being studied!¹² As with several previous studies, they found that higher volume was associated with lower 30 day mortality from repair of abdominal aortic aneurysm, oesophagectomy, pancreaticoduodectomy, major lung resection for lung cancer, and not from colorectal cancer surgery. However, the reduction in mortality from all these procedures except colorectal cancer was also associated with higher volumes in the other procedures, sometimes more than with its own volume. Although the study can be criticised for using volume thresholds rather than treating volume as a continuous measure and for inadequate case mix adjustment compared with some of the best, the findings raise serious questions about what we are really measuring when such associations are calculated. Simple empirical associations are not a sufficient basis for policy making in this field. We need to understand more about what is going on in units with different levels of quality. As the authors suggest, we need to “revisit the conceptual framework underlying volume based regionalisations”.

This is not just a technical debate. The drive to increase volumes can have adverse effects which would need to be outweighed by greater benefits to be justifiable. For example, hospitals worried about accreditation or losing business might inflate volumes by artificially lowering the threshold for treatment, admitting patients with less need and so reducing efficiency. More importantly, the policy of concentrating services can result in reduced local access for services, more travel time, and possibly suboptimal local treatment. We have been urged to “move ahead” on the question of volume and outcome;¹³ however, perhaps with the exception of a few high risk procedures where the evidence is clear, we still do not know enough about the meaning of the relationships found to be able confidently to use volume as an effective policy instrument.