The medical profession has, until recently, largely dictated standards of medical practice. If doctors completed their training and became licensed by the state they were trusted by the general public to provide clinical care with minimal obligation to show that they were achieving acceptable levels of performance.

Several factors have caused this situation to change. A societal trend towards greater openness in public affairs has been fuelled by the ready availability of information in many areas of life outside of the health sector. A slow realisation of wide variation in practice standards^{1, 2} and occasional dramatic public evidence of deficiencies in quality of care^{3, 4} have led to demands by the public and government for greater openness from healthcare providers. The availability of computerised data and major advances in methods of measuring quality⁵ have allowed meaningful performance indicators to be developed for public scrutiny. The result has been advocacy for the use of standardised public reports on quality of care as a mechanism for improving quality and reducing costs.^6–8

Publication of data about performance is not, however, new. In the 1860s Florence Nightingale highlighted the differences in mortality rates of patients in London hospitals,⁹ and in 1917 an American surgeon complained that fellow surgeons failed to publish their results because of fear that the public might not be impressed with the results.¹⁰

In most developed countries there is now an increasing expectation that healthcare providers should collect and report information on quality of care, that purchasers should use the information to make decisions on behalf of their population, and that the general public has a right to access that information. Organisations in the US have been publishing performance data, in the form of “report cards” or “provider profiles”, for over a decade. The scientific rigour and publication format of this information is highly variable, in part because many different organisations have contributed to the process and content. Fewer examples exist of public disclosure of performance data in other countries, though this situation is changing rapidly. For example, recent government policy in the UK suggests that public disclosure will form a central part of a coordinated strategy for quality improvement in the NHS.^{11, 12}

This article outlines the possible benefits and risks of such a strategy. It describes the US experience of public disclosure and the limited empirical evidence of its influence on the clinical and managerial practices of professionals and organisations, as well as the impact on quality of care for health service users. Drawing on this experience, recommendations are made to contribute to the growing debate in other countries, particularly the UK, about the potential advantages and problems associated with public disclosure.

Purpose of public disclosure

Public disclosure is advocated by some proponents with great enthusiasm but often with no clear conceptualisation of its purpose or implications. In principle it could serve to facilitate regulation and public accountability, to assist effective purchasing decisions, to facilitate informed consumer choice, and to influence provider behaviour.

The most commonly cited reason for public disclosure in the US is to promote consumer choice, based on the assumption that an informed consumer can influence market forces.¹³ In the past, most of the available information in the US has been about cost and therefore market driven changes have been financially focused. In theory, making meaningful quality information available to consumers will encourage market competition based on quality and either drive out low quality providers or stimulate them to improve.

Securing central control has been cited as one of the principal reasons for publishing performance data.¹⁴ The need for central accountability in the UK NHS has been increased by the replacement of a unitary system with semi-autonomous business units in the form of trusts and primary care groups (PCGs). Regulation is most effective if based on valid information; making this information public is compatible with open democratic government and makes it harder for interest groups to ignore. The relevance of market forces in the cooperative ethos of the new UK NHS is unclear. Although individual consumers may have little influence on large healthcare providers, group purchasers—such as health authorities and PCGs in the UK, and businesses, government, and managed care organisations in the US—might be in a position to make greater use of information about performance.

The potential of public disclosure to influence provider behaviour is based on the assumption that organisations and professionals have an intrinsic desire to improve practice but that barriers (time, competing priorities, lack of knowledge, deficient infrastructure) prevent the expected or desired improvement. Using performance data for internal audit purposes has not resulted in the anticipated change in clinical practice or consistent improvements in the quality of care,¹⁵ and it could be argued that publishing the information will increase providers' sensitivity to their performance by reminding, refocusing, or shaming them into action.

Examples of US public reporting systems

Many different organisations have contributed to the report card movement in the US, including federal and state governments, employers, consumer advocate groups, the media, private enterprises, and coalitions. The scientific rigour of the reports is highly variable. Report cards are usually published in the form of short brochures,^{16, 17} but dissemination on the internet is now common, allowing users to access more information at different levels of complexity.^18–20 Information is available about structural quality measures (for example, number of specialists, qualifications of and complaints about individual doctors), process measures (for example, preventative screening rates), and outcome measures (for example, post-operative mortality, patient satisfaction).

Perhaps the best known reporting system in the US is the Health Plan Employer Data Information Set (HEDIS), produced by the National Committee for Quality Assurance (NCQA), a not-for-profit partnership of private purchasers, health plans, and consumers.²¹ HEDIS represents an attempt to standardise how plans measure and report performance data and is based on both administrative and clinical data. Since the introduction of the first set of indicators in 1991, the set of indicators has become larger. The 1995 version, HEDIS 2.5, contained nine measures directly related to quality; the current version, HEDIS 3.0, contains 14 measures; and a pilot set of new indicators contains a further 25 measures of quality.

Comparative HEDIS data from volunteering health plans are published as the Quality Compass.²¹ The second edition published in 1997 contains information from over 330 plans, representing three quarters of all health maintenance organisation enrolees. The HEDIS indicators have not escaped criticism. They represent a considerable cost and administrative burden to health plans and, because participation is voluntary, concern has been expressed that only plans with above average performance would be willing to provide information. It has also been claimed that the NCQA may be subject to conflicts of interest because of the nature of the organisations contributing to indicator development.⁷ Some concerns have been expressed in the past that the data are not risk adjusted, emphasise process over outcome measures of quality, preventative over curative indicators, and that the data collection methods are not standardised.⁸ The NCQA has taken a pragmatic approach to indicator development and has responded to these criticisms through an ongoing research and development initiative resulting in annual releases of new and revised indicators and measurement methods.²²

Other high profile US reporting systems have focused on inhospital mortality. The New York cardiac surgery reporting system publishes hospital and surgeon specific risk adjusted coronary artery bypass surgery (CABG) mortality data.²³ It is one of the most sophisticated and most studied of the reporting systems. Clinical and administrative databases are used to collect information on age, sex, type of coronary artery disease, presence of myocardial ischaemia, level of ventricular function, presence of other diagnoses, severity of atherosclerotic process, previous heart operations, and the degree of emergency of the operation. These data are used to construct a multivariate risk adjustment model to compare mortality rates among hospitals and individual surgeons.

Similar work has been done in Pennsylvania,²⁴ and other states are following this example. The Californian Hospitals Outcome Project publishes inpatient mortality rates for acute myocardial infarction and is about to publish CABG mortality data.²⁵ The myocardial infarction report differs from the New York and Pennsylvania reports in that it is based on routinely collected clinical data extracted from hospital records rather than using data collected specifically for the purpose.

Controversy

The public disclosure of performance data has been described as essential,²⁶ desirable,⁸ inevitable,²⁷ and potentially dangerous.^{28, 29} No commentators have totally rejected the process despite sometimes vehement criticisms of current initiatives.³⁰ Report cards have been described as untimely,³¹ backward looking, unable to predict future performance, judgmental, and at odds with the principles of continuous quality improvement.^{32, 33} The unintended consequences of publication have been highlighted, including manipulation of data and an inappropriate focus on what is being measured, to the detriment of other areas of activity.^{14, 33} The debate is often highly polarised and both advocates and antagonists sometimes use the same data to support their opposing arguments.

The controversy can be illustrated by describing the New York cardiac surgery reporting system. Some authorities have questioned the conclusions of the quasi-experimental study that described dramatic improvements in post-CABG mortality after publication of performance data in New York State.³⁴ Similar improvements in outcome were reported in areas where performance data were used for internal purposes only.³⁵ Others have suggested that the improvement in mortality was the result of outmigration of high risk individuals to other states,³⁶ refusal to operate on those with highest risk of death,³⁷ unreliable data,^{29, 38} inadequate risk adjustment,^{37, 38} or over reporting of risk factors.

These criticisms are being addressed in various ways. Some of them are based on a misunderstanding of the New York study and have been countered by the New York research team.²³ Others represent inevitable disagreements among commentators on the interpretation of observational data. Specific criticisms about outmigration and reduced access for high risk patients have been discounted by an independent research team.³⁹

Research evidence

Despite the large amount of activity, surprisingly little research has been conducted into the utility, acceptability, and impact of public disclosure in the US. The following brief summary is based on a detailed review of original articles, reviews, correspondence, documents, and websites.⁴⁰ Medline and EMBASE were searched using the following MeSH headings: report cards, public performance reports, provider profiling, public/consumer/patient information, and consumer reports. The reference lists of all articles were searched and authors of published studies and other experts in the field were asked to recommend relevant published and unpublished studies. Most of the studies have used descriptive or quasi-experimental designs. No randomised controlled trial has been done to compare the relative effectiveness of public disclosure and use of the same information for internal quality improvement purposes.

Firstly, consumers and purchasers of health services express wide ranging and sometimes contradictory demands for comparative information about the performance of providers.^{41, 42} A small proportion search out or read the information, however,^42–45 and a significant proportion fail to understand the reports that are now available.^{43, 44, 46–48} Current evidence suggests that performance reports have a small, but possibly increasing, impact on the decision making of individual consumers and purchasers.^{42–44, 49, 50}

Secondly, doctors are sceptical about, and make little use of, the publicly disclosed performance information. Most are aware of and read the reports, but only a small proportion discuss them with their patients or consider them to have any impact on their clinical practice.^{37, 51} This is supported by an observational study which showed no significant difference in bed occupancy among hospitals with high and low mortality rates after publication of their inpatient mortality data.⁵²

Thirdly, hospitals do appear to respond to the public release of performance data. In some instances a negative and defensive response was observed, usually in the form of criticism of the reports.⁵³ In others, however, the information was received positively and incorporated into the institution's quality improvement strategy,^{31, 54} and resulted in measurable improvements in the quality of patient care.^{55, 56}

Finally, there is limited evidence that publication of performance data may contribute to an improvement in health outcomes. A dramatic reduction in mortality after CABG surgery was observed in New York after the publication of comparative risk adjusted hospital and surgeon specific mortality data.^{34, 39} The controversy surrounding the impact of the New York CABG reports has been described earlier in this article, but current evidence suggests that the reports are associated with, and perhaps have stimulated, a reduction in mortality.⁵⁷

Lessons to be drawn from the US experience

An increased level of public disclosure of healthcare performance data is probably inevitable in most developed countries. This is despite considerable technical problems, possible misinterpretation of the results, predictable resistance of some interest groups, and current paucity of evidence of its impact. The US health system is markedly consumer orientated, operates in a relatively unregulated healthcare market, and has ready access to routine data. These factors might lead analysts in countries with more socialised and data deficient health systems to discount the US experience of public disclosure. This would be a mistake and would lead to them repeating many of the errors that have occurred in the US over the past decade. The relatively high level of central regulation in, for example, the UK NHS provides an opportunity for the UK to implement public disclosure in a more controlled and sensitive fashion than was possible in the US. Addressing the issues highlighted in box 1 would facilitate this process.

The current lack of empirical research into the US experience of public reporting will dictate that other countries will operate, at least initially, in an evidence sparse policy zone. Research funding bodies should ensure that this situation is rectified as rapidly as possible. More information is needed about the type of information that users and purchasers want and can use, and the impact of public disclosure on quality of care. Further research is also required into the relative merits of publishing process and outcome quality indicators,^{58, 59} the level of risk adjustment necessary to make fair and meaningful comparisons between outcomes of providers,^{60, 61} the most practical reporting level (for example health authority, trust, PCG, or individual clinician),⁶² the financial cost, and how best to manage the adverse and unintended consequences of public release. The UK national performance framework represents one of the first attempts anywhere in the world to develop a systematic and comprehensive public reporting system. Publication of the framework provided a unique opportunity to provide experimental evidence of the relative benefits and disadvantages of public disclosure and use of the same data for internal quality improvement purposes. Although this opportunity has been missed for the first group of indicators, it could still be pursued as the indicator set is developed.

The process of public disclosure can and should be actively managed, not allowed to happen in an uncoordinated fashion, and not led by interest groups that have little understanding of the science of quality measurement and reporting. This will require a sensitive balance between scientific rigour, dogma, and common sense. Publishing misleading information would be dangerous, but waiting for researchers to develop perfect measures of quality would also be misguided. Public disclosure should be seen as one tool in a quality improvement strategy, which should be integrated with other tools such as educational initiatives, clinical audit, peer review, and incentives. Public disclosure is best regarded as a way of increasing the heat on organisations or professionals to allow a catalyst (information on performance) to facilitate a chemical reaction (quality improvement). Clinicians should work with managers, policy analysts, and researchers to determine the optimum level of heat required to produce the most desirable outcomes.

The extent to which organisations and health professionals accept the principles and practicalities of public disclosure is central to its success. Using disclosure as a stick to beat those who are working hard and are committed to improving the service for their patients is likely to have significant adverse consequences. Public disclosure will be seen by some as a new and exciting way to drive quality improvement. Managing its implementation to maximise the benefits and reduce the risks will be a major challenge.