It doesn’t seem long ago that many health professionals and managers were voicing passionate opposition to the public release of comparative information about provider performance. By and large, these voices have now gone quiet. In their place we hear more thoughtful discussions about how best to publish information in a way that engages the various audiences, maximises the benefits of disclosure, and minimises the potential for adverse consequences.

Why such a dramatic change? The argument in favour of publishing information about performance has been won in policy terms principally because of the contribution that disclosure can make to increasing the accountability of provider organisations. In more philosophical terms, many of the opponents of disclosure have been won over by arguments about the “right to know” of citizens in a democratic society.

Those who look for more instrumental reasons to justify the costly and complex task of publishing performance information have to search a little harder for supportive evidence. The original expectation in the United States, the home of so-called “report cards”, was that informed and empowered consumers would use comparative information to select high quality providers or to demand better performance from lower quality providers. In fact, there is little empirical evidence that consumers use data in this way, and even less evidence that purchasers do so on behalf of consumers.¹ In the UK it seems that most members of the public do not want to view health care as a market commodity and that they have a strong sense of responsibility for their local providers which leads them to distrust comparative information produced by external bodies.² Furthermore, since people tend to make decisions about providers only when they need services, they are less likely to feel empowered to make rational decisions.

In contrast, provider organisations do seem to be sensitive to the publication of comparative performance data.³ For example, a report card on the structure and process of obstetric care was associated with the institution of car seat programs, follow up care, transfer agreements with tertiary care centres, and breastfeeding education at hospitals that were publicly reported not to offer these services.⁴ In a non-randomised controlled trial, a public report card on hospital complications was associated with markedly increased quality improvement activity (relative to either a private report card or none at all), especially among hospitals that received poor ratings.⁵ It is less clear whether the process changes stimulated by public report cards actually lead to improved outcomes. A now defunct public reporting program in Cleveland apparently shifted some deaths to post-discharge settings, leading to reduced 30 day mortality for two conditions, increased 30 day mortality for one condition, and no change for three conditions.⁶ New York’s cardiac surgery reporting program stimulated a dramatic decrease in risk adjusted mortality, largely through focused interventions at high mortality hospitals,⁷ but northern New England achieved similar benefits through private reporting and sharing of best practices.⁸

The big question—which we are only just starting to address—is the nature of these associations and whether there is a causal and unidirectional relationship between the publication of comparative information and improvements in the quality of care. Put simply, is public disclosure a cost effective way to improve the quality of patient care?

While there are potential mechanisms to explain how public disclosure might drive improvement—refocusing providers’ attention on quality of care rather than financial performance, capitalising on the sensitivity of providers to their reputations and the innate competitiveness of clinicians and mangers—it has proved difficult to design evaluative studies. Most of the published research to date has used observational designs and most of this has been carried out in the United States.

In this issue of QSHC Ito and Sugawara⁹ contribute to the mounting body of observational evidence linking disclosure to performance using accreditation data from the Japanese health system. They report that small and medium sized hospitals that self-disclosed their accreditation reports received significantly higher scores on “efforts to meet community needs” and “medical/healthcare” than non-disclosing hospitals; no such difference was observed among large hospitals. There are methodological flaws with the study, including the unrepresentative sample of hospitals that volunteered for accreditation and the lack of standardisation of quality measures based on accreditation assessments. Nonetheless, their findings are consistent with those of other studies which have reported uniformly lower “effectiveness of care” among health plans that allow the National Committee for Quality Assurance to publicly disclose their Health Plan Employer Data and Information Set (HEDIS) scores than among health plans that do not, largely because poorly scoring plans withdraw from the public disclosure program.¹⁰ Similarly, hospitals that participate in California’s voluntary cardiac surgery reporting scheme have lower unadjusted death rates than non-participating hospitals.¹¹

It is difficult to judge from these studies whether public disclosure is something that only good hospitals are willing to do, or whether it actually leads to improved performance. This is a critical question because of its policy implications. Should public disclosure be mandatory because we expect it to stimulate quality improvement? Or should it remain voluntary, with confidential efforts to share best practices across hospitals and public efforts to applaud volunteers for their honesty and courage?

The challenge now facing the research community is to produce experimental evidence of the impact of disclosure on quality of care. This is a complicated task. There are problems with optimising the intervention because the content and presentation format of many report cards are deficient, and in determining the most appropriate outcomes. Examining patient movements between providers or insurance groups is relatively easy, but examining the impact on quality of care is more problematic. Even the strongest advocate of disclosure must accept that the effect size of publishing information is likely to be small and hence the sample size and costs of an experimental study are likely to be enormous.

Despite the increasing commitment to public disclosure of provider performance data, there is still much that we do not know about its cost effectiveness, impact, and mechanisms of action. The paper by Ito and Sugawara gives us one more piece in the jigsaw, but there are still significant challenges for the research community.