Background Public report card designers aim to provide comprehensible provider performance information to consumers. Report cards often display classifications of providers into performance tiers that reflect whether performance is statistically significantly above or below average or not statistically significantly different from average. To further enhance the salience of public reporting to consumers, report card websites often allow a user to compare a subset of selected providers on tiered performance rather than direct statistical comparisons of the providers in a consumer's personal choice set.

Objective We illustrate the differences in conclusions drawn about relative provider performance using tiers versus conducing statistical tests to assess performance differences.

Methods Using publicly available cross-sectional data from Medicare Hospital Compare on three mortality and three readmission outcome measures, we compared each provider in the top or bottom performance tier with those in the middle tier and assessed the proportion of such comparisons that exhibited no statistically significant differences.

Results Across the six outcomes, 1.3–6.1% of hospitals were classified in the top tier. Each top-tier hospital did not statistically significantly differ in performance from at least one mid-tier hospital. The percentages of mid-tier hospitals that were not statistically significantly different from a given top-tier hospital were 74.3–81.1%. The percentages of hospitals classified as bottom tier were 0.6–4.0%. Each bottom-tier hospital showed no statistically significant difference from at least one mid-tier hospital. The percentage of mid-tier hospitals that were not significantly different from a bottom-tier hospital ranged from 60.4% to 74.8%.

Conclusions Our analyses illustrate the need for further innovations in the design of public report cards to enhance their salience for consumers.