Table 2

Requirements, steps forward and remaining challenges for robust and useful composite indicators

Requirement	Steps forward	Remaining challenges
Transparency The principles and theory underlying the composite indicator must be clear	Being clear about who is involved in making decisions in developing the composite indicator.	Many stakeholders may be involved. The design may evolve in unexpected ways over time.
	Fully describing the decision-making process, reporting the reasons and justifications for the decisions made.
Purpose-led design The composite indicator must plausibly measure what it sets out to measure	Selecting individual measures to cover the full range of services intended to be measured by the composite.	Identifying appropriate individual measures. Appropriate measures may not exist for all areas included in the composite.
	Choosing weights that reflect the relative importance of the different quality measures.	Balancing the weighting system against competing priorities.
Technical reproducibility The composite indicator must be reproducible using the raw data and the published methodology	Providing clear and comprehensive technical documentation.
	Reporting full definitions of the individual underlying measures and how they are combined.	Individual measures may only be available from sources that do not fully document the details, but these measures should not be used in the composite.
	Publishing the code used in data processing and statistical analysis.
Statistical fitness Individual measures must be adequately adjusted for case-mix, have acceptable statistical reliability and be appropriately standardised to consistent scales	Performing appropriate statistical case-mix adjustment.	Accurate patient-level data may not exist for important case-mix factors. Adequate statistical case-mix adjustment may not be possible. Interpretable results may require further processing.
	Using reporting periods long enough to give acceptable reliability.	Longer reporting periods may be necessary to increase reliability, but impedes use in driving quality improvement.
	Standardising measures to consistent scales in a principled way that preserves the useful information in the underlying measures.	Understanding what good and bad performance in the real world looks like on each measure.