Table 3

Inter-rater reliability between holistic overall ratings of the same record by paired reviewers of different staff types

Reviewer pairs	Condition	Site^*	No. of paired reviews	ICC between ranked scores (95% CI)	Weighted mean ICC^† (95% CI)
Doctor vs doctor	Heart failure	B^‡	49	0.67 (0.54 to 0.79)	0.52 (0.41 to 0.62)
	COPD	G	48	0.33 (0.05 to 0.56)
	Heart failure	F	18	−0.03 (−0.48 to 0.43)
	Heart failure	E^§	12	−0.44 (−0.80 to 0.15)
Nurse/clinical vs nurse/clinical	Heart failure	D	21	0.74 (0.47 to 0.89)	0.46 (0.34 to 0.59)
	COPD	D	49	0.37 (0.10 to 0.58)
	COPD	J	26	0.27 (−0.12 to 0.59)
	Heart failure	H	48	0.22 (−0.07 to 0.47)
Non-clinical audit staff vs non-clinical audit staff	COPD	A	48	0.47 (0.22 to 0.66)	0.47 (0.22 to 0.66)

COPD, chronic obstructive pulmonary disease; ICC, intraclass correlation coefficient.
↵* Only sites with more than one reviewer of the same staff type are included in this table.
↵† Mean ICC per staff type, weighted by inverse variances to account for differing numbers of paired reviews.
↵‡ A single ICC was calculated for the three doctors at site B.
↵§ The doctors at site E were non-specialist doctors.