Table 3

Generalisability (G) study: variance components*

Effect	df	Variance component (95% CI)†	Variance component, %	Component description
Person (p)	27	0.004 (0.001 to 0.012)	2.0%	Variability in participant proficiency
Scenario (s)	5	0.001 (0.000 to 0.011)	0.7%	Variability in scenario difficulty
Rater (r)	5	0.002 (0.000 to 0.017)	0.6%	Variability in rater stringency
Item (i)	8	0.049 (0.021 to 0.186)	22.2%	Variability in item difficulty
p × s	135	0.009 (0.006 to 0.013)	0.9%	Variability in participant performance between scenarios (case specificity)
p × r	135	0.002 (0.001 to 0.004)	4.1%	Variability in rater stringency between participants
p × i	216	0.009 (0.006 to 0.012)	3.9%	Variability in participant performance between items (item specificity)
r × s	25	0.000 (0.000 to 0.001)	0.1%	Variability in rater stringency between scenarios
i × s	40	0.001 (0.000 to 0.003)	5.4%	Variability in items between scenarios
r × i	40	0.012 (0.008 to 0.020)	0.5%	Variability in rater stringency between items
r × s × p	675	0.004 (0.003 to 0.006)	2.0%	Variability in rater stringency between stations and participants
i × s × p	1080	0.032 (0.028 to 0.036)	1.3%	Variability in item difficulty between stations and participants
r × i × p	1080	0.003 (0.001 to 0.005)	14.7%	Variability in rater stringency between items and participants
r × i × s	200	0.003 (0.002 to 0.004)	1.3%	Variability in rater stringency between items and stations
r × i × s × p	5400	0.088 (0.002 to 0.004)	40.3%	Residual error

*Analysis based on a fully crossed design with 163 responses to six clinical scenarios from 28 Adult and Child Neurology residents, each of which were independently scored by six faculty raters using the Quality Improvement Knowledge Application Tool Revised.
†Variance components are estimates of the magnitude of variability of each effect in the model.
df, degree of freedom.