Summary of reviewer rates of agreement for the independent and joint reviews (number of index consultations rated as such by both reviewers/number rated as such by either; % agreement)
| No new diagnosis | New accurate diagnosis (no MDO) | MDO implicated | Overall agreement | Kappa* (95% CI) | |
| Independent reviews (n=2064) | 845/1143 (73.9%) | 760/1138 (66.8%) | 35/207† (16.9%) | 1640/2064 (79.4%) | 0.63 (0.60 to 0.66) |
| Joint reviews (n=2057‡) | 858/1121 (76.5%) | 835/1106 (75.5%) | 89/105† (84.8%) | 1782/2057 (86.6%) | Not relevant |
*Kappa 95% CI estimated using 1000 bootstrap samples.
†Indicates that when reviewing independently, 35 consultations were coded as including an MDO by both reviewers, while another 172 (total=207) were coded as an MDO by one reviewer but as no new diagnosis or a new accurate diagnosis by the other; resulting in 16.9% agreement (35/207×100%). After the joint reviews 89 consultations were agreed by the reviewers to include an MDO, with another 16 (total=105) coded an MDO by one but not the other; giving 84.8% agreement.
‡Excludes seven index consultations unintentionally omitted from the joint review exercise.
MDO, missed diagnostic opportunity.