Measures of clinical agreement for nominal and categorical data: The kappa coefficient

https://doi.org/10.1016/0010-4825(92)90063-SGet rights and content

Abstract

The desire to determine the extent inter-rater measurements obtained in a clinical setting are free from measurement error and reflect true scores has spurned a renewed interest in assessment of reliability. The kappa coefficient is considered the statistic of choice to analyze the reliability of nominal and categorical types of data recorded on the same patient by more than one clinician. This paper presents a simple computer program written in PASCAL that can be used in a clinical environment to quickly determine the reliability of nominal or categorical data. This computer program calculates both weighted and non-weighted kappa coefficients with their corresponding standard errors as well as bias-correcting jackknife estimates of kappa for use with small sample sizes.

References (20)

  • J. Bartko et al.

    On the methods and theory of reliability

    J. Nervous Mental Dis.

    (1976)
  • K. Francis

    Reliability

    Phys. Ther.

    (1986)
  • A.D. Delitto et al.

    Reliability in the clinical setting

    Res. Newslett. Am. Phys. Ther. Assoc.

    (1991)
  • J.L. Fleiss

    Measuring nominal scale agreement among many raters

    Pychol. Bull.

    (1971)
  • J.L. Fleiss et al.

    Large sample variance of kappa in the case of different sets of raters

    Psychol. Bull.

    (1971)
  • J.R. Landis et al.

    The measurement of observer agreement for categorical data

    Biometrics

    (1977)
  • S. Haley

    Kappa coefficient calculation using multiple ratings per subject: a special communication

    Phys. Ther.

    (1989)
  • J. Ubersax

    A generalized kappa coefficient

    Ed. Psychol. Measure.

    (1982)
  • J.L. Fleiss et al.

    The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability

    Ed. Psychol. Measure.

    (1973)
  • J. Cohen

    Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit

    Psychol. Bull.

    (1968)
There are more references available in the full text version of this article.

Cited by (97)

View all citing articles on Scopus
View full text