Basic principles of ROC analysis

https://doi.org/10.1016/S0001-2998(78)80014-2Get rights and content

The limitations of diagnostic “accuracy” as a measure of decision performance require introduction of the concepts of the “sensitivity” and “specificity” of a diagnostic test. These measures and the related indices, “true positive fraction” and “false positive fraction”, are more meaningful than “accuracy”, yet do not provide a unique description of diagnostic performance because they depend on the arbitrary selection of a decision threshold. The receiver operating characteristic (ROC) curve is shown to be a simple yet complete empirical description of this decision threshold effect, indicating all possible combinations of the relative frequencies of the various kinds of correct and incorrect decisions. Practical experimental techniques for measuring ROC curves are described, and the issues of case selection and curve-fitting are discussed briefly. Possible generalizations of conventional ROC analysis to account for decision performance in complex diagnostic tasks are indicated. ROC analysis is shown to be related in a direct and natural way to cost/benefit analysis of diagnostic decision making. The concepts of “average diagnostic cost” and “average net benefit” are developed and used to identify the optimal compromise among various kinds of diagnostic error. Finally, the way in which ROC analysis can be employed to optimize diagnostic strategies is suggested.

References (21)

  • DorfmanDD et al.

    Maximum-likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—rating method data

    J. Math Psych

    (1969)
  • TurnerDA et al.

    Brain scanning with the Anger multiplane tomographic scanner as a secondary examination

    Radiology

    (1976)
  • GoodenoughDJ et al.

    Implications of a “noisy” observer to data processing techniques

  • GreenDM et al.
  • GreenDM et al.

    Signal Detection Theory and Psychophysics

  • GreenDM et al.

    Signal Detection Theory and Psychophysics

  • DorfmanDD et al.

    Maximum likelihood estimation of parameters of signal detection theory—a, direct solution

    Psychometrika

    (1968)
  • MetzCE et al.

    Quantitative evaluation of visual detection performance in medicine: ROC analysis and determination of diagnostic benefit

  • GoodenoughDJ

    Radiographic Applications of Signal Detection Theory

  • StarrSJ et al.

    Visual detection and localization of radiographic images

    Radiology

    (1975)
There are more references available in the full text version of this article.

Cited by (0)

The Franklin McLean Memorial Research Institute is operated by the University of Chicago for the U.S. Department of Energy under Contract EY-76-C-02-0069.

1

From the Department of Radiology, University of Chicago, and the Franklin McLean Memorial Research Institute, Chicago, Ill.

View full text