Methodologic Standards for the Development of Clinical Decision Rules in Emergency Medicine,☆☆,,★★

https://doi.org/10.1016/S0196-0644(99)70309-4Get rights and content

Abstract

The purpose of this review is to present a guide to help readers critically appraise the methodologic quality of an article or articles describing a clinical decision rule. This guide will also be useful to clinical researchers who wish to answer 1 or more questions detailed in this article. We consider the 6 major stages in the development and testing of a new clinical decision rule and discuss a number of standards within each stage. We use examples from emergency medicine and, in particular, examples from our own research on clinical decisions rules for radiography in trauma. [Stiell IG, Wells GA: Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med April 1999;33:437-447.]

Section snippets

INTRODUCTION

Reports of clinical decision rules are becoming increasingly common throughout the medical literature and particularly within emergency medicine journals. Clinical decision rules (prediction rules) are designed to help physicians with diagnostic and therapeutic decisions at the bedside. We define a clinical decision rule as a decisionmaking tool that is derived from original research (as opposed to a consensus-based clinical practice guideline) and incorporates 3 or more variables from the

1. IS THERE A NEED FOR THE DECISION RULE?

Clinicians should ask themselves whether there really appears to be a need for a particular decision rule, or whether the rule appears only to represent the analysis of a convenient set of data. Is there a demonstrated inefficiency or variation in current medical practice, and does there appear to be the potential for improved efficiency through guidelines or a decision rule?

2. WAS THE RULE DERIVED ACCORDING TO METHODOLOGIC STANDARDS?

Research methodology standards for the derivation of a clinical decision rule were first reviewed in 1985 in a landmark paper by Wasson et al.9 Feinstein,10 one of the fathers of clinical epidemiology, later added to the literature of evidence-based patient assessment in his book Clinimetrics . More recently, our research group at the University of Ottawa Clinical Epidemiology Unit assessed clinical decision rule articles in 4 major medical journals and proposed modifications to Wasson et al’s

3. HAS THE RULE BEEN PROSPECTIVELY VALIDATED AND REFINED?

Unfortunately, many clinical decision rules are not prospectively assessed to determine, in a new patient population, their accuracy, reproducibility, acceptance to clinicians, or potential effect on practice. This validation process is very important because many statistically derived rules or guidelines fail to perform well when tested in a new population.53, 54, 55 The reason for this poor performance may be statistical—overfitting or instability in the original derived model56—or may be

4. HAS THE RULE BEEN SUCCESSFULLY IMPLEMENTED INTO CLINICAL PRACTICE?

Very few clinical decision rules have been implemented and shown to alter clinical practice in what has been termed “the next painful step” for evaluating decision aids.69 A decision rule that has been shown to be valid and reliable still faces significant barriers to implementation in terms of both patient and physician acceptance.48, 49, 70 Emergency physicians are concerned about the medicolegal consequences of missing a diagnosis. Patients must place their trust in a physician whom they

5. WOULD USE OF THE RULE BE COST-EFFECTIVE?

If an implementation trial does, in fact, show that the decision rule alters clinical behavior, then a formal economic evaluation conducted by a health economist might be conducted.74, 75 The objective of such a study would be to clearly demonstrate the health care savings that might be associated with widespread use of the decision rule. Economic assessment is concerned with choosing between alternative uses of resources. Resources are limited and choices must be made.

Three basic concepts are

6. HOW WILL THE RULE BE DISSEMINATED AND IMPLEMENTED?

For a clinical guideline to have widespread effect on health care delivery, there must be an active plan for dissemination and implementation. We are well aware that the simple passive diffusion of original study results (through publication in medical journals or presentation at scientific meetings) is unlikely to significantly alter clinical practice in emergency medicine or in any other discipline.76, 77 Strategies to ensure the dissemination and implementation of clinical research are

References (88)

  • J Svenson

    Need for radiographs in the acutely injured ankle

    Lancet

    (1988)
  • EC Pigman et al.

    Evaluation of the Ottawa clinical decision rules for the use of radiography in acute ankle and midfoot injuries in the emergency department: An independent site assessment

    Ann Emerg Med

    (1994)
  • GM Lucchesi et al.

    Sensitivity of the Ottawa Rules

    Ann Emerg Med

    (1995)
  • AH Anis et al.

    Cost-effectiveness analysis of the Ottawa ankle rules

    Ann Emerg Med

    (1995)
  • JM Grimshaw et al.

    Effect of clinical guidelines on medical practice: A systematic review of rigorous evaluations

    Lancet

    (1993)
  • A Laupacis et al.

    Clinical prediction rules: A review and suggested modifications of methodological standards

    JAMA

    (1997)
  • MJ Fine et al.

    A prediction rule to identify low-risk patients with community-acquired pneumonia

    N Engl J Med

    (1997)
  • IG Stiell et al.

    Decision rules for the use of radiography in acute ankle injuries: Refinement and prospective validation

    JAMA

    (1993)
  • CJ McDonald

    Guidelines you can follow and can trust: An ideal and an example

    JAMA

    (1994)
  • IG Stiell et al.

    Implementation of the Ottawa ankle rules

    JAMA

    (1994)
  • JH Wasson et al.

    Clinical predition rules: Have they come of age?

    JAMA

    (1996)
  • I Stiell et al.

    A multicentre trial to introduce clinical decision rules for the use of radiography in acute ankle injuries

    BMJ

    (1995)
  • JH Wasson et al.

    Clinical prediction rules: Application and methodological standards

    N Engl J Med

    (1985)
  • AR Feinstein

    Clinimetrics

    (1987)
  • IG Stiell

    Cranial computed tomography for nontrauma patients: Critical appraisal of a decision rule

    Acad Emerg Med

    (1997)
  • LF McCaig

    National Hospital Ambulatory Medical Care Survey: 1992 emergency department summary

    Advance Data

    (1994)
  • National Center for Health Statistics

    National Hospital Ambulatory Medical Care Survey 1992

    (1994)
  • IG Stiell et al.

    Use of radiography in acute ankle injuries: Physicians’ attitudes and practice

    Can Med Assoc J

    (1992)
  • IG Stiell et al.

    Use of radiography in acute knee injures: Need for clinical decision rules

    Acad Emerg Med

    (1995)
  • IG Stiell et al.

    Variation in emergency department use of cervical spine radiography for alert, stable trauma patients

    Can Med Assoc J

    (1997)
  • ID Graham et al.

    Emergency physicians’ attitudes toward the use of clinical decison rules for radiography

    Acad Emerg Med

    (1998)
  • C Madden et al.

    High-yield selection criteria for cranial computed tomography after acute trauma

    Acad Emerg Med

    (1995)
  • MS Kramer et al.

    Clinical biostatistics: LIV. The biostatistics of concordance

    Clin Pharmacol Ther

    (1982)
  • JR Landis et al.

    The measurement of observer agreement for categorical data

    Biometrics

    (1977)
  • JL Fleiss

    Statistical Methods for Rates and Proportions

    (1981)
  • C Roberts et al.

    A matrix of kappa-type coefficients to assess the reliability of nominal scales

    Stat Med

    (1998)
  • JL Fleiss

    The design and analysis of chemical experiments

  • VF Flack et al.

    Sample size determinations for the two rater kappa statistics

    Psychometrika

    (1988)
  • SD Walter et al.

    Sample size and optimal designs for reliability studies

    Stat Med

    (1998)
  • A Donner

    Sample size requirements for the comparison of two or more coefficients of interobserver agreement

    Stat Med

    (1998)
  • A Donner et al.

    Testing the homogeneity of kappa statistics

    Biometrics

    (1996)
  • VF Flack et al.

    Sample size determinations using logistic regression with pilot data

    Stat Med

    (1993)
  • FY Hsieh

    Sample size tables for logistic regression

    Stat Med

    (1989)
  • JHL Friedman

    A recursive partitioning decision rule for nonparametric classification

    IEEE Trans Comput

    (1977)
  • Cited by (0)

    Dr Stiell is a career scientist of the Medical Research Council of Canada.

    ☆☆

    Address for reprints:Ian G Stiell, MD, MSc, FRCPC, Clinical Epidemiology Unit, Ottawa Hospital Loeb Health Research Institute, 1053 Carling Avenue, Ottawa, Ontario, Canada, K1Y 4E9.

    0196-0644/99/$8.00 + 0

    ★★

    47/1/94993

    View full text