The practice of emergency medicine/original research
The Presence of Outcome Bias in Emergency Physician Retrospective Judgments of the Quality of Care

https://doi.org/10.1016/j.annemergmed.2010.10.004Get rights and content

Study objective

In peer review and malpractice litigation, biased assessment of the quality of care can have a profound effect. We determine the effect of knowledge of outcome on emergency physicians' ability to assess care quality.

Methods

Emergency physicians completed a Web-based survey containing 6 case scenarios written to fall along a spectrum of quality of care. Participants were randomized to receive either no case outcomes or a mixture of good and bad outcomes. For each scenario, participants rated the quality of care categorically (poor, below average, average, good, outstanding) and on a 0- to 100-point scale. We examined how the scenario's outcome affected judgments about the quality of the process of care and whether certain individuals are more prone to outcome bias.

Results

Five hundred eighty-seven participants completed the survey. For each scenario, quality ratings were highest when the outcome was good and lowest when the outcome was bad. The difference between ratings for “good outcome” and “no outcome provided” was bigger than the difference between “no outcome provided” and “bad outcome.” In cases of intermediate quality, outcome bias shifts ratings by a magnitude equivalent to 1 qualitative step in quality (eg, from good to average). The outcome bias effect is smaller for scenarios for which care is unambiguously good or bad. We found no evidence that outcome bias was concentrated in individuals.

Conclusion

Emergency physicians demonstrate outcome bias in cases of intermediate quality more than in cases in which the quality of care is clear. Outcome bias tends to inflate ratings in the presence of a positive outcome more than it penalizes scenarios with negative ones.

Introduction

In 1980, Donadebian1 proposed the triad of structure, process, and outcome to characterize the quality of medical care and argued that process measures were essential because “the most direct route to an assessment of the quality of care is an examination of that care.” Despite 30 years of contentious arguments about the relative merits of process and outcome measures of quality, whether the perspective is medical quality, legal culpability, or cost-effectiveness, process measures indisputably have an important role in judging the quality of medical care.

With regard to certain measures (eg, was the patient with acute myocardial infarction given an aspirin?), judging the process of care may be straightforward. However, as judgments become more subjective (eg, did the physician appreciate the laxity of the anterior cruciate ligament?), they become more vulnerable to a series of biases. The first of these is the human tendency to reduce complex processes to individual errors in cognition.2 The second is that context can bias raters; a physician may be reluctant to find fault with a colleague's care, whereas a plaintiff's attorney may be more than willing to do so. Third, knowledge of the patient's outcome may impede a rater's ability to judge process objectively. Outcome bias, the propensity to assign blame more readily when the outcome is bad, has been described in radiology, anesthesiology, internal medicine, and surgery.3, 4, 5, 6, 7

In certain peer review and medicolegal situations, the biased assessment of quality of the process of care can have profound negative emotional, occupational, and financial consequences for the treating physician who may be inappropriately blamed for morbidity or mortality beyond his or her control. Given that assessments of process are more often triggered by the occurrence of a bad outcome than by the occurrence of a good one, and given the high-risk nature of emergency medicine, we thought it important to determine the extent to which outcome bias can alter assessments of the quality of process of care in emergency medicine. We designed a study to determine to what extent knowledge of outcome affects emergency physicians' ability to assess the quality of the process of care, and what factors affect degree of outcome bias.

Section snippets

Materials and Methods

We performed a Web-based survey of members of the American Academy of Emergency Medicine. A list of potential participants was generated by academy staff, who subsequently sent out an e-mail solicitation for participation. The e-mail informed subjects that they had been randomly selected to aid in a study of quality assurance. Participants were then given a hyperlink to 6 brief clinical scenarios and asked to judge the quality of care provided.

All scenarios were created by the authors, and each

Results

We had originally planned to send our request to a sample of American Academy of Emergency Medicine members, but because of a miscommunication the request for participation, in the form of a single e-mail, was sent to all 4,785 members. After 2 weeks, we checked the database and, finding 622 responses, stopped enrollment. We dropped 35 incomplete sessions, leaving 587 complete cases for evaluation. Eighty-five percent of respondents were men. Thirty-two percent were younger than 40 years, 34%

Limitations

This study has a number of limitations, the most notable being that we used hypothetic scenarios instead of actual cases. Furthermore, to reduce respondent burden we provided brief synopses of the cases rather than actual medical records. It is possible that respondents given a complete record would interpret the facts differently. For example, when given a case with a bad outcome, a rater might scrutinize the entire record, searching for clues that should have alerted the physician that the

Discussion

Whether care was good, bad, or indeterminate, retrospective assessments of the quality of the process of care should be unaffected by the patient's outcome. Unfortunately, those who judge the quality of care are seldom unaware of that outcome, and our study shows that emergency physicians' judgments of the process of care are affected by their knowledge of the outcome. In cases of intermediate quality, outcome bias shifts ratings by a magnitude equivalent to 1 qualitative step in quality (eg,

References (10)

  • R.L. Wears et al.

    Replacing hindsight with insight: toward better understanding of diagnostic failures

    Ann Emerg Med

    (2007)
  • A. Donabedian

    Methods for deriving criteria for assessing the quality of medical care

    Med Care Rev

    (1980)
  • L. Berlin

    Outcome bias

    AJR Am J Roentgenol

    (2004)
  • T.A. Brennan et al.

    Incidence of adverse events and negligence in hospitalized patientsResults of the Harvard Medical Practice Study I

    N Engl J Med

    (1991)
  • T.A. Brennan et al.

    Relation between negligent adverse events and the outcomes of medical-malpractice litigation

    N Engl J Med

    (1996)
There are more references available in the full text version of this article.

Cited by (0)

Supervising editor: Donald M. Yealy, MD

Dr. Yealy was the supervising editor on this article. Dr. Schriger did not participate in the editorial review or decision to publish this article.

Author contributions: MG and JAT conceived of the study. MG, DLS, and JAT designed the study. MG and DLS performed the data collection and statistical analyses. MG drafted the article, and all authors contributed substantially to its revision. MG was responsible for data collection and analysis and takes responsibility for the paper as a whole.

Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article that might create any potential conflict of interest. See the Manuscript Submission Agreement in this issue for examples of specific conflicts covered by this statement. Dr. Schriger is funded in part by an unrestricted grant from the Korein Foundation.

Publication dates: Available online January 12, 2011.

Earn CME Credit: Continuing Medical Education is available for this article at: www.ACEP-EMedhome.com.

Reprints not available from the authors.

Please see page 324 for the Editor's Capsule Summary of this article.

Provide feedback on this article at the journal's Web site, www.annemergmed.com.

A podcast for this article is available at www.annemergmed.com.

View full text