Article Text

Teaching about how doctors think: a longitudinal curriculum in cognitive bias and diagnostic error for residents
  1. James B Reilly1,
  2. Alexis R Ogdie2,
  3. Joan M Von Feldt3,
  4. Jennifer S Myers4
  1. 1Division of Nephrology, Department of Medicine, Allegheny General Hospital, West Penn Allegheny Health System, Pittsburgh, Pennsylvania, USA
  2. 2Division of Rheumatology, Department of Medicine, Center for Epidemiology and Biostatistics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, USA
  3. 3Division of Rheumatology, Department of Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, USA
  4. 4Division of General Internal Medicine, Department of Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, USA
  1. Correspondence to Dr James B Reilly, Allegheny General Hospital, 320 E. North Ave., 7th Floor South Tower, Pittsburgh, PA 15212, USA; jreilly1{at}


Background Trends in medical education have reflected the patient safety movement's initial focus on systems. While the role of cognitive-based diagnostic errors has been increasingly recognised among safety experts, literature describing strategies to teach about this important problem is scarce.

Methods 48 PGY-2 internal medicine residents participated in a three-part, 1-year curriculum in cognitive bias and diagnostic error. Residents completed a multiple-choice test designed to assess the recognition and knowledge of common heuristics and biases both before and after the curriculum. Results were compared with PGY-3 residents who did not receive the curriculum. An additional assessment in which residents reviewed video vignettes of clinical scenarios with cognitive bias and debiasing techniques was embedded into the curriculum.

Results 38 residents completed all three parts of the curriculum and completed all assessments. Performance on the 13-item multiple-choice knowledge test improved post-curriculum when compared to both pre-curriculum performance (9.26 vs 8.26, p=0.002) and the PGY-3 comparator group (9.26 vs 7.69, p<0.001). All residents correctly identified at least one cognitive bias and proposed at least one debiasing strategy in response to the videos.

Conclusions A longitudinal curriculum in diagnostic error and cognitive bias improved internal medicine residents’ knowledge and recognition of cognitive biases as measured by a novel assessment tool. Further study is needed to refine learner assessment tools and examine optimal strategies to teach clinical reasoning and cognitive bias avoidance strategies.

  • Cognitive biases
  • Continuing education, continuing professional development
  • Diagnostic errors
  • Graduate medical education
  • Medical education

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.


Since the Institute of Medicine's report, ‘To Err is Human’, efforts to improve patient safety by focusing on system improvements have been at the forefront of healthcare.1 ,2 Although 10–20% of medical errors are related to a diagnostic error,3 ,4 they have received less attention in the patient safety movement because they are infrequently reported and difficult to remedy. Unlike systems errors, which are highly visible, diagnostic errors are frequently ‘invisible’. Additionally, since many are caused in whole or in part by how physicians think,5 ,6 efforts to improve them seem beyond the reach of traditional quality and safety improvement efforts.7–9

Not surprisingly, patient safety education for medical students and residents has reflected the initial focus of the patient safety movement and emphasised a systems approach to medical error analysis.10–12 This approach remains relevant and important, but it is equally important for trainees to appreciate and learn how errors in clinical reasoning, also called cognitive errors, can contribute to misdiagnosis. Experts have called for greater attention to be paid to the science of clinical decision-making and diagnostic reasoning, including the ‘dual-process’ model and the concepts of heuristics and bias,13–18 as a means to better understand the cognitive aspects of diagnostic errors. While controversy may exist regarding the best way to educate physicians about these concepts, educating physicians about the science of decision-making, especially during medical school and residency when trainees are still forming clinical habits, may have potential to reduce diagnostic error. More recently, the educational community is responding to this perceived need, and there is renewed interest in designing curricula to teach trainees about this topic.19–21 A recent review by Graber et al7 has summarised the literature on cognitive strategies to reduce diagnostic errors among practitioners and trainees, yet published examples of well executed, longitudinal curricula (more than a single session) on cognitive bias and diagnostic errors that are integrated into patient safety training remain scarce.

In response we designed a longitudinal curriculum in cognitive bias and diagnostic error for internal medicine residents that utilised multiple educational strategies. Our overall goal is to increase residents’ awareness of their own clinical decision-making behaviours, with an emphasis on the use of heuristics. The learning objectives in our curriculum were to raise awareness of the prevalence of diagnostic errors, differentiate cognitive from systems errors and describe the interplay between the two, and identify common cognitive biases in clinical practice. Additionally, residents were asked to reflect on their own diagnostic errors or delays in diagnosis, and those of others, and to articulate debiasing strategies.

The objective of this study was to examine the ability of the curriculum to improve knowledge and recognition of cognitive bias. It was hypothesised that residents who participated in the curriculum would demonstrate improved knowledge of the contributions of cognitive bias to diagnostic error, recognise these biases and heuristics within simulated clinical vignettes and suggest strategies to avoid bias in the future.


Study population

Forty-eight PGY-2 internal medicine residents from the Perelman School of Medicine at the University of Pennsylvania participated in a required curriculum in diagnostic error and cognitive bias in 2010. Twenty-two were female, 36 were in the categorical internal medicine programme, nine in the primary care programme, and four in the physician-scientist programme. Eleven reported having additional degrees: five PhD and six Masters degrees. Forty-two, PGY-3 internal medicine residents, who had recently completed the PGY-2 years in our programme but did not receive the curriculum, were a contemporaneous comparator group. Of these, 29 were female, 39 were in the categorical programme, three were in the primary care programme, and none were in the physician-scientist programme (since those residents enter fellowship after PGY-2). Seven reported additional degrees (all Masters). Participants were required to receive the curriculum but were free to decline study inclusion. The Institutional Review Board of the University of Pennsylvania approved this study.


The curriculum consisted of three parts completed in sequence over one academic year (figure 1). Part one consisted of a 40-min lecture and a 20-min small group, case-based discussion. The lecture, led by two authors (JM and JVF), framed the problem of diagnostic errors in patient safety, introduced the dual-process model of medical decision making,18 contrasted cognitive errors from systems errors, and defined several common heuristics and cognitive biases. The small group discussions occurred in nine groups of 5–6 residents led by trained faculty facilitators. Each group read a case of a delayed diagnosis while facilitators helped the residents identify and differentiate systems and cognitive errors using a modified root cause analysis discussion format. The authors, as well as several faculty and chief residents trained by one of the authors (JR), served as facilitators.

Figure 1

Timeline and curriculum description. RCA, root cause analysis; D-KAT, Diagnostic error Knowledge Assessment Test.

Part two occurred 4 months later, consisting of a brief review of the lecture material presented in part one followed by a 1-h small group session. In this session, each resident was asked to reflect and write a brief narrative about a clinical experience in which a cognitive bias led to a missed or delayed diagnosis. A semi-structured discussion followed in which each resident read his or her narrative aloud and the group discussed cognitive biases, contextual factors surrounding the diagnostic error, and strategies that could have been employed to recognise and counteract the bias before the error was made. Given the sensitive nature of the topic and semi-structured format of the discussion, small group facilitators were trained in a 1-h session consisting of a review of the core content and a ‘mock’ narrative session in which potentially difficult or distracting responses from residents were role-played by the authors. Facilitators were also provided with a written guide created by one of the authors (JR) that included a script for the discussion and suggestions on how to anticipate and respond to problematic scenarios.

The third portion of the curriculum was an online module that the residents completed at the end of the academic year. The module consisted of three discrete components. The first component required residents to read a short case and then complete an interactive matching activity to identify, differentiate and categorise cognitive errors from systems errors using a modified fishbone diagram created by the investigators for the curriculum. The second component employed three linked video vignettes that depicted clinical scenarios in an emergency room, an outpatient primary care office and the hospital (table 1). Videos were created by the authors and utilised one standardised patient and six volunteer physician actors.

Table 1

Video-based clinical vignette question items for cognitive bias and diagnostic error curriculum

Assessment tools

Since no published tools exist to assess physician knowledge of cognitive biases in medicine, one was designed. The Diagnostic error Knowledge Assessment Test (D-KAT, see online supplementary appendix 1) was a 13-item, single-best-answer, multiple-choice test. Five items tested general knowledge of diagnostic error prevalence, cognitive contributions to diagnostic error, definition of heuristics, and the dual processing model of decision making. Eight items consisted of short clinical vignettes that were written to highlight a single cognitive bias that was taught in the curriculum. The test was piloted with six internal medicine residents and two medical students and revised in response to feedback. The D-KAT was completed by the residents immediately before part one of the curriculum and embedded in the third part of the curriculum as the final component of the online module, serving as the post-test. Internal consistency was measured using Cronbach's α (0.42).

After each of the three video vignettes (table 1), residents were asked to type a free-text response to a question that related to the preceding segment. The first question, administered immediately after presentation of video 1, asked residents to identify three cognitive biases illustrated in the video (at least eight were present). The second question, administered immediately after presentation of video 2, asked the residents to explain how the practitioner exhibited diagnostic momentum in the video. The third question, administered immediately after presentation of video 3, asked the residents to identify the debiasing strategies employed by the physician actors. The web module was programmed such that residents could not go back in the module and revise their answer to a previous question.


The primary outcome was performance on the post-curriculum D-KAT, compared with the pre-curriculum score on the same tool. Post-curriculum scores were also compared to the comparator group of PGY-3 residents who completed the D-KAT without receiving the curriculum. The comparator and post-curriculum groups completed the test at the same point in their academic training (end of second year), minimising the maturation effect and validating our comparator group. Accuracy of resident responses to the simulated case questions in the web-based module was examined as a secondary outcome.

Pre–post differences in residents’ scores for the post-curriculum D-KAT, as compared to their pre-curriculum scores, were tested for significance using a paired t test. The difference between the median of the post-curriculum scores to the median score of the PGY-3 comparator group was tested for significance using the Wilcoxon rank sum test. Short answers from the web curriculum were graded independently by two authors (JR and AO). Inter-rater agreement was calculated using Cohen's κ statistic (κ 0.73, 0.64 and 0.25 for questions 1, 2 and 3, respectively). All discrepancies were resolved by subsequent discussion between the two graders.

Qualitative analysis was performed on the written narratives and small group discussions in part two of the curriculum, and the results of this analysis are described elsewhere.22


All those who participated in the curriculum and comparator groups did provide informed consent for the study. Of the 48 PGY-2 residents enrolled, 38 (79%) completed all three components of the longitudinal curriculum and were included in the analysis. Of those who did not complete the entire curriculum, two were not present for part one of the curriculum, five did not complete part two or three of the curriculum due to family leave or early entrance into fellowship, and the remainder were otherwise unable to complete part three of the curriculum (web module) by the close of the study despite multiple reminders.

Of the 38 residents who completed the entire curriculum, the mean score on the post-curriculum 13-item, multiple-choice knowledge test was 9.26 (SD 1.45; median 9) compared to a pre-curriculum mean of 8.26 (SD 1.45; median 8, t=−3.01, df=74, p=0.002) and the PGY-3 group mean of 7.69 (SD 2.01, median 8, z=−3.73, p=0.0002).

In the video vignette short-answer assessment, all residents (100%) were able to correctly identify at least one bias, 36 (95%) identified two biases and 23 (61%) identified three biases. Thirty-four residents (89%) accurately described how the physician in the second video exhibited diagnostic momentum. Lastly, all residents were able to identify at least one appropriate debiasing strategy, 37 (97%) identified at least two strategies and 23 (61%) participants correctly identified three or more strategies.


We describe a model for a year-long, longitudinal curriculum in diagnostic errors and cognitive bias, administered to internal medicine residents in their second year of training. Residents who participated in this curriculum improved their recognition and knowledge of common cognitive biases and heuristics, measured by a novel assessment tool. This improvement was likely to be related to the curriculum as opposed to the accumulation of clinical experience, as a comparator group from the previous residency class performed less well. Furthermore, in the final component of this curriculum, participants demonstrated the ability to apply this knowledge to identify both cognitive biases and potential debiasing strategies in simulated clinical scenarios.

Developing awareness of heuristics could decrease diagnostic errors by allowing physicians to recognise them and implement debiasing strategies to prevent errors. This study builds on the growing body of literature describing cognitive interventions to reduce diagnostic error.7 Previous studies include interventions that incorporate meta-cognition and reflective practice,23–25 as well as interventions that teach cognitive ‘forcing’25–28 or other strategies29 to help learners increase awareness of their own cognitive processing and recognise shortcomings. All of these studies were inconclusive in terms of the effectiveness of education to reduce diagnostic errors. These cognitive interventions meant to improve clinical reasoning7 are largely ‘stand alone’ experiences; studies of the execution of longitudinal curricula designed to help learners recognise common heuristics and biases that could contribute to diagnostic error have not been published.

Our educational intervention has several strengths. First, the intervention was embedded within the trainees’ required curriculum, not as a ‘stand alone’ experience. Second, we used several teaching methods including video vignettes, reflective narrative writing, and several case-based activities designed to help learners differentiate the systems and cognitive components of diagnostic error. Third, we delivered our curriculum with a core group of faculty who received training in conducting the sessions, due to the sensitive nature of many diagnostic errors. Finally, our unique approach was designed to be executed longitudinally, showing greater concordance with educational principles of adult learning30 by allowing for a graduated, repetitive exposure to the cognitive psychology of clinical decision making errors in a psychologically safe environment (their peer group, facilitated by trusted faculty). Merriam et al describe three keys to transformational learning: experience, critical reflection and development. Our longitudinal curriculum provided: (a) a learning opportunity relevant and applicable to the residents’ set of experiences, (b) a component of critical reflection, and (c) a critical thinking exercise with the web curriculum and D-KAT. Merriam et al30 note that ‘the ability to think critically, which is mandatory to … transformation, is itself developmental’. We chose second year residents as the target for this curriculum because they are entering a critical period in their development, in which they begin to assume the role of clinical team leader and make more independent diagnostic decisions. Although even earlier training in the cognitive psychology of misdiagnosis is needed in medical education, our intent was that this training would influence the diagnostic skills of the residents during a period where medical decision-making habits are being formed and solidified.

This study has limitations that raise additional questions and offer future directions for educators and researchers in the field of diagnostic errors and clinical reasoning. This study was performed in one residency programme at a single institution and had a relatively small number of participants, which may limit the generalisability of our findings. We did not formally assess resident satisfaction with the curriculum. All of the educational interventions and assessments occurred in a classroom or web-based, simulated environment as opposed to an actual clinical practice setting. Therefore, although we hope that the knowledge and skills gained in these settings will be transferred to clinical practice, this was not measured. We attempted to increase the clinical relevance and knowledge transfer of this topic by having our learners write and discuss personal stories about their real experiences with diagnostic errors.22 Next, because our video vignettes were integrated formally into the curriculum as an instructional tool, we could not assess the comparator group using the videos. Our primary assessment method was the D-KAT, administered before and after the curriculum. Although the pre- and post-tests were administered 9 months apart, familiarity with the instrument may have contributed to improved performance. Further, it is possible that pre-curriculum performance on the D-KAT reflected a considerable baseline understanding of cognitive errors, given the relatively modest improvement after the completion of the curriculum. The D-KAT instrument had a low internal consistency (Cronbach's α=0.42) and has not been previously validated. We theorise that the observed α is at least in part a reflection of the multiple constructs that the test evaluates and the relatively small sample size in this pilot study. Further refinement and testing of the instrument is required in order to determine if this tool can accurately measure knowledge of heuristics, cognitive bias and related concepts in diagnostic error, and iterative refinement cycles could improve the test's psychometric properties.

Mindful of the strengths of the intervention and the limitations of our study, we propose several next steps. We would continue to revise individual items within the D-KAT to improve their psychometric properties and the internal consistency of the instrument. Next, having utilised our video vignettes both as learning tools and as assessment tools in this study, we found them to have great potential. Future curricula in diagnostic errors should utilise and validate video-based question items as assessment tools by utilising an intervention and control group. Additionally, we feel that the concepts of heuristics, bias and diagnostic error are not unique to internal medicine, and therefore the clinical scenarios used in our curriculum should also be adapted to meet the needs of other medical specialties.

Diagnostic errors are difficult to measure, and previous study of cognitive interventions among clinicians are inconclusive.7 Controversy exists regarding the utility of teaching about these issues as a direct means of improving diagnosis, given the subconscious nature of bias and heuristics. However, we do believe that teaching trainees about their cognitive vulnerabilities and promoting reflection on their past mistakes are essential to the ongoing development of good diagnostic habits, providing a long term benefit that is difficult to measure. We think it is also likely to increase trainees’ willingness to adopt other tools that have shown early promise in reducing diagnostic errors. For example, newer studies of checklists have shown potential to improve diagnostic skills such as ECG interpretation,31 ,32 and this benefit may actually be larger for trainees compared to seasoned clinicians.33

Education in the field of diagnostic errors is nascent. More research is needed to understand how to develop and improve clinical reasoning across the educational continuum and how to measure the effects of these efforts on diagnostic accuracy. Given the prevalence of diagnostic errors in medicine and their underlying cognitive causal factors, we believe that it is critical for educators to include this content in their patient safety curriculum and develop novel ways to teach it. To meet this need, we offer this longitudinal curriculum as a model for integrating instruction on cognitive bias and diagnostic error into residency training.


We wish to thank Lisa Bellini, MD and the Penn Internal Medicine Residents who participated in the curriculum. We are also grateful to the following colleagues who participated in our faculty development sessions and served as small group facilitators throughout the curriculum: Matt Rusk, MD, Steve Gluckman, MD, Karen Warburton, MD, Todd Barton, MD, David Aizenberg, MD, Stephen Kim, MD and Jodi Lenko, MD. Finally, we thank Jennifer Kogan, MD, who participated as faculty and also provided feedback on an earlier version of this manuscript, and Judy Shea, PhD for her advice on study design.


Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Files in this Data Supplement:


  • Contributors ARO provided particular expertise for the statistical analysis and description of the results and methods, in addition to providing revisions in all other sections. JMvF provided substantial assistance in the initial writing and revision of every section of the paper, especially the discussion. JSM wrote the introduction section and provided substantial revisions throughout each section, including the discussion. JBR was responsible for the discussion, provided substantial revisions in all sections, and assumed responsibility for overall coherence of the manuscript, submission of the manuscript and ongoing correspondence and revisions. The final submission is approved by each author.

  • Funding This work was supported by the Sam Martin Education Pilot Award from the Division of General Internal Medicine at the Perelman School of Medicine at the University of Pennsylvania. ARO is supported by the American College of Rheumatology Research and Education Foundation Rheumatology Investigator Award. During this project, JBR was supported by NIH Institutional Training Grant T32-DK 07006-37 and the Center for Healthcare Improvement and Patient Safety at the University of Pennsylvania. JSM is supported in part by a grant from the Josiah Macy Jr. Foundation.

  • Ethics approval This study was approved by the Institutional Review Board of the University of Pennsylvania.

  • Provenance and peer review Not commissioned; externally peer reviewed.