You are here

Article Text

Article menu
Innovations in education
Teaching about how doctors think: a longitudinal curriculum in cognitive bias and diagnostic error for residents
  1. James B Reilly1,
  2. Alexis R Ogdie2,
  3. Joan M Von Feldt3,
  4. Jennifer S Myers4
  1. 1Division of Nephrology, Department of Medicine, Allegheny General Hospital, West Penn Allegheny Health System, Pittsburgh, Pennsylvania, USA
  2. 2Division of Rheumatology, Department of Medicine, Center for Epidemiology and Biostatistics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, USA
  3. 3Division of Rheumatology, Department of Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, USA
  4. 4Division of General Internal Medicine, Department of Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania, USA
  1. Correspondence to Dr James B Reilly, Allegheny General Hospital, 320 E. North Ave., 7th Floor South Tower, Pittsburgh, PA 15212, USA; jreilly1{at}wpahs.org

Abstract

Background Trends in medical education have reflected the patient safety movement's initial focus on systems. While the role of cognitive-based diagnostic errors has been increasingly recognised among safety experts, literature describing strategies to teach about this important problem is scarce.

Methods 48 PGY-2 internal medicine residents participated in a three-part, 1-year curriculum in cognitive bias and diagnostic error. Residents completed a multiple-choice test designed to assess the recognition and knowledge of common heuristics and biases both before and after the curriculum. Results were compared with PGY-3 residents who did not receive the curriculum. An additional assessment in which residents reviewed video vignettes of clinical scenarios with cognitive bias and debiasing techniques was embedded into the curriculum.

Results 38 residents completed all three parts of the curriculum and completed all assessments. Performance on the 13-item multiple-choice knowledge test improved post-curriculum when compared to both pre-curriculum performance (9.26 vs 8.26, p=0.002) and the PGY-3 comparator group (9.26 vs 7.69, p<0.001). All residents correctly identified at least one cognitive bias and proposed at least one debiasing strategy in response to the videos.

Conclusions A longitudinal curriculum in diagnostic error and cognitive bias improved internal medicine residents’ knowledge and recognition of cognitive biases as measured by a novel assessment tool. Further study is needed to refine learner assessment tools and examine optimal strategies to teach clinical reasoning and cognitive bias avoidance strategies.

  • Cognitive biases
  • Continuing education, continuing professional development
  • Diagnostic errors
  • Graduate medical education
  • Medical education

https://doi.org/10.1136/bmjqs-2013-001987

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Since the Institute of Medicine's report, ‘To Err is Human’, efforts to improve patient safety by focusing on system improvements have been at the forefront of healthcare.1 ,2 Although 10–20% of medical errors are related to a diagnostic error,3 ,4 they have received less attention in the patient safety movement because they are infrequently reported and difficult to remedy. Unlike systems errors, which are highly visible, diagnostic errors are frequently ‘invisible’. Additionally, since many are caused in whole or in part by how physicians think,5 ,6 efforts to improve them seem beyond the reach of traditional quality and safety improvement efforts.7–9

    Not surprisingly, patient safety education for medical students and residents has reflected the initial focus of the patient safety movement and emphasised a systems approach to medical error analysis.10–12 This approach remains relevant and important, but it is equally important for trainees to appreciate and learn how errors in clinical reasoning, also called cognitive errors, can contribute to misdiagnosis. Experts have called for greater attention to be paid to the science of clinical decision-making and diagnostic reasoning, including the ‘dual-process’ model and the concepts of heuristics and bias,13–18 as a means to better understand the cognitive aspects of diagnostic errors. While controversy may exist regarding the best way to educate physicians about these concepts, educating physicians about the science of decision-making, especially during medical school and residency when trainees are still forming clinical habits, may have potential to reduce diagnostic error. More recently, the educational community is responding to this perceived need, and there is renewed interest in designing curricula to teach trainees about this topic.19–21 A recent review by Graber et al7 has summarised the literature on cognitive strategies to reduce diagnostic errors among practitioners and trainees, yet published examples of well executed, longitudinal curricula (more than a single session) on cognitive bias and diagnostic errors that are integrated into patient safety training remain scarce.

    In response we designed a longitudinal curriculum in cognitive bias and diagnostic error for internal medicine residents that utilised multiple educational strategies. Our overall goal is to increase residents’ awareness of their own clinical decision-making behaviours, with an emphasis on the use of heuristics. The learning objectives in our curriculum were to raise awareness of the prevalence of diagnostic errors, differentiate cognitive from systems errors and describe the interplay between the two, and identify common cognitive biases in clinical practice. Additionally, residents were asked to reflect on their own diagnostic errors or delays in diagnosis, and those of others, and to articulate debiasing strategies.

    The objective of this study was to examine the ability of the curriculum to improve knowledge and recognition of cognitive bias. It was hypothesised that residents who participated in the curriculum would demonstrate improved knowledge of the contributions of cognitive bias to diagnostic error, recognise these biases and heuristics within simulated clinical vignettes and suggest strategies to avoid bias in the future.

    Methods

    Study population

    Forty-eight PGY-2 internal medicine residents from the Perelman School of Medicine at the University of Pennsylvania participated in a required curriculum in diagnostic error and cognitive bias in 2010. Twenty-two were female, 36 were in the categorical internal medicine programme, nine in the primary care programme, and four in the physician-scientist programme. Eleven reported having additional degrees: five PhD and six Masters degrees. Forty-two, PGY-3 internal medicine residents, who had recently completed the PGY-2 years in our programme but did not receive the curriculum, were a contemporaneous comparator group. Of these, 29 were female, 39 were in the categorical programme, three were in the primary care programme, and none were in the physician-scientist programme (since those residents enter fellowship after PGY-2). Seven reported additional degrees (all Masters). Participants were required to receive the curriculum but were free to decline study inclusion. The Institutional Review Board of the University of Pennsylvania approved this study.

    Intervention

    The curriculum consisted of three parts completed in sequence over one academic year (figure 1). Part one consisted of a 40-min lecture and a 20-min small group, case-based discussion. The lecture, led by two authors (JM and JVF), framed the problem of diagnostic errors in patient safety, introduced the dual-process model of medical decision making,18 contrasted cognitive errors from systems errors, and defined several common heuristics and cognitive biases. The small group discussions occurred in nine groups of 5–6 residents led by trained faculty facilitators. Each group read a case of a delayed diagnosis while facilitators helped the residents identify and differentiate systems and cognitive errors using a modified root cause analysis discussion format. The authors, as well as several faculty and chief residents trained by one of the authors (JR), served as facilitators.

    Figure 1
    Figure 1

    Timeline and curriculum description. RCA, root cause analysis; D-KAT, Diagnostic error Knowledge Assessment Test.

    Part two occurred 4 months later, consisting of a brief review of the lecture material presented in part one followed by a 1-h small group session. In this session, each resident was asked to reflect and write a brief narrative about a clinical experience in which a cognitive bias led to a missed or delayed diagnosis. A semi-structured discussion followed in which each resident read his or her narrative aloud and the group discussed cognitive biases, contextual factors surrounding the diagnostic error, and strategies that could have been employed to recognise and counteract the bias before the error was made. Given the sensitive nature of the topic and semi-structured format of the discussion, small group facilitators were trained in a 1-h session consisting of a review of the core content and a ‘mock’ narrative session in which potentially difficult or distracting responses from residents were role-played by the authors. Facilitators were also provided with a written guide created by one of the authors (JR) that included a script for the discussion and suggestions on how to anticipate and respond to problematic scenarios.

    The third portion of the curriculum was an online module that the residents completed at the end of the academic year. The module consisted of three discrete components. The first component required residents to read a short case and then complete an interactive matching activity to identify, differentiate and categorise cognitive errors from systems errors using a modified fishbone diagram created by the investigators for the curriculum. The second component employed three linked video vignettes that depicted clinical scenarios in an emergency room, an outpatient primary care office and the hospital (table 1). Videos were created by the authors and utilised one standardised patient and six volunteer physician actors.

    View this table:
    Table 1

    Video-based clinical vignette question items for cognitive bias and diagnostic error curriculum

    Assessment tools

    Since no published tools exist to assess physician knowledge of cognitive biases in medicine, one was designed. The Diagnostic error Knowledge Assessment Test (D-KAT, see online supplementary appendix 1) was a 13-item, single-best-answer, multiple-choice test. Five items tested general knowledge of diagnostic error prevalence, cognitive contributions to diagnostic error, definition of heuristics, and the dual processing model of decision making. Eight items consisted of short clinical vignettes that were written to highlight a single cognitive bias that was taught in the curriculum. The test was piloted with six internal medicine residents and two medical students and revised in response to feedback. The D-KAT was completed by the residents immediately before part one of the curriculum and embedded in the third part of the curriculum as the final component of the online module, serving as the post-test. Internal consistency was measured using Cronbach's α (0.42).

    After each of the three video vignettes (table 1), residents were asked to type a free-text response to a question that related to the preceding segment. The first question, administered immediately after presentation of video 1, asked residents to identify three cognitive biases illustrated in the video (at least eight were present). The second question, administered immediately after presentation of video 2, asked the residents to explain how the practitioner exhibited diagnostic momentum in the video. The third question, administered immediately after presentation of video 3, asked the residents to identify the debiasing strategies employed by the physician actors. The web module was programmed such that residents could not go back in the module and revise their answer to a previous question.

    Analysis

    The primary outcome was performance on the post-curriculum D-KAT, compared with the pre-curriculum score on the same tool. Post-curriculum scores were also compared to the comparator group of PGY-3 residents who completed the D-KAT without receiving the curriculum. The comparator and post-curriculum groups completed the test at the same point in their academic training (end of second year), minimising the maturation effect and validating our comparator group. Accuracy of resident responses to the simulated case questions in the web-based module was examined as a secondary outcome.

    Pre–post differences in residents’ scores for the post-curriculum D-KAT, as compared to their pre-curriculum scores, were tested for significance using a paired t test. The difference between the median of the post-curriculum scores to the median score of the PGY-3 comparator group was tested for significance using the Wilcoxon rank sum test. Short answers from the web curriculum were graded independently by two authors (JR and AO). Inter-rater agreement was calculated using Cohen's κ statistic (κ 0.73, 0.64 and 0.25 for questions 1, 2 and 3, respectively). All discrepancies were resolved by subsequent discussion between the two graders.

    Qualitative analysis was performed on the written narratives and small group discussions in part two of the curriculum, and the results of this analysis are described elsewhere.22

    Results

    All those who participated in the curriculum and comparator groups did provide informed consent for the study. Of the 48 PGY-2 residents enrolled, 38 (79%) completed all three components of the longitudinal curriculum and were included in the analysis. Of those who did not complete the entire curriculum, two were not present for part one of the curriculum, five did not complete part two or three of the curriculum due to family leave or early entrance into fellowship, and the remainder were otherwise unable to complete part three of the curriculum (web module) by the close of the study despite multiple reminders.

    Of the 38 residents who completed the entire curriculum, the mean score on the post-curriculum 13-item, multiple-choice knowledge test was 9.26 (SD 1.45; median 9) compared to a pre-curriculum mean of 8.26 (SD 1.45; median 8, t=−3.01, df=74, p=0.002) and the PGY-3 group mean of 7.69 (SD 2.01, median 8, z=−3.73, p=0.0002).

    In the video vignette short-answer assessment, all residents (100%) were able to correctly identify at least one bias, 36 (95%) identified two biases and 23 (61%) identified three biases. Thirty-four residents (89%) accurately described how the physician in the second video exhibited diagnostic momentum. Lastly, all residents were able to identify at least one appropriate debiasing strategy, 37 (97%) identified at least two strategies and 23 (61%) participants correctly identified three or more strategies.

    Discussion

    We describe a model for a year-long, longitudinal curriculum in diagnostic errors and cognitive bias, administered to internal medicine residents in their second year of training. Residents who participated in this curriculum improved their recognition and knowledge of common cognitive biases and heuristics, measured by a novel assessment tool. This improvement was likely to be related to the curriculum as opposed to the accumulation of clinical experience, as a comparator group from the previous residency class performed less well. Furthermore, in the final component of this curriculum, participants demonstrated the ability to apply this knowledge to identify both cognitive biases and potential debiasing strategies in simulated clinical scenarios.

    Developing awareness of heuristics could decrease diagnostic errors by allowing physicians to recognise them and implement debiasing strategies to prevent errors. This study builds on the growing body of literature describing cognitive interventions to reduce diagnostic error.7 Previous studies include interventions that incorporate meta-cognition and reflective practice,23–25 as well as interventions that teach cognitive ‘forcing’25–28 or other strategies29 to help learners increase awareness of their own cognitive processing and recognise shortcomings. All of these studies were inconclusive in terms of the effectiveness of education to reduce diagnostic errors. These cognitive interventions meant to improve clinical reasoning7 are largely ‘stand alone’ experiences; studies of the execution of longitudinal curricula designed to help learners recognise common heuristics and biases that could contribute to diagnostic error have not been published.

    Our educational intervention has several strengths. First, the intervention was embedded within the trainees’ required curriculum, not as a ‘stand alone’ experience. Second, we used several teaching methods including video vignettes, reflective narrative writing, and several case-based activities designed to help learners differentiate the systems and cognitive components of diagnostic error. Third, we delivered our curriculum with a core group of faculty who received training in conducting the sessions, due to the sensitive nature of many diagnostic errors. Finally, our unique approach was designed to be executed longitudinally, showing greater concordance with educational principles of adult learning30 by allowing for a graduated, repetitive exposure to the cognitive psychology of clinical decision making errors in a psychologically safe environment (their peer group, facilitated by trusted faculty). Merriam et al describe three keys to transformational learning: experience, critical reflection and development. Our longitudinal curriculum provided: (a) a learning opportunity relevant and applicable to the residents’ set of experiences, (b) a component of critical reflection, and (c) a critical thinking exercise with the web curriculum and D-KAT. Merriam et al30 note that ‘the ability to think critically, which is mandatory to … transformation, is itself developmental’. We chose second year residents as the target for this curriculum because they are entering a critical period in their development, in which they begin to assume the role of clinical team leader and make more independent diagnostic decisions. Although even earlier training in the cognitive psychology of misdiagnosis is needed in medical education, our intent was that this training would influence the diagnostic skills of the residents during a period where medical decision-making habits are being formed and solidified.

    This study has limitations that raise additional questions and offer future directions for educators and researchers in the field of diagnostic errors and clinical reasoning. This study was performed in one residency programme at a single institution and had a relatively small number of participants, which may limit the generalisability of our findings. We did not formally assess resident satisfaction with the curriculum. All of the educational interventions and assessments occurred in a classroom or web-based, simulated environment as opposed to an actual clinical practice setting. Therefore, although we hope that the knowledge and skills gained in these settings will be transferred to clinical practice, this was not measured. We attempted to increase the clinical relevance and knowledge transfer of this topic by having our learners write and discuss personal stories about their real experiences with diagnostic errors.22 Next, because our video vignettes were integrated formally into the curriculum as an instructional tool, we could not assess the comparator group using the videos. Our primary assessment method was the D-KAT, administered before and after the curriculum. Although the pre- and post-tests were administered 9 months apart, familiarity with the instrument may have contributed to improved performance. Further, it is possible that pre-curriculum performance on the D-KAT reflected a considerable baseline understanding of cognitive errors, given the relatively modest improvement after the completion of the curriculum. The D-KAT instrument had a low internal consistency (Cronbach's α=0.42) and has not been previously validated. We theorise that the observed α is at least in part a reflection of the multiple constructs that the test evaluates and the relatively small sample size in this pilot study. Further refinement and testing of the instrument is required in order to determine if this tool can accurately measure knowledge of heuristics, cognitive bias and related concepts in diagnostic error, and iterative refinement cycles could improve the test's psychometric properties.

    Mindful of the strengths of the intervention and the limitations of our study, we propose several next steps. We would continue to revise individual items within the D-KAT to improve their psychometric properties and the internal consistency of the instrument. Next, having utilised our video vignettes both as learning tools and as assessment tools in this study, we found them to have great potential. Future curricula in diagnostic errors should utilise and validate video-based question items as assessment tools by utilising an intervention and control group. Additionally, we feel that the concepts of heuristics, bias and diagnostic error are not unique to internal medicine, and therefore the clinical scenarios used in our curriculum should also be adapted to meet the needs of other medical specialties.

    Diagnostic errors are difficult to measure, and previous study of cognitive interventions among clinicians are inconclusive.7 Controversy exists regarding the utility of teaching about these issues as a direct means of improving diagnosis, given the subconscious nature of bias and heuristics. However, we do believe that teaching trainees about their cognitive vulnerabilities and promoting reflection on their past mistakes are essential to the ongoing development of good diagnostic habits, providing a long term benefit that is difficult to measure. We think it is also likely to increase trainees’ willingness to adopt other tools that have shown early promise in reducing diagnostic errors. For example, newer studies of checklists have shown potential to improve diagnostic skills such as ECG interpretation,31 ,32 and this benefit may actually be larger for trainees compared to seasoned clinicians.33

    Education in the field of diagnostic errors is nascent. More research is needed to understand how to develop and improve clinical reasoning across the educational continuum and how to measure the effects of these efforts on diagnostic accuracy. Given the prevalence of diagnostic errors in medicine and their underlying cognitive causal factors, we believe that it is critical for educators to include this content in their patient safety curriculum and develop novel ways to teach it. To meet this need, we offer this longitudinal curriculum as a model for integrating instruction on cognitive bias and diagnostic error into residency training.

    Acknowledgments

    We wish to thank Lisa Bellini, MD and the Penn Internal Medicine Residents who participated in the curriculum. We are also grateful to the following colleagues who participated in our faculty development sessions and served as small group facilitators throughout the curriculum: Matt Rusk, MD, Steve Gluckman, MD, Karen Warburton, MD, Todd Barton, MD, David Aizenberg, MD, Stephen Kim, MD and Jodi Lenko, MD. Finally, we thank Jennifer Kogan, MD, who participated as faculty and also provided feedback on an earlier version of this manuscript, and Judy Shea, PhD for her advice on study design.

    References

    1. ↵
      1. Kohn LT,
      2. Corrigan J,
      3. Donaldson MS
      . To err is human: building a safer health system. Washington, DC: National Academy Press, 2000.
    2. ↵
      1. Shortell SM,
      2. Singer SJ
      . Improving patient safety by taking systems seriously. JAMA 2008;299:445–7.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Leape LL,
      2. Brennan TA,
      3. Laird N,
      4. et al
      . The nature of adverse events in hospitalized patients. Results of the Harvard Medical Practice Study II. N Engl J Med 1991;324:377–84.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Shojania KG,
      2. Burton EC,
      3. McDonald KM,
      4. et al
      . Changes in rates of autopsy-detected diagnostic errors over time: a systematic review. JAMA 2003;289:2849–56.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Schiff GD,
      2. Hasan O,
      3. Kim S,
      4. et al
      . Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med 2009;169:1881–7.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      Diagnosing Diagnosis Errors: Lessons from a Multi-institutional Collaborative Project. Schiff GD, Kim S, Abrams R, et al. In: Henriksen K, Battles JB, Marks ES, et al, editors. Advances in Patient Safety: From Research to Implementation (Volume 2: Concepts and Methodology). Rockville (MD): Agency for Healthcare Research and Quality (US); 2005 Feb.
    7. ↵
      1. Graber ML,
      2. Kissam S,
      3. Payne VL,
      4. et al
      . Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf 2012;21:535–57.
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Graber ML,
      2. Wachter RM,
      3. Cassel CK
      . Bringing diagnosis into the quality and safety equations. JAMA 2012;308:1211–12.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Wachter RM
      . Why diagnostic errors don't get any respect—and what can be done about them. Health Aff (Millwood) 2010;29:1605–10.
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Wong BM,
      2. Etchells EE,
      3. Kuper A,
      4. et al
      . Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med 2010;85:1425–39.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Walton M,
      2. Woodward H,
      3. Van Staalduinen S,
      4. et al
      . The WHO patient safety curriculum guide for medical schools. Qual Saf Health Care 2010;19:542–6.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Kirch DG,
      2. Boysen PG
      . Changing the culture in medical education to teach patient safety. Health Aff (Millwood) 2010;29:1600–4.
      OpenUrlAbstract/FREE Full Text
    13. ↵
      1. Eva KW,
      2. Norman GR
      . Heuristics and biases—a biased perspective on clinical reasoning. Med Educ 2005; 39:870–2.
      OpenUrlCrossRefPubMed
    14. ↵
      1. Kassirer JP,
      2. Wong JB,
      3. Kopelman RI
      . Learning clinical reasoning. 2nd edn. Baltimore, MD: Lippincott Williams & Wilkins Health, 2010.
    15. ↵
      1. Elstein AS
      . Thinking about diagnostic thinking: a 30-year perspective. Adv Health Sci Educ Theory Pract 2009; 14(Suppl 1):7–18.
      OpenUrlCrossRefPubMed
    16. ↵
      1. Norman G
      . Dual processing and diagnostic errors. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):37–49.
      OpenUrlCrossRefPubMed
    17. ↵
      1. Croskerry P
      . The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med 2003; 78:775–80.
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Croskerry P
      . Clinical cognition and diagnostic error: applications of a dual process model of reasoning. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):27–35.
      OpenUrlCrossRefPubMed
    19. ↵
      1. Graber ML
      . Educational strategies to reduce diagnostic error: can you teach this stuff? Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):63–9.
      OpenUrlCrossRefPubMed
    20. ↵
      1. Cosby KS,
      2. Croskerry P
      . Patient safety: a curriculum for teaching patient safety in emergency medicine. Acad Emerg Med 2003;10:69–78.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Trowbridge RL
      . Twelve tips for teaching avoidance of diagnostic errors. Med Teach 2008;30:496–500.
      OpenUrlCrossRefPubMed
    22. ↵
      1. Ogdie AR,
      2. Reilly JB,
      3. Pang WG,
      4. et al
      . Seen through their eyes: residents’ reflections on the cognitive and contextual components of diagnostic errors in medicine. Acad Med 2012;87:1361–67.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Mamede S,
      2. van Gog T,
      3. van den Berge K,
      4. et al
      . Effect of availability bias and reflective reasoning on diagnostic accuracy among internal medicine residents. JAMA 2010; 304:1198–203.
      OpenUrlCrossRefPubMed
    24. ↵
      1. Coderre S,
      2. Wright B,
      3. McLaughlin K
      . To think is good: querying an initial hypothesis reduces diagnostic error in medical students. Acad Med 2010;85:1125–9.
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Sibbald M,
      2. de Bruin AB
      . Feasibility of self-reflection as a tool to balance clinical reasoning strategies. Adv Health Sci Educ Theory Pract 2012;17:419–29.
      OpenUrlCrossRefPubMed
    26. ↵
      1. Sherbino J,
      2. Yip S,
      3. Dore KL,
      4. et al
      . The effectiveness of cognitive forcing strategies to decrease diagnostic error: an exploratory study. Teach Learn Med 2011;23:78–84.
      OpenUrlCrossRefPubMed
    27. ↵
      1. Eva KW,
      2. Hatala RM,
      3. Leblanc VR,
      4. et al
      . Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information. Med Educ 2007;41:1152–8.
      OpenUrlPubMedWeb of Science
    28. ↵
      1. Sibbald M,
      2. McKinney J,
      3. Cavalcanti RB,
      4. et al
      . Cardiac examination and the effect of dual-processing instruction in a cardiopulmonary simulator. Adv Health Sci Educ Theory Pract 2013;18(3):497–508.
    29. ↵
      1. Wolpaw T,
      2. Papp KK,
      3. Bordage G
      . Using SNAPPS to facilitate the expression of clinical reasoning and uncertainties: a randomized comparison group trial. Acad Med 2009;84:517–24.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Merriam SB,
      2. Caffarella RS,
      3. Baumgartner LM
      . Learning in adulthood: a comprehensive guide, San Francisco, CA, USA: Jossey-Bass, 2007.
    31. ↵
      1. Sibbald M,
      2. de Bruin AB,
      3. van Merrienboer JJ
      . Checklists improve experts’ diagnostic decisions. Med Educ 2013; 47:301–8.
      OpenUrlCrossRefPubMed
    32. ↵
      1. Sibbald M,
      2. de Bruin AB,
      3. Cavalcanti RB,
      4. et al
      . Do you have to re-examine to reconsider your diagnosis? Checklists and cardiac exam. BMJ Qual Saf 2013;22:333–8.
      OpenUrlAbstract/FREE Full Text
    33. ↵
      1. Sibbald M,
      2. De Bruin AB,
      3. van Merrienboer JJ
      . Finding and fixing mistakes: do checklists work for clinicians with different levels of experience? Adv Health Sci Educ Theory Pract 2013 Apr 27. [Epub ahead of print].

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors ARO provided particular expertise for the statistical analysis and description of the results and methods, in addition to providing revisions in all other sections. JMvF provided substantial assistance in the initial writing and revision of every section of the paper, especially the discussion. JSM wrote the introduction section and provided substantial revisions throughout each section, including the discussion. JBR was responsible for the discussion, provided substantial revisions in all sections, and assumed responsibility for overall coherence of the manuscript, submission of the manuscript and ongoing correspondence and revisions. The final submission is approved by each author.

    • Funding This work was supported by the Sam Martin Education Pilot Award from the Division of General Internal Medicine at the Perelman School of Medicine at the University of Pennsylvania. ARO is supported by the American College of Rheumatology Research and Education Foundation Rheumatology Investigator Award. During this project, JBR was supported by NIH Institutional Training Grant T32-DK 07006-37 and the Center for Healthcare Improvement and Patient Safety at the University of Pennsylvania. JSM is supported in part by a grant from the Josiah Macy Jr. Foundation.

    • Ethics approval This study was approved by the Institutional Review Board of the University of Pennsylvania.

    • Provenance and peer review Not commissioned; externally peer reviewed.