Purpose: Experienced clinicians derive many diagnoses intuitively, because most new problems they see closely resemble problems they've seen before. The majority of these diagnoses, but not all, will be correct. This study determined whether further reflection regarding initial diagnoses improves diagnostic accuracy during a high-stakes board exam, a model for studying clinical decision making.
Method: Keystroke response data were used from 500 residents who took the 2010 American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. Data included time to initial response on each question, whether the answer was correct, and whether or not the resident changed her or his initial response. The focus was on 80 diagnosis questions that comprised realistic clinical vignettes with multiple-choice single-best answers. Cognitive skill (ability) was measured using overall exam scores. Case complexity was determined using item difficulty (proportion of examinees that correctly answered the question). A hierarchical generalized linear model was used to assess the relationship between time spent on initial responses and the probability of correctly answering the questions.
Results: On average, residents changed their responses on 12% of all diagnosis questions (or 9.6 questions out of 80). Changing an answer from incorrect to correct was almost twice as likely as changing an answer from correct to incorrect. The relationship between response time and accuracy was complex.
Conclusions: Further reflection appears to be beneficial to diagnostic accuracy, especially for more complex cases.