Background: Accurately assessing how physicians perform in practice remains an unresolved psychometric challenge. Neither chart reviews nor patient surveys indicate when physicians overlook important information, which can result in a missed opportunity for a correct diagnosis and appropriate plan of care. Standardized patient (SP) assessments provide an opportunity for direct observation of clinical behavior and are increasingly used in licensure examinations. (SPs who are sent incognito are termed unannounced standardized patients [USPs].) One study showed that physicians had particular difficulty adapting care to individual patient context ("contextual error"). In a subsequent study with the same actors, SP cases, and outcomes, an intervention was deployed to reduce contextual error among medical students. In an exploratory reanalysis of data from the two studies, clinicians' assessments of SPs and USPs were compared.
Methods: Participants in the first study were 65 board-certified internists visited by USPs; the 59 participants in the second were fourth-year medical students examining SPs in a clinical performance center.
Results: Attending physicians measured with USPs significantly underperformed medical students measured with SPs in the probing of biomedical red flags (odds ratio [OR] = 0.45 [0.30 to 0.67]) and contextual red flags (OR = 0.66 [0.45 to 0.99]) and in planning appropriate care (OR = 0.43 [0.27 to 0.67]).
Conclusions: Across these two studies, attending physicians underperformed medical students on the same outcomes, measured with the same patient cases presented by the same actors. Studies that seek to assess elicitation and incorporation of patient information by physicians as measures of individualization of care should weigh the benefits and costs of direct observation by USPs.