You are here

Article Text

Article menu
Narrative review
Cognitive interventions to reduce diagnostic error: a narrative review
  1. Mark L Graber1,2,3,
  2. Stephanie Kissam3,
  3. Velma L Payne4,5,
  4. Ashley N D Meyer6,7,
  5. Asta Sorensen3,
  6. Nancy Lenfestey3,
  7. Elizabeth Tant3,
  8. Kerm Henriksen8,
  9. Kenneth LaBresh3,
  10. Hardeep Singh6,7
  1. 1VA Medical Center, Northport, New York, USA
  2. 2Department of Medicine, SUNY Stony Brook, New York, USA
  3. 3RTI International, Research Triangle Park, North Carolina, USA
  4. 4School of Biomedical Informatics, University of Texas Health Science Center in Houston, Houson, Texas, USA
  5. 5National Center for Cognitive Informatics and Decision Making in Healthcare, University of Texas Health Science Center in Houston, Houston, Texas, USA
  6. 6Houston VA HSR&D Center of Excellence, and the Center of Inquiry to Improve Outpatient Safety Through Effective Electronic Communication, Michael E. DeBakey Veterans Affairs Medical Center, Houston, Texas, USA
  7. 7Section of Health Services Research, Department of Medicine, Baylor College of Medicine, Houston, Texas, USA
  8. 8Center for Quality Improvement and Patient Safety, Agency for Healthcare Research and Quality, Rockville, Maryland, USA
  1. Correspondence to Dr Mark L Graber, RTI International, c\o 1 Breezy Hollow, St James, NY 11780, USA; mgraber{at}rti.org

Abstract

Background Errors in clinical reasoning occur in most cases in which the diagnosis is missed, delayed or wrong. The goal of this review was to identify interventions that might reduce the likelihood of these cognitive errors.

Design We searched PubMed and other medical and non-medical databases and identified additional literature through references from the initial data set and suggestions from subject matter experts. Articles were included if they either suggested a possible intervention or formally evaluated an intervention and excluded if they focused solely on improving diagnostic tests or provider satisfaction.

Results We identified 141 articles for full review, 42 reporting tested interventions to reduce the likelihood of cognitive errors, 100 containing suggestions, and one article with both suggested and tested interventions. Articles were classified into three categories: (1) Interventions to improve knowledge and experience, such as simulation-based training, improved feedback and education focused on a single disease; (2) Interventions to improve clinical reasoning and decision-making skills, such as reflective practice and active metacognitive review; and (3) Interventions that provide cognitive ‘help’ that included use of electronic records and integrated decision support, informaticians and facilitating access to information, second opinions and specialists.

Conclusions We identified a wide range of possible approaches to reduce cognitive errors in diagnosis. Not all the suggestions have been tested, and of those that have, the evaluations typically involved trainees in artificial settings, making it difficult to extrapolate the results to actual practice. Future progress in this area will require methodological refinements in outcome evaluation and rigorously evaluating interventions already suggested, many of which are well conceptualised and widely endorsed.

  • Patient safety
  • human error
  • medical error
  • measurement/epidemiology
  • decision support
  • computerised
  • decision-making
  • information technology
  • trigger tools
  • primary care
  • diagnostic errors
  • health services research
  • healthcare quality improvement
  • evidence-based medicine
  • quality improvement
  • audit and feedback
  • cognitive biases

https://doi.org/10.1136/bmjqs-2011-000149

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Although the rate of diagnostic error in practice is unknown, experts estimate it to be in the range of 10%–15%.1 Diagnostic errors are of great concern in all specialties and those characterised by high levels of stress, workload and distractions are particularly vulnerable. Errors are more likely when the level of uncertainty is high, if clinicians are unfamiliar with the patient, and when there are atypical or non-specific presentations of a common disease or ‘distracting’ comorbid conditions.2

    Diagnostic errors reflect the complex interplay of system-related and cognitive factors, typically with multiple root causes identifiable in a single case.3–6 Cognitive errors can be found in the majority of cases.4 ,7 Given the dominant role that cognitive shortcomings play in contributing to diagnostic error, it is appropriate to begin considering what could be done to help minimise the likelihood of these errors. We therefore conducted an analytic review of the literature to identify interventions to reduce the likelihood of cognitive errors or error-related harm in healthcare. Interventions relating to system-related factors were discussed in a companion publication.8

    Methods

    Our search strategy has been previously described.8 Briefly, we sought articles, books and conference presentations relating to the prevention, reduction or mitigation of diagnostic errors in PubMed and several other medical and non-medical databases. We pursued references from these sources and asked authorities in the field of applied cognition and decision-making to recommend additional readings. Articles and books were included in this analysis if they contained results from an intervention trial or suggested an intervention to reduce cognitive-related diagnostic error. Publications that focused on development or refinement of specific diagnostic tests or technologies, or solely on the aetiology or epidemiology of error, or dealt primarily with provider satisfaction or preferences were excluded.

    A full-text review using an approach described by Gordon and Findley9 was performed on the 42 empirical studies that tested an intervention. Nineteen quality-based criteria were independently extracted from each article using a data extraction form (online appendix A). Items answered with ‘yes’ or ‘no’ included literature review described, clear objectives reported, study design reported, appropriate design to address objectives, control group used, subjects randomised, blinding used, intervention clearly described, resources described, outcomes match objectives, statistical tests used, statistical tests appropriate, data collection replicable, study replication possible and limitations discussed. Additional items assessed were the study design, subject characteristics and number of subjects. Based on these items, we assigned an ‘Outcomes Rating’ and ‘Strength of Conclusions’ rating to each article (detailed instruments in online appendix B). The Outcomes Rating was based on Kirkpatrick's hierarchy9 ,10 that we slightly modified for use in assessing diagnostic errors. This hierarchy demonstrates the level of impact of each intervention on diagnostic errors (eg, Level 2b refers to an intervention in which an acquisition of concepts might impact diagnostic error, whereas Level 4b refers to an intervention that directly reduces diagnostic error). The Strength of Conclusions of each study was rated on a numerical scale (1–5) in accordance with Best Evidence in Medical Education guidelines.9 ,11 This rating is not an assessment of the overall methodological quality, but is a measure of how well the conclusions made are supported by the data presented.

    Two reviewers with expertise in cognitive psychology (ANDM and VLP) assessed each of the intervention studies independently. We assessed agreement between the reviewers for the Outcomes Rating and the Strength of Conclusions with Cohen's κ statistic. Differences were resolved by discussion between the two reviewers and in cases of disagreement, another investigator (SK) reviewed and rated the article. In these cases, we used consensus among these three reviewers to determine the final ratings.

    Based on a prior classification scheme,1 all articles were assigned to one of three natural categories: (1) Interventions that increase medical knowledge and experience; (2) Interventions that improve clinical reasoning; and (3) Interventions that involve getting help. Articles were further subdivided into more specific types of interventions (such as ‘focused training on specific content areas’, ‘develop simulation exercise to expose clinicians to a greater number and variety of cases presentations’, etc.) to facilitate the synthesis of the findings (tables 1–4).

    View this table:
    Table 1

    Interventions to increase medical knowledge and experience

    View this table:
    Table 2

    Interventions to improve intuitive (system 1) and deliberate (system 2) processes in decision-making

    View this table:
    Table 3

    Interventions: help from other people

    View this table:
    Table 4

    Interventions: help from decision support

    Results

    We identified 141 sources (articles, books and conference papers) for full review. Of these, 42 sources (tables 1–4) reported empirical studies of an intervention to reduce cognitive-based diagnostic error (and sometimes also additional suggestions for interventions), 100 sources contained only suggestions (table 5) and one had both. Some sources reported more than one suggestion.

    View this table:
    Table 5

    Intervention suggestions

    During the full-text review of the empirical studies assessing cognitive interventions, agreement between reviewers on the Outcomes Rating was substantial (κ=0.70). Similar agreement was obtained for the Strength of Conclusions (κ=0.70). There were three articles with disagreements that were resolved by discussions with a third reviewer. We categorised the intervention studies into one of three mutually exclusive categories: (1) Interventions to increase clinicians' knowledge and experience, (2) Interventions to improve clinical reasoning and decision-making skills or (3) ‘Get help’, interventions that assist clinicians with tools or access to other clinicians or experts. For each of these sections, we use the suggested intervention and background literature to first provide context, following which we discuss the tested interventions. The Outcomes Ratings and Strength of Conclusions ratings for each intervention article are included in tables 1–4.

    1) Increase knowledge and experience

    Diagnostic error could potentially be reduced by increasing physician's structured knowledge and experience, the essential basis of expertise.143 By definition, experts will tend to make the fewest errors, have the best degrees of calibration and excel in efficient diagnosis.50 ,144 ,145 Medical educators similarly agree with the concept of increasing experience as the key to developing expertise.143 ,146 ,147 The interventions in this domain are summarised in table 1 and are organised into the following three categories.

    Training focused on specific content areas

    An effect of training on diagnostic reliability is illustrated in radiology, where certain certification programmes are based on demonstrating competency. For instance, radiologists in the UK must review 5000 mammograms a year for certification, as opposed to 480 in the USA, which may in part account for the large difference in diagnostic accuracy noted between the two countries.54 In certain programmes, radiologists also receive additional training in cancer detection where they attend disease-related meetings, receive feedback on cancer detection rates and attend a 2-week course led by specialists at high-volume mammography screening sites.55 Similar measures, including regular peer review and participation in the American College of Radiology's RADPEER™ system, have been proposed for the USA.148

    Interventions to increase the knowledge base of practicing clinicians through continuing medical education activities have generally not led to substantial improvement in measured performance.56 ,57

    Interventions

    We identified three formal studies of training interventions related to diagnostic error.12–14 One notable study was a highly content-specific intervention to improve recognition of subarachnoid haemorrhage. This low-cost training programme on sudden onset headache for community-based physicians reduced the baseline diagnostic error rate (12%) by 77% and improved interactions between neurosurgeons and local physicians.12

    Simulation

    The ability to provide realistic simulations through both scenarios and simulated patients offers the potential to improve skills in clinical reasoning63 and the opportunity to expose trainees or physicians to a greater number and variety of case presentations. Simulation is a well-established approach to improving manual, procedural skills, but has not yet been evaluated extensively in its ability to improve cognitive skills or decision-making related to diagnosis. It also remains to be demonstrated that simulation can replace experience in actual practice.

    Interventions

    We identified only two interventions in this domain, both involving trainees. Carlson et al16 demonstrated improved diagnostic accuracy by the combined use of simulation with a diagnosis support tool and Bond et al15 used simulation successfully to introduce the use of cognitive forcing strategies to emergency medicine residents.

    Feedback as a way to improve expertise, calibration and error awareness

    Deliberate practice, with immediate and focused feedback, is viewed as an essential prerequisite to developing expertise in any domain.144 ,145 Moreover, lack of feedback is a dominant factor that sustains overconfidence, thought to be a major factor in causing diagnostic error.1 A systematic review of feedback across all medical areas (not solely diagnosis) concluded that feedback improves performance in selected settings, especially if the feedback is intensive.65 Feedback is most useful if it incorporates instruction and information on why a given answer was correct or not.66 ,67 For example, psychology trainees improved their diagnoses if feedback provided details on why they were right or wrong.18 ,149 Using feedback to improve diagnostic performance has been most convincingly demonstrated in radiology through programmes such as the ‘PERFORMS’ system in the UK54 and the RADPEER programme in the USA.

    Although clinicians received immediate and dramatic feedback on their diagnostic performance from autopsies, the rate of autopsies is declining.150 Local ‘Morbidity and Mortality’ conferences73 and creative new venues such as the ‘Web M&M’ series sponsored by the Agency for Healthcare Research and Quality (http://www.webmm.ahrq.gov/) are alternative venues where feedback is provided.151 In this spirit, Eva74 has advocated for incorporating diagnostic error review into medical school and postgraduate training. Alpert and Hillman discuss other types of data that should be part of such feedback, such as the results of professional audits, peer reviews and risk management programmes.68

    Interventions

    Our search yielded two studies on feedback to improve diagnostic performance.17 ,18 Both studies showed benefits of feedback on later diagnostic accuracy. The positive impact noted by Wood and Tracey18 was possibly explained by the provision of detailed feedback to trainees on the reasons their initial diagnoses were correct or not.

    Category 1 summary

    The empiric studies identified were positive, but generally used trainees and specific content limiting the ability to generalise the impact of results to actual practice.

    2) Improve clinical reasoning

    According to the currently popular paradigm, diagnoses are made by some interacting combination of intuitive, automatic processing (system 1) and deliberate, rational consideration (system 2).152 Interventions to reduce diagnostic error have been suggested in each of these areas, and many authors have advocated for the benefits of general training in clinical reasoning.1 ,23 ,47 ,49 ,53 ,75–78 The interventions in this domain are presented in table 2.

    Improve intuitive processing: debiasing

    Many, and perhaps most, medical diagnoses are derived intuitively, acknowledging that most conditions are common and present in typical, easily recognised, fashion. Coderre et al153 found that intuitive diagnoses are more likely to be correct compared with diagnoses derived by hypothetico-deductive reasoning, and this concept is also consistent with the substantial literature regarding expertise.

    Experts in the field of naturalistic decision-making emphasise that intuitive judgements cannot be taught because they emerge subconsciously from the amassed experience of the decision-maker and his or her ability to access this knowledge instantaneously and effectively.48 ,50 However, others have argued that intuition can be encouraged, strengthened and improved.51 ,101 ,151 Brawn highlighted several strategies to encourage use of intuition such as showcasing examples of how intuition was used in discovery and insight situations.98 Noddings and Shore100 suggest that intuition can be developed by first acknowledging intuition and its role in decision-making, demonstrating its capacity and successes, and by sharing how intuition is used, especially by experienced role models. Hogarth151 recommends a series of novel educational interventions to teach and improve intuition, including creating increased motivation to learn by exposure to one's own errors and constantly seeking to improve one's learning skills by reviewing and revising skills in observation, sense-making and hypothesis testing.

    Croskerry and others have argued that clinicians would make fewer errors if they learned the potential shortcomings (biases) of intuitive decision-making so as to understand and avoid them.76 ,77 ,91 Interventions to avoid both affective bias (engendered by our inherent discomfort with certain types of patients or interactions) and cognitive bias (due to the known shortcomings and pitfalls of subconscious thought) have been suggested.

    Similar debiasing interventions were suggested by Fischhoff85 and included: (1) warning about the possibility of bias; (2) describing how the bias distorts good decisions; (3) letting the individual make a bias-related judgement error and giving them feedback; and (4) repeating these cycles with extended coaching. Larrick87 reported an example of successful debiasing by keeping it focused on a particular context and a particular bias.

    Experimental evidence suggests that hindsight bias can be reduced by considering alternatives.154 In one such study, subjects were asked to choose between two answers to a difficult question,93 where some were asked to give the reasons they made their choice and others were asked to give reasons both for and against their choice. Considering both alternatives improved accuracy and reduced the tendency for subjects to be overconfident in their answers.92 Similarly, physicians evaluating a difficult test case were more likely to trust a diagnosis when asked to consider alternatives.154

    Although debiasing is potentially attractive, several authors have expressed scepticism if this approach will work based on the intrinsic difficulty of changing the subconscious processing individuals use in decision-making.86 ,155 ,156

    Interventions

    Our search yielded two studies. Sherbino and colleagues19 tested an effort to improve clinical reasoning of trainees by teaching them cognitive forcing strategies to counteract biases. The study lacked baseline data (no measure prior to intervention) or a control group, and the results were generally negative. In addition, the reported retention of the cognitive forcing strategies that were the subject of the intervention was short-lived. Eva et al encouraged the use of combined strategies (pattern recognition plus deliberate consideration) in teaching students to read electrocardiograms (ECR), and found this improved their diagnostic performance in part by avoiding biases.20

    Improving metacognition and reflection

    Improving metacognition, the ability to reflect on one's own thought processes, is an appealing approach to reduce cognitive error.77 ,78 ,103 Metacognition could potentially alert clinicians to possible flaws in their reasoning and help detect errors. A related and widely endorsed recommendation is to practice reflectively,82 ,94 ,96 ,97 ,102 recently referred to as the diagnostic ‘time out’.83 Reflective practice promotes metacognition and incorporates four distinct elements: Seeking out alternative explanations, exploring the consequences of alternative diagnoses, being open to tests that would differentiate the various possibilities and accepting uncertainty. This process, essentially getting a second opinion from your own conscious mind, has the potential to avoid many of the inherent pitfalls of heuristic thought.82

    Several tools have been suggested that might be helpful to promote metacognition and reflective practice, including Trowbridge's ‘12 Tips’ and Leonidas' ‘Ten Commandments’.83 ,157 Using a diagnosis checklist, by promoting conscious review and reflection, has also been advocated as a way to avoid pitfalls in clinical reasoning.106 ,157

    Interventions

    Two studies were identified. Mamede and colleagues found that conscious reflection decreased the tendency towards availability bias,21 and Coderre et al demonstrated that reflection on an initial diagnosis was helpful if the initial diagnosis was wrong, and did not lead to new errors if the initial diagnosis was correct.22 A limitation of both studies is that the additional time spent on problem solving may be what is driving the result, not conscious reflection per se. Also, both studies involved trainees in a laboratory environment, so that the positive results would have to be reconfirmed in practice settings. It is therefore inconclusive whether these techniques successfully reduce diagnostic errors.

    Consider alternatives

    A central element of reflective practice is reviewing alternative diagnoses, an approach widely endorsed as a valid approach to improved decision-making107 which we consider separately in this section. In this approach, clinicians should invoke what has been called the universal antidote, ‘Could this be something else?’ and use appropriate tests to exclude the alternatives, rather than ordering tests that simply confirm original suspicions.107 Others79 have also suggested that clinicians ‘jot down, in advance, outcomes that would support one's initial conclusions and also those that would disconfirm them’ or consider alternatives.111 A related strategy is to assume the perspective of an outside observer,86 prompting evaluation of the decision-making strategy that was used and whether or not it was flawed. Military planners have used ‘prospective hindsight’ to teach this principle: one looks into the future to see that the working diagnosis is not correct: What was missed and what else should have been considered?97 ,112

    Interventions

    Our search yielded one study that tested an intervention in this category. Wolpaw et al23 attempted to improve clinical reasoning and decision-making skills through a six-step training programme for medical students to express their diagnostic reasoning process. The impact of this technique on diagnostic errors is inconclusive since the study did not assess the reduction of diagnostic errors, but only assessed frequency and thoroughness of their skills in presenting a patient case. The study only measured the presence/amount of reasoning and not accuracy thereof, and so it is unclear how this intervention would improve diagnostic accuracy.

    Improve rational processing

    Rational, deliberate review and consideration combine the use of evidence-based knowledge158 ,159 with two normative approaches, the use of expected value decision-making160 to choose among a group of possible diagnoses and Bayesian analysis to incorporate test results in considering a single diagnosis. Kassirer et al47 describes the process of clinical reasoning as generating initial hypothesis which are then investigated by diagnostic tests and Bayesian analysis until an appropriate threshold (Treat, Don't Treat) is reached. Kassirer suggests that the essential skills of clinical reasoning can and should be taught to medical students from their first days,126 and reviewers have concluded that conscious review can be taught effectively.121 ,122 Trainees taught principles of evidence-based medicine are more likely to use Bayesian techniques to interpret clinical findings.158 In efforts to reduce surgical cognitive errors, Brannick et al108 oriented surgical trainees to Reason's major error types using an educational video and role-playing emphasising errors. Although actual surgical error rates were the same as in untrained controls after a month, attention to detail improved.108

    Category 2 summary

    We noted a major discrepancy between the breadth and enthusiasm for these interventions in the background studies, but a paucity of actual interventions. For all three categories, there is very limited evidence addressing diagnostic accuracy or errors. The studies identified involved trainees in laboratory-like settings, limiting the ability to generalise the findings to real practice.

    3) Get help: use other people and decision support tools

    Given the constraints of human cognition,78 physicians may be able to augment their innate cognitive abilities by obtaining advice and help from others. All of the tested interventions in this category are detailed together in online appendix C and were organised in the following categories.

    Second opinions

    Interventions

    Several studies have demonstrated that second reviews of surgical pathology or cytology specimens find a small but important group of errors,24–28 and a growing number of healthcare systems now require second readings in case types known to have substantial rates of inter-observer variability. Most of these studies do not, however, include data on patient outcomes (table 3).

    Second readings in radiology also improve test sensitivity. Duijm et al31 found that multiple independent readers (radiologists or technicians) increased cancer detection rates with only a slight decrease in specificity, and Kwek et al32 found that second reading increased cancer detection by 5%.

    The impact of second readings has been mixed in other settings. Second reading of Emergency Room (ER) imaging studies was helpful in one study,30 but in another, besides identifying previously missed abnormalities, the second reading introduced new misinterpretations leading to inappropriate changes in management.29 Canon et al33 measured the impact of independent double reading of barium enemas and found no effect on the sensitivity of polyp detection and an increased rate of false positives.

    Thus, the overall impact of ‘second opinions’ on diagnostic errors appears to be mixed. Sensitivity appears to improve in most but not all studies, but the second readings tend to introduce new errors that detract from the specificity of the diagnostic test. Results could potentially be both reliable and generalisable because of the relatively large number of cases reviewed in these studies, and the use of expert reviewers. Cost–benefit analyses will be needed to determine whether the costs of second readings and the seemingly inevitable increment in false positives are offset by the increased rate of case finding.

    Groups and librarians

    Groups can make better decisions than its individual members if the members are allowed to function independently.86 ,161 Diagnosing challenging cases within teams or with peers would take advantage of this strategy. A recent novel approach leverages the use of librarians who are experienced and skilled in identifying information, evidence, and knowledge relevant to diagnostic alternatives or testing strategies.129–131

    Interventions

    One study by Christensen et al34 studied team-based decisions. This was a well designed, controlled study, but the results were negative: performance did not improve by using the team.

    A randomised trial of embedded clinical informaticians at one university demonstrated a positive impact on the clinical care provided,35 although self-reported perceptions were used in place of actual outcomes.

    Decision support

    Most studies of decision support tools have evaluated impact on process measures, user satisfaction and utility in a limited sense,141 and are not consistently positive. A systematic review of decision support systems in 1998 identified only a single study focusing on diagnosis,133 and in this study, using a decision support tool in an emergency room on patients with joint or bone injuries actually led to more missed fractures.134

    Using linear prediction models (actuarial decision-making, algorithms) has been shown to yield better ‘decisions’ than most decision-makers, including experts, in a wide range of settings.111 Wedding and colleagues79 ,132 ,162 report that actuarial diagnosis was more accurate than clinical judgement in patients with neuropsychiatric conditions. However, clinicians tend to disregard advice from these tools or not use them even when they are readily available.163 ,164 The importance of embedding decision support in the physician's workflow has been repeatedly emphasised, for example, by incorporating decision support logic in computer-based order entry systems. A systematic review of this approach identified 11 controlled trials, seven of which reported improved professional practice141 on ordering diagnostic tests.

    Hamm and Zubialde and more recently Schiff and Bates have called attention to many other ways in which the electronic medical record can enhance clinical reasoning.102 ,142 Besides providing clear access to the necessary data, good records help clinicians organise their thoughts, enhance collaborative thinking, enhance efficiency and promote feedback. Another promising type of clinical decision support enabled by electronic records is the graphic display of timeline data to assist in the interpretation of diagnostic test results and to help detect subtle trends.137

    Interventions

    A recent review identified 10 newer studies, each focused on a specific clinical condition135 and of these only four studies had positive results: one reported improved ability to detect and diagnose mood disorders in outpatients,136 two improved diagnosis in acute coronary syndromes36 ,37 and one evaluated diagnosis of acute abdominal conditions on a surgical service, which improved provider performance but not patient outcomes38 (table 4). Finally, De Simone et al39 describe a system that receives clinical information from the patient directly and synthesises that information to aid the clinician in diagnosing the cause of headaches. Overall, the studies were all sound and results seem to be generalisable by virtue of testing a range of subjects and case types.

    Another approach to supporting the diagnosis of specific conditions in general is the Infobutton functionality, described by Cimino.40 The only available study of Infobuttons included subjects from varying levels (attending physicians, residents, medical students, nurses) who were mostly satisfied with the tool. The impact of the tool on diagnostic accuracy and patient outcomes was not assessed.

    Computer-aided detection systems

    Interventions

    Five studies have examined the use of computer-aided detection systems to aid radiologic diagnosis. Peldschus et al165 studied the effectiveness of an automated computer-aided detection system for chest CT studies and found both new positives and false positives. Berbaum et al166 found that use of a computer-aided detection system in chest radiography could not counteract the satisfaction-of-search effect (being able to find additional defects beyond the first one) in 16 subjects. In another study, Kakeda et al demonstrated a significant beneficial effect of using computer-aided diagnosis support to help analyse chest radiographs.167

    In mammography, Jiang et al168 found that computer-aided diagnosis reduced inter-observer variability, but in another study computer-assisted mammography interpretation had no beneficial effects on cancer detection and significantly increased the false positive rate of the studies and the biopsy rate.169 A recent commentary on computer-assisted detection noted that while use of this technology is increasingly the norm, the jury is still out on its utility.170 All of the intervention studies reviewed were solid in design and in the interpretation of the results and conclusions, but the ability to generalise is limited due to studies in just two domains (chest x-rays and mammograms).

    Computer-aided interpretation systems

    Interventions

    Two studies focused on technology to improve ECG interpretation. Daudelin and Selker171 reported using an ECG-based acute cardiac ischaemia predictive instrument to improve triage decision-making in the ER. Olsson et al172 studied the use of an artificial neural network trained to automatically detect ECGs indicating possible transmural ischaemia and found that this decision support tool was effective in improving inexperienced interns' interpretation of ECGs.

    General decision support tools for medical diagnosis

    Computer aided decision support tools have also been developed to assist specifically with differential diagnosis. Anecdotally these tools succeed, in a small fraction of searches, in suggesting a difficult or obscure diagnosis that was previously missed. The clinician inputs the patient's key findings, and these programmes suggest possible diagnoses. Some programmes help refine these choices by further suggestions of questions to ask, findings to look for or tests to perform. Berner et al evaluated the first generation products (QMR, DXplain, Iliad and Meditel) using test scenarios and all the products were effective in providing useful suggestions.173 However, the correct diagnosis appeared on the suggestion list only half to three-fourths of the time, and all of the programmes generated a large number of extraneous conditions.174 Some of these initial products are no longer available, although DXplain has been maintained and updated.

    Interventions

    Of the many newer web-based decision support tools, ‘ISABEL’ has been the most extensively evaluated. Compared with first generation tools, ISABEL displays much improved sensitivity in both paediatric settings41 ,42 and in analysing adult case scenarios, in which the sensitivity approached 100%.43–45 ‘Google’ searching has also been evaluated in medical settings, but suggests the correct diagnosis in only 58% of difficult cases.46

    Category 3 summary

    Overall, the technique of ‘getting help’ during the diagnostic process may be beneficial. The use of decision support resources has been studied more extensively than any other intervention, and these approaches, if used, show promise in their potential to reduce diagnostic errors. More research is needed regarding the use of second reviews, teams and librarians.

    Discussion

    Reducing harm from diagnostic errors requires interventions to improve the cognitive processes that underlie clinical reasoning. We identified a reasonably large literature on potential interventions and organised these interventions into three categories: (1) Increasing knowledge and expertise, (2) Improving intuitive and deliberate consideration and (3) Getting help from colleagues, consultants and tools.

    We found that most interventions in the literature were simply ideas or suggestions. Many of these are well conceptualised and widely endorsed, and seem ripe to be tested in experimental or real-world clinical settings. A major finding in each of the three categories was a large discrepancy between the broad and enthusiastic recommendations for the various interventions, but a relative paucity of actual trials. Of the few studies that reported true interventions, few included robust designs or metrics. Typically, the interventions involved an observational study design and measured outcomes before and after an intervention with a small number of trainees or clinicians and/or healthcare sites, without a control group.

    Our findings also affirm that the science of outcome measurement in this area is underdeveloped. Educational interventions in particular are difficult to evaluate in terms of changing attitudes and behaviours in practice. One major issue is the difficulty of demonstrating that diagnosis can be improved by any approach in real-world settings. Definitions of diagnostic error are not standardised and error designations are typically subjective judgements, often confounded by hindsight bias. Measurement instruments and methods to evaluate cognitive intervention effects are not well developed. Additionally, because diagnostic error reflects the interplay of system-related and patient-dependent factors, the true effect of a purely cognitive intervention might be difficult to ascertain. All of these factors pose challenges in the design of future interventions in this area.

    The major limitation of this review is the likelihood that we overlooked conceptual ideas to improve decision-making from both medical and non-medical fields. Medical diagnosis is essentially a special case of decision-making under conditions of uncertainty, and ideas for improving these decisions can arise from almost any discipline, including the social sciences, business fields and military scholars.

    A clear challenge going forward is to identify the advances in these areas that might be applicable to improving the reliability of medical diagnosis. Despite the many shortcomings of these studies, our review identified promising ideas for reducing diagnostic error in each of the three major categories.

    Increase knowledge

    At the present time, disease-specific training is the only intervention that is both supported by evidence and seems implementable. In the future, simulation offers potential both in terms of teaching clinicians about diagnostic error and error-prevention strategies, as well as serving as a method to rapidly build expertise through exposure to many types of disease variants. Feedback also offers the potential to reduce errors by helping develop expertise. Feedback is also the key to reducing overconfidence, which in turn could open the door for clinicians to appreciate the possibility of their own errors and take actions to avoid them. Deliberate feedback is embedded in many approaches that seek to improve individual and team performance outside of medicine.

    Improve clinical reasoning

    Although some of the interventions to improve reasoning have been successful with trainees, most have yet to be implemented or evaluated in practice. Reflective practice and active metacognitive review may have great potential to reduce diagnostic error, and the tools to promote these practices need to be further developed and evaluated in practice. These approaches expand the number of conditions to be considered and effectively address many of the major causes of cognitive error, including context errors, framing bias and premature closure. However, the cost of trade-offs is not clear. For example, will the broadened consideration of alternative diagnoses lead to inappropriate or costly testing, divert attention away from the correct diagnosis or be deleterious in another way?

    Get help

    Decision support for diagnosis has the unique advantage that it can be implemented at the system level, without requiring some new skill or behaviour to be learnt by clinicians. Still, clinicians need to be willing to take advantage of these resources, and error reduction will critically hinge on how well the support functionality is incorporated into everyday workflow and how clinicians will deal with the specificity problem. Using informaticians, working more effectively in groups, taking full advantage of the comprehensive electronic health record and relying more on actuarial tools (algorithms) may be effective strategies. Second opinions and consultations bring fresh eyes to examine a case, a powerful and effective way to find and correct diagnostic errors.

    Conclusions

    In conclusion, there is a surprisingly wide range of possible approaches to reducing the cognitive contributions to diagnostic error. Not all the suggestions have been tested, and of those that have, the evaluations typically involved trainees in artificial settings, making it difficult to extrapolate the results to actual practice.

    The field is immature and progress in reducing diagnostic error will require considerable research to evaluate the relative merits of these different ideas, refinements in the methodology of defining and measuring outcomes in preventing diagnostic error and harm, and leveraging advances in other aspects of medical decision-making and cognitive sciences that may make medical diagnosis more reliable.

    Acknowledgments

    We gratefully acknowledge administrative and literature research assistance from Ms Grace Garey, Mary Lou Glazer, Diane Martin1 and Wendy Isser.

    References

    1. ↵
      1. Berner E,
      2. Graber M
      . Overconfidence as a cause of diagnostic error in medicine. Am J Med 2008;121:S2–23.
      OpenUrlCrossRefPubMed
    2. ↵
      1. Kostopoulou O,
      2. Delaney BC,
      3. Munro CW
      . Diagnostic difficulty and error in primary care - a systematic review. Fam Pract 2008;25:400–13.
      OpenUrlAbstract/FREE Full Text
    3. ↵
      1. Kostopoulou O
      . Do GPs report diagnostic errors? Fam Pract 2008;25:1–2.
      OpenUrlFREE Full Text
    4. ↵
      1. Graber ML,
      2. Franklin N,
      3. Gordon R
      . Diagnostic error in internal medicine. Arch Intern Med 2005;165:1493–9.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Schiff GD,
      2. Hasan O,
      3. Kim S,
      4. et al
      . Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med 2009;169:1881–7.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Zwaan L,
      2. de Bruijne M,
      3. Wagner C,
      4. et al
      . Patient record review of the incidence, consequences, and causes of diagnostic adverse events. Arch Intern Med 2010;170:1015–21.
      OpenUrlCrossRefPubMed
    7. ↵
      1. Kachalia A,
      2. Gandhi TK,
      3. Puopolo AL,
      4. et al
      . Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from four liability insurers. Ann Emerg Med 2007;49:196–205.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Singh H,
      2. Graber ML,
      3. Kissam SM,
      4. et al
      . System-related interventions to reduce diagnostic errors: a narrative review. BMJ Qual Saf 2012;21:160–70.
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Gordon M,
      2. Findley R
      . Educational interventions to improve handover in health care: a systematic review. Med Educ 2011;45:1081–90.
      OpenUrlCrossRefPubMed
    10. ↵
      1. Barr H,
      2. Freeth D,
      3. Hammick M,
      4. et al
      . Evaluations of Interprofessional Education: A United Kingdom Review of Health and Social Care. http://www.caipe.org.uk/silo/files/evaluations-of-interprofessional-education.pdf
    11. ↵
      BEME. Best Evidence in Medical Education. BEME Collaboration, 2003. http://www.bemecollaboration.org
    12. ↵
      1. Fridriksson S,
      2. Hillman J,
      3. Landtblom AM,
      4. et al
      . Education of referring doctors about sudden onset headache in subarachnoid hemorrhage: a prospective study. Acta Neurol Scand 2001;103:238–42.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Rezvyy G,
      2. Parniakov A,
      3. Fedulova E,
      4. et al
      . Correcting biases in psychiatric diagnostic practice in Northwest Russia: comparing the impact of a general educational program and a specific diagnostic training program. BMC Med Educ 2008;8:15.
      OpenUrlCrossRefPubMed
    14. ↵
      1. Gutmark R,
      2. Halsted MJ,
      3. Perry L,
      4. et al
      . Use of computer databases to reduce radiograph reading errors. J Am Coll Radiol 2007;4:65–8.
      OpenUrlCrossRefPubMed
    15. ↵
      1. Bond WF,
      2. Deitrick LM,
      3. Arnold DC,
      4. et al
      . Using simulation to instruct emergency medicine residents in cognitive forcing strategies. Acad Med 2004;79:438–46.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Carlson J,
      2. Abe M,
      3. Bridges D,
      4. et al
      . The impact of a diagnostic reminder system on student clinical reasoning during simulated case studies. Simul Healthc 2011;6:11–18.
      OpenUrlPubMed
    17. ↵
      1. Tudor GR,
      2. Finlay DB
      . Error review: can this improve reporting performance? Clin Radiol 2001;56:751–4.
      OpenUrlCrossRefPubMed
    18. ↵
      1. Wood DS,
      2. Tracey TJG
      . A brief feedback intervention for diagnostic overshadowing. Train Educ Prof Psychol 2009;3:218–25.
      OpenUrlCrossRef
    19. ↵
      1. Sherbino J,
      2. Dore K,
      3. Siu E,
      4. et al
      . The effectiveness of cognitive forcing strategies to decrease diagnostic error—an exploratory study. Teach Learn Med 2011;23:78–85.
      OpenUrlCrossRefPubMed
    20. ↵
      1. Eva KW,
      2. Hatala RM,
      3. LeBlanc VR,
      4. et al
      . Teaching from the clinical reasoning literature: combined reasoning strategies help novice diagnosticians overcome misleading information. Med Educ 2007;41:1152–8.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Mamede S,
      2. van Gog T,
      3. van den Berge K,
      4. et al
      . Effect of availability bias and reflective reasoning on diagnostic accuracy among internal medicine residents. JAMA 2010;304:1198–203.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Coderre S,
      2. Wright B,
      3. McLaughlin K
      . To think is good: querying an initial hypothesis reduces diagnostic error in medical students. Acad Med 2010;85:1125–9.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Wolpaw T,
      2. Papp KK,
      3. Bordage G
      . Using SNAPPS to facilitate the expression of clinical reasoning and uncertainties: a randomized comparison group trial. Acad Med 2009;84:517–24.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Raab SS,
      2. Grzybicki DM,
      3. Mahood LK,
      4. et al
      . Effectiveness of random and focused review in detecting surgical pathology error. Am J Clin Pathol 2008;130:905–13.
      OpenUrlAbstract/FREE Full Text
    25. ↵
      1. Raab SS,
      2. Stone CH,
      3. Jensen CS,
      4. et al
      . Double slide viewing as a cytology quality improvement initiative. Am J Clin Pathol 2006;125:526–3.
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Manion E,
      2. Cohen MB,
      3. Weydert J
      . Mandatory second opinion in surgical pathology referral material: clinical consequences of major disagreements. Am J Surg Pathol 2008;32:732–7.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Nordrum I,
      2. Johansen M,
      3. Amin A,
      4. et al
      . Diagnostic accuracy of second-opinion diagnoses based on still images. Hum Pathol 2004;35:129–35.
      OpenUrlCrossRefPubMed
    28. ↵
      1. Hamady ZZ,
      2. Mather N,
      3. Lansdown MR,
      4. et al
      . Surgical pathological second opinion in thyroid malignancy: impact on patients' management and prognosis. Eur J Surg Oncol 2005;31:74–7.
      OpenUrlCrossRefPubMed
    29. ↵
      1. Benger JR,
      2. Lyburn ID
      . What is the effect of reporting all emergency department radiographs? Emerg Med J 2003;20:40–3.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. Espinosa JA,
      2. Nolan TW
      . Reducing errors made by emergency physicians in interpreting radiographs: longitudinal study. BMJ 2000;320:737–40.
      OpenUrlAbstract/FREE Full Text
    31. ↵
      1. Duijm LE,
      2. Groenewoud JH,
      3. Fracheboud J,
      4. et al
      . Additional double reading of screening mammograms by radiologic technologists: impact on screening performance parameters. J Natl Cancer Inst 2007;99:1162–70.
      OpenUrlAbstract/FREE Full Text
    32. ↵
      1. Kwek BH,
      2. Lau TN,
      3. Ng FC,
      4. et al
      . Non-consensual double reading in the Singapore breast screening project: benefits and limitations. Ann Acad Med Singapore 2003;32:438–41.
      OpenUrlPubMed
    33. ↵
      1. Canon CL,
      2. Smith JK,
      3. Morgan DE,
      4. et al
      . Double reading of barium enemas: is it necessary? AJR Am J Roentgenol 2003;181:1607–10.
      OpenUrlPubMedWeb of Science
    34. ↵
      1. Christensen C,
      2. Larson JR Jr.,
      3. Abbott A,
      4. et al
      . Decision making of clinical teams: communication patterns and diagnostic error. Med Decis Making 2000;20:45–50.
      OpenUrlAbstract/FREE Full Text
    35. ↵
      1. Mulvaney SA,
      2. Bickman L,
      3. Giuse NB,
      4. et al
      . A randomized effectiveness trial of a clinical informatics consult service: impact on evidence-based decision-making and knowledge implementation. J Am Med Inform Assoc 2008;15:203–11.
      OpenUrlAbstract/FREE Full Text
    36. ↵
      1. Pozen MW,
      2. D'Agostino RB,
      3. Selker HP,
      4. et al
      . A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease. N Engl J Med 1984;310:1273–8.
      OpenUrlCrossRefPubMedWeb of Science
    37. ↵
      1. Selker HP,
      2. Beshansky JR,
      3. Griffith JL,
      4. et al
      . Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute cardiac ischemia. A multicenter, controlled clinical trial. Ann Intern Med 1998;129:845–55.
      OpenUrlCrossRefPubMedWeb of Science
    38. ↵
      1. Bogusevicius A,
      2. Maleckas A,
      3. Pundzius J,
      4. et al
      . Prospective randomised trial of computer-aided diagnosis and contrast radiography in acute small bowel obstruction. Eur J Surg 2002;168:78–83.
      OpenUrlCrossRefPubMedWeb of Science
    39. ↵
      1. De Simone R,
      2. Coppola G,
      3. Ranieri A,
      4. et al
      . Validation of AIDA Cefalee, a computer-assisted diagnosis database for the management of headache patients. Neurol Sci 2007;28(Suppl 2):S213–16.
      OpenUrlCrossRefPubMed
    40. ↵
      1. Cimino JJ
      . Use, usability, usefulness, and impact of an infobutton manager. AMIA Annu Symp Proc 2006:151–5.
    41. ↵
      1. Ramnarayan P,
      2. Winrow A,
      3. Coren M,
      4. et al
      . Diagnostic omission errors in acute paediatric practice: impact of a reminder system on decision-making. BMC Med Inform Decis Mak 2006;6:37.
      OpenUrlCrossRefPubMed
    42. ↵
      1. Ramnarayan P,
      2. Cronje N,
      3. Brown R,
      4. et al
      . Validation of a diagnostic reminder system in emergency medicine: a multi-centre study. Emerg Med J 2007;24:619–24.
      OpenUrlAbstract/FREE Full Text
    43. ↵
      1. Ramnarayan P,
      2. Kapoor RR,
      3. Coren J,
      4. et al
      . Measuring the impact of diagnostic decision support on the quality of clinical decision making: development of a reliable and valid composite score. J Am Med Inform Assoc 2003;10:563–72.
      OpenUrlAbstract/FREE Full Text
    44. ↵
      1. Ramnarayan P,
      2. Roberts GC,
      3. Coren M,
      4. et al
      . Assessment of the potential impact of a reminder system on the reduction of diagnostic errors: a quasi-experimental study. BMC Med Inform Decis Mak 2006;6:22.
      OpenUrlCrossRefPubMed
    45. ↵
      1. Graber ML,
      2. Mathew A
      . Performance of a web-based clinical diagnosis support system for internists. J Gen Intern Med 2008;23(Suppl 1):37–40.
      OpenUrlPubMedWeb of Science
    46. ↵
      1. Tang H,
      2. Ng JH
      . Googling for a diagnosis–use of Google as a diagnostic aid: internet based study. BMJ 2006;333:1143–5.
      OpenUrlAbstract/FREE Full Text
    47. ↵
      1. Kassirer JP,
      2. Wong J,
      3. Kopelman R
      . Learning Clinical Reasoning. 2nd edn. Baltimore, MD: Lippincott Williams and Wilkins, 2010.
    48. ↵
      1. Gigerenzer G
      . Adaptive Thinking: Rationality in the Real World. New York: Oxford University Press, 2000.
    49. ↵
      1. Elstein AS
      . Thinking about diagnostic thinking: a 30-year perspective. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):7–18.
      OpenUrlCrossRefPubMed
    50. ↵
      1. Klein G
      . Sources of Power: How People Make Decisions. Cambridge, MA: The MIT Press, 1998.
    51. ↵
      1. Klein G
      . The Power of Intuition. New York, NY: Doubleday, 2003.
      1. Bazerman MH,
      2. Moore DA
      . Improving Decision Making. Judgment in Managerial Decision Making. John Wiley and Sons, 2009:179–99.
    53. ↵
      1. Norman G
      . Dual processing and diagnostic errors. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):37–49.
      OpenUrlCrossRefPubMed
    54. ↵
      1. Smith-Bindman R,
      2. Chu PW,
      3. Miglioretti DL,
      4. et al
      . Comparison of screening mammography in the United States and the United kingdom. JAMA 2003;290:2129–37.
      OpenUrlCrossRefPubMedWeb of Science
    55. ↵
      1. Singh H,
      2. Sethi S,
      3. Raber M,
      4. et al
      . Errors in cancer diagnosis: current understanding and future directions. J Clin Oncol 2007;25:5009–18.
      OpenUrlAbstract/FREE Full Text
    56. ↵
      1. Bowen JL
      . Educational strategies to promote clinical diagnostic reasoning. N Engl J Med 2006;355:2217–25.
      OpenUrlCrossRefPubMedWeb of Science
    57. ↵
      1. Davis D,
      2. O'Brien MA,
      3. Freemantle N,
      4. et al
      . Impact of formal continuing medical education: do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? JAMA 1999;282:867–74.
      OpenUrlCrossRefPubMedWeb of Science
      1. Freidlander ML,
      2. Phillips SD
      . Preventing anchoring errors in clinical judgment. J Consult Clin Psychol 1984;52:366–71.
      OpenUrlCrossRefPubMedWeb of Science
      1. Gentner D,
      2. Lowenstein G,
      3. Thompson L
      . Learning and transfer: a general role for analogic encoding. J Educ Psychol 2003;95:393–408.
      OpenUrl
      1. Hershberger PJ,
      2. Markert RJ,
      3. Part HM,
      4. et al
      . Understanding and addressing cognitive bias in medical education. Adv Health Sci Educ 1996;1:221–6.
      OpenUrlCrossRefWeb of Science
      1. Parmley MC
      . Confirmation Bias on Diagnostic Decision Making. Drexel University, 2006.
      1. Hogarth R
      . Educating Intuition. University of Chicago Press, 2001.
    63. ↵
      1. Bond W,
      2. Kuhn G,
      3. Binstadt E,
      4. et al
      . The use of simulation in the development of individual cognitive expertise in emergency medicine. Acad Emerg Med 2008;15:1037–45.
      OpenUrlCrossRefPubMedWeb of Science
      1. Schiff GD
      . Minimizing diagnostic error: the importance of follow-up and feedback. Am J Med 2008;121:S38–42.
      OpenUrlPubMed
    65. ↵
      1. Jamtvedt G,
      2. Young JM,
      3. Kristoffersen DT,
      4. et al
      . Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback. Qual Saf Health Care 2006;15:433–6.
      OpenUrlAbstract/FREE Full Text
    66. ↵
      1. Papa FJ,
      2. Aldrich D,
      3. Schumacker RE
      . The effects of immediate online feedback upon diagnostic performance. Acad Med 1999;74:S16–18.
      OpenUrlCrossRefPubMedWeb of Science
    67. ↵
      1. Stone ER,
      2. Opel RB
      . Training to improve calibration and discrimination: the effects of performance and environment feedback. Organ Behav Hum Decis Process 2000;83:282–309.
      OpenUrlCrossRefPubMedWeb of Science
    68. ↵
      1. Alpert HR,
      2. Hillman BJ
      . Quality and variability in diagnostic radiology. J Am Coll Radiol 2004;1:127–32.
      OpenUrlCrossRefPubMed
      1. Arkes H
      . Impediments to accurate clinical judgment and possible ways to minimize their impact. J Consult Clin Psychol 1981;49:323–30.
      OpenUrlCrossRefPubMedWeb of Science
      1. Humble JE,
      2. Keim RT,
      3. Hershauer JC
      . Information systems design: an empirical study of feedback effects. Behav Inf Technol 1992;11:237–44.
      OpenUrlCrossRef
      1. Pulford BD,
      2. Colman AM
      . Overconfidence: feedback and item difficulty effects. Pers Individ Differ 1997;23:125–33.
      OpenUrlCrossRefWeb of Science
      1. Subbotin V
      . Outcome feedback effects on under- and overconfidence judgments (general knowledge tasks). Organ Behav Hum Decis Process 1996;66:268–76.
      OpenUrlCrossRef
    73. ↵
      1. Fischer MA,
      2. Mazor KM,
      3. Baril J,
      4. et al
      . Learning from mistakes. Factors that influence how students and residents learn from medical errors. J Gen Intern Med 2006;21:419–23.
      OpenUrlCrossRefPubMedWeb of Science
    74. ↵
      1. Eva KW
      . Diagnostic error in medical education: where wrongs can make rights. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):71–81.
      OpenUrlCrossRefPubMed
    75. ↵
      1. Eva KW
      . What every teacher needs to know about clinical reasoning. Med Educ 2005;39:98–106.
      OpenUrlCrossRefPubMedWeb of Science
    76. ↵
      1. Croskerry P
      . Cognitive forcing strategies in clinical decision making. Ann Emerg Med 2003;41:110–20.
      OpenUrlCrossRefPubMedWeb of Science
    77. ↵
      1. Croskerry P
      . The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med 2003;78:775–80.
      OpenUrlCrossRefPubMedWeb of Science
    78. ↵
      1. Croskerry P
      . Diagnostic Failure: A Cognitive and Affective Approach in Advances in Patient Safety: From Research to Implementation. Rockville, MD: Agency for Healthcare Research and Quality (Publication No. 050021), 2005;2:241–54.
      OpenUrl
    79. ↵
      1. Wedding D,
      2. Faust D
      . Clinical judgment and decision making in neuropsychology. Arch Clin Neuropsychol 1989;4:233–65.
      OpenUrlAbstract/FREE Full Text
      1. Groves M,
      2. O'Rourke P,
      3. Alexander H
      . Clinical reasoning: the relative contribution of identification, interpretation and hypothesis errors to misdiagnosis. Med Teach 2003;25:621–5.
      OpenUrlCrossRefPubMedWeb of Science
      1. Pines JM
      . Profiles in patient safety: confirmation bias in emergency medicine. Acad Emerg Med 2006;13:90–4.
      OpenUrlCrossRefPubMedWeb of Science
    82. ↵
      1. Moulton CE,
      2. Regehr G,
      3. Mylopoulos M,
      4. et al
      . Slowing down when you should: a new model of expert judgement. Acad Med 2007;82:S109–16.
      OpenUrlPubMedWeb of Science
    83. ↵
      1. Trowbridge RL
      . Twelve tips for teaching avoidance of diagnostic errors. Med Teach 2008;30:496–500.
      OpenUrlCrossRefPubMed
      1. Kuhn GJ
      . Diagnostic errors. Acad Emerg Med 2002;9:740–50.
      OpenUrlCrossRefPubMedWeb of Science
    85. ↵
      1. Fischhoff B
      . Debiasing. In: Kahneman D, Slovic P, Tversky A, eds. Judgment Under Uncertainty. New York: Cambridge University Press, 1982:422–44.
    86. ↵
      1. Milkman KL,
      2. Chugh D,
      3. Bazerman MH
      . How can decision making be improved? Perspect Psychol Sci 2009;4:379–83.
      OpenUrlAbstract/FREE Full Text
    87. ↵
      1. Larrick RP
      . Debiasing. In: Koehler DJ, Harvery N, eds. Blackwell Handbook of Judgment and Decision Making. Oxford, UK: Blackwell, 2004.
      1. Koriat A,
      2. Bjork RA,
      3. Sheffer L,
      4. et al
      . Predicting one's own forgetting: the role of experience-based and theory-based processes. J Exp Psychol Gen 2004;133:643–56.
      OpenUrlCrossRefPubMedWeb of Science
      1. Renner C,
      2. Renner M
      . But I thought I knew that: using confidence estimation as a debiasing technique to improve Classroom performance. Appl Cogn Psychol 2001;15:23–32.
      OpenUrlCrossRef
      1. Estrada CA,
      2. Isen AM,
      3. Young MJ
      . Positive affect facilitates integration of information and decreases anchoring in reasoning among physicians. Organ Behav Hum Decis Process 1997;72:117–35.
      OpenUrlCrossRefWeb of Science
    91. ↵
      1. Scott IA
      . Errors in clinical reasoning: causes and remedial strategies. BMJ 2009;338:b1860.
      OpenUrlFREE Full Text
    92. ↵
      1. Slovic P,
      2. Fischoff B
      . On the psychology of experimental surprises. J Exp Psychol Hum Percept Perform 1977;3:544–51.
      OpenUrlCrossRefWeb of Science
    93. ↵
      1. Lichtenstein S,
      2. Fischoff B,
      3. Bishop B
      . Calibration of probabilities: The state of the art to 1980. In: Kahneman D, Slovic P, Tversky A, eds. Judgment Under Uncertainty: Heuristics and Biases. New York: Cambridge University Press, 1982.
    94. ↵
      1. Schon DA
      . Educating the Reflective Practioner. San Francisco: Jossey-Bass, 1987.
      1. Greenhalgh T
      . Intuition and evidence–uneasy bedfellows? Br J Gen Pract 2002;52:395–400.
      OpenUrlAbstract/FREE Full Text
    96. ↵
      1. Mamede S,
      2. Schmidt HG,
      3. Rikers R
      . Diagnostic errors and reflective practice in medicine. J Eval Clin Pract 2007;13:138–45.
      OpenUrlCrossRefPubMedWeb of Science
    97. ↵
      1. Singh H,
      2. Petersen LA,
      3. Thomas EJ
      . Understanding diagnostic errors in medicine: a lesson from aviation. Qual Saf Health Care 2006;15:159–64.
      OpenUrlAbstract/FREE Full Text
    98. ↵
      1. Brawn R
      . The formal and the intuitive in science and medicine. In: Atkinson T, Claxton G, eds. The Intuitive Practitioner. Buckingham UK: Open University Press, 2000:149–64.
      1. Gregory G
      . Developing intuition through management education. In: Atkinson T, Claxton G, eds. The Intuitive Practitioner. Buckingham, UK: Open University Press, 2000.
    100. ↵
      1. Noddings S,
      2. Shore PJ
      . Awakening the Inner Eye: Intuition in Education. New York: Teachers College Press, 1984.
    101. ↵
      1. Quirk M
      . Intuition and Metacognition in Medical Education. New York, NY: Springer Publishing Co., 2006.
    102. ↵
      1. Hamm RM,
      2. Zubialde J
      . Physicians' expert cognition and the problem of cognitive bias. Med Decis Making 1995;22:181–213.
      OpenUrl
    103. ↵
      1. Croskerry P
      . The cognitive imperative: thinking about how we think. Acad Emerg Med 2000;7:1223–31.
      OpenUrlCrossRefPubMedWeb of Science
      1. Leonidas L
      . Ten commandments to reduce cognitive errors. In: Graber, ML. Educational interventions to reduce diagnostic error. Adv Health Sci Educ 2009;14:63–9.
      OpenUrlCrossRefWeb of Science
      1. Gawande A
      . The Checklist Manifesto: How to Get Things Right. 1st edn. New York: Metropolitan Books, 2010.
    106. ↵
      1. Ely JW,
      2. Graber M,
      3. Croskerry PC
      . Checklists to reduce diagnostic errors. Acad Med 2011;86:7.
      OpenUrlWeb of Science
    107. ↵
      1. Taleb N
      . The Black Swan. New York: Random House, 2007.
    108. ↵
      1. Brannick MT,
      2. Fabri PJ,
      3. Zayas-Castro J,
      4. et al
      . Evaluation of an error-reduction training program for surgical residents. Acad Med 2009;84:1809–14.
      OpenUrlCrossRefPubMedWeb of Science
      1. Arkes H
      . Why medical errors can't be eliminated: uncertainties and the hindsight bias. Chron High Educ 2000;46:591–600.
      OpenUrl
      1. Schwenk CR
      . Cognitive simplification processes in strategic decision-making. Strategic Manag J 1984;5:111–28.
      OpenUrlCrossRef
    111. ↵
      1. Baron J
      . Thinking and Deciding. 3rd edn. New York, NY: Cambridge University Press.
    112. ↵
      1. Mitchell DJ,
      2. Russo JE,
      3. Pennington N
      . Back to the future: temporal perspective in the explanation of events. J Behav Decis Making 1989;2:25–38.
      OpenUrlCrossRef
      1. Mussweiler T,
      2. Strack F,
      3. Pfeiffer T
      . Overcoming the inevitable anchoring effect: considering the opposite compensates for selective accessibility. Pers Soc Psychol Bull 2000;26:1142–50.
      OpenUrlAbstract/FREE Full Text
      1. Lord CG,
      2. Lepper MR,
      3. Preston E
      . Consider the opposite: a corrective strategy for social judgment. J Pers Soc Psychol 1984;37:1231–43.
      OpenUrl
      1. Arzy S,
      2. Brezis M,
      3. Khoury S,
      4. et al
      . Misleading one detail: a preventable mode of diagnostic error? J Eval Clin Pract 2009;15:804–6.
      OpenUrlCrossRefPubMed
      1. Gorman M,
      2. Gorman M
      . A comparison of disconfirmatory, confirmatory and control strategies on Watson's 2-4-6 task. Q J Exp Psychol 1984;36A:629–48.
      OpenUrlCrossRef
      1. Hirt E,
      2. Markman K
      . Multiple explanation: a consider-an-alternative strategy for debiasing judgments. J Pers Soc Psychol 1995;69:1069–86.
      OpenUrlCrossRefWeb of Science
      1. Mumma G,
      2. Steven W
      . Procedural debiasing of primary/anchoring effects in clinical-like judgments. J Clin Psychol 1995;51:841–53.
      OpenUrlCrossRefPubMedWeb of Science
      1. Strauss SE,
      2. Glasziou P,
      3. Richardson WS,
      4. et al
      . Evidence-Based Medicine, 4th edn. How to Practice and Teach. New York, NY: Churchill Livingstone, 2011.
      1. Sox HC
      . Medical Decision Making. Butterworth-Heinman, 1988.
    121. ↵
      1. Dobbie AE,
      2. Schneider FD,
      3. Anderson AD,
      4. et al
      . What evidence supports teaching evidence-base medicine? Acad Med 2000;75:1184–5.
      OpenUrlCrossRefPubMedWeb of Science
    122. ↵
      1. Khan KS,
      2. Coomarasamy A
      . A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ 2006;15:59.
      OpenUrl
      1. Croskerry P
      . Clinical cognition and diagnostic error: applications of a dual process model of reasoning. Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):27–35.
      OpenUrlCrossRefPubMed
      1. Ullman DG
      . Making Robust Decisions: Decision Management for Technical, Business, and Service Teams. Victoria, BC: Trafford Publishing, 2006.
      1. Pauker SC,
      2. Kassirer JP
      . The threshold approach to clinical decision making. N Engl J Med 1980;302:1109–17.
      OpenUrlCrossRefPubMedWeb of Science
    126. ↵
      1. Kassirer JP
      . Teaching clinical reasoning: case-based and coached. Acad Med 2010;85:1118–24.
      OpenUrlCrossRefPubMedWeb of Science
      1. Tatarka CJ
      . Overcoming biases in military problem analysis and decision-making. Mil Intelligence 2002;28:8–10.
      OpenUrl
      1. Raab SS
      . Improving patient safety by examining pathology errors. Clin Lab Med 2004;24:863.
      OpenUrl
    129. ↵
      1. Zipperer L,
      2. Sykes J
      . The role of librarians in patient safety: gaps and strengths in the current culture. J Med Libr Assoc 2004;92:498–500.
      OpenUrlPubMed
    130. ↵
      1. Albert KM
      . Integrating knowledge-based resources into the electronic health record: history, current status, and role of librarians. Med Ref Serv Q 2007;26:1–19.
      OpenUrlPubMed
    131. ↵
      1. Zipperer L,
      2. Sykes J
      . Engaging as partners in patient safety: the experience of librarians. Patient Saf Qual Healthc. March/April 2009;6:28–33, 32–3.
      OpenUrl
    132. ↵
      1. Wedding D
      . Comparison of statistical and actuarial models for predicting lateralization of brain-damage. Clin Neuropsychol 1983;5:15–20.
      OpenUrl
    133. ↵
      1. Hunt DL,
      2. Haynes RB,
      3. Hanna SE,
      4. et al
      . Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA 1998;280:1339–46.
      OpenUrlCrossRefPubMedWeb of Science
    134. ↵
      1. Klassen TP,
      2. Ropp LJ,
      3. Sutcliffe T,
      4. et al
      . A randomized controlled trial of radiograph ordering for extremity trauma in a pediatric emergency department. Ann Emerg Med 1993;22:1524–9.
      OpenUrlCrossRefPubMed
    135. ↵
      1. Garg AX,
      2. Adhikari NK,
      3. McDonald H,
      4. et al
      . Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. J Am Med Assoc 2005;293:1223–38.
      OpenUrlCrossRefPubMedWeb of Science
    136. ↵
      1. Cannon DS,
      2. Allen SN
      . A comparison of the effects of computer and manual reminders on compliance with a mental health clinical practice guideline. J Am Med Inform Assoc 2000;7:196–203.
      OpenUrlAbstract/FREE Full Text
    137. ↵
      1. Radecki RP,
      2. Medow MA
      . Cognitive debiasing through sparklines in clinical data displays. AMIA Annu Symp Proc 2007:1085.
      1. Reyna VF,
      2. Nelson WL,
      3. Han PK,
      4. et al
      . How numeracy influences risk comprehension and medical decision making. Psychol Bull 2009;135:943–73.
      OpenUrlCrossRefPubMedWeb of Science
      1. Bhandari G,
      2. Hassanein K,
      3. Deaves R
      . Debiasing investors with decision support systems: an experimental investigation. Decis Support Syst 2008;46:399–410.
      OpenUrlCrossRefWeb of Science
      1. Cook MB,
      2. Smallman HS
      . Human factors of the confirmation bias in intelligence analysis: decision support from graphical evidence landscapes. Hum Factors 2008;50:745–54.
      OpenUrlAbstract/FREE Full Text
    141. ↵
      1. Kawamoto K,
      2. Lobach DF
      . Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior. AMIA Annu Symp Proc 2003:361–5.
    142. ↵
      1. Schiff G,
      2. Bates DW
      . Can electronic clinical documentation help prevent diagnostic errors? N Engl J Med 2010;362:1066–9.
      OpenUrlCrossRefPubMedWeb of Science
    143. ↵
      1. Norman G
      . Building on experience–the development of clinical reasoning. N Engl J Med 2006;355:2251–2.
      OpenUrlCrossRefPubMedWeb of Science
    144. ↵
      1. Chi MTH,
      2. Glaser R,
      3. Farr MJ
      . The Nature of Expertise. Hillsdale, NJ: L. Erlbaum Associates, 1988.
    145. ↵
      1. Ericsson KA
      . Deliberate practice and acquisition of expert performance: a general overview. Acad Emerg Med 2008;15:988–94.
      OpenUrlCrossRefPubMedWeb of Science
    146. ↵
      1. Norman G
      . Research in clinical reasoning: past history and current trends. Med Educ 2005;39:418–27.
      OpenUrlCrossRefPubMedWeb of Science
    147. ↵
      1. Bordage G
      . Why did I miss the diagnosis? Some cognitive explanations and educational implications. Acad Med 1999;74:S138–43.
      OpenUrlCrossRefPubMedWeb of Science
    148. ↵
      1. Lee JK
      . Quality—a radiology imperative: interpretation accuracy and pertinence. J Am Coll Radiol 2007;4:162–5.
      OpenUrlCrossRefPubMed
    149. ↵
      1. Wood D
      . An Intervention for Diagnostic Overshadowing [e-book]. US: ProQuest Information & Learning. Ipswich, MA: PsycINFO, 2005.
    150. ↵
      1. Shojania KG,
      2. Burton EC,
      3. McDonald KM,
      4. et al
      . The autopsy as an Outcome and Performance Measure (Evidence Report/Technology Assessment No. 58; AHRQ Publication No. 03–E002). Rockville, MD: Agency for Healthcare Research and Quality, 2002.
    151. ↵
      1. Hogarth RM
      . Educating Intuition. Chicago: University of Chicago Press, 2001.
    152. ↵
      1. Croskerry P
      . A universal model of diagnostic reasoning. Acad Med 2009;84:1022–8.
      OpenUrlCrossRefPubMedWeb of Science
    153. ↵
      1. Coderre S,
      2. Mandin H,
      3. Harasyn PH,
      4. et al
      . Diagnostic reasoning strategies and diagnostic success. Med Educ 2003;37:695–703.
      OpenUrlCrossRefPubMedWeb of Science
    154. ↵
      1. Arkes HR,
      2. Faust D,
      3. Guilmette TJ,
      4. et al
      . Eliminating the hindsight bias. J Appl Psychol 1988;73:305–7.
      OpenUrlCrossRefWeb of Science
    155. ↵
      1. Graber M
      . Metacognitive training to reduce diagnostic errors: ready for prime time? Acad Med 2003;78:781.
      OpenUrlCrossRefPubMedWeb of Science
    156. ↵
      1. Fioratou E,
      2. Flin R,
      3. Glavin R
      . No simple fix for fixation errors: cognitive processes and their clinical applications. Anaesthesia 2010;65:61–9.
      OpenUrlCrossRefPubMedWeb of Science
    157. ↵
      1. Graber ML
      . Educational strategies to reduce diagnostic error: can you teach this stuff? Adv Health Sci Educ Theory Pract 2009;14(Suppl 1):63–9.
      OpenUrlCrossRefPubMed
    158. ↵
      1. Sackett DL,
      2. Strauss SE,
      3. Richardson WS,
      4. et al
      . Evidence-Based Medicine: How to Practice and Teach. New York: Churchill Livingstone, 1997.
    159. ↵
      1. Sox HC Jr.,
      2. Blatt MA,
      3. Higgins MC,
      4. et al
      . Medical Decision Making. Stoneham, MA: Butterworth-Heinemann, 1988.
    160. ↵
      1. Janis IL,
      2. Mann L
      . Decision Making: A Psychological Analysis of Conflict, Choice, and Commitment. New York: Free Press, 1977.
    161. ↵
      1. Surowiecki J
      . The Wisdom of Crowds. New York, NY: Anchor Books, 2005.
    162. ↵
      1. Wedding D
      . Clinical and statistical prediction in neuropsychology. Clin Neuropsychol 1983;5:49–55.
      OpenUrl
    163. ↵
      1. Smith WR
      . Evidence for the effectiveness of techniques to change physician behavior. Chest 2000;118:8S–17S.
      OpenUrlCrossRefPubMedWeb of Science
    164. ↵
      1. Militello L,
      2. Patterson ES,
      3. Tripp-Reimer T,
      4. et al
      . Clinical reminders: Why don't people use them? Proceedings of the human factors and Ergonomics Society - 48th Annual Meeting, New Orleans, Louisiana, 2004;48:1651–5.
      OpenUrlAbstract/FREE Full Text
    165. ↵
      1. Peldschus K,
      2. Herzog P,
      3. Wood SA,
      4. et al
      . Computer-aided diagnosis as a second reader: spectrum of findings in CT studies of the chest interpreted as normal. Chest 2005;128:1517–23.
      OpenUrlCrossRefPubMedWeb of Science
    166. ↵
      1. Berbaum KS,
      2. Caldwell RT,
      3. Schartz KM,
      4. et al
      . Does computer-aided diagnosis for lung tumors change satisfaction of search in chest radiography? Acad Radiol 2007;14:1069–76.
      OpenUrlCrossRefPubMed
    167. ↵
      1. Kakeda S,
      2. Moriya J,
      3. Sato H,
      4. et al
      . Improved detection of lung nodules on chest radiographs using a commercial computer-aided diagnosis system. AJR Am J Roentgenol 2004;182:505–10.
      OpenUrlCrossRefPubMedWeb of Science
    168. ↵
      1. Jiang Y,
      2. Nishikawa RM,
      3. Schmidt RA,
      4. et al
      . Potential of computer-aided diagnosis to reduce variability in radiologists' interpretations of mammograms depicting microcalcifications. Radiology 2001;220:787–94.
      OpenUrlCrossRefPubMedWeb of Science
    169. ↵
      1. Fenton JJ,
      2. Taplin SH,
      3. Carney PA,
      4. et al
      . Influence of computer-aided detection on performance of screening mammography. N Engl J Med 2007;356:1399–409.
      OpenUrlCrossRefPubMedWeb of Science
    170. ↵
      1. Hall FM
      . Breast imaging and computer-aided detection. N Engl J Med 2007;356:1464–6.
      OpenUrlCrossRefPubMedWeb of Science
    171. ↵
      1. Daudelin DH,
      2. Selker HP
      . Medical error prevention in ED triage for ACS: use of cardiac care decision support and quality improvement feedback. Cardiol Clin 2005;23:601–14, ix.
      OpenUrlCrossRefPubMed
    172. ↵
      1. Olsson SE,
      2. Ohlsson M,
      3. Ohlin H,
      4. et al
      . Decision support for the initial triage of patients with acute coronary syndromes. Clin Physiol Funct Imaging 2006;26:151–6.
      OpenUrlCrossRefPubMedWeb of Science
    173. ↵
      1. Berner ES,
      2. Webster GD,
      3. Shugerman AA,
      4. et al
      . Performance of four computer-based diagnostic systems. N Engl J Med 1994;330:1792–6.
      OpenUrlCrossRefPubMedWeb of Science
    174. ↵
      1. Kassirer JP
      . A report card on computer-assisted diagnosis—the grade: C. N Engl J Med 1994;330:1824–5.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • The authors of this paper are solely responsible for its content, and disclosed no competing interests. The findings and interpretations in the paper do not represent the opinions or recommendations of the institutions with which the authors are affiliated, the Agency for Healthcare Research and Quality, or the US Department of Health and Human Services, Department of Veterans Affairs.

    • Funding This study was funded by the Agency for Healthcare Research and Quality (AHRQ) ACTION II Task Order #8, Contract No. HHSA290200600001 and in part by the Houston VA HSR&D Center of Excellence (HFP90-020).

    • Competing interests None.

    • Provenance and peer review Not commissioned; externally peer reviewed.