Existing research methods

Multiple research methods, largely retrospective, have been used to examine the epidemiology of diagnostic error (for an extensive overview on incidence, see Graber's article in this issue)10 Commonly used data sources include reports of malpractice claims,6 ,11 chart reviews of selected diagnoses or hospital admissions2 ,12 and autopsy reports.13 ,14 While these studies have given us a general appreciation of the burden of the problem, these estimates of incidence of the problem are far from precise or generalisable. Retrospective methods also introduce hindsight bias in judgments about error determination. Nevertheless, one advantage of retrospective methods is the potential availability of longitudinal data that spans the continuum of care of the patient (ie, outpatient visits, multiple subsequent hospital admissions, outcomes, etc). This allows researchers to retrospectively track the evolution of the diagnostic process over time and link it to diagnostic outcomes (ie, final diagnosis).2

Prospective methods have advantages such as reduced hindsight bias but used infrequently to study the epidemiology of diagnostic error. One example is the use of ‘standardised patients’ who present to providers in routine clinical settings without revealing their actual purpose.15 ,16 One study involved 23 rheumatologists who were visited by a standardised patient with known psoriatic arthritis. The diagnosis was missed or wrong in nine visits (39%). However, such methods are resource intensive. Other types of prospective methods involve presentation of hypothetical cases to assess physician cognitive error rates.17 ,18 However, these methods are mainly used to determine how varying circumstances or characteristics of physicians or patients influence error rates, and do not help determine incidence rates representative of a certain population.

Although previous studies have given us a general appreciation of the burden of diagnostic error, the estimated incidence rates of diagnostic errors vary greatly, from about 1% to as high as 55% in certain diseases or patient groups.1 This large variation can in part be explained by four factors:

1. Setting of the error and patient population: errors in certain specialty or ancillary care settings, such as radiology, appear to be less common than errors in clinical medicine. Also, diagnostic errors are more likely to occur with older or more complex patients compared with younger patients who have fewer comorbidities.1

2. Study methods: higher error rates appear to be related in part to use of data sources that over-represent severe or lethal outcomes, such as autopsy studies and malpractice claims. Conversely, other data sources, such as chart reviews, might not uncover all diagnostic errors due to lack of documented information (ie, ascertainment bias).

3. Type of diagnosis: certain diagnoses are likely more easily missed or confused, especially at an earlier stage of the illness, for example, tuberculosis or HIV.

4. Definition of diagnostic error: some studies define diagnostic error as any diagnosis that is wrong, delayed or missed, regardless of whether an error in the diagnostic process occurred.19 ,20 Other studies define diagnostic errors only when there is clear evidence of ‘missed opportunities’ for earlier diagnosis in a patient's care.12 ,21 Therefore, the rates of diagnostic error found in different studies are likely to vary substantially according to criteria used to define ‘error’.

Directions for future research on the epidemiology of diagnostic error

Obtaining more precise rates of error is an enormous challenge, given the heterogeneous data and methodological and logistical challenges noted above. Nevertheless, such an undertaking is necessary to establish base rates in common settings of care (primary care, general medical hospitals, etc), to both attract needed attention to the urgency and magnitude of the problem as well as prioritise and evaluate intervention efforts. Such a project could have an important impact in ways similar to the IOM's landmark report on medical errors and patient safety,22 which propelled the field of patient safety forward. Suggestions for specific methodological and research approaches to advance this agenda include:

1. Synthesise existing studies to estimate rates of diagnostic error: the true incidence of diagnostic error remains unknown. As more sophisticated methods for detecting diagnostic error are being developed, an early first step would be to apply rigorous methods to synthesise findings across studies in order to estimate incidence rates of diagnostic error. We recommend that incidence rates of diagnostic errors should be determined separately for the different settings (eg, inpatient hospital setting, primary care and outpatient setting) to account for differences in patient populations and diagnostic techniques. A synthesis of existing studies would also help further clarify the effect of methodological differences among the studies.

2. Evaluate incidence rates over time: longitudinal studies are needed to determine trends related to the incidence of diagnostic error, to identify any fluctuations in types of diagnostic errors in different settings and to understand what types of diagnoses might be more or less vulnerable to diagnostic error over time (eg, in response to the adoption of new diagnostic procedures and technologies). In addition, within organisations and systems that are implementing strategies to reduce error, it is important to know whether general patient safety interventions (such as improved safety culture, patient engagement), or certain targeted interventions (such as techniques to improve test result follow-up) actually result in lower rates of diagnostic error. Conducting studies using similar definitions, measurement methods and time intervals (eg, every 5 years) can provide insights into which processes, settings, patient groups or diseases are in greatest need of attention over time, and which interventions appear to have the greatest impact on patient safety.

3. The use of Electronic Health Records (EHRs) to identify diagnostic error: currently, manual chart reviews are considered the gold standard for detecting diagnostic errors. This method is labour-intensive and extensive resources are required.23 Several recent studies illustrate the use of EHR-based algorithms to identify patient charts with a higher probability of a diagnostic error.4 ,7 ,24 Such methods should now be explored further and refined for use in other institutions. Additional methods using more sophisticated algorithms should also be developed to proactively identify diagnostic errors before patient harm. This technique could also lend itself to study causes of error as well as facilitate tracking of error rates over time and testing the impact of interventions.

Currently used research methods

Because many aspects of the diagnostic process are not easily captured using current study methods, the causes of diagnostic error are difficult to identify. Ideally, the diagnostic process is studied prospectively in a real clinical setting, but this poses substantial practical difficulties (eg, ethical issues, time constraints, measuring clinician thought processes unobtrusively, observer bias). Consequently, most studies conducted in naturalistic settings have examined the causes of diagnostic error retrospectively (eg, through record review which sometimes is followed by interviews).2 ,20 ,21 ,25 These studies have revealed vulnerabilities in clinical workflow processes or organisational issues that contribute to error (eg, policies that are ambiguous about who is responsible for test results follow-up).26 They have also yielded illustrative patterns of cognitive failures or biases, (ie, errors in the reasoning process of the physician),19 as well as the complexity of interaction between system and provider factors.25 Prospective evaluations have been generally restricted to artificial settings (eg, case vignette studies)27 and offer a complementary cognitive perspective.

Prior studies have revealed that the diagnostic step of ‘information synthesis’ was particularly prone to error.1 ,4 ,19 ,20 Retrospective studies conducted in naturalistic settings can identify contributory factors and types of diagnostic breakdowns7; however, they are seldom able to reveal the root causes of diagnostic error due to limited information about causality. For instance, in reviewing an error case in which a diagnosis was missed despite the availability of laboratory results to support the diagnosis, it may be difficult or impossible to know whether the physician lacked the appropriate knowledge to arrive at a correct diagnosis, or was influenced by a cognitive bias, distraction or faulty heuristic, or simply did not have time or access to the result.18 ,28 ,29

Prospective studies in controlled settings could be potentially useful to examine causal relationships. Experimental designs, in particular, can isolate the effects of certain biases or the mechanisms underlying the reasoning process.18 ,30 However, the next step for any such line of research would be to better understand how problems detected in the laboratory translate into real-world behaviours and problems. A key will be enabling a ‘culture of safety’ that will permit clinicians to honestly reflect on errors they have been involved in, and examine and test ways they could be prevented in their daily work in the future.

Directions for future research on causes of diagnostic error

1. Linking naturalistic and experimental studies: studies on the causes of error in naturalistic settings and experimental (case vignette-based, simulation-based, etc.) clinical research studies each carry distinct advantages and limitations. To maximise the knowledge gained from these disparate methods, we believe that more can be done to encourage linkages between research paradigms. The results of naturalistic studies could inform the hypotheses of experimental studies (eg, to better understand causal factors), and in turn, findings from these experimental studies could be tested for their applicability and validity in a naturalistic setting. For example, frequently missed lesions in x-ray evaluations that have been found in studies on malpractice claims could be studied in a clinical research setting (eg, by studying the eye-movements of radiologists when reviewing the x-rays). By replicating these errors in the clinical research setting, we can study why they occur and how the process could be improved. ‘Triangulating’ methodological approaches in this manner could accelerate our understanding of the pathogenesis and prevention of diagnostic error.

2. Testing newer models to study diagnostic error: newer models can be used to integrate and align the objectives of naturalistic observational studies and experimental studies. Advanced techniques for ‘root cause analysis’ developed for accident investigation outside of medicine have barely begun to be used in healthcare and offer promising returns. Another potentially fruitful avenue of research is application of the ‘situational awareness’ model, which has been widely studied in the human factors literature, specifically in the field of aviation.31 It can be applied to examine how practice environments and systems influence physicians’ cognitive processes, helping or hindering their ability to differentiate information relevant to diagnosis while maintaining a broader general overview of the patient.32 Situational awareness emphasises clinicians’ understanding of ‘what's going on’ and ‘what's likely to happen next’ as key elements in effective diagnostic decision making.33

Current intervention research

A recent systematic review evaluated the effects of patient safety strategies that focused on diagnostic error and found that only few strategies had an effect in terms of diagnostic error reduction or reduction of patient harm.34 While this review identified over 100 studies that tested the effects of interventions on the diagnostic process, two other recent narrative reviews showed that there are a large number of interventions that have been developed to reduce diagnostic errors, but have not been yet tested for their effectiveness. Specifically, these reviews evaluated the existing literature on outcomes of system-related and cognitive interventions.35 ,36 System-related interventions are those that are focused on addressing organisational process errors, and vulnerabilities, whereas cognitive interventions focus on improving physicians’ perceptual and thought processes (eg, educational interventions, cognitive de-biasing strategies; see related papers in supplement). Whereas many of the suggested interventions were judged to have great potential, only few were actually tested and implemented in practice. This paucity of published outcome studies underscores the urgent need to not only conceptualise and develop interventions but to systematically evaluate their effectiveness.

Directions for future research to improve diagnosis

• 1. Patient engagement: strategies to promote shared decision making and patient engagement in healthcare hold great promise for reducing errors and error-related harm.37–39 The patient is key stakeholder at every step in the diagnostic process and serves as the main source of information during the diagnostic process.40 The patient also has access to information that can help the physicians maintain and regain situation awareness. For example, patients may support physicians by accurately informing them in a timely fashion about changes in their clinical situation. Patients are uniquely suited to detect and report errors that occur during their care processes. Thus patients are an enormous and virtually untapped resource for reducing diagnostic error;41–43 research studies that examine this potential are few. Therefore, studies are warranted to examine ways in which patients can influence their physicians’ decisions by providing more accurate or focused information, asking questions, questioning diagnoses and discussing medical tests and diagnoses they may be concerned about.44 To be able to involve the patient in their medical process, the physician's role is changing as well. The physician needs to facilitate patient involvement by creating an environment in which patients can contribute, without shifting responsibility for accurate diagnosis to the patient. Specifically, the physician needs to create opportunities for patient involvement, for example, provide the possibility for patients to upload the patient history (eg, family history) online in advance as well as creating an atmosphere in which the patient feels comfortable to ask questions.

  Furthermore, research should explore the potential diagnostic benefits of patients’ critical review of their own medical record.45

• 2. EHRs: EHRs are now being developed and implemented in many countries, and the focus on improving the diagnostic process through the use of EHRs is timely. Research has already shown that the EHRs can be used for detection of diagnostic error,5 ,7 and the next step is to apply these methods to the implementation and evaluation of interventions to improve the diagnostic process. Schiff and Bates created a conceptual model and made several suggestions for preventing or minimising diagnostic error with use of EHRs and other health information technologies.46 Some of the most promising suggestions are:

• Improving access to information. An EHR provides longitudinal information about the patient's health status, including prior outpatient visits and hospitalisations. Although the ready availability of this information should facilitate the diagnostic process, research is needed to understand how best to facilitate display and use of this information to improve the diagnostic process. In particular, it will be necessary to identify and correct unintended consequences of EHR adoption, including ‘information overload.’47 Incorporation of tools in the EHR to facilitate diagnosis. Checklists have been shown to offer great potential for reducing errors in general48 ,49 and can be applied to the diagnostic process.50 Although there is limited evidence thus far to support the use of checklists for diagnostic error reduction,46 ,51 the EHR provides new opportunities to incorporate checklists into clinical practice to facilitate the diagnostic reasoning process.

• Exploring newer forms of Health Information technology (HIT): New information technologies, such as the IBM supercomputer Watson, have potential to improve the diagnostic process52 by using natural language processing. Natural language processing allows systematic analysis of human natural language into medical concepts, which eventually can result in a shortlist of differential diagnoses. Although we are far from broad implementation of such techniques in clinical practice, research is needed to evaluate which techniques have the potential to reduce error. An extensive review of ways to reduce diagnostic error using HIT is described in the article of El-Kareh et al in this special issue.

• 3. Delineation of specific diagnostic process pitfalls: research on the frequency and causes of errors could reveal specific areas of improvement that can be targeted through focused interventions. For example, a commonly seen error in malpractice claims for missed or delayed breast cancer diagnosis is failure to pursue the appropriate additional diagnostic evaluation for a woman with a breast lump after her diagnostic mammogram returns as ‘negative’.6 Rather than recognising the imperfect sensitivity of this test in the face of a palpable mass and proceeding to do the recommended next test (eg, ultrasound-guided aspiration cytology), physicians have on occasion instead reassured the patient about the ‘normal’ mammogram and stopped the diagnostic workup.53 Attributing rectal bleeding to ‘hemorrhoids’ in a patient at high risk for colorectal cancer, rather than conducting additional diagnostic testing to rule it out, illustrates another common diagnostic pitfall.12 Specific pitfalls that contribute to greater-than-expected burden of diagnostic errors could be targeted through training and system redesign.

In conclusion, research on diagnostic error, although still nascent, has evolved significantly as compared with the state of knowledge a decade ago. In this paper, we highlight areas where research methods could propel the field forward in promising directions related to epidemiology, contributory factors and interventions. The recently founded Society of Improving Diagnosis in Medicine and the annual Diagnostic Error in Medicine conferences are platforms to bring together a multidisciplinary group of people (including clinicians, patients, researchers and educators) to advance the research agenda. This agenda could also be integrated with research agendas of other organisations (eg, National Institutes of Health, specialty societies as well as national research agendas, eg, the UK's large research agenda on cancer). Furthermore, funding in this area should be stimulated to operationalise this research agenda. Using well coordinated, multidisciplinary approaches, and with appropriate research support, the foundation laid thus far can lead to future methodological advancement in the field and to reduction of diagnostic errors.