Article Text

Download PDFPDF

Research to improve diagnosis: time to study the real world
Free
  1. Sumant R. Ranji1,
  2. Eric J Thomas2
  1. 1 Medicine, University of California, San Francisco, California, USA
  2. 2 Internal Medicine, University of Texas John P and Katherine G McGovern Medical School, Houston, Texas, USA
  1. Correspondence to Dr Sumant R. Ranji, Department of Medicine, University of California San Francisco at San Francisco General Hospital, 1001 Potrero Avenue Box 0862, San Francisco, CA 94110, USA; sumant.ranji{at}ucsf.edu

Statistics from Altmetric.com

More than a decade ago, diagnostic errors were named as the ‘next frontier’ in patient safety.1 In spite of this, our understanding of the epidemiology of diagnostic errors and approaches to preventing them has only recently begun to mature. Recent systematic reviews document an unacceptably high burden of diagnostic error in both outpatients2 and hospitalised patients.3 Given this landscape, clinicians and healthcare leaders would benefit from understanding which approaches to preventing diagnostic error are effective. A systematic review of strategies to reduce diagnostic error published in 20134 identified several promising strategies, such as technology-based system interventions like computerised diagnostic decision support, but acknowledged a need for higher quality evaluation of these interventions.

In the 8 years since the earlier review was published, interest in diagnostic safety has greatly increased, thanks in part to the publication of influential reports by the US National Academies of Sciences, Engineering, and Medicine5 (NASEM), in 2015, and the US National Quality Forum, in 2019, and an increase in publications on diagnostic error. The systematic review by Dave and colleagues6 in this issue of BMJ Quality & Safety largely replicates the prior review, and therefore provides important insights into progress from 2012 through the end of 2019, given the two reports noted above and developments such as technology-based tools for improving diagnosis. The authors used the same search criteria as the original review and followed the accepted standards for conducting and reporting a systematic review. In contrast to the earlier study, Dave and colleagues’ review attempted to assess the effectiveness of interventions on patient-level outcomes, such as diagnostic accuracy for a targeted condition; studies that only measured changes in diagnostic processes, or simulation-based studies, were excluded.7 8

The authors found 20 studies of diagnostic safety strategies, and categorised them into the same six groups as in the previous study: technique (n=8); technology-based system interventions (n=5); educational interventions (n=2); personnel changes (n=2); structured process changes (n=2); and additional review approaches such as audit and feedback (n=1). ‘Technique’ interventions were defined as ‘Changes in equipment, procedures and clinical approaches that target diagnostic performance in clinical practice’. Technology-based system interventions were ‘Implementation at the system level of technology-based tools, such as computer-assistive diagnostic aids, decision-support algorithms, text message alerting and pager alerts’. Most included studies were retrospective cohort studies; only four were randomised controlled trials (RCTs). Nearly all studies (18 of 20 overall and 3 of 4 RCTs) reported some improvement in diagnostic accuracy, although the specific outcomes being measured varied widely, precluding comparison of effect size; no interventions were associated with improvement in mortality. Since the earlier 2013 review, only one new study (a retrospective cohort study) was identified that used additional review (audit and feedback) to improve diagnosis, despite this being the most frequently used intervention in the earlier systematic review; no studies used combinations of intervention types. The authors used the Cochrane Collaboration’s Risk of Bias tool to assess the methodological quality of the included literature, and also conducted a sensitivity analysis excluding lower-quality studies. Most of the included studies had at least ‘moderate’ risk of bias (1 of 4 RCTs and 14 of 16 non-randomised studies), but exclusion of the two studies with ‘high’ risk of bias did not change the primary results of the review.

Present limitations in the diagnostic error literature

Dave and colleagues’ review serves an important purpose by comprehensively updating the published literature around interventions to improve diagnosis. Yet, their synthesis also highlights the limitations in the diagnostic safety literature—among them, the variation in outcomes and inconsistent methodological quality—which limits the generalisability of the studies included in the review. Unfortunately, after another 8 years of research, practising clinicians and organisational leaders have gained little additional evidence about which interventions are effective at preventing diagnostic errors.

This lack of progress highlights a familiar issue in the patient safety literature, where interventions to preventing adverse events have often been slow to develop. Part of the explanation is that interventions to improve patient safety are by definition implemented in complex adaptive systems, with systems embedded within other systems, and response to any individual change difficult to predict. Interventions to improve safety must therefore be developed with a theory of change9—an overarching rationale that links an intervention to the specific outcomes, and specifies the contextual factors10 that influence the probability of success. The 2015 National Academies of Science and Medicine report Improving Diagnosis in Health Care stated that ‘The diagnostic process is a complex and collaborative activity that unfolds over time and occurs within the context of a health care work system.’ The complex nature of diagnosis heightens the importance of developing robust programme theories to inform intervention development, and the need to take the context of the intervention into account at both the development and implementation stages. This is particularly true for improving diagnosis, since the diagnostic process requires a high cognitive load for diagnosticians and takes place across varying clinical environments. The lack of attention to theory of change and healthcare system context in quality improvement studies has been identified as a problem for the patient safety field over the past decade11; as the diagnostic safety field continues to develop, those interested in improving diagnosis will need to avoid the mistakes of the past.

How should attention to theory of change and healthcare system context translate to intervention development? Interventions to help improve diagnosis for similar patients cannot be designed or implemented without understanding the social and cultural contextual factors in which a problem such as a delayed diagnosis took place. A case of delayed diagnosis of sepsis, based on similar cases at our hospitals, illustrates these and other factors that need to be addressed in future research to improve diagnosis (see box 1). A theory of change for improving diagnosis of sepsis would focus on identifying the primary barriers to accurate diagnosis in the emergency department, using qualitative methods. This would likely identify difficulty in communicating with patients with limited English proficiency, inability to obtain collateral information due to visitor restrictions and availability bias due to the COVID-19 surge as some of the major contributors to the delayed diagnosis. Interventions to address these barriers could include improving the availability of interpreters in the emergency department, facilitating video-conferencing with patients’ families (or being willing to liberalise morbidity and mortality conference based on this case). In other words, we would typically need a combination of the intervention types found in the systematic review by Dave and colleagues, to effectively prevent such a delayed diagnosis.

Box 1

Example of diagnostic error case

At the height of a COVID-19 surge, an elderly Spanish-speaking man with a history of prostate surgery presents to an emergency department with fever, dyspnoea and abdominal pain. His family cannot accompany him due to COVID-19-related visitor restrictions, and as a result, the emergency department clinicians are unaware of his recent surgery and history of prostate problems. He is assessed as likely having COVID-19 pneumonia and placed in respiratory isolation while awaiting testing results. He is given oxygen and symptomatic treatment. A few hours later, with the COVID-19 test result still pending, his clinical status deteriorates with hypotension and respiratory distress. The admitting hospitalist, who speaks the patient’s primary language, is able to obtain a full history and realises that the patient’s presentation is more consistent with sepsis due to a urinary tract infection. She promptly initiates broad-spectrum antibiotic therapy and fluid resuscitation, but the patient continues to deteriorate and requires admission to the intensive care unit. Further workup reveals Escherichia coli in blood and urine cultures; COVID-19 testing is negative. The patient requires mechanical ventilation and vasopressor support, but eventually survives and is discharged from the hospital.

The role of context in the diagnostic process is especially important when planning interventions to improve care for vulnerable and marginalised populations. Many hospitals—including ours—already have technology-based system interventions (such as electronic health record (EHR) alerts based on physiological variables) to help clinicians identify patients with sepsis, with the goal of initiating evidence-based treatment in a timely fashion. There is some evidence in simulation studies that the use of clinician decision support systems (CDS) that generate a list of possible diagnoses can help improve diagnostic accuracy.12 However, prior research on the effectiveness of CDS at preventing other types of adverse events has shown that CDS typically have a small impact on the quality and safety of care. This is because CDS are often implemented without accounting for the social and technical context in which they will be used.13 It is possible—perhaps even likely—that widespread use of CDS will have similarly small effects on improving diagnostic accuracy if they are deployed without sufficient consideration of the complex environments in which clinicians make diagnoses.

Furthermore, as the US Agency for Healthcare Research and Quality Making Healthcare Safer III report noted, ‘The diagnoses generated by CDS tools are only as good as the information that is put into the system; if the initial assessment of the patient (eg, physical examination finding) is incorrect, the output is likely to be incorrect.’14 In the case described above, a language barrier and the inability of family to accompany the patient resulted in inadequate information being gathered during the initial clinician–patient interaction—which likely would have limited the ability of a CDS in this case to help generate the correct diagnosis. There is considerable interest in using ‘big data’ derived from EHRs to develop decision support algorithms to improve diagnosis, but concern has been raised that such artificial intelligence-based approaches may actually worsen disparities in diagnosis15 in part through the mechanisms illustrated in this case—clinical data on patients with language barriers or who receive care in resource-limited settings may be under-represented in EHR-based data sets, potentially leading to inaccuracies in diagnostic algorithms. The real world is more complex, where it is not just the individual clinicians’ cognitive processes that need to be targeted by a unimodal intervention like the CDS, but also the surrounding system context to improve diagnosis. Addressing inequities in diagnosis will therefore require additional qualitative work that engages patients and families in order to understand the context in which misdiagnosis occurs and inform the development of interventions that improve diagnosis for all patients.

Recommendations for future, real-world research

As the diagnostic safety field develops, researchers should be mindful of the challenges encountered in efforts to improve other safety problems, as learning lessons from prior efforts to improve safety will help inform interventions to improve diagnostic safety. Fundamentally, diagnosis will only be improved by recognising the complex sociotechnical nature of the diagnostic process, so the following suggested recommendations can be used to design and test future approaches (box 2).

Box 2

Recommendations for designing studies to improve diagnosis

Planning the study

  • Develop and report a programme theory for how the intervention will improve the diagnostic process.

  • Describe the health system context in which the team seeks to improve diagnosis.

Designing the intervention

  • Engage patients, families and caregivers in intervention design.

  • Target improving both individual clinicians’ cognitive processes and improving systems to facilitate accurate diagnosis.

Reporting and measurement

  • Use the most rigorous study design feasible to assess the impact of the intervention.

  • Report the intervention with enough detail to allow for replication in other sites or settings of care.

  • Use qualitative and realist research methods to help determine how and why the intervention worked (or did not work).

First, researchers should plan studies using programme theories that inform interventions and how they will improve diagnosis. For example, the NASEM report outlines a conceptual model for the overall process of diagnosis; researchers developing interventions targeting specific aspects of this process (for example, communication of the diagnosis to the patient) should develop and report a programme theory for how their intervention will result in an improvement. The planning process should also include describing the healthcare system context in which they seek to improve diagnosis16 and incorporating key contextual features into the intervention design. Second, interventions should seek to target both the cognition and decision-making of individual clinicians, and the systems in which they work. At the moment, we may not have enough knowledge about which cognitive interventions and which systems-based interventions should be combined, but this should be the goal. For example, we may be close to being able to study the combination of an intervention that improves differential diagnoses and one that proactively alerts clinicians and provides guidance to evaluate abnormal test results that indicate important diagnoses (eg, new anaemia in an adult that may indicate colon cancer). The process of designing interventions must also engage patients and caregivers who can tell us about breakdowns that only they are aware of, as well as some of the implications of these breakdowns, to inform how to best target the intervention.17–19 Third, researchers should use the most rigorous study designs feasible,20 and interventions and their implementation should be described so they can be replicated.21 Studies should measure both diagnostic processes (ranging from patient recognition and communication of symptoms, to access to care, to clinician decision-making, testing and follow-up, to communication of diagnoses to patients, to helping patients understand and treat their diagnoses) and patient-level outcomes (correct diagnosis, correct treatment, harm and health status). Finally, we strongly recommend using both quantitative and qualitative methods to fully understand diagnostic improvement interventions in a real-world context. Traditional quantitative research is needed to measure the effectiveness of interventions, and qualitative and realist research approaches are needed to help understand how and why they work in complex systems.

This work will require more research funding from governments, foundations and perhaps the private sector, and will need to target a wide variety of diseases,22 sites of care and patient populations. Future progress should be assessed by both traditional systematic reviews such as those performed by Dave and colleagues, and by realist reviews that seek to determine what interventions work for which patients, under what conditions and how the interventions work.

The systematic review by Dave and colleagues is a signpost indicating the limitations of current approaches. The increased recent attention to improving diagnosis is overdue, but to achieve its potential, the diagnostic safety field will need a different approach to improving diagnosis—one that incorporates lessons from the efforts of the past.

Ethics statements

Patient consent for publication

Ethics approval

This study does not involve human participants.

References

Footnotes

  • Twitter @sumantranji, @EJThomas_safety

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Provenance and peer review Commissioned; internally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles