Framework rationale

We envision the need for the Safer Dx framework because current organisational processes, procedures and policies do not necessarily facilitate or focus on measurement of diagnostic safety, which is essential for creating future feedback and learning to improve patient care. The Safer Dx framework is expected to facilitate feedback and learning to help accomplish two short-term goals: (1) refine the science of measuring diagnostic error and (2) make diagnostic error an organisational priority by securing commitment from organisational leadership and refocusing the organisation's clinical governance structure. High quality diagnostic performance requires both a well-functioning healthcare system along with high-performing individual providers and care teams within that system.17 Therefore, the framework posits that ideal measurement and monitoring techniques should facilitate feedback and learning at all of these levels.

Operationalising the Safer Dx framework

Our framework follows Donabedian's Structure–Process–Outcome model,18 which approaches quality improvement in three domains: (1) structure (characteristics of care providers, their tools and resources, and the internal physical/organisational setting along with the external regulatory environment); (2) process (both interpersonal and technical aspects of activities that constitute healthcare); and (3) outcome (change in the patient's health status or behaviour). We made several modifications to adapt these three domains to diagnostic error-related measurement work.

Challenges of real-world diagnostic error measurement

While the Safer Dx framework could provide a useful impetus for using both retrospective and prospective methods to improve the ‘basic science’ of measuring diagnostic safety,16 several challenges remain. Most measurement methods are still evolving and have limitations. In this section, we illustrate challenges of certain current methods to routinely measure diagnostic errors.

As Graber et al30 have emphasised, there is ‘not a single HCO in the United States that is systematically measuring the rate of diagnostic error in its clinics, hospitals, or emergency departments.’ One of the reasons is that HCOs do not know how to find these errors. Nevertheless, organisations will need to start somewhere before we arrive at better measurement and monitoring techniques.

While there have been some early successes with voluntary reporting by providers,31 ,32 there are no standardised mechanisms or incentives for providers to report diagnostic errors. Further, diagnostic error incident reports from providers are not included in the standardised error reporting protocols in the common formats developed by the Agency for Healthcare Research and Quality.33 It has also become obvious that quality improvement team support and local champions are needed to sustain reporting behaviour.32 At present, few HCOs allocate the ‘protected time’ that is essential for clinicians to report, analyse and learn from safety events. A recently implemented programme at Maine Medical Center was found to be resource intensive and faced challenges in sustainability as well as in identifying the underlying contributory factors and developing and implementing strategies to reduce error.30 Even if these challenges could be overcome, voluntary reporting alone cannot address the multitude of complex diagnostic safety concerns.34 Many times, physicians are unaware of errors they make.35 Although accounts from providers necessarily play an important role in understanding breakdowns in the diagnostic process, provider reporting can only be one aspect of a comprehensive measurement strategy. Patient reports could be a source of measurements about events not discovered through other means36 and recent studies have collected data about diagnostic errors from patients. For example, of 1697 patients surveyed in seven primary care practices, 265 reported physician mistakes and 227 of these involved a diagnostic error.37 Although patient reporting has significant future potential, it has not been validated as a method to measure diagnostic error yet.

While medical record reviews are considered valuable for detecting diagnostic errors, non-selective record review is burdensome and resource intensive. To improve the yield of record reviews, we have developed ‘triggers’38 ,39 to alert personnel to potential patient safety events, which enables selective targeted review of high-risk patients.40 ,41 The robust electronic health record (EHR) and large clinical data warehouses within the Department of Veterans Affairs (VA) healthcare system have allowed us to pursue research on electronic trigger development, enabling us to identify preventable conditions likely to result in harm.40 ,42 Triggers can be used both retrospectively and prospectively, but they are yet to move beyond the research environment into real-world practice.

The use of administrative data (eg, use of billing data as proxy for evolution of clinical diagnosis) to identify possible diagnostic errors is not sufficiently sensitive to detect events of interest. Recent work has tried to refine approaches to use administrative datasets for measurement of diagnostic errors related to stroke and myocardial infarction.43 ,44 However, this approach alone is unlikely to become a viable basis for rigorous measurement in and of itself in part because of the absence of clinical details related to diagnostic evolution and corresponding care processes.

Prospective methods such as use of unannounced standardised patients help to overcome limitations of other methods, such as chart reviews, which often lack certain details including those related to patient context.45 However, direct observations and simulations, though relatively rigorous, have distinct drawbacks; direct observations are expensive, and simulations may not generalise well to real-world practice.13 An alternative strategy is to develop means of gaining more valid and actionable insights from existing data sources in a proactive fashion.15 For example, many organisations are conducting safety huddles46 and reviews of patient presentations, though we are unaware of any efforts to monitor these sources for diagnosis-related issues. Similarly, we found that peer review may be a useful tool for healthcare organisations when assessing their sharp-end clinical performance, particularly for diagnosis-related safety events.47 However, current peer review systems are inadequate and need to include root cause analysis approaches and performance improvement thinking to detect and address system issues.48 This enhancement of existing peer review programmes could greatly improve processes for diagnostic self-assessment, feedback and improvement.47

Once HCOs start to measure diagnostic safety, ideally using multiple methods (both retrospective and prospective), the findings would lead to further refinement of the science of measuring diagnostic error. This could lead to creation of standardised methods for measuring and monitoring diagnostic error and determining the effects of intervention or improvement. To make this into a reality, external rules, regulations and policies would be necessary to make diagnostic error an organisational priority and to secure commitment from the organisational leadership. Once we have the focus of the organisation's clinical governance structure, the Safer Dx framework suggests that two additional challenges must be overcome for conducting and using measurements at the institutional level.

First, information must be directed to the right people in the appropriate organisational roles. Currently, most institutions do not have dedicated personnel who can analyse and act upon this information. Risk and safety managers routinely collect quality and safety data from multiple sources and are often privy to data unavailable to others, but they have little guidance on how to approach diagnostic errors.49 Second, measurements must generate feedback and learning to improve calibration. However, currently the mechanisms of generating and delivering feedback at the organisational level and provider level remain unknown.

To facilitate measurement and learning, policy makers working on monitoring and quality improvement and researchers would need to address the following questions about giving feedback: (1) what type of content is appropriate for feedback (eg, a return visit based on administrative data which may or may not be related to error vs a clear preventable diagnostic adverse event determined only after a comprehensive review of an incident), (2) what should be the method of delivery (eg, verbal vs written; in person vs by phone; individual vs group; anonymous vs identified), (3) what should be the timing and frequency (eg, delayed until an event is clear vs real-time as details become available) and (4) how do we create feedback that is unambiguous, non-threatening and non-punitive. Use of the Safer Dx framework should lead to shared accountability beyond the clinicians directly involved, including improving the processes and systems they practice in. Because the framework also focuses on value as an outcome, it should help address the challenge of ensuring we mitigate unintended consequences due to feedback-related hypervigilance, for example, due to increased testing/treatment.

Suggested application of the Safer Dx framework and next steps

Use of the Safer Dx framework accounts for the complexity of diagnostic error and will lead to addressing the sociotechnical context while understanding and preventing error. We envision that it will help overcome some of the challenges in exploration, measurement and ultimately improvement of the diagnostic care process within a technology-enabled healthcare system. We have used the approach suggested by this framework in our current work on measuring and reducing delays from missed follow-up of abnormal diagnostic test results in EHR-enabled healthcare. The Safer Dx framework has enhanced our conceptual understanding of the problem and informed intervention development. For example, we often suspected specific aspects of the EHR led to missed follow-up (such as the software or the user interface)50 but we learnt that the user interface is only one, albeit an important, part of the problem. Conversely, we identified and fixed a software configuration error that prevented providers at our site from receiving notifications of abnormal faecal occult blood test results,51 but in doing so we also found problems related to organisational factors, workflow and human–computer interaction, all of which needed to be addressed to improve follow-up.52 In a different study, we found that the same technical intervention to improve test result follow-up in two different VA facilities led to improved outcomes in only one facility.53 The findings remind us about the importance of accounting for the local sociotechnical context as well as the external financial and regulatory environment in which safety innovations are embedded.54

The Safer Dx framework also encourages its users to leverage the transformational power of state-of-the-art health information technology and the EHR in developing interventions and feedback for improvement. Based on the Safer Dx framework, organisations should be harnessing a wealth of electronic data and these data need to be put to use to provide actionable information on diagnostic error. As an example, with funding from the VA,55 we are now developing an automated measurement and surveillance intervention to improve timely diagnosis and follow-up of cancer-related abnormal findings and applying the Safer Dx framework. We are using a large, national data warehouse to trigger medical records with evidence of potential delays in follow-up of cancer-related abnormal test results40 and have continually worked over past 2 years to make measurements more valid and reliable. Additionally, to integrate near real-time surveillance and communication of information about at-risk patients into the point of care, we are adhering to many of the organisational learning elements of the Safer Dx framework. Safer Dx encourages the involvement of organisation's existing safety/quality improvement (QI) infrastructure, including risk managers and safety managers, and to more effectively leverage these personnel to take action and improve collective mindfulness. Through qualitative work, we are working with our organisational partners to ensure that the data we provide them will be used meaningfully, will lead to improvement in collective mindfulness and learning, and be fed back to the providers on the frontlines to improve diagnostic processes. The framework reminds us that various internal and external policies and measurement procedures which, if not designed and implemented with the entire sociotechnical context in mind, can affect diagnosis performance and measurements.

The Safer Dx framework could also help jump-start the improvement of diagnostic safety measurement by informing the design of proactive risk assessment strategies at the organisational or practice level. As used in other high-risk settings and industries, proactive risk assessment involves self-assessment to identify specific areas of vulnerability and ensure readiness to apply preventive measures to promote safety and reduce adverse events. Stakeholders, including policymakers, QI/safety personnel or researchers, could help develop tools for risk assessment based on the framework. For example, with support from the Office of the National Coordinator for Health Information Technology, we previously used a rigorous, iterative process to develop a set of nine self-assessment tools to optimise the safety and safe use of EHRs using our eight-dimension sociotechnical model.56–58 These tools, referred to as the Safety Assurance Factors for EHR Resilience guides, are each organised around a set of ‘recommended practices’.59 The goal is also to change local culture and foster the implementation of risk-reducing practices. Similar tools could be developed based on Safer Dx framework and use a multifaceted approach to assess risk proactively and drive improvements in the quality and safety of diagnosis.58

Last, policymakers working on diagnostic error-related measurement issues would need more evidence on what structural, process and outcomes-related issues need to be measured for improving diagnostic safety. This groundwork could inform the development of future performance measures at a national level with partnership with the National Quality Forum.

Summary

Diagnostic errors are considered harder to tackle8 and remain elusive to improvement efforts in part because they are difficult to define and measure.60 A new conceptual framework (the Safer Dx framework) can be used by a variety of stakeholders including researchers, clinicians, policymakers and healthcare organisations to stimulate the development of local, regional and national strategies and tools for measuring and monitoring diagnostic error. The resulting knowledge would be instrumental to lay the foundation for both retrospective and more proactive measurement of diagnostic errors. The feedback and learning that would result will help develop subsequent interventions that lead to safer diagnosis, improved value of healthcare delivery and improved patient outcomes.