Article Text
PDF
Abstract
Objective Diagnostic errors in primary care are harmful but poorly studied. To facilitate the understanding of diagnostic errors in real-world primary care settings that use electronic health records (EHRs), this study explored the use of the situational awareness (SA) framework from aviation human factors research.
Methods A mixed-methods study was conducted involving reviews of EHR data followed by semi-structured interviews of selected providers from two institutions in the USA. The study population included 380 consecutive patients with colorectal and lung cancers diagnosed between February 2008 and January 2009. Using a pre-tested data collection instrument, trained physicians identified diagnostic errors, defined as lack of timely action on one or more established indications for diagnostic work-up for lung and colorectal cancers. Twenty-six providers involved in cases with and without errors were interviewed. Interviews probed for providers' lack of SA and how this may have influenced the diagnostic process.
Results Of 254 cases meeting inclusion criteria, errors were found in 30 of 92 (32.6%) lung cancer cases and 56 of 167 (33.5%) colorectal cancer cases. Analysis of interviews related to error cases revealed evidence of lack of one of four levels of SA applicable to primary care practice: information perception, information comprehension, forecasting future events, and choosing appropriate action based on the first three levels. In cases without error, application of the SA framework provided insight into processes involved in attention management.
Conclusions A framework of SA can help analyse and understand diagnostic errors in primary care settings that use EHRs.
- Diagnostic error
- decision-making
- patient safety
- primary care
- medical errors
- human factors
- cancer
- electronic health records
- diagnostic delays
- information technology
- trigger tools
- health services research
- healthcare quality improvement
- teamwork
- safety culture
Statistics from Altmetric.com
- Diagnostic error
- decision-making
- patient safety
- primary care
- medical errors
- human factors
- cancer
- electronic health records
- diagnostic delays
- information technology
- trigger tools
- health services research
- healthcare quality improvement
- teamwork
- safety culture
Introduction
Diagnostic errors are among the most common, expensive, and harmful types of errors in outpatient settings,1–6 and were recently recognised by WHO as a priority area.7 Despite their global significance,8–14 these errors are challenging to study and have received inadequate attention.15–18 Additionally, the science of understanding and analyzing outpatient diagnostic errors is poorly developed.19–21 Although patient and system factors are known to contribute to diagnostic error,22 little is known about how diagnostic decision-making errors occur in routine primary care settings.23 Preventive strategies are thus relatively underdeveloped.24
The diagnostic process (ie, detecting signs/symptoms and coordinating diagnostic procedures) is dependent on several external factors (work-system factors) and extends beyond the cognition of a single provider.22 Thus, cognition in the primary care environment has become increasingly ‘distributed’ and dependent on evolving situations and work-system factors such as teamwork, technology and organisational factors.25–29 However, the role of these work-system factors and how they affect cognition has not been adequately addressed. For instance, there is little understanding of how attention is managed and diagnostic plans are created and executed in the context of longitudinal outpatient care that spans weeks or months.
We previously described the application of the situational awareness (SA) conceptual framework from aviation human factors research to advance the science of diagnostic decision-making in primary care.30 In aviation, SA is defined as a person's ‘perception of the elements in the environment within a volume of space and time, the comprehension of their meaning, and the projection of their status in the near future’.31 Loss of or failure to gain SA is commonly implicated in aviation accidents related to pilot errors and may be precipitated by system-related and cognitive factors.32–35 Maximising SA has been an important focus of aviation safety research.36 An application of four levels of SA to the primary care diagnostic process (figure 1)30 illustrates the potential of this framework to improve understanding of diagnostic errors. The four levels include information perception, information comprehension, forecasting future events, and choosing appropriate action (resolution) based on the first three levels. SA has also been defined as a ‘non-technical skill’ in anaesthesia.37 38 In this context, SA components include detection and interpretation of situational clues from multiple, dynamically changing sources (including from patients, other members of the care teams, and from displays and equipment); thinking about what might happen in the future and adapting to evolving situations; and keeping track of and utilising special elements of knowledge.37 Thus, the unit of analysis is not only the provider but the interaction of the provider within the environment.38
An adaptation of Endsley's model of situational awareness to medical diagnosis. Note: Information from lower levels feeds forward to higher levels, and information from higher levels feeds back to lower levels.
Missed and delayed cancer diagnosis is a leading type of diagnostic error.2 4 39 In previous work, we found that substantial delays in cancer diagnosis often resulted from failure to take appropriate action on obvious alarm features in clinical presentation (including diagnostic information documented in the electronic health record (EHR)),40–42 a scenario that strongly suggests lack of SA.40 41 In the present study, we defined diagnostic errors as missed opportunities to make a timely diagnosis and explored the relationship between the lack of SA and diagnostic errors in two common cancers (colorectal and lung). Our aim was to evaluate whether the SA framework can be used to analyze outpatient diagnostic errors using a ‘distributed cognition’25–27 approach that included provider–work system interaction, including interaction with the EHR.
Methods
We used a mixed-method approach to explore application of the SA framework to the understanding of diagnostic errors. We first identified cases of diagnostic errors through record reviews using previously developed data collection methods.40 41 Because decision-making processes are not always documented clearly, record reviews were followed by qualitative semi-structured interviews to better understand providers' thoughts and actions in selected cases.
Setting
Sites included two large integrated health systems in the USA, each with almost 2 million outpatient visits annually. Both sites used EHRs. Primary care was delivered by about 175 staff (both academic and non-academic) primary care providers (PCPs) and residents. Local institutional review boards approved the study.
Inclusion/exclusion criteria
Using tumour registry data, we identified new cases of primary colorectal cancer (CRC) and lung cancer diagnosed at the study sites between February 2008 and January 2009 (N=380). We excluded cancer recurrences, cases diagnosed at other institutions, and cases diagnosed in the inpatient setting with no previous primary care contact (eg, when the patient's first presentation to the emergency room led to hospitalisation).
Data collection
Chart reviews to identify errors
In the absence of formal definitions for missed and delayed cancer diagnosis, we defined diagnostic error as lack of timely action on one or more established indications for diagnostic work-up in CRC and lung cancer (table 1).40 41 For example, failure to initiate colonoscopy despite documented presence of iron deficiency anaemia was considered an error. ‘Timeliness’ was defined based on expert opinion and selected literature.41 43 44 Justified decisions to not pursue further diagnostic work-up after recognition of alarm symptoms (eg, for patients with reduced life expectancy) were not considered errors.
Established indications for cancer-related diagnostic work-up and defined time periods
A trained physician used a pre-tested data collection instrument to identify diagnostic errors through detailed review of the EHR.40 41 We did not collect information on patient harm to reduce hindsight bias.45 After chart review, two physician experts discussed all identified cases of diagnostic error to confirm the presence or absence of error. Cases unanimously judged to contain errors were considered for interviews. We also interviewed selected PCPs in the non-error group to assess their SA.
Interviews
A social scientist conducted semi-structured, open-ended interviews of PCPs between June 2008 and June 2009. To facilitate recall, we excluded PCPs involved in errors >1 year prior to diagnosis. We selected cases in which PCPs were closely involved and excluded those in which PCPs were residents. We invited PCPs to participate in an interview about the diagnostic process concerning specific patients of theirs who had recently been diagnosed with cancer. The identity of the patient was not specified at the invitation, and we refrained from discussing the purposes of the study in terms of ‘errors’. To increase the likelihood of finding new meaningful information across individuals and diagnostic scenarios, we used maximum variation sampling46 to select cases with different types of established indications. Selection decisions were made by team consensus. Potential participants were recruited through an email invitation, followed by a reminder email and phone call. PCPs were offered a $50 gift certificate as an incentive to participate.
The interview guide focused on discussing details of the case (see box 1). PCPs had the EHR available to them to prompt their memories during interviews. The interviewer asked providers to give an account of patient care from initial presentation to the time of diagnosis. Providers were prompted to give details about their diagnostic decision-making, such as their perception of clinical clues and reasons for particular actions. Interviews were conducted in the PCPs' offices and lasted between 15 and 60 min. Content was audio-recorded and later transcribed and de-identified.
Sample interview protocol questions
1. Patient history/presentation
I'd like to specifically focus on the patient presentation. S/he was diagnosed with cancer on date XX/XX/XX. Take me through the process that led to his diagnosis.
Probe examples
Can you talk about some of the reasons the patient came in?
Was there anything specific that caught your attention?
Talk about your plan of action.
2. Specific events
On this date XX/XX/XX, _______ happened. Can you take me through that event?
Probe examples
What is the general procedure?
What do you usually do?
Talk about the factors led to (or could lead to) the delay?
After going through the case, is there anything you think could have prevented the delay (example from case)?
3. General questions
In general, can you talk about factors that you think might affect clinical decision making? (Example: factors that help/facilitate or impede/delay the process.)
Analysis
We analysed data using a framework-analysis approach,47 a qualitative research method that allows for the inclusion of existing concepts as well as emergent themes.48 Based on previous work,30 40 41 we anticipated that chart review and interview data would suggest failure to act on established indications for work-up in diagnostic error cases.
Framework analysis consists of five stages: familiarisation, thematic analysis (thematic framework), indexing (coding), charting, and mapping and interpretation. First, the qualitative team (four study investigators) familiarised themselves with all data by reading, rereading, and summarising each case. Multiple readings allowed investigators to gain an overall impression of the interviews and chart reviews and to begin creating a thematic framework incorporating both emergent and a priori themes. From this framework the qualitative team created a coding scheme to summarise recurrent observations and themes in the data. Two investigators independently applied the coding scheme to all interview transcripts using the software package Atlas.ti 5.0.49 Coders subsequently met to reach consensus about discrepant coding decisions.
The research team reviewed the EHR and interview transcript for each case and collaboratively determined whether there was evidence for lack of SA. If so, the team defined the lowest level at which this occurred (higher levels depend on the success of the lower levels, consistent with the SA literature).50 For example, lack of SA at level 1 occurred when there was failure of perception of some information by the provider (see online appendix). Finally, the coded data and levels of SA were analyzed across cases and by themes to explore emergent patterns. All conclusions were drawn from multiple team meetings and group consensus.
Results
Sample size
Across both sites, 254 cancer cases met inclusion criteria. On chart review, 30 of 92 lung cancer cases (32.6%) and 56 of 167 CRC cases (33.5%) met criteria for error. Table 2 shows the distribution of types of cancers and respective indications for work-up. Of 55 PCPs invited to interview, 26 agreed to participate, 10 declined (most often due to lack of time), five left the institution before contact, and 14 did not respond to multiple interview requests. Providers had been in practice for a median of 12 (range 2–44) years. The final sample consisted of 31 cases (17 cases containing 18 errors and 14 with no errors) discussed with 26 PCPs (three PCPs discussed two cases, and one PCP discussed three cases). There was no statistical difference in median years in practice between providers in error versus no-error groups.
Types of established indications in cases of diagnostic error*
SA and error
In cases of diagnostic error, the model was useful to understand how SA was lacking in certain aspects of the dynamic provider–work system interaction.
Perception: level 1 SA (n=5)
All cases associated with delayed response or lack of response to predefined established indications were categorised as lack of level 1 SA. The EHR notification system, the primary means by which providers received abnormal test results, featured prominently in these cases. For instance, a provider who did not order colonoscopy until 6 months after an abnormal Hemoccult discussed how he might have missed the test result:It may have been an error, I got it [test result alert] but I just wasn't able to remember to put the colonoscopy in … But it would have not missed my eye when he [the patient] came the next time that's why I gave it [colonoscopy referral]. There is no defense other than between then and now he didn't see me … It could have come on a day when I was off and it was somebody doing an alert for me. I really cannot tell.(Provider 2)
Certain providers did not perceive alarm features recorded in the EHR. For instance, one provider acknowledged the importance of a patient's documented weight loss over several months but failed to see this clue until the patient brought it to his attention:They [x-rays] were all normal. That was up to March, until March and then July he came in and said ‘ok I am losing weight.’ Ok so CAT Scan on you because… it's a little bit worrisome now. (Provider 13)
Comprehension: level 2 SA (n=4)
Cases judged as lacking level 2 SA were those in which comprehension of the current situation was affected; they led to delayed action, inaction, or inappropriate action. In one such case a provider recognised rectal bleeding in an older patient but attributed this to haemorrhoids and referred the patient to a proctologist without performing additional work-up. In another case with rectal bleeding, another provider did not take any follow-up action because he attributed this to prostate cancer radiation:Bright-red blood per rectum on occasion but as far as I knew it was due to his radiation. So everything that he had, kind of leaned towards prostate cancer and things you get from prostate cancer and radiation and metastatic to bone and all … in most patients I take care of who have metastatic prostate cancer all these symptoms are related to that. And not a new cancer… (Provider 6)
Forecasting: level 3 SA (n=5)
When providers' thinking about what could happen in the near future was affected, we categorised these cases as lacking level 3 SA. Forecasting was thus applied not only in terms of disease progression and outcomes, but also in terms of how the work system28 29 in which the provider was practicing in might operate. Most of the providers involved took inappropriate actions, rather than delayed or no actions. For instance, one provider, knowing he would be out of the office for a few months, failed to hand over a patient being followed for two previous abnormal x-rays. Instead, he assumed that the patient would return to complete the third x-ray prior to his upcoming leave:And so what happened was, I went out and there was a discontinuation in care … when you're talking to patients and you give them follow-up appointments you do expect them to keep them although sometimes they don't for whatever reason … but in hindsight, I guess with two abnormal, we probably should've just went on and got the CAT scan so that was an error in my judgment… (Provider 3)
Choosing appropriate action: level 4 SA (n=4)
Delayed or no response tended to occur when providers lacked a sense of urgency related to the clue. For instance, despite knowledge of microcytic anaemia, a provider did not pursue actions such as ordering iron studies. In other cases, two providers with knowledge of diagnostic clues waited until their patients' return visits to order the subsequent diagnostic work-up:Well usually I'll want to see the patient back so I can explain to them what the problem is and that may sometimes be a delay … my preference is to talk to them [in person] because sometimes it may be that they didn't do the diet or you know something like, they're having rectal bleeding and I need to get them in quicker. So I prefer to see them … (Provider 26)
Lack of level 4 SA also led to inappropriate responses. For example, one provider ordered a CT scan of the abdomen and tumour markers to work up rectal bleeding.
SA in cases without errors (n=14)
We also evaluated application of the SA model in cases without errors. These cases involved patients who were symptomatic as well as those in whom only the cancer screening test was abnormal. In all asymptomatic cases (n=5; all CRC), providers immediately ordered the next appropriate step and notified patients as soon as the screen was found abnormal. For patients with symptoms (n=9), we noticed that providers not only perceived and comprehended symptoms as suspicious and took immediate action, but also took additional steps to ensure that patients maintained future follow-up (table 3), a potential strategy to combat diagnostic error.51 Thus, the model was applicable and showed the dynamic nature of the interaction of the providers and their environments (including patients) and provided insight into processes involved in attention management.
Additional steps taken by primary care providers to ensure that patients maintained future follow-up
Discussion
We found that the SA framework can be used to analyse outpatient diagnostic errors in the context of a distributed cognition approach that includes provider–work system interaction, including interaction with the EHR. The SA framework has already proven useful in other complex distributed environments such as aviation and, more recently, in anaesthesia. Because the diagnostic process in the longitudinal, distributed environment of primary care is poorly understood, application of the SA framework might help overcome some of the challenges faced in the application of current diagnostic decision-making models.52–55
Previous work on diagnostic error has emphasised concepts related to medical problem-solving and normative decision-making.54 56–61 Much of this literature has focused on unusual scenarios (uncommon diseases or atypical presentations of common diseases)62 and clinical problem-solving under experimental conditions.52 53 63 Models of deliberate decision-making processes, though useful, do not fully account for the reality that providers frequently miss more typical diagnostic ‘alarm features’ in real-world practice environments. Human factors-based frameworks for addressing safety in complex systems accommodate both higher-level decision-making processes and ‘simple’ risks such as forgetting or overlooking important information. One finding of this exploratory study was that providers implicitly depend on ‘prompts’ within the system (of which EHR alerts are but one example) to remember when to perform required actions. This is an example of distributed cognition (information processing distributed within the overall social-technical system rather than being held solely within the mind of the individual provider). When these expected prompts fail to occur for various reasons, the provider may fail to act. Thus, whereas many errors manifest in an apparently straightforward manner (eg, ‘dropping the ball’ on a missed test result), the circumstances that influence their frequency or consequences may be complex, and potentially modifiable. Application of the SA framework may help in better understanding the nature and circumstances around which errors occur in naturalistic socio-technical environments.29 64
Retrieval of diagnostic information from several distributed sources65 will become increasingly challenging as healthcare becomes more integrated through EHRs or health information exchanges.66 67 Thus, measuring and maximising various components of providers' SA should be an essential consideration to reduce diagnostic errors. Because SA has been studied within teams in aviation, it can also offer useful insights for primary care team interaction.68 The framework also encompasses cognitive activities suggested by experimental work involving diagnosis, including ‘generating/triggering a diagnostic hypothesis’ or ‘formulating a working-diagnosis’.52 56 However, by defining these activities as a series of cognitive processes involving errors in real-world practice, the framework lays a foundation for understanding contributory factors that might affect them.
The SA framework has features similar to previous taxonomies of data gathering and interpretation22 and other diagnostic care processes.39 Use of the SA framework might also help us understand how both intuitive and more deliberate processing are involved in the context of the provider–work system interaction, tying well to the dual-process model of diagnostic reasoning.54 For instance, later stages of the SA model reflect the more deliberative processing that must take place for successful diagnosis. Because of the emphasis on distributed cognition and provider–work system interaction, analysis using the SA approach could be complementary to other taxonomies. For instance, it could provide insight by distinguishing among levels of SA failures, an important concept because there may be different strategies to address risks (ie, through training or system redesign) depending on the nature of the failure.
We see two potential advantages of applying the SA framework alone or in combination with other analytical approaches. First, EHRs will house much of the diagnostic data providers will use to make diagnosis in the future and more automation is inevitable, hence this type of human factors perspective is essential to make further advances in the area. Second, SA has been measured in simulated settings in other fields and we envision that innovative mechanisms to measure levels of SA in simulated and live healthcare settings (eg, using standardised patients)69 can help advance the understanding of breakdowns in the diagnostic process in real-world settings. The current analytic frameworks do not allow that.
For optimal performance, certain aspects of the socio-technical work system need to be anticipated. In this context, we found forecasting to be of particular interest in diagnostic error scenarios. For instance, not being able to predict system vulnerabilities in healthcare could increase the risk that patients will ‘fall through the cracks’. Anecdotally, we noted that in non-error cases providers mentioned certain proactive strategies to minimise risk of loss of follow-up (eg, personally scheduling colonoscopies by calling gastroenterology, reinforcing need for follow-up to patients and/or family members).70 Cognitive and behavioural scientists should further explore these concepts.71 In aviation, high levels of SA might also be associated with ‘dynamic scepticism’, an attitude of questioning the validity of previous assumptions by constantly evaluating incoming data.72 This concept also merits further exploration. For instance, PCPs might become less sensitive to subtle clues because much of what they deal with is management of many routine or chronic conditions. Dynamic scepticism could be taught as a countermeasure providers might take to help anticipate and prevent problems.
Ongoing research in aviation and other industries about improving SA might be useful to design interventions to reduce diagnostic error.33 73 For instance, system design could have a powerful influence on how information is presented (for example, automated graphical display of weight if the EHR detects weight loss) to maximise perception. Level 1 SA failures, for instance, could be averted through improved alerting mechanisms, display designs, and task management strategies. Similarly, future research can identify cognitive constructs, mental models, and other EHR-related determinants (such as decision support systems) that facilitate comprehension.74 Novel training strategies using simulation could be implemented to maximise forecasting, which we found to be of particular concern.69 75 76 Measurement of SA could be incorporated in the design and evaluation of information systems so that provider SA can be maximised to reduce diagnostic error.
Our results have several limitations. First, our results do not show any causal relationship between SA and diagnostic error or diagnostic performance. Failure of SA does not imply a provider error and it is possible that factors unrelated to SA lead to problems in error cases and to successful outcomes in non-error cases. Underlying work-system problems might also be responsible for both loss of SA and errors. At present, we can only posit that loss of SA might be a risk factor of certain diagnostic errors. Second, as SA models have mostly been applied to more volatile situations, such as in aviation and anaesthesia, its applicability to primary care needs to be further validated. Third, we did not use any experimental techniques to measure SA but used consensus to determine lack of SA. Fourth, despite a large sample for chart reviews, the interview sample was small. Though adequate for concept exploration, our sample might not have represented PCPs and diagnostic scenarios optimally and may not be generalisable outside our study settings. Lastly, we could not collect reasons why providers did not choose to participate despite multiple invitations. Despite these limitations, because diagnostic errors occur universally, these concepts and methodologies might be useful to others.77
In summary, we found that a human factors-based framework can help analyse and understand outpatient diagnostic errors using a distributed cognition approach that includes provider–work system interaction, including providers' interaction with the EHR. Future research should explore strategies to measure and improve SA through multidisciplinary interventions.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Data supplement 1 - Online appendix
Footnotes
Funding This study was supported by an NIH K23 career development award (K23CA125585) to Dr Singh, and in part by the Houston VA HSR&D Center of Excellence (HFP90-020). These sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
Competing interests None.
Ethics approval Ethics approval was provided by Baylor College of Medicine.
Provenance and peer review Not commissioned; externally peer reviewed.