Article Text
PDF
Abstract
Background Large numbers of claim files present a potentially valuable source of information to get insight on possibilities for prevention of claims. Therefore, the feasibility of root cause analyses on incidents leading to liability claims at The Netherlands' largest medical liability insurer was assessed.
Methods Feasibility was defined by validity, reliability and applicability. Claim files from diagnostic errors in emergency departments of Dutch hospitals were selected. All closed and settled claim files from the year 2001 and 2002 were used.
Results Fifty incidents occurring at 31 emergency departments were found in 47 files. 114 root causes were found, on average 2.3 per incident. 78% of the incidents were related to missed fractures, luxations or tendon lesions. Zero technical, 29% organisational, 66.7% human-related and 4.4% patient-related factor errors were found. Inter-rater agreement for classification of root causes was good (κ=0.78). Preventive measures following from Prevention and Recovery Information System for Monitoring and Analysis (PRISMA) classification–action matrix were improving completion of available sources of information, improving structures of communication and training staff to prevent failures due to the wrong implementation of tasks or due to errors in reasoning.
Discussion Validity of the root cause profile of diagnostic claims is considered moderate because of a lack of information about technical and organisational causes of errors. Therefore, the root cause profile was incomplete for organisational factors in comparison with other studies. However, with regard to the diagnostic reasoning process, the profile was stable. The feasibility of PRISMA for retrospective analyses of closed claims may be improved if system-based reasoning by the liability insurer and hospital staff is enhanced.
- Malpractice
- diagnostic errors
- emergency service hospital
- root cause analysis
- feasibility
- patient safety
- emergency department
Statistics from Altmetric.com
- Malpractice
- diagnostic errors
- emergency service hospital
- root cause analysis
- feasibility
- patient safety
- emergency department
Learning from incidents by means of root cause analysis is one of the key elements of programmes or institutions aiming to improve patient safety.1 Insight on root causes simultaneously gives insight on possibilities for prevention of patient safety incidents. In the past decades, several initiatives to assess root causes of incidents have been developed, such as the Prevention and Recovery Information System for Monitoring and Analysis (PRISMA).2 The PRISMA method was developed originally to detect human failures in the industry.3 It has recently been adjusted for and applied in healthcare.4 Analysis of consecutive incidents with PRISMA results in a quantitative database of root causes of incidents and process deviations.4 This will make it possible to aggregate results on a higher level and to identify preventive measures from a broader perspective.
PRISMA has proven to be successful in detecting latent management and organisational failures in institutions that may lead to errors and thereby harm patients, clients or employees.5 In The Netherlands, the Healthcare Inspectorate and various blood transfusion and anaesthesiology departments have introduced this method to define the underlying causes of near misses and adverse events.1 ,5 In these studies, PRISMA was applied soon after the occurrence of a patient safety incident. Whether it is feasible for research on root causes of incidents leading to liability claims is unknown. Malpractice claim files present a potentially valuable source of information about errors6 because they contain detailed information about the statement of claim, medical reports, experts' opinions from both sides, plaintiffs' pre-event and post-event condition, the patients' record and, in some cases, verdicts of the Healthcare Inspectorate and/or court. In particular, the medical expert's opinions from both sides are a valuable source of extra information in comparison to what is usually available when drawing root cause analyses using documentation only within hospitals, where medical records would be the primary source of information. The aim of this study was to assess the feasibility of using PRISMA on closed malpractice claims to support prevention of medical claims.
Methods
Subjects/study population
We used claim files of the largest medical liability insurer for hospitals in The Netherlands, where approximately 1000 claims are reported annually. The claims files have a standardised structure that contains the liability claims formulated by the patient, the medical file of the patient, the opinion of the medical experts of the liability insurer, which is a concise summary of the clinical file, the opinions of the care givers involved, the correspondence between accuser and defender and the liability insurer and, finally, information about payments that were made. In addition, sometimes a report of an external medical expert was present. Furthermore, if available, verdicts pronounced by the hospital complaints commissions were used, as were the verdicts of the Healthcare Inspectorate and courts.
We selected diagnosis-related settled and closed claim files that were reported between 1 January 2001 and 31 December 2002 on 20 March 2006. Files were selected from a standardised database. After adverse events related to surgical procedures, diagnosis-related adverse events were the most frequent reason to sue for a liability claim at the liability insurer involved in this study. Two-thirds of these claims occur at the emergency department (ED). Diagnosis related was defined as a wrong, missed or delayed diagnosis. Furthermore, incorrect assessments of patients' conditions and inappropriate follow-ups after diagnostic tests were included.
PRISMA-medical
PRISMA-medical is a relatively simple method to analyse incidents (or, in this case, incidents/adverse events that led to liability claims). Only basic knowledge about medicine and insight on the hospital structure is needed. In this case, both investigators doing the analyses were health scientists. Both investigators practiced with 10 diagnosis-related claims from years other than 2001 and 2002 until face validity was reached. Those 10 claim files were discussed in the presence of a third person (AH). A consensus about the root causes was soon achieved. These 10 files were excluded from the definitive set of claims.
PRISMA, which is based on the theory of system approach of Reason7 and the SRK model of Rasmussen8, consists of three components that were applied as follows. First the incident was described by means of making a causal tree after the information is gathered from the claim files. In this case, the main investigator used a self-constructed information gathering questionnaire to systematically collect information from the files. This information questionnaire consisted of information about dates, care givers involved, causes and patient characteristics. Furthermore, a short description of the incident was asked for. As a result, in every analysis, the same information was searched for. The top of the tree represented the incident that triggered the analysis. Direct and indirect causes are added by means of structural questioning on why the incident happened. Answers on the why questions must have been present in the claim files. When no objective fact could have been attributed to the tree, the drawing of the tree stopped. Otherwise, conclusions would be drawn on the investigator's own interpretation instead of what was really going on. In addition, the formation of the tree stopped when causes were outside the scope of the organisation.
Second, the root causes were classified according to the Eindhoven classification model (ECM) of system failure. The ECM distinguishes four types of failures: technical, organisational, human-related and patient-related failures (table 1). The four categories are refined in a subclassification of 20 root causes. In this study, we added a code for organisational problems regarding the communication of x ray results (OR), and a code for the situation in which residents did not consult their supervisor (OS) as we believed the causes were very common but their underlying causes were unclear. However, to leave them out of the analysis would imply a loss of possible indications for improvement, as these topics would not receive a code. Current ECM did not sufficiently cover these types of failures. Third, based on the frequencies of ECM codes of root causes, a classification–action matrix was made. Each type of failure defined by the ECM corresponds with measures in a given classification–action matrix on the aspects of technique, procedures, information/communication, training, motivation, escalation and reflection (see figure 1).
Frequencies (%) and types of root causes of diagnosis-related incidents at emergency departments that resulted in malpractice claims in 2001 and 2002
Example of a PRISMA analysis.
Analysis
Root cause analyses were done by one investigator (IvN) and separately classified according to the ECM by two investigators (IvN and ME). Feasibility was studied by assessing the method's internal validity, reliability and applicability. Validity was assessed by determining whether results were presumably based on reality, by comparing our results with those of other research regarding root causes on diagnosis-related incidents. In addition, we assessed face validity of the recommendations resulting from the classification–action matrix by interviewing two risk managers of the liability insurer about the suitability of our recommendations in practice. Reliability was assessed as the inter-rater agreement regarding classification of root causes by means of calculating the Cohen's κ with correction for coincidence between the two investigators. Applicability was assessed as the clocking time spent per case and the delay between occurrence, detection and reporting of incidents.
Descriptive analyses were done for gender, age, the average number of days in delay between missing the diagnosis and the right treatment, number of patients with permanent damage, number of patients who died, the number of residents consulting their supervisor, incident time and day and finally the presentation at the ED. Incidents were categorised as missed or delayed diagnosis regarding fractures, luxations and tendon lesions; missed diagnosis other than fractures; and incorrect assessments of a patient's condition.
Results
Fifty closed and settled files were found, 22 from 2001 and 28 from 2002. In both years, four cases were still open. The claims originated from 31 EDs with a maximum of four claims per ED. Three of the 50 files were excluded because they were not settled in light of a diagnostic-related adverse event.
Most patients were men. Patients' age and the delay between missing the diagnosis and starting the right treatment varied widely. Almost 60% suffered from permanent damage. The incidents most often occurred during office hours and relatively more frequently during weekends. Seventy-eight per cent of the missed diagnoses were related to fractures, luxations and tendon lesions (table 2).
Patient and incident characteristics of closed and settled malpractice claims reported in 2001 and 2002
Fifty incidents were found in which 114 root causes could be identified, an average of 2.3 per incident (range 0–8, median 2) (table 1). No technical failures were found. In 19 cases, an unusual presentation of the disease was classified as a context factor and not taken into account as root causes. Inter-rater agreement was high (κ=0.78). Root causes were mainly deduced from reports from medical experts and the care givers who were involved. The top five of the PRISMA profile (figure 2) indicates that several preventive measures are necessary.
Prevention and Recovery Information System for Monitoring and Analysis (PRISMA) profile of claim files related to diagnostic failures (percentage of root causes with 95% CI*).
To decrease knowledge-based failures (HKK), information sources have to be completed and communication structures (eg, between supervisors and residents or radiologists and residents) have to be improved. For example, it might be very useful to write down at the beginning of a shift who is supervising who. Furthermore, written instructions when to consult a supervisor under what conditions must be clear to all. In addition, protocols regarding the treatment of musculoskeletal disorders and other conditions must be available to all healthcare workers at all times, and they must be regularly updated.
To accomplish a decline in frequencies in the human rule-based errors due to incorrect qualification and interventions, adequate training has to be established—for example, in the form of a resident orientation programme before their employment at the ED. The last code that is suitable for measures is the x ray code (OR), which could imply that there should be an optimisation of procedures—for example, daily x ray review meetings in which to check interpretation, and its corresponding disease management, of x rays from the previous day by a radiologist, a senior physician or a supervisor and the residents. The face validity of these measures is high as risk managers of the liability insurer confirm that these problems are commonly seen by them.
Discussion
When evaluating 50 incidents of diagnostic claim files, the root causes appeared to be mainly human related. Furthermore, unusual presentation of diseases often served as a context factor that contributed to misdiagnosis.
The preventive measures that resulted from this study concerned completion of available information resources and communication structures, and training of staff to prevent failures due to the wrong implementation of a task or due to errors in reasoning. These measures may only be valid when they are based on a complete profile and have face validity. Regarding this completeness, the results found in this study may give an incorrect representation of reality. First, other studies reported, on average, more root causes per incident, especially when one might not consider our added codes (OR and OS) as root causes.5 ,9 Readers who want to compare our results to future studies using PRISMA should neglect the codes OR and OS.
Second, these studies also reported more system-related or latent root causes. However, discrepancies with respect to other studies may be attributed to different classification techniques. For example, Graber et al9 conducted a study into diagnostic failures at the internal medicine department. They defined failures after their aetiology into cognitive, system-related and no-fault errors. When we apply the same incident definition to our data, it becomes clear that they identified more system-related errors than we did: 19% versus 6% of all incidents were based on system-related errors only, and 65% versus 50% of the incidents were caused by at least one system-related error.
Although our results are unstable for organisational factors, they may well be stable concerning human-related and patient-related factors. Kachalia et al10 found that cognitive factors were the leading causes of missed diagnoses, in fact in 96% of cases. The difference between their 96% and our 66.7% human-related factors may be because they also included contributing factors, for instance context factors or intermediate causes, in their calculation of human-related factors. These contributing factors may be defined as direct causes in our study and therefore not reflected in the percentage of human-related factors as we considered only root causes.
The PRISMA-medical tool indeed shows that there are other factors besides human factors that lie at the heart of diagnostic errors. Furthermore, the numerous human-related root causes are not surprising in view of the fact that the diagnostic reasoning process is mainly a cognitive process. This implies that objective reasoning and valid and logical thinking is not always present depending on the, in this case, mostly inexperienced, physicians' cognitive and affective disposition to respond to disease manifestation and patient characteristics, especially in a context of uncertainty.11 The numerous amount of unusual presentation of conditions as context factors suggest that diagnosis-related incidents resulting in claims are affecting these dispositions to respond.
Although, in this study, organisational failures were probably underestimated, the proportion of disease-related and human-related failures may give a reliable estimate of these causal factors leading to diagnostic failures at the EDs. Since 70% of EDs are represented in our sample, we believe the results are generalisable to EDs in The Netherlands. However, because of the specific patient population and care applied in the EDs, our results are not generalisable to other hospital departments or to other types of claims, such as surgical or medication-related claims.
The face validity of the recommendations resulting from the root cause analyses was supported by the fact that the root causes and proposed solutions were recognised as the main problems at EDs by the liability insurer's risk managers. As a result, the proposed recommendations have been translated into real measures at EDs. They seem to have their impact on patient safety and healthcare quality in The Netherlands; however, empirical data are not available. An advantage of the PRISMA procedure over the practice experience from risk managers is that problems and its possible solutions are prioritised.
Another part of feasibility is the reliability of the method. Our κ of 0.78 shows that inter-rater agreement concerning the classification of root causes is high. However, there is currently no valid method available for incident description. To achieve as reliable trees as possible, the investigators therefore have practised with 10 dossiers. In addition, the main investigator used a self-constructed information-gathering questionnaire. As a result, in every analysis, the same information was searched for.
The application of PRISMA-medical is relatively time consuming; however, the main problem was the considerable length of time involved between the occurrence of incidents, their detection and their reporting at the liability insurer (and for diagnostic incidents, we counted the delay between occurrence and detection to that). Additional information about the context and organisational and technical information can therefore hardly be gathered. If PRISMA will be applied soon after the occurrence of the incident, this problem may not exist. However, the small number and great variability in topic and nature of incidents that are reported in, for example, 3 weeks (ie, 5%), will make it nearly impossible to draw conclusions from PRISMA analyses.
We conclude that the main problem of PRISMA's feasibility is twofold. First, the delay between incident occurrence, detection and reporting at the insurer, and second its moderate validity. This moderate validity also lies within the fact that the liability insurer's medical advisors and the care givers involved in the incidents were not trained in system-related reasoning. It becomes clear from applying structural root cause analysis on claim files that medical advisors and care givers do not contemplate latent failures, which is a missed opportunity to identify prevention strategies for hospitals (departments) and medical liability insurers. Feasibility might increase as medical advisors, risk managers and care givers, keep in mind that healthcare organisations are complex organisations and, as a result, question more why things happen in light of this complexity. However, a shift from a single-person focus to a system approach needs a cultural change in the field of medical liability insurance, which cannot be generated readily, in particular because root cause analyses are not necessary to make decisions about the liability of a hospital or healthcare workers. Still, efforts to establish a culture change may be given the benefit of the doubt for several reasons. First, the learning effect might be considerable if measures could be made on an organisational level, although, at present, the liability insurer has a preventive policy with high face validity. Second, individual physicians will feel less threatened when not blamed in person. As a result, they might be more open regarding patient safety incidents, which will finally enhance patient safety by means of gaining more learning aspects from incidents and creating a culture of patient safety with probably less patient safety incidents and consequently fewer malpractice claims.
Problems with feasibility may also derive from the nature of the liability claims we have used in this research as diagnostic claims have a greater delay in detection as, for example, missed myocardial infarcts, and they are mainly cognitive in essence. With a further development into system-related reasoning, which is at present triggered by all patient safety efforts in The Netherlands, the balance between human-related and system-related root causes in the PRISMA profile may become valid, which makes it possible to tailor measures. Therefore, re-assessment of the applicability of root cause analysis on recent and other types of claim files is recommended.
Footnotes
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.