Methods

PRISMA-Medical

Prevention Recovery Information System for Monitoring and Analysis (PRISMA) was originally developed to manage human error in the chemical process industry, but in the last decade, it was also applied in the transportation sector, as well as in healthcare (PRISMA-Medical).8 12 13 16 17 The main goal of PRISMA is to build a quantitative database of incidents (including near misses) and process deviations, in order to facilitate the development and evaluation of system-based preventive strategies. Three main steps can be identified in the PRISMA-Medical method: (1) the causal tree incident description method; (2) classification of root causes by the Eindhoven classification model (ECM); and (3) formulation of structural measures for improvement. In this study, we focus on the first two steps.

Causal trees provide a visual interpretation of the chain of events leading to an incident, without hypothesising about possible causes. They present critical activities and decisions during the development of an incident in chronological order and show how activities and decisions are logically related to each other. Causal trees support the fact that nearly all incidents have more than one cause. By continuing to ask “why” of each event (beginning with the top event), a structure of causes and consequences arises, until the root causes are identified at the bottom of the tree. These root causes are subsequently classified by linking them to one of the categories of the ECM (online Appendix A). In some incidents, recovery factors can also be identified. In this study, we focus on the failure factors.

The ECM includes both active failures and latent conditions. Active failures are mainly represented by human error. The human section of the model is based on the SRK model developed by Rasmussen, which distinguishes three levels of behaviour: (1) skill-based behaviour; (2) rule-based behaviour; and (3) knowledge-based behaviour.18 The medical version of PRISMA also distinguishes patient-related factors. The latent conditions in the ECM involve technical and organisational errors.1 In total, the ECM distinguishes 20 different types of failure factors, which have been linked to 20 classification codes (online Appendix A). Technical and organisational factors are considered first when classifying root causes, and human failures are considered last. This sequence helps to counteract the tendency to start and stop analysis at the level of the end-user and leave the technical and organisational context of an incident unquestioned. The standardised classification of the causes of incidents through a coding system enables the analysis of multiple incident types or incidents from multiple units at the same time.16 17

Implementation of NEOSAFE

From February through June 2005, NEOSAFE has been implemented in 8 of the 10 Dutch level III NICUs (14–24 beds per NICU) and one paediatric surgical ICU. In these units, a total of approximately 3500 neonates are admitted each year. Voluntary, non-punitive incident reporting was introduced to establish specialty-based learning.9 An incident was defined as “any event which could have reduced, or did reduce the safety margin for the patient”.19 Before the introduction of NEOSAFE, only severe or catastrophic NICU incidents were collected through mandatory reporting to a central hospital committee, without performing systematic analysis routinely.

A multidisciplinary patient safety committee, consisting of at least one physician and three nurses, was formed in each unit. Committee members were recruited on a voluntary basis. Unit employees were encouraged to communicate openly after incidents and to report incidents to the committee non-anonymously, to enable contact if any additional information was needed during analysis. Personnel were asked to fill in a two-paged incident report form (containing both closed and open items) immediately after the discovery of an incident. Incidents were either self-reported or reported by personnel who discovered the incident.

In December 2004, members of the committees participated in a 2-day PRISMA-Medical course. With the introduction of NEOSAFE, patient safety committees started using PRISMA-Medical to identify root causes of incidents pertaining to medication, arterial and venous lines and mechanical ventilation. Based on previous literature and expert opinion, these incident categories were thought to have the greatest influence on patient safety in the NICU.19 20 21 To increase reliability of results, each analysis was conducted by two members of the committee, who were PRISMA-trained and familiar with the department and its processes. Committees were encouraged to analyse incidents within 2 weeks after reporting. Due to the organisational structure in the NICU, the composition of these couples was subject to continuous changes. The patient safety committees managed an electronic database (MS Access) of reported incidents and results of subsequent PRISMA analysis. Patient and staff confidentiality was ensured by excluding personal identification from the electronic database. To stimulate incident reporting and safety awareness, the committees provided all NICU employees with a summary of incident reports and planned preventive actions on a regular basis. From July 2005 (t = 0), incident reports have been aggregated for specialty-based analysis. The local medical research ethics committee (METC Zwolle) waived the need for ethical approval, as the study only involved the registration of incidents and their causes. The present study describes the results of the PRISMA analysis of incidents reported between 1 July 2005 and 30 June 2006.

Reliability testing

In a pilot study in September 2005 (test 1, t = 3 months), the committees applied ECM classification codes to a total of nine root causes in predefined causal trees, sent by email by the central investigator (CS). The predefined causal trees were constructed by an expert couple (authors CS and TvdS) and were based on randomly selected incidents reported to one of the NICUs (one from each incident category). In November 2005 (t = 5 months), a PRISMA-Medical refresher course and consensus meeting was held to evaluate implementation of PRISMA-Medical. As a result of this evaluation, the ECM was further illustrated with examples specific to the NICU (“ECM-NICU”).

At the next NEOSAFE evaluation meeting in June 2006 (test 2, t = 12 months), ECM classifications were performed on new test cases. The test cases consisted of two sets of 45 root causes (15 root causes from each incident category) randomly selected from 1240 root causes identified in the central NEOSAFE database during the first 6 months. Analyst couples were formed within each committee. Couples were divided into group A and group B. Using the ECM-NICU, each couple in group A classified the first set of 45 root causes and each couple in group B classified the second set, while an expert couple (authors CS and TvdS) classified all causes.

In both tests, pairwise agreement for each rater couple versus the expert and agreement among all raters was measured at the three levels of ECM classification (online Appendix A). Inter-rater reliability was determined by calculating generalised κ values for each level of ECM classification.22 23 We used a macro in SPSS V.12.0.1 for Windows to define the number of cases and number of raters. The classification by Landis and Koch was used for interpretation of κ values (κ<0.00 = poor; 0.00–0.20 = slight; 0.21–0.40 = fair; 0.41–0.60 = moderate; 0.61–0.80 = substantial; 0.81–1.00 =  (almost) perfect).24

Results

For organisational reasons, one committee did not supply PRISMA results during the study period and was therefore excluded from further analysis. Several of the remaining eight units reported shortage of time in handling the large number of incidents reported after the introduction of the voluntary reporting system. Therefore, units expecting time-management problems were instructed to analyse every third report to get a representative sample of PRISMA analyses. During the study period, 981 out of 1786 eligible incidents (55%) were analysed for underlying root causes. In total, 2313 root causes were identified and classified, giving an average of 2.4 root causes for every incident. Table 1 shows the results of PRISMA analysis for each incident category.

On average, 64% of all identified causes were classified as human failure, almost one-third of root causes represented technical and organisational failure (9% and 22%, respectively), and another 3% were patient-related factors. Most technical failures were found among incidents with mechanical ventilation, whereas the causes of incidents with medication were most often classified as human failure (fig 1). As can be seen from fig 2, verification failures were most often found in this incident category, followed by failures in monitoring and intervention. Besides human failures, a peak in protocol failures was found among incidents with IV lines (over 10% of all identified causes), and another 8% of causes of incidents with IV lines were classified as patient-related factors. Among incidents with mechanical ventilation, 9% of the causes were classified as technical failures that were beyond the control and responsibility of the unit.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Distribution of root causes (main level) for each incident category (July 2005–June 2006).

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

PRISMA profile of root causes (sublevel) identified in each incident category (July 05–June 06). *See online Appendix A for an explanation of the ECM codes.

In test 1, seven out of eight committees were able to return the results of test cases in time (table 2). Multirater agreement was moderate at the sublevel and SRK level and substantial at the main level of ECM classification (table 3).

In test 2, 20 members from seven patient safety committees participated in the test. One committee was unable to join the NEOSAFE meeting and therefore did not participate in this test. Each group of test cases was performed by five pairs of raters. In both groups A and B, although the extent of agreement between each couple and the expert varied, substantial agreement was reached among all analyst couples at the main level of classification, whereas the SRK level showed moderate agreement. At the sublevel of classification, moderate agreement was reached among analyst couples in group A. In group B, however, agreement at this level was fair (table 4).

Discussion

Our results show that the PRISMA-Medical method has a great potential to contribute to the identification of system failures that lead to incidents in the NICU. As both active failures (human failure) and latent conditions (technical and organisational failures) that caused the incident were discovered, the total profile of root causes identified through PRISMA analysis can be used to provide a more realistic view of how the system is actually working. The variation in causal coding between incident categories suggests that different approaches are needed for each incident category to prevent recurrence. However, our practical experience shows that feasibility of PRISMA-Medical for specialty-based analysis strongly depends on the availability of time and PRISMA-trained personnel. A possible solution to this problem is the selection and prioritisation of certain incident types, such as high-risk incidents, for PRISMA analysis.7 Moreover, in the early stages of incident reporting, caution is needed when interpreting results of PRISMA analysis for the development of specialty-based preventive strategies, as inter-rater reliability tests pointed out that agreement among patient safety committees depends on the level of classification. Although substantial agreement was reached at the main level of ECM classification, and agreement among the committees within the SRK level was acceptable, discrimination between rule-based errors (qualification, coordination, verification, intervention and monitoring) was more difficult to assess. Given the fact that some rater couples scored better than others (table 3: raters 2, 5 and 7), agreement probably also depends on other factors such as level of training or time working in the unit.

Our study is the first extensive, multi-centred reliability study that is based on both major and minor incidents. Few other studies have investigated reliability of PRISMA-Medical. A study among three healthcare inspectors and a PRISMA expert in The Netherlands found acceptable reliability scores at all levels of ECM classification, with increased reliability after repeated education (M. Habraken, 2005: Better care for incidents in healthcare, MSc thesis, Eindhoven Univ of Technology). However, this study was based only on severe, mostly fatal medication incidents, which is quite different from our study design.25

The value of PRISMA-Medical for specialty-based incident analysis has been described previously. A US study in transfusion medicine reported that PRISMA-Medical is a very useful tool in sense making of individual and specialty-based, aggregate causal data.7 12 17 The percentage of human causes in our study (64%) was rather high compared with their study (46%) and compared with a study on incidents reported to The Netherlands Health Care Inspectorate (42%).17 25 Although this may reflect the actual situation in the NICU, it may also be due to remaining “person-oriented” biases during the root cause analysis. If the latter is the case, one should expect to observe a decrease in the percentage of human failures after repeated education. Moreover, although several causes were identified for each incident, we did not test inter-rater reliability of the Causal tree description method. Therefore, future research should develop methods to test the reliability of the causal tree description method as well. A pilot study has already shown the importance of this issue.26

This study has some limitations. First, only 55% of all eligible reported incidents were analysed. Although our sample size is quite high, and we tried to minimise selection bias by analysing every third report in case of time-management problems, this may have affected the final profile of root causes. Second, valid comparisons between the first and second reliability test were difficult to assess because of natural selection of analyst couples, as a reflection of the clinical setting. Third, the 2-day PRISMA course was a shortened version of the original 3-day course. This may have affected the reliability of classification negatively, especially during the time before the PRISMA refresher course (table 3).

In conclusion, with some restraints, PRISMA-Medical proves to be both feasible and acceptably reliable for specialty-based identification and classification of multiple causes of incidents in the NICU. The system approach states that preventive actions should primarily aim at the structural (latent) system failures to prevent inevitable human failures, which are usually at the end of the incident cascade, leading to patient injury.1 In this light, preventive strategies in the NICU should be aimed at the technical and organisational weaknesses first, rather than at human failure. Future research should study the predictive validity of PRISMA by investigating the effect of system-based interventions on the prevalence of these system failures, as well as on patient harm. Moreover, future studies should also examine the role of the PRISMA Error Recovery Factors in the prevention of incidents.