Root Cause Analysis (RCA) is a method designed to investigate errors in high-risk industries such as petrochemical, aviation and railway industries. It was adopted in healthcare1 during the 1990s by the US Veterans Affairs Hospitals for identifying the root causes of clinical mishaps, and for devising strategies that will minimise recurrence. RCA was adopted by the National Patient Safety Agency (NPSA) in England and Wales in 2001, arising from a report to the Chief Medical Officer “An Organisation with a Memory”2 and the subsequent response “Building a Safer NHS for Patients.”3 A training programme for England and Wales was launched in March 2003.

RCA methods require healthcare staff to analyse incidents, and to formulate recommendations that aim to prevent recurrence. These recommendations can then be shared with colleagues and with other health organisations where comparable kinds of work are done. The RCA technique aims to avoid personal blame of healthcare staff involved in critical incidents, and to focus on the “systems problems.”4 The term “systems problems” pinpoints those routine and habitual facets of clinical or managerial practice, and the physical environment and equipment, and care pathways, if subjected to scrutiny and explicit intervention, may be redesigned to reduce system vulnerabilities and support system barriers and recovery routines to avoid or minimise risk.

Improving the effectiveness of RCA training is of huge importance, given the centrality of these methods both in the UK and worldwide. In England and Wales, RCA is the tool of choice for serious untoward incidents and homicides in mental health services (http://www.npsa..nhs.uk/health/resources/7steps). RCAs are required by the Healthcare Commission, with similar requirements by, for example, the Clinical Excellence Commission in New South Wales, Australia. The progressive introduction of “critical incident monitoring schemes” into healthcare systems places still greater emphasis on transparent monitoring of adverse events. In England and Wales, all NHS trusts are required to report incidents to the Strategic Health Authorities (SHAs) and the National Reporting and Learning System (NRLS), and undertake analyses of patterns of incidents, as well as investigate more serious incidents fully.

The NPSA developed a 3-day RCA training programme delivered by pairs of the 34 Patient Safety Managers (PSMs) employed by the NPSA to work with local health boards in Wales and SHAs and their geographically associated NHS trusts in England. There are also internet-based self-study tools and materials to assist with the teaching of RCA and conduct of RCAs (http://www.npsa.org.uk). The NPSA trained over 7000 staff in 2 years, offering at least eight fully funded places to each of the 607 trusts in England and Wales.

The Department of Health’s Patient Safety Research Programme commissioned an evaluation of the National Patient Safety Agencies’ (NPSA) national programme known as the Networked RCA training over a 9-month period from January to September 2005.

METHODS

The evaluation was commissioned in the last year of the programme, and as almost all training courses were due to complete by April 2005, the evaluation had to be a post-training design. The study reports a survey conducted with two groups of participants immediately after the course (Time 1; T1) and at least 6 months after the course (Time 2; T2). Results of case studies of the application of the programme in NHS trusts will be reported elsewhere.5 This paper describes the knowledge outcomes achieved and the personal beliefs about RCA practice and training within their trusts of course participants.

A self-report survey was developed by the authors and is described in appendix A, with the questionnaires in appendices B and C. The questionnaires describe the characteristics of the participant’s job and organisation, their role in risk-management and patient-safety investigations (PSI) and in house training in RCA, their use of RCA.

The core course content and expertise from the course designers were used to identify key factual knowledge which was tested by multiple-choice questions. Personal beliefs about the practice of RCA were measured using constructs from a social cognition theory, the Theory of Planned Behaviour6 as described in appendix A. This measures the attitudes towards RCA (its purpose), the social norm for conducting RCA (managerial, peer, organisational influences) on the practice of RCA, and the perceived behavioural control and self-confidence in conducting RCA, all of which are expected to influence intention to conduct RCA. Stepwise multiple regression was used in each sample to ascertain the extent that intention is predicted by each of these different beliefs. In addition, specific questions were asked about the confidence that participants have in conducting RCA within their particular trust context and about barriers to the conduct of RCA.

Recruitment of participants

PSMs briefed participants on the last day of the course (T1) and administered the questionnaire, which could be completed immediately, or returned by post or via a website. A further sample who had completed a course 6 months previously (T2) were contacted by the researchers by post.

Response rates

T1: 374 questionnaires were received from 734 attendees (51.0%). T2: 2312 questionnaires were administered, yielding 350 returns (15.1%).

RESULTS

Participants’ employer, job role and experience of RCA

The two samples are broadly representative of the trust types in the NHS, that is with most staff employed in primary care trusts or the acute sector, and with a smaller proportion from mental health and ambulance trusts. In both samples, two job types accounted for most of the jobs: Risk Manager (RM) (T1, 115/374 30.7%: T2: 145/350 41.5%) and non-clinical staff with RM responsibilities (T1 83/374 22.0%, T2 87/374, 23.3%). There were numerous other job titles, although these were mainly clinical roles. At T1, 79.6% (258/324) had conducted a PSI prior to the programme. In the T2 sample, 58.3% (204/346) had conducted RCAs since the programme.

Knowledge outcomes from RCA training

Results at T1 for understanding the human factors model in relation to the healthcare context showed high percentages of correct answers concerning defining violation errors (83.7%), barriers to error (77.5%), contributory factors (77.3%) and types of error (61.5%). But on a case vignette in which the specific type of error had to be identified, only 42.5% did this correctly. Similarly, analysis of care and service delivery problems (causes) in a case vignette were judged correctly by only 51% of participants. Arguably, the course can teach identification and use of terms more readily than application to real-world scenarios, where other judgements and experience are brought to bear. Testing understanding of RCA practice included correctly identifying critical components of techniques; 89.7% were correct in relation to change analysis, 83.5% were correct in relation to cognitive interviewing.

Knowledge test results at T2 were obtained from two questions regarding identification of errors where 84% (294/350) were able to judge the correct factual item, whereas on a test using a case vignette for identifying violations, only 101/350 (29.0) judged this correctly.

The correct result of a case-study scenario to test understanding of the procedure for managing staff involved in an incident was given by only 49.5% of participants (a non-blame/supportive outcome). Of note was that 47.1% indicated that punitive (counselling and/or disciplinary) actions should result. This is not only incorrect as far as the approach taught, but suggests that participants are reflecting organisational cultures of their trust which do not embrace the “fair blame” ethos espoused on the course.

The results of the knowledge tests show generally high correct responses for factual information, but much lower levels of knowledge when concepts are tested using vignettes. It is likely the vignettes test the more real-world application of the concept, and so present a tougher test of the learning outcomes.

Beliefs about RCA, training others in RCA and personal motivation to conduct RCA

The programme was assessed to be helpful or very helpful by 88.9% (316/356) at T1 and 73.6% (253/344) at T2, which suggests a high level of overall satisfaction.

The commonest model for training others within their trust was cascade training by those trained on the NPSA course (T1: 245/374, 65.5%; T2: 200/333, 60.1%), with 30.7% (115/374) at T1 and 42.3% (141/333) expecting to send others on NPSA courses (often those close to the NPSA offices). At T2, there is evidence that NPSA materials are recognised as an important part of local training for 26.7% (89/333) of participants. Fewer than 10% of the whole sample rely on external consultants. Detailed results are presented in appendix D.

Given the centrality of cascade methods of training, we examined the confidence of participants in their role as trainers. At T1, 22.7% (84/370) would find it easy to cascade RCA training to others, rising to 38% (133/350) at T2. This may reflect both that not all staff perceive they are or will be involved in training and that training others in RCA requires skills and materials beyond those available on the course.

Personal beliefs about RCA practice

Table 1 shows the scale mean scores for concepts derived from the Theory of Planned Behaviour, in relation to the conduct of RCA.

The results show that in both samples, three-quarters of participants intend to use RCA for a serious untoward incident in their trust in the next 6 months. This is supported by overwhelmingly positive attitudes to RCA and the importance of training others in RCA to improve patient safety, with three-quarters in both samples also recognising a positive social norm towards RCA (ie, expectations of managers’ support for their use of RCA). However, belief in the person’s control and confidence in conducting RCA is high for only 42% after the course, and nearly 60% 6 months after the course. Stepwise multiple regression was used to ascertain the extent that the attitudes, social norm and perceived behavioural control predict intention, as suggested by the Theory of Planned Behaviour. At T1 the result was significant (F 5.93,df 4,305, p<0.0001;R² 0.072, t = 3.2 p<0.0001), but this explained only 7% of the variance. At T2 a stronger association was found (F = 25.8, df 4,304, p<0.001) which accounted for 25% of the variance in intention, with perceived behavioural control making the greatest contribution (t = 5.9, p<0.0001), and social norm was modestly significant (t = 3.5, p<0.001). Adding other variables, such as prior experience of RCA in the past 6 months, and the individual’s total barriers scores, added little to prediction, supporting the powerful effect of the psychological variables. Caution must be used in interpreting these results, as the model is more appropriately applied to sequential data rather than cross-sectional samples. Also, the ceiling effect of the high endorsement of attitude may have accentuated the effect of the other two variables, but given the overall effect, it appears promising to target support activities that increase personal control over the conduct of RCA by tackling specific organisational barriers in order to increase intention to undertake RCA.

In order to understand the barriers that might need to be addressed to improve motivation to conduct RCAs, we asked participants to rank a list of barriers to RCA in their trust. The results are presented in table 2.

The results show there are quite similar common concerns at both times, that is, time-related to conduct of RCA by the participant and for other trust employees to be involved. Issues of the lack of opportunity for personal and organisational learning in RCA practice are more salient at T2, which could be remedied by continued training and learning sets. Some trust system issues appear to be frequent concerns at both times, particularly having an adequate system for training other staff in the trust in RCA. Issues of decision process for triggering RCA, feedback and responsibility for implementing learning from RCAs are middle-ranking issues at both times, which require operational procedures. A barrier that might be expected to directly influence motivation to conduct RCA is whether actions from RCA are implemented. This was rated the second highest barrier at T2 but 19th at T1. Similarly, confidence in personal use of RCA is ranked 18th at T1 but ranked fourth at T2. This again suggests that with greater experience, participants became aware of their need for continuing development in the practice of RCA.

Additional items enable us to examine in more depth participants’ beliefs in whether RCA will impact on patient safety and trust practices. Just 46% (172/374) of participants are confident that RCA will reduce patient safety incidents in their trust at T1, and only 62% (217/350) at T2. Also, at T1 only 23.2% (148/332) felt it would be easy to implement RCA as standard practice in their trust, and 43.0% (148/344) at T2. This suggests the reality of implementation may not be addressed sufficiently by the initial RCA course, and in trusts subsequently.

Questions were asked of the T2 sample about the outcome and reporting of RCAs, which again may affect motivation to conduct RCA. The results are primarily based on the 59% of respondents who had conducted RCAs since the course. Most participants believed the outcomes and learning were reported within the trust (87.6%), but only a third (32.9%) thought they were reported to external bodies. Reporting included some form of internal dissemination and improvement strategies for three-quarters of participants. Further data are presented in appendix E. This is lower than optimal, since it is difficult to conceive of an RCA that would not result in internal learning points.

DISCUSSION

The study has a number of design weaknesses associated with self-report studies, although these were ameliorated as far as possible, by pretesting items for validity and by ensuring anonymity. Measures were developed for the study, as none existed, and further validation and reliability tests would be warranted before they can be used in similar studies. The design was cross-sectional, which means that differences between T1 and T2 may be an artefact of different samples rather than attributable to time since the RCA course. Response rates were also low, especially at T2, so the representativeness of the samples cannot be confidently ensured, and results are to be interpreted cautiously. However, the study is the first worldwide that we are aware of to test acquired knowledge of RCA among staff who are key to delivery of patient safety improvement and who work in trusts typical of the spectrum of public healthcare.

The study has shown that the course achieved high levels of participant satisfaction, as is often the case for training delivered by respected experts on a novel topic. But it also achieved high levels of objectively assessed factual knowledge. However, rather less than half of participants were able to get the correct answer when asked questions using vignettes that simulate some of the real-world application of knowledge, suggesting that learning would be enhanced by exposure to model anonymised RCAs to build competence, and by specific use of pedagogical approaches such as Problem-Based Learning7 which explicitly build in learning on application into practise. We contend, however, that such approaches would require explicit support and feedback by experts. Such approaches might work better within the context of the organisation in which RCA will be applied, rather than a national scheme.

Most respondents are confident about applying RCA after the course and believe there is a strong social norm favouring RCA locally. Also, attitudes are positive about the purported benefits of RCA, and most intend to conduct RCA. However, it is a concern that only about a quarter at T1 and half at T2 believed they have personal control over the conduct of RCAs, which may reflect the reality of other work priorities in their role. There are many findings here that reflect the likely determinants of whether they will be motivated to conduct RCA continue to improve their practice which are under the control of managers in the trust, rather than the NPSA as a training agent. It must be remembered that conducting RCAs is arduous and often lays staff open to criticism and conflict with powerful others.8 Our results suggest that even enthusiasts for RCA need convincing that organisational learning is achieved from the investment made, emotionally by the staff themselves, and in terms of organisational resources deployed, in order that RCA can become a real force for improving patient safety. Our results are very similar in this respect to recent findings from evaluation of RCA training in New South Wales910 and a small qualitative study of UK participants,11 which also suggests that there is scope for cross-national research on the optimal methods of training and implementing RCA.