Concern about patient safety expressed in studies^1–3 and inquiries^4–7 has led to a series of policy responses. Multiple authoritative reports^8,9,10,11 have argued that changes in systems and cultures, particularly organisational and clinical practices, are needed. System-wide safety improvement initiatives have been designed and set in motion. Although they vary in terms of structure and approach in different countries, they largely consist of three strategies: to educate cohorts of policy makers, managers and clinicians in safety improvement; to develop mechanisms for monitoring and managing incidents and adverse events; and to train stakeholders in methods to deal with and learn from the most serious kinds of events and near misses.^12–14

The procedure usually advocated for identifying the underlying reasons for adverse events is root cause analysis (RCA). This process was imported from aviation, and initially applied to healthcare by the US Department of Veterans Affairs.¹⁵ Its National Center for Patient Safety¹⁶ devised a Severity Assessment Code. In Australia in 2002, New South Wales (NSW) Health (the main government agency administering the NSW state health services) adopted the Veterans Affairs model of RCAs and RCA training that has also been embraced in varying degrees by the National Health Services of England, Wales, Northern Ireland and Scotland. In NSW, it is now mandatory for health managers to set up an RCA team to investigate Severity Assessment Code 1 incidents. These are serious events that could recur. Incidents with lower ratings such as near misses are investigated at the manager’s discretion.

Usually an RCA team meets on three occasions. Initially, the team identifies what is known and what is unknown. What is known is documented, and what is unknown is investigated, a flow chart of events is developed, the root causes are determined and recommendations made. The process focuses attention on system problems and avoids personal blame. NSW Health and the Clinical Excellence Commission (the main body responsible for systemic safety and quality issues) developed the 2-day NSW Safety Improvement Program (SIP) based on the 3-day course of the National Center for Patient Safety. The SIP covers seven topics: incident identification; prioritisation of incidents; systematic notification of incidents to those who need to know; investigation using RCA depending on the severity of the incident; analysis and action regarding recommendations; feedback of aggregated data to the system; and open disclosure—that is, appropriate acknowledgement and discussion of adverse events. Most of the course time is devoted to the process of conducting RCAs. This involves equal amounts of presentation of information and group work in which eight people practise skills and enact RCAs with help from the faculty. Between 2002 and 2004, 24 courses, attended by some 2500 senior health practitioners, were conducted across NSW.

The limited evidence available suggests that incident management and error reduction can be facilitated through such system-wide change programmes.^15,16 Little research has been conducted on the outcomes of RCAs for participants and broader systems. The RCA process is complex, and outcomes from adopting this procedure are not fully understood. Iedema et al¹⁷ conducted discourse analysis of RCA teams in action. They concluded that conducting an RCA “is difficult work … the talk vacillates between interpersonal and ideational issues on the one hand, and, when addressing interpersonal issues it swings back and forth between affective and critical talk” (p ). The authors speculate whether the adoption of RCA “will lock the clinical gaze into a micro-sociology of error [and thereby make it difficult] to influence matters superordinate to the specifics of practice and the design of clinical treatments; that is, the over-arching governance and restructuring of hospital care” (p ).

Research questions include which professional groups have been trained, and of those who are trained what characterises those who go on to participate in RCAs? The professional backgrounds of those who attended the SIP training in NSW were broadly representative of the demographic profile of the Australian healthcare workforce, which comprises 25.8% men and 74.2% women, and professionals from medicine (13.3%), nursing (54.2%), allied health (10.5%) and others (22.0%).¹⁸ There was no difference in the likelihood of members of the various clinically based health professions (doctors, nurses and allied health) participating in, or leading RCA teams.¹⁹ No significant differences were found between the experiences of these professions when conducting RCAs, although there were differences in their attitudes towards RCAs and the SIP, with doctors being least favourably inclined and nurses being the most.¹⁹ Are other characteristics beside profession associated with likelihood of participating in RCAs? Research on health services has shown that a range of social variables is associated with embracing new initiatives.^20,21 Other unanswered questions include:

• What types of problems are encountered during the conduct of RCA?

• Do such problems diminish as participants become more experienced in conducting RCA?

• What is participants’ commitment to the RCA process?

• What safety skills do they consider they learn from their safety training and have they applied these in their workplaces?

AIMS AND HYPOTHESES

Our aim was to investigate the characteristics, attitudes and experiences of health professionals who, after attending SIP courses, went on to conduct RCAs in their workplace. We formulated the following hypotheses.

1. The first hypothesis examined whether the SIP initiative had succeeded in involving a demographically representative group of course participants in RCAs. It asserted that the demographic characteristics of SIP attendees who did, or did not, go on to participate in RCAs differed.

2. The second hypothesis claimed that respondents would experience problems in their workplaces when conducting RCAs owing to deficits in organisational and healthcare system supports, as has been found during the introduction of many health initiatives.²²

3. The third hypothesis was that less experienced participants conducting RCAs would report more problems when conducting RCAs than more experienced participants, as practice improves performance.²³

4. The fourth hypothesis asserted that those who performed RCAs would report positive attitudes regarding both the benefits of and the broader safety issues encompassed by SIP. Leape and Berwick²⁴ argued that there was widespread motivation among health practitioners to improve patient safety and to embrace opportunities to do so.

METHODS

Setting

NSW, Australia’s most populous state, has a population of 6.7 million. The Australian health sector consumes about 9.7% of gross domestic product, and its profile is typical of member countries of the Organisation for Economic Co-operation and Development such as Canada and Britain. About two thirds of health expenditure is publicly funded. Administratively, the NSW public health system is divided into 11 area health services.

Questionnaire

A questionnaire was developed²⁵ to examine four domains: (1) satisfaction with the SIP course; (2) skills learnt from the SIP and transferred to the workplace; (3) attitudes regarding the benefits of the SIP and RCAs; and (4) experiences when conducting RCAs.¹⁹ This paper reports responses to sections 2–4 of the questionnaire. The questionnaire items comprised Likert scales, factual queries and open-ended questions requiring text comments. The items are described in the Results section.

Procedure

The questionnaire was administered via a dedicated website. NSW Health provided 1325 email addresses of SIP attendees. Participants received an email inviting them to participate, and were provided with a link to the questionnaire and assured their responses would be de-identified. Emails were dispatched in mid-December 2004, with 21 January 2005 being the cut-off date for inclusion. A factor reducing the rate of return was the absence of staff during the Christmas and summer holidays. An unknown, small proportion of participants who were contacted had email access, but not internet services. The questionnaire was sent by post to the few participants who notified us.

We used χ² analyses and t tests for independent means to compare subgroups in the study. Pearson’s correlation coefficients were calculated to test for associations between variables. Some respondents did not answer all questions, so numbers in analyses vary. Text responses were transcribed and content analyses categories developed from examination of responses received, using the grounded theory procedures suggested by Glaser and Strauss.²⁶ Answers were coded by two raters who were each blinded to the other’s assessment. Discrepancies were discussed and categorisation was determined.

Sample

This survey yielded 463 (34.9% of SIP participants) responses, of which 252 were from health professionals who had been members of RCA teams. Demographic data available from the evaluations that participants completed at the conclusion of each SIP course indicated that these 463 respondents comprised a representative sample of those who attended courses.²⁷ The 252 respondents who had conducted RCAs comprise the sample that is the primary focus of this paper. Of these 252 respondents, 197 were longer-standing graduates of the SIP, having attended one of the 20 courses held between November 2002 and November 2003, 51 had recently attended courses conducted after March 2004 and four respondents did not report the date of their courses. Table 1 shows the demographic characteristics of the sample.

RESULTS

Characteristics of course attendees performing and leading RCAs

The average number of RCAs conducted by the 252 respondents was 3.7 (standard deviation 6.1). Over half (52.8%) of the respondents had led an RCA, whereas 85.7% had been members of, while not leading, a team. Thus 38.5% had experienced both roles.

We compared the characteristics of these 252 SIP attendees who had performed RCAs with those of the 211 attendees who had not. These did not differ between the groups (table 1)—for example, there was no evidence of RCA procedures being dominated by men, more experienced professionals or practitioners less involved in clinical work.

Problems encountered conducting RCAs

Table 2 lists seven barriers to conducting RCAs. The most frequently encountered barrier was lack of time: 75.0% of participants indicated that this always or sometimes occurred. Other barriers included lack of resources (45.0%), unwilling colleagues (44.5%), lack of feedback and data (38.3%), difficulty with teams (34.2%), interprofessional differences (26.9%) and unsupportive management (16.7%).

Respondents were asked, “Following the RCA(s) you were involved in, were your recommendations implemented?” Most (50.6%) of the respondents said “Partly”, 24.7% said “Unsure”, 18.6% said “Yes” and 6.1% replied “No”. When asked, “Do you think a follow-up training session (after you have actually undertaken an RCA) would be beneficial?”, 84.8% said “Yes”, 6.4% said “Uncertain” and 8.8% replied “No”. Those desiring follow-up were asked, “What would be your preferred method of receiving follow-up training?” Most (78.0%) participants wanted face-to-face follow-up, 15.0% preferred email and 7.0% gave other answers such as telephone and printed material.

Problems of experienced and less experienced participants

Respondents who had attended recent or earlier SIP courses were compared. Not surprisingly, those who had attended earlier SIP courses were more likely to have participated in RCAs (61.0%) than those who attended recent courses (39.8%; χ² = 16.56, df = 1, p = 0.000). However, when recent graduates who had performed RCAs were compared with earlier attendees who had done so, we found no significant mean difference in the number of RCAs they had participated in (3.71 v 3.67, p = 0.97), the number of RCAs they had led (1.47 v 1.49, p = 0.97) or the number of RCAs they had participated in but not led (2.24 v 2.18, p = 0.91). By the time of our study, the smaller group of recent graduates who had been involved in RCAs had as much experience as the earlier group, thus obscuring possible practice effects on problems encountered. The relationship of RCA experience with problems experienced was examined by calculating Pearson’s correlation coefficients between the numbers of RCAs conducted by the 252 participants and their rating scores on the seven barriers (table 2). Correlation coefficients, which ranged from −0.086 to 0.058, did not show any significant association between the number of RCAs conducted and problems experienced (p values ranged from 0.165 to 0.928) Thus, over the time span since attending a SIP, we found no evidence that practice in conducting RCAs decreased with respect to the problems experienced.

Attitudes regarding RCAs

Table 3 shows respondents’ attitudes about the conduct and benefits of RCAs. In their evaluation, 72.6% of the respondents either agreed or agreed strongly that, although time consuming, RCAs are a good use of staff resources. The specific benefits of RCAs are listed from the most to the least endorsed items: improving work practices (88.4%), improving patient safety (87.9%), facilitating teamwork (80.3%), improving communication about patient care (79.8%), improving patient outcomes (77.4%) and improving professional standing (53.6%). Improving professional standing elicited most disagreement (11.5%) and uncertainty (34.9%). Fewer respondents (2.0–3.6%) disagreed that RCAs would result in the other five benefits, or expressed uncertainty about these benefits (9.5–19.8%). Regarding the conduct of RCAs, 39.2% of respondents expressed a preference for having only colleagues with a clinical background on teams and 46.0% wanted consumers also to be part of the RCA team.

Attitudes towards the SIP

Table 4 shows assessments of the skills acquired by the participants from the SIP. Most (91.6%) participants agreed or strongly agreed that they felt better trained to deal with issues of patient safety, and 81.2% felt that they were more able to improve work processes in clinical care. Most (72.9%) participants thought that the SIP would build a leadership group for advancing safe healthcare, and 74.6% agreed that the benefits from the SIP justified the investment. Most (96.8%) participants indicated that they had definitely or partly applied knowledge gained from the SIP at work, and 79.6% reported changes to their error-reporting practices. The SIP courses definitely or partly imparted sufficient understanding to 94.0% of respondents to allow them to conduct an RCA, and 92.8% expressed similar sentiments regarding their skills to be involved in or lead an RCA. When respondents were asked whether they would recommend the SIP course to colleagues, 71.7% replied “Definitely”, 25.5% replied “Probably”, 1.2% said “Unsure” and 1.6% said “No”.

Three questions required text comments. Respondents were asked what changes they had made to their work practices since attending the SIP, and 108 commented. Showing greater understanding and awareness of safety issues was mentioned by 37.1% of the respondents, better reporting of adverse incidents by 11.1% and adopting a “no blame approach” by 4.6%. Some respondents (11.1%) indicated that they had attempted to make changes to their work practices but encountered organisational resistance. Others recorded that they had not made changes: 20.4% of respondents said that this was unnecessary as their work safety practices were already good and 6.5% simply wrote “No”. There was a mixture of other answers (9.3%).

Of the 174 who responded to the question “Considering the health system’s investment in the SIP, are the benefits you see worth the investment?”, half wrote that organisational or cultural changes were necessary if the benefits were to be sustained, 33.8% gave unqualified favourable responses, 9.5% considered the investment not worthwhile and 6.8% considered it was too early to tell. The box gives examples of their comments.

Of the 107 respondents who answered the question “Are there any other comments you would like to make about the Safety Improvement Training Program?”, 28.0% noted difficulties of implementing what had been learnt, 25.2% praised the programme, 16.8% wanted to change the way RCAs are implemented, 11.2% said that information about RCAs conducted and their results should be disseminated in the healthcare system (without identifying data), 10.3% said more health practitioners should undertake SIP training, 5.6% wanted hands-on experience of conducting RCAs before (or even during) the course and 2.8% desired follow-up training.

DISCUSSION

Ever since the earliest studies of healthcare error,^28–32 commentators have argued that systems-wide changes are needed to deal with safety and quality concerns. The NSW SIP initiative is one example of how such changes are being pursued.

The results show that for most participants, the educational objectives of the NSW health system’s safety improvement initiatives have been met. Up to 3 years after attending a course, most of these senior healthcare practitioners held very positive attitudes regarding the SIP, and considered that they had acquired and applied a range of safety skills including conducting RCAs.

Assessing the hypotheses

The first hypothesis was not supported. We found no identifiable differences for the six demographic variables between SIP participants who had already performed RCAs and those who had not. Whatever selection processes were involved, representation of SIP graduates in compositions of RCA teams was achieved.

Difficulties in implementation did occur, as witnessed by the barriers encountered when conducting RCAs and answers to the open-ended questions. This provides a measure of support for the second hypothesis. The SIP seems to have been successful in preparing management to support staff conducting RCAs, as only 2% of respondents always encountered unsupportive management. However, unwilling colleagues, difficulty with team members and interprofessional conflict caused greater problems. The significant differences in attitudes and in latent and surface conflict between health professional groups observed in past research^33,34 were in evidence here. However, the major difficulties were the failure of work schedules to provide time for RCA activities and work infrastructure to provide the necessary resources to enable teams to perform their job.

It was difficult to test the third hypothesis to the effect that recent SIP graduates would encounter more problems than earlier graduates, as the smaller percentage of the recent SIP graduates who had conducted RCAs had performed as many RCAs as earlier graduates. However, we found no significant association between the number of RCAs performed and ratings of problems experienced. Thus in the period (up to 3 years since attending a SIP course) covered by the study, no evidence suggested that a practice effect reduced the problems encountered.

The fourth hypothesis was largely supported. Respondents reported skills acquired and applied—for example, error reporting practices reportedly changed in four out of five of the sample participants. More than nine in ten respondents considered that the course had given them skills for leading or participating in RCAs. Their reports of high transfer rates of skills are supported by findings from other research studies of the SIP.³⁵ These include focus groups of course participants, an ethnographic study of RCA teams in operation, interviews with faculty members who presented the courses and the utilisation of the incident-reporting system put in place by NSW Health. Most respondents were keen to recommend the course. Most respondents endorsed many benefits of RCAs and the SIP, and considered the investment worthwhile despite the costs in staff time and resources.

The future

Most respondents expressed the need for follow-up training in RCA. This would seem to reflect the complexity of the process and an ongoing desire to deal with issues that arise with various cases rather than dissatisfaction with the original training. Further difficulties identified involved lack of communication. Respondents (38.3%) cited lack of information and feedback as a problem when conducting RCAs. Almost a quarter were unsure whether recommendations of their RCA teams were implemented. Final comments by 11.2% reiterated the need to disseminate de-identified information generated by RCAs. Some respondents expressed a degree of frustration and cynicism regarding the commitment of NSW Health to the SIP. It is a case not only of safety being progressed but needing to be seen to be progressed.

NSW Health has monitored the implementation of the SIP through a range of research evaluations,³⁵ and action is occurring designed to address many of these problems. The Reportable Incident Review Steering Committee oversees the process of policy development and information feedback. Standardising and improving the quality of RCA reports will facilitate analysis of incidents. The adoption of the Incident Information Management System (based on the Advanced Incident Management System)³⁶ is designed to make the reporting of adverse events easier and improve their analysis, thereby enabling better communication throughout the system.

Although the available evidence indicates that the sample was representative of the senior healthcare practitioners who attended SIP courses,²⁵ we do not know what self-selection may have occurred regarding attendance at these courses. There were anecdotes of doctors “getting out” of attending the course.³⁵ This would be in keeping with the findings that doctors have less positive attitudes towards the SIP than nurses,¹⁹ and are less likely to report incidents.^37,38 However, almost no differences regarding responses by profession were found in the barriers encountered when conducting RCAs or in the rates of implementation of RCA recommendations.¹⁹

One important finding was that 46.0% of respondents wanted consumers to be part of investigative RCA teams (a preference stronger than the desire to restrict membership of RCA teams to colleagues with a clinical background (39.2%)). This strength of belief may be considered to be relatively high. However, compared with other western nations, Australia is egalitarian in its organisational practices,³⁹ as measured by the low power distance (the degree to which members of society think power should be concentrated at upper levels of organisations).

The implications for health policy and practice are that skills can be taught and motivation aroused for improving healthcare safety as shown in the predominantly positive responses to the survey. However, if the enactment of such skills and their concomitant procedures is hindered by lack of workplace and system support, discouragement occurs. When the wider system fails to reinforce safety behaviours by neglecting to implement RCA recommendations, or by not communicating recommendations to team members or the wider healthcare community, doubt can arise that any more than superficial system changes are occurring.

CONCLUSION

To date, the NSW Health SIP has been largely successful in terms of its endorsement by experienced health professionals who participated in it and developed expertise in conducting RCA. Most believe that there are tangible benefits of conducting RCAs, but implementation has not been without challenges. They are keen for the ongoing success and enhancement of the improvements in safety that they have already observed, but many feel that further action and cultural change within the healthcare system is necessary for the momentum to be maintained.