Article Text
Abstract
Aim Following the introduction of an electronic Incident Information Management System (IIMS) in New South Wales, Australia, the authors investigated enablers and barriers to the use of IIMS and factors associated with increased, static and decreased reporting rates.
Methodology An online and paper-based, anonymous survey of 2185 health practitioners collected information about their reporting behaviour and experiences of enablers/barriers: training, system accessibility, ease of use, system security, feedback, perceived value of IIMS and workplace safety culture.
Findings The 79.3% of respondents who reported on IIMS were distinguished from non-reporters by having undertaken IIMS training and evaluating this highly. Users reporting more incidents post-IIMS were more likely than those with static or decreased reporting rates to evaluate their training highly and to have experienced all enablers. Users reporting fewer incidents were least likely to do so. The relative likelihood of the three reporting groups experiencing various enablers was similar. Those most frequently experienced by all groups were system security and accessibility. Barriers most frequently encountered were more culturally embedded—for example, poor workplace safety culture. The ‘more’ reporting group actually reported most, and the ‘static’ group least, incidents.
Limitations/implications The sample was large but not randomly selected, which limits the generalisability of findings.
Practical implications Interventions to increase reporting should target provision of training that endorses and fosters conditions shown to enhance reporting rates.
Originality Enablers to incident reporting have been shown to be associated not only with reporting per se but also with changes to reporting patterns and rates.
- Adverse incidents
- electronic incident reporting
- culture
- barriers
- enablers
- incident reporting
- adverse event
Statistics from Altmetric.com
- Adverse incidents
- electronic incident reporting
- culture
- barriers
- enablers
- incident reporting
- adverse event
Introduction
Incident reporting is generally considered to be a core initiative in improving patient safety.1 2 Between 2004 and 2006, the New South Wales Health Department (NSW Health) and the Clinical Excellence Commission introduced an electronic Incident Information Management System (IIMS) across the State, based on Runciman's Advanced Incident Management System.3 4 As part of an evaluation of IIMS, we investigated factors associated with health practitioners using IIMS to report an incident and the effect of the implementation of IIMS on their incident reporting rates. What characterised groups whose reporting rates increased, remained static or decreased post-IIMS? We used a multimethod research design. In this study, a large-scale questionnaire survey is reported. We also conducted focus groups of health practitioners and interviews with Directors of Clinical Governance and Quality and Safety Managers in the 11 Area Health Services (AHSs), and with senior staff in NSW Health.5
Demographic variables have been demonstrated to be associated with the likelihood of using incident reporting systems. Braithwaite et al found that nurses, compared with other health professionals, were most likely to report incidents on IIMS, followed by allied health professionals, administrative staff and doctors.6 Lower incident reporting rates of doctors have been observed in other studies.7 8 Health professionals with a managerial component to their role were significantly more likely to report incidents on IIMS than their non-managerial counterparts.9 However, professional and managerial status were not associated with reporting rates following the introduction IIMS.
Enablers of, and barriers to, incident reporting
There has been substantial theoretical and empirical research into factors which either inhibit or enable incident reporting.10 11 Barriers which inhibit reporting can be the flip-side, or absence of, enabling conditions. Seven enablers/barriers were the focus of the present study: training in, and knowledge of, incident reporting;12–15 ease of access to the system;16 ease of use of system;17 belief in the security and confidentiality of the system;18 19 receiving feedback about incident reports;20 21 workplace safety reporting culture—for example, fear of reprisals;18 19–22 and value of or scepticism towards incident reporting.23–25
Little is known about the introduction of training and other enabling factors during the implementation of a new reporting system such as IIMS. Are they associated with an increase in health practitioners' reporting rates? What proportions of health practitioners increase, maintain or decrease their rates postimplementation? Of course, practitioners with any of these reporting patterns may in fact be reporting many or few incidents. We do not know whether groups whose reporting patterns change following the implementation of a reporting system vary in their relative exposure to particular enablers/barriers. That is, are the enablers most and least frequently experienced by groups with different reporting patterns much the same or very different? Such information has important implications when deciding which groups and enablers to target in order to enhance and increase reporting.
Aims and hypotheses
We aimed to investigate these questions. Our first hypothesis was that IIMS training history (undertaking IIMS training, type of training and evaluating the training received as helpful) was associated with using the IIMS system. We predicted that attending courses would be most likely, and learning from a CD or DVD least likely, to be linked to reporting on IIMS as Westbrook et al9 found that health practitioners rated the former most, and the latter least, highly.
The second hypothesis concerned the three reporting patterns of IIMS users; those who increased (the ‘more’ group), maintained (the ‘static’ group) or decreased (the ‘fewer group’) their reporting rates post-IIMS. We predicted that the ‘more’ group would have the greatest experience of the seven enabling factors (IIMS training, accessibility, ease of use, security, feedback, workplace culture supportive of reporting and perception of the value of IIMS) while the ‘fewer’ group would have the least experience of these factors.
We developed two further, exploratory hypotheses which are expressed in null form. The third hypothesis was that there was no difference in the actual incident-reporting rates on IIMS of those who increased, maintained or decreased their reporting rates post-IIMS.
The fourth hypothesis was that there was no relationship between the relative likelihood of the three reporting groups experiencing the enablers and barriers. Were the enablers most and least frequently reported by three groups the same or very different?
Method
Sample and setting
The sample consisted of 2185 health professionals drawn from across the New South Wales (NSW) public health system. Australia's publicly funded health sector has a profile similar to that of other OECD countries such as Britain and Canada. In total, Australian health care consumes about 9.7% of GDP. NSW has the largest population of the States, with 6.8 million inhabitants. Its health system is administered by 11 AHSs. It is estimated that about one in seven members of the workforce of 106 000 received information about the survey in time to participate. The respondents' professional backgrounds were medicine (5.5%), nursing (54.5%), allied health (18.5%) and other (eg, administrative, IT staff). Compared with the Australian health workforce (doctors (13.3%), nurses (54.2%) allied health (10.5%) and other (22.0%)),26 doctors were under-represented and allied health staff over-represented in the survey sample. Just over half the sample (n=1095) had a managerial component in their role. Several questions were directed only at managers because of their involvement in the processing of IIMS reports following submission.
Questionnaire
A questionnaire, which drew from work in similar studies,8 13–20 27–29 was developed to investigate health professionals' attitudes towards and experiences of IIMS. The questions relevant to the present paper asked about respondents' IIMS training, their actual incident reporting rates (see table 2 for items), whether they were reporting more, the same number or fewer incidents since the introduction of IIMS, and their experiences of enablers and barriers to reporting (see table 3). The reliability of the questionnaire was 0.83 as measured by Cronbach α. This measure of internal consistency is based on average correlation between items. A value approaching 1 indicates that the set of items measures a single unidimensional construct.
Procedure
Chief executives of the AHSs were requested to circulate information about the survey during 2006 via their intranet and email systems. Most participants accessed the questionnaire through a purpose-designed website. Questionnaires were completed anonymously, and the response data were accessed electronically by the researchers. Paper copies of the questionnaire were also provided at work sites for staff who preferred not to use the internet or had limited access to it. The amount of publicity given the study, and hence the response rates, differed by AHS.
The IIMS training experiences of participants who had reported incidents on IIMS and their actual reporting rates were compared with those of non-reporters using χ2 analyses. Respondents who had used IIMS were divided into three groups (‘more,’ ‘static’ and ‘fewer’) according to whether they said that they were reporting more, the same number or fewer incidents since the introduction of IIMS. These groups' IIMS training experiences and their responses to the items investigating barriers and enablers were compared using χ2 analyses. To facilitate comprehension of the large number of χ2 results in our tables, the ‘agree strongly’ and ‘agree’ responses and the ‘disagree strongly’ and ‘disagree’ responses were combined in tables 1 and 2. In table 3, the ‘strongly agree’ and ‘agree’ responses were combined and compared with the ‘disagree strongly/disagree/neutral’ answers. Some respondents failed to answer all items, so numbers in analyses vary. Percentages in the tables are based on the number of participants answering the item.
To determine whether there was a relationship between the relative likelihood of the three reporting groups experiencing the six enablers, each group's mean agreement scores for the enabler types (ie, the average percentage of respondents in the group agreeing with a subset of enabler items) were ranked as shown in table 3 (eg, all groups had the highest mean agreement score for ‘Security of IIMS’ items so were each assigned a rank score of 1 for this enabler). The Kendall coefficient of concordance W was calculated between the three groups' rankings to determine whether they were significantly related. Significance level for all analyses was set at p<0.05.
Results
Comparison of reporters and non-reporters on IIMS
The majority of respondents (79.3%) had reported incidents on IIMS. As table 1 shows, IIMS users were significantly more likely to have received training in its use. Among survey participants who had received training, users were more likely than non-users to rate their training highly, while non-users were more likely to give neutral or negative responses. The type of IIMS training received was not related to whether or not participants had reported on IIMS.
Comparisons of IIMS users with increased, static or decreased reporting rates
Of the IIMS users who answered the question on their reporting rates, 22.7% said they were reporting more incidents since the introduction of IIMS, 55.6% had maintained their pre-IIMS rate, and 21.8% were reporting fewer incidents. Table 2 indicates that among IIMS users, neither receiving training nor the type of training received was associated with incident reporting pattern post-IIMS. However, respondents who decreased their reporting rates post-IIMS were significantly more likely to make unfavourable or neutral evaluations of their training. Only 1097 of the participants answered the question about the number of incidents they had reported on IIMS by citing a number. Some of these were not specific—for example, ‘about 100,’ ‘more than 20.’ Ninety respondents wrote descriptions such as ‘numerous’ and ‘too many to count,’ which could not be classified in table 2. Thus, there was a bias among those with lower rates to cite precise numbers. Members of the ‘more’ reporting post-IIMS group were in fact reporting more incidents than other groups. There were higher percentages of ‘more’ group members in the 6–10, 11–20 and 21+ incident categories. The ‘fewer’ group had the second highest reporting rate with less members in the lowest and more in the highest number of incident categories than the ‘static’ group. The ‘static’ group members were reporting fewest incidents.
Table 3 indicates that on all 18 items examining the relationship between reporting rates post-IIMS and six enablers/barriers to incident reporting, there were significant differences in the answers of the three groups with varying reporting patterns. The ‘more’ group were more likely than the other two to assert that the IIMS system was accessible, easy to use and secure, provided feedback and was of value, and that their workplace safety culture favoured incident reporting. The ‘fewer’ group was less likely than the ‘static’ group to agree with 17 of the items. Thus, there was evidence that increased reporting was associated with the presence of enabling factors, and their absence was linked to a decrease in reporting.
Another way of comparing the results in table 2 is in terms of the percentage of each reporting group experiencing the presence of the various enablers. Instances in which over 50% of a group had experienced an enabler are shown in table 3 in bold script. Over half the ‘more’ reporting group agreed with 10 (55.6%) of the 18 items. Over half of the ‘same’ reporting group agreed with six (33.3%) of the items. There were no instances of half the decreased reporting group agreeing with an item.
The Kendall W of 0.835 (χ2=12.52, df 5, p=0.028) calculated from the three reporting groups' rankings of the six types of enablers revealed a significant relationship between the relative enabling experiences of the groups. This indicates that there was a similarity between the rank order of the types of enablers encountered by the three groups. All groups agreed most with items in the areas of security followed by accessibility. Thus, although the ‘more’ group experienced IIMS as most accessible, and the ‘fewer’ group experienced it as least accessible, accessibility was the enabler second most likely to be encountered by all groups The ranking of groups' experiences of the other types of enablers varied somewhat. The ‘more’ and ‘static’ reporting groups agreed least with items about experiencing workplace cultures that encouraged reporting. The ‘fewer’ reporting group was least likely to agree with items asserting the value of IIMS and the ease of incident reporting (see table 3).
Discussion
The first hypothesis was largely supported. Receiving training in IIMS and appraising that training highly were significantly associated with the likelihood of using IIMS, though type of training undertaken was not.
The second hypothesis was largely supported. Those who increased their reporting rates were more likely to rate their IIMS training highly and to have experienced all six other enabling factors. Those who reported fewer incidents after the implementation of IIMS were the group least likely to have experienced these factors. Among users, having received training and type of training were not associated with reporting pattern post-IIMS.
The third, null hypothesis was rejected. Reporting pattern post-IIMS proved to be associated with actual number of incidents reported on IIMS. Participants who said they had increased their reporting post-IIMS did in fact report more incidents than other groups. Interestingly, the ‘fewer’ group that had decreased their reporting post-IIMS were actually reporting more incidents than the ‘static’ group.
The fourth, null hypothesis was rejected. There was a significant relationship between the relative likelihood of members of the three reporting groups experiencing the various enabling factors. The security and accessibility of IIMS were the most frequently reported enablers by all groups. Over half of the members of the ‘more’ and ‘static’ reporting groups agreed that these occurred. The implementation of IIMS has been most successful regarding these two domains. It has been least successful in terms of culturally embedded characteristics which are more difficult to change, such as workplace reporting cultures and the perceived value of the reporting system.
The importance of training as an enabler of incident reporting is apparent. For the total sample training, especially if evaluated highly by participants, predicted the use of IIMS. Among users (the subsample that had reported incidents on IIMS) those who evaluated their training highly were more likely to report more incidents and to increase their reporting rates above their pre-IIMS level. It was apparent from the findings of other studies in our multimethod appraisal5 that many participants wanted further training or ongoing refresher courses.
Our study was retrospective. Further prospective investigation is required to clarify the relationship between training and reporting. The evaluation of training may have been enhanced or decreased over time as a result of successful or difficult experiences using IIMS. There needs to be investigation as to why some participants did not receive training. Was training unavailable? How many actively avoided attendance at training? Another issue that we did not investigate was the length of time between training and the installation of IIMS in the facility. In some instances, lengthy time lapses occurred, making it difficult for practitioners to apply their training.
The finding that the ‘fewer’ group was actually reporting more incidents than the ‘static’ group is seemingly paradoxical. Members of the ‘fewer’ group must have been relatively frequent reporters prior to IIMS, as post-IIMS, they still reported at rates higher than the ‘static’ group that maintained its pre-IIMS rate. The ‘fewer’ group reported being less likely than the other two groups to encounter each of the enabling factors. There is a sense of this group having poor experiences with the IIMS system which have made them less inclined or less able to report incidents. Interviews with senior staff indicated that implementation of IIMS was judged to be more successful in some AHSs than others. Senior staff varied in their enthusiasm for IIMS and in the provision of enablers—for example, feedback.5 There was marked variation within some AHSs, particularly in terms of access to computers and the IIMS programme.5 These may well be among the factors accounting for the negative experiences that led the ‘fewer’ reporting group to report less. The ‘fewer’ group may also contain more staff with poorer computer skills who found the transition from paper to electronic reporting difficult.
Similar proportions of respondents said they had increased or decreased their incident reporting rates, which does not suggest an overall increase in reporting rates. However, there is evidence30 that incident reporting rates have increased significantly since the introduction of IIMS. In our results, some of these increases were obscured, as many respondents who did not cite a number of incidents typically wrote a word indicating a very high rate (eg, ‘numerous’). Additionally, reporting rates of over 20 were grouped together, which particularly affected the increased reporting group.
A limitation of the research was that the generalisability of the findings is reduced by the use of a non-random sample. Health practitioners in some AHSs were less likely to be informed about the survey or were given less time to participate before the survey closed. Most participants were self-selected, so the sample is not representative of the health workforce as witnessed by the low participation rate of doctors and the over-representation of allied health professionals. On the other hand, the survey was completed by 2185 participants.
Conclusion
In conclusion, we have examined reporting patterns following the introduction of an electronic incident reporting system across a health system. This implementation was well supported by providing training programmes and support for users. These data support past findings indentifying common enablers and barriers to reporting. We have shown that the experiences of these enablers are significantly associated with increased, decreased or maintained levels of reporting, and actual reporting rates. Enablers have a stronger mediating effect on incident reporting behaviours than previously recognised.
References
Footnotes
Funding NSW Health Department; Clinical Excellence Commission, NSW, Australia.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.
Linked Articles
- Quality lines