Article Text
PDF
Abstract
Aim To prototype a system for identifying and monitoring those organisational processes that give rise to latent conditions that can contribute to failures in a dispensary environment.
Methods A proactive risk-monitoring system was prototyped during a 9-month period within the dispensary at Hereford Hospital. The system is used to identify empirically a preliminary set of Basic Problem Factors through qualitative analysis of narratives submitted by pharmacy staff about problems they encountered during their daily work. These factors are monitored and rated based on staff perceptions elicited through a questionnaire. At the concept stage, the system idea was discussed at two stakeholder workshops to ensure plausibility. A Plan–Do–Study–Act approach was used to prototype the system and to evaluate the perceived usability and perceived completeness of the system.
Results After four Plan–Do–Study–Act cycles, staff were satisfied with the usability of the questionnaire and the choice of factors being monitored. In total, 11 Basic Problem Factors were identified from the narratives, 10 of which have been monitored over a period of 6 months using a questionnaire. The differences in staff perceptions were statistically not significant. The qualitative and quantitative results led to improvements that included a review of all IT equipment in the department and the clean-up of the work environment.
Conclusion A system for identifying and monitoring organisational processes that give rise to latent conditions that may contribute to failures was prototyped at the dispensary at Hereford Hospital. This contributes to the organisation's efforts towards creating a proactive safety culture.
- Incident reporting
- PDSA
- safety culture
Statistics from Altmetric.com
Introduction
Incident reporting is a key mechanism for organisational learning in the NHS and has received a large amount of attention during the past 10 years.1–3 In addition to many local incident-reporting systems, the National Patient Safety Agency operates a National Reporting and Learning System to identify risks and opportunities to improve patient safety across the NHS. There has been considerable research into barriers to reporting, such as lack of training in the use of incident reporting, usability problems of the systems that have to be used, uncertainty about what constitutes a reportable incident, blame culture and fear of consequences, lack of feedback and the absence of learning relevant to local practices.4–8
In 2008, the Health Foundation commissioned the Safer Clinical Systems (SCS) programme involving four NHS organisations to develop systems approaches to delivering more reliable and safer care. At Hereford Hospital, one of the aims of their SCS work was to prototype and to implement a local Proactive Risk Monitoring System for Organisational Learning (PRIMO) to complement incident reporting. The project aim was the result of a very practical need: very few incident reports were available at the start of SCS, and the learning that could be extracted from these in terms of error-producing conditions and latent factors was minimal. The PRIMO approach is intended to operate alongside incident reporting, but its aim is to elicit a rich contextual picture of the local work environment, to move away from negative and threatening notions of errors and mistakes, and to encourage active participation and ownership with clear feedback and demonstrable learning for local work practices. The system was prototyped and piloted as a service improvement project within the dispensary of the hospital pharmacy, and the paper describes the principles of the system and the results of the pilot.
Intervention overview: PRIMO
The PRIMO system consists of a number of elements: staff narratives, questionnaire and action plan. Staff narratives are used to identify empirically a set of Basic Problem Factors for subsequent monitoring to ensure that these factors are relevant to the local work environment. Monitoring takes place using a questionnaire to elicit perceptions from staff about the amount of hassle that these factors cause to their daily work in order to construct a risk profile over time. Once the risk profile starts to stabilise, high-ranking problem factors can be investigated for subsequent improvement. The action plan details both short-term and longer-term actions as a result of the qualitative analysis of narratives and the questionnaire results. Ongoing submission of staff narratives and their review is used to identify problem factors that should be included in the questionnaire as a result of a changing local context. The flow of the PRIMO system is represented in figure 1.
Proactive Risk Monitoring (PRIMO) overview.
Methods
Setting
Hereford Hospitals NHS Trust is the main provider of acute services across Herefordshire and parts of Wales, and has a capacity of 259 inpatient beds. The pharmacy department employs 50 staff, the majority of whom work in the dispensary on a rotational basis. Eight staff are based permanently in the dispensary.
Intervention (PRIMO) measures
Qualitative analysis of staff narratives
All pharmacy staff who worked regularly in the dispensary (receptionists, pharmacy assistants, pharmacy technicians, pharmacists) were invited to submit a narrative about ‘something that caused you hassle during your work during the previous week.’ Staff were encouraged to use their own language and style. No further guidance or restrictions were provided. Narratives were entered into a database with identifiers removed. The narratives were discussed and annotated (for clarification) collaboratively by a domain expert (CI) and a safety expert (MAS). The narratives were then coded by one author (MAS) with codes generated from the narratives through comparative qualitative analysis. Codes were grouped into higher-level categories (the set of Basic Problem Factors) through theoretical comparison, in particular with Reason's set of General Failure Types (GFT) developed for the Oil & Gas industry; see table 1.9 The final set of codes and categories was reviewed by the domain expert (CI). Staff were encouraged to continue submitting narratives every other month. Submission of narratives is voluntary.
General failure types identified in the oil and gas industry9
Quantitative monitoring of basic problem factors
The set of Basic Problem Factors identified in the narratives was selected for continuous monitoring through a questionnaire to be completed by all pharmacy staff with regular duties in the dispensary on a bi-monthly basis. The questionnaire elicits from staff their perception about the extent to which the Basic Problem Factors caused them problems during their work during the previous week. Staff could rate this on a five-item Likert scale ranging from 1 (not at all) to 5 (to a large extent). For further differentiation, each factor was broken down into constituent dimensions identified from the narratives and the literature; see figure 2 for an example from the questionnaire.
Example (Equipment & IT) from the prototype questionnaire.
Evaluation
Stakeholder workshops to ensure plausibility of the PRIMO concept
The concept was presented to representatives of all four SCS sites (hospitals located in England and Scotland) and subsequently at Hereford to pharmacy staff. Participants were selected using convenience sampling, while ensuring a spread of staff roles. Participants were asked to discuss the feasibility and applicability of the approach as well as the meaningfulness in a healthcare setting of the set GFTs identified by Reason9 (see table 1).
Prototyping of questionnaire
The questionnaire for monitoring the set of Basic Problem Factors was prototyped using a pragmatic Plan–Do–Study–Act (PDSA) approach.10 We started with one volunteer close to the project (pharmacy director) and iteratively increased this to all members of pharmacy staff with regular duties in the dispensary. Participation was voluntary. The PDSA cycles evaluated the time taken to complete the questionnaire (start & end time to be filled in by the participants), the perceived usability of the questionnaire (participants indicated on the questionnaire any questions they were struggling with) and the perceived completeness of the questionnaire (participants indicated on the questionnaire any other problem factors that were relevant to them but not included in the questionnaire).
Results
Intervention (PRIMO) results
Qualitative analysis of the narratives
Sixteen narratives were submitted (by 13 out of 34 members of staff that had regular duties in the dispensary) initially. The narratives varied in length from one paragraph to five pages. The qualitative analysis of the narratives revealed references (ie, coded at least once) to 11 Basic Problem Factors as shown in table 2 along with the corresponding lower-level codes generated from the narratives. Some of the factors identified, such as training, procedures and communication, map directly onto Reason's framework. Issues surrounding staffing levels, demand management and workload, and teamwork and attitudes were felt to be of particular importance in order to reflect the narratives adequately and were included as separate Basic Problem Factors. These changes reflect differences between a dispensary environment and technical environments, such as Oil and Gas, or railways and aviation maintenance.
Set of Basic Problem Factors identified empirically from the narratives
Questionnaire
After three PDSA cycles of distributing the questionnaire during a 6-month period (excluding the first PDSA that was completed with one person) the mean scores shown in table 3 were obtained (1: did not cause any hassle; 5: caused a lot of hassle). The observed differences in results were statistically not significant.
Mean scores for each Basic Problem Factor elicited during the initial 6-month period (scores ranging from 1: no hassle to 5: a lot of hassle)
Each Basic Problem Factor had been broken down into three constituent dimensions. These dimensions were derived from the analysis of the narratives (see lower-level codes, table 2) or from the literature in those cases where less than three dimensions had been identified. The dimensions receiving the highest (worst) scores are shown in table 4.
Basic Problem Factor dimensions scored highest (worst) during the initial 6-month period (scores ranging from 1: no hassle to 5: a lot of hassle)
Application and action plan
What is the hassle in the dispensary? The quantitative results need to be treated with caution as data were collected only over a limited period of time during the pilot study, and the observed differences were statistically not significant. While keeping this limitation in mind, it was decided to select some of those factors and dimensions for investigation and improvement that had received the worst mean scores. These included unavailable equipment and the immediate work environment. Examples of how these problems manifest themselves or are perceived in practice can be extracted directly from the database of narratives (see box 1 for examples relating to unavailable equipment and IT) in order to get an initial understanding of the type of problems that appear to be of relevance.
Examples extracted from staff narratives relating to unavailable equipment and IT
‘A bad day last Thursday, tracker (log of prescriptions arrived, dispensed and completed in pharmacy) consistently going down so couldn't get information for ourselves or ward on the state of TTOs. Extra work trying to look through the different areas the TTO may be. Later on, the Ascribe system was very slow, and even rebooting did not help. I rang [X] […]—whose number has changed—and he said he thought all the IT systems were having problems as he was as well. Low and behold yesterday (Monday) tracker was failing again. No sooner had we reloaded it, it failed again. Monday is one of our worst days, and we also had an audit starting this week with extra information to go on the tracker.’
‘First on my list was to complete the retail sale order that arrived yesterday. After entering all of the data onto the computer, I went to print my work—the printer did not work. I asked dispensary technician if she could look at it for me; she said it was an IT issue. Already 20 min behind at this point, I phoned IT. They said they were busy and would call me back as soon as possible.’
‘I went to put my order into a carrier bag—there wasn't any there. At this point I was feeling increasingly anxious at the amount of time I was spending on this, and concerned that if any further delays happened I would be running late for my ACT slot and cause delays for my colleagues.’
‘A technician rang from the ward; he wanted a couple of green profiles podding down to [ward X]—a 2 s job, well so I thought. When I got there, the draw was bare, someone had obviously used the last one without photocopying anymore. So I had to leave an already-short-staffed dispensary to go to the copier to copy some green profiles, which takes a little longer because you have to swap the paper in the machine, etc.’
‘10:00—Phoned IT helpdesk to chase up a label printer which I had reported as not working the previous day—10 min’
‘What a frustrating day. I was really busy. I had the previous day's data basing to catch up on plus all today's, staff arriving in to book on and off the tracker with the added information required for the audit, and just when you want the computers to be working at their optimum best, the access database system was on a go-slow. After every two or three items, it crashed, and you had to close it down and reopen, which took forever. I rang IT, and they said they were aware of the problem and trying to sort it, but it was a problem throughout the hospital, not just happening to us!!’
The information provided by PRIMO has then been used to trigger improvements on two levels:
Longer-term actions: a deeper understanding of the extent of IT problems was gained through a review of relevant narratives and discussions with staff. This triggered a review of all IT equipment in the dispensary and associated communication channels to the IT department. The IT department also provided feedback and clarification on the procedures for reporting IT problems.
Short-term actions: in order to close the feedback loop8 the information elicited from staff should lead to demonstrable learning and improvement ideally within a shorter time period than is intended with the longer-term actions directed at improving organisational processes. To this end, ‘Easy wins’ were derived each month from the specific hassle or problems reported in the narratives or the questionnaire. Demonstrable progress was made, thus avoiding the feeling of a purely bureaucratic exercise that many staff have with incident reporting.11 Staff are encouraged to take the lead on actions that result from the narratives. For example, in response to reported problems with the work environment, actions led to the provision of a dedicated space in the dispensary for the checking pharmacist and the storage of appropriate forms at the point where they are needed.
Evaluation
Stakeholder workshops
The first workshop was attended by 23 participants from the four SCS sites (including nursing staff, consultants, pharmacists and technicians, improvement leads and managerial staff). The second workshop was attended by eight pharmacy staff from Hereford Hospital (pharmacists, pharmacy technicians and pharmacy assistants). Concerns and frustrations with incident reporting found in the published literature were echoed, and the suggestion of a complementary approach was welcomed. The need for a pilot study set in healthcare was emphasised by participants to allow for both the local context and healthcare-specific features, and to develop an evidence base to justify the selection of problem factors and specific questions in the questionnaire.
Prototyping of questionnaire
The set of Basic Problem Factors identified during the qualitative analysis of staff narratives (see table 2) was selected for inclusion in the questionnaire. These factors were included regardless of the frequency with which they occurred in the narratives to ensure maximum coverage initially. The Basic Problem Factor relating to Management of Change was perceived as not being relevant by participants during the first two PDSA cycles, and it was excluded from subsequent versions of the questionnaire in order to reduce the time taken to complete the questionnaire. This may change in the future as the organisation is undergoing major restructuring.
After four PDSA cycles, the questionnaire produced no further suggestions from staff about the need to improve usability or to include additional factors. The questionnaire now monitors 10 factors with three dimensions each, totalling 30 questions. An example from the questionnaire relating to Equipment and IT is presented in figure 2. At present, the average time taken to fill in the questionnaire is 21 min (range: 5–40 min). The time taken to complete the questionnaire is an important determinant for the success and the uptake of the approach, as staff may find it difficult to free up longer periods of time.
Discussion
The project arose from the desire to utilise proactively, and with as much contextual detail as possible, the knowledge and expertise of front-line staff about organisational deficiencies that may cause latent conditions in the work environment. The aim was also to create a process that is owned by staff rather than being perceived purely as a management tool.
In the elicitation of narratives, staff are deliberately encouraged to write in their own style and language to place less burden on the author of the narrative and to retain as much contextual and emotional/attitudinal information as possible. The focus of the narratives is on hassle rather than incidents or adverse events, since staff may feel uncomfortable to report errors, and to alleviate fears of negative consequences to the reporter.
The questionnaire will build up over time a risk profile that provides an indication as to which organisational factors are perceived to be particularly problematic or deficient in the dispensary and thus merit further investigation and improvement most urgently. Once a factor is selected for investigation in this way, the database of narratives can be queried for all occurrences of this particular problem factor. In this way, the problem factor becomes fully contextualised once again, and this provides a good starting-point for subsequent in-depth analysis and improvement work.
The factors that are included in the questionnaire for monitoring should be reviewed periodically to ensure that they adequately reflect the local work environment. Factors that are perceived consistently as not causing any significant hassle could be considered for removal from the questionnaire. Factors emerging from the ongoing submission of narratives could be included.
While the narratives and the resulting questionnaire have been tailored to the dispensary environment, the overall PRIMO process should be generalisable to other environments. It is a strength of the approach that the factors that are being monitored are identified empirically from the elicitation of narratives by staff and are therefore close to the respective work environment. The use of staff narratives appears to be an effective process for eliciting contextual knowledge of problems that staff encounter, and this approach may merit further investigation in other safety-improvement projects.
Limitations
As this was a pilot study to prototype a novel proactive approach to organisational learning, the results obtained remain preliminary. The observed differences in quantitative results were statistically not significant, and it was not possible to determine whether any improvement action led to measurably different perceptions and associated scores. Once the process has been in use for an extended period of time, and a more stable risk profile has emerged, the feasibility and utility of the process should be evaluated more rigorously.
The choice of the PDSA approach, a pragmatic improvement methodology leading to a series of rapid prototypes, led to a questionnaire that was meaningful and useful as perceived by staff. No further formal evaluation of the extent to which the Basic Problem Factors are independent of one another or the weighting of each constituent dimension was undertaken. Further work should investigate the psychometric properties of the questionnaire in order to assess its reliability and validity.
Of greatest importance to healthcare practitioners is the generalisability and transferability of the approach to other settings. This was not part of the scope of the pilot study and needs to be validated in a larger-scale research project.
Acknowledgments
We would like to thank members of staff at Hereford Hospital who participated in this pilot. We are also grateful to the reviewers of the paper for their constructive comments.
Footnotes
Funding The work was commissioned by the Health Foundation (Registered Charity Number: 286967).
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.