Article Text
PDF
Abstract
Objective To describe the washout effect after stopping a prevention checklist for ventilator-associated pneumonia (VAP).
Methods VAP rates were prospectively monitored for special cause variation over 42 months in a paediatric intensive care unit. A VAP prevention bundle was implemented, consisting of head of bed elevation, oral care, suctioning device management, ventilator tubing care, and standard infection control precautions. Key practices of the bundle were implemented with a checklist and subsequently incorporated into the nursing and respiratory care bedside flow sheets to achieve long-term sustainability. Compliance with the VAP bundle was monitored throughout. The timeline for the project was retrospectively categorised into the benchmark phase, the checklist phase (implementation), the checklist washout phase, and the flowsheet phase (cues in the flowsheet).
Results During the checklist phase (12 months), VAP bundle compliance rose from <50% to >75% and the VAP rate fell from 4.2 to 0.7 infections per 1000 ventilator days (p<0.059). Unsolicited qualitative feedback from frontline staff described overburdensome documentation requirements, form fatigue, and checklist burnout. During the checklist washout phase (4 months), VAP rates rose to 4.8 infections per 1000 ventilator days (p<0.042). In the flowsheet phase, the VAP rate dropped to 0.8 infections per 1000 ventilator days (p<0.047).
Conclusions Salient cues to drive provider behaviour towards best practice are helpful to sustain process improvement, and cessation of such cues should be approached warily. Initial education, year-long habit formation, and effective early implementation demonstrated no appreciable effect on the VAP rate during the checklist washout period.
- Paediatric critical care
- ventilator-associated pneumonia
- checklists
- form fatigue
- human factors engineering
- inattentional blindness
- continuous quality improvement
- human factors
- PDSA
- significant event analysis
- statistical process control
Statistics from Altmetric.com
- Paediatric critical care
- ventilator-associated pneumonia
- checklists
- form fatigue
- human factors engineering
- inattentional blindness
- continuous quality improvement
- human factors
- PDSA
- significant event analysis
- statistical process control
Introduction
Many healthcare quality improvement (QI) projects aim for a seamless transition from rapid successful implementation to long-term, low-drag sustainability. However, implementation of a new safety strategy can create novel issues and burdens which need to be addressed if positive change is to be built into standard workflow effectively. While much has been published on effective implementation of care bundles through checklists, and some on long-term efficacy of checklists, little is published about what happens when such safety strategies end or change.
Many national organisations—such as the Institute for Medicine and Institute for Healthcare Improvement—have long advocated designing systems with human capabilities and limitations in mind as a strategy to increase effectiveness and reduce errors. Instead of trying to perfect human behaviour and thinking, focus has shifted to creating layers of protection to detect and prevent opportunities for errors through human factors engineering (HFE), especially in perceptual, cognitive, and physical domains.1
While the application of HFE to healthcare is not a new phenomenon,2 it is steadily garnering broader application.3 The known fallibility of human memory and attention can be anticipated and mitigated—both when executing mundane tasks (such as washing hands) or when striving to comply with detailed evidence-based guidelines (such as care bundles to prevent central line infections). Designing error-free systems requires deep understanding of human characteristics and limitations, and checklists have emerged as a common and effective tool for helping humans perform more reliably in many fields.
The particular context of this analysis is ventilator-associated pneumonia (VAP), a common, costly, and preventable problem associated with significant morbidity and mortality.4–7 In the paediatric intensive care unit (PICU), VAP is the second most common type of hospital-acquired infection (HAI) and has been shown to be mitigated by the implementation of a best practice bundle.8 While our experience redemonstrates the effectiveness of a VAP prevention bundle in a PICU setting, the specific aim of this analysis is to better elucidate what enables and cripples successful transition from implementation to sustainability from a HFE perspective, rather than a clinical one.
In the planned crossover from a VAP prevention checklist to cues in the routine workflow, an unintended 4-month phase without behavioural cues provided an opportunity to observe the post-checklist washout effect, as well as the comparative effectiveness of the two cueing strategies employed—namely checklists versus bedside flowsheets.
Methods
The Institutional Review Board waived oversight for this QI project and analysis.
Setting
Our PICU is a 16-bed medical-surgical unit within a 200-bed academic quaternary-care children's hospital (The University of Michigan's C.S. Mott Children's Hospital PICU). During the study period, the PICU admitted approximately 1000–1200 patients annually, with 800–900 annual ventilator days, and exhibited a predicted risk of mortality ranging from 3.6% to 5.6% (Pediatric Risk of Mortality III model). The PICU provides continuous physician, nursing, and respiratory therapist coverage for all children beyond the newborn age through young adulthood.
Definitions and data collection
All mechanically ventilated patients from January 2007 to July 2010 were included in this analysis. VAP cases were prospectively screened for and adjudicated by the infection control department. Clinical VAP diagnoses by ICU personnel were incorporated into PICU care but were not considered a reliable metric for stable surveillance. Infection control made all VAP determinations based on microbiological data, official radiology reports, and a standardised chart extraction. VAP was defined in accordance with criteria established by the Centers for Disease Control and the National Healthcare Surveillance Network (NHSN). The surveillance processes and personnel were consistent throughout the study period. The VAP rate was expressed as infections per 1000 mechanical ventilator days, and data were prospectively plotted and monitored on control charts. VAP bundle checklist compliance data were collected real time in all patients during the checklist phase (February 2008 to January 2009), and captured retrospectively from periodic samples of bedside flow sheets in the audit period (June 2009 to July 2010).
Process-improvement initiative
A unit-based VAP working group was formed in the fall of 2007 and consisted of a respiratory therapy manager, a clinical nurse specialist, an infection preventionist, and a paediatric critical care physician. This team focused on adaptation and implementation of a proven paediatric-specific VAP eradication bundle.8 The VAP eradication bundles used derived generally from Institution for Healthcare Improvement recommendations, targeted literature review, and expert clinician consensus.8–11 The local adaptation of the VAP bundle is summarised in table 1. The VAP working group undertook staff education/re-education, equipment procurement/modification, and employed the Institute for Healthcare Improvement's model for improvement by use of rapid-cycle small tests of change. A stretch goal of eradicating VAP was set with a minimum goal of reducing VAP rates by 50% within 1 year of adopting the VAP bundle. This VAP initiative was one of numerous parallel initiatives directed at broadly improving quality and safety. Other initiatives included checklists, education, team training, prioritisation of safety, data transparency, and performance visibility. On an annual basis, the Safety Attitudes Questionnaire (SAQ) was administered in the PICU as part of a routine unit assessment. The SAQ, a psychometrically validated instrument, includes 64 questions that capture the attitudes and perspectives of staff on specific safety issues at a granular level.12
Ventilator-associated pneumonia prevention bundle
Project timeline
The study period of 3.5 years was retrospectively divided into four intervals for the purpose of this analysis (table 2). Performance and compliance data were shared back with frontline staff and unit leadership throughout.
Ventilator-associated pneumonia (VAP) project timeline and interval performance
Statistical analysis
Prospective statistical process control was the preferred method for real-time surveillance and actionable insight into the data for the VAP improvement team. Throughout implementation and maintenance of the VAP prevention bundle, rates and bundle compliance were plotted on control charts (u-charts and p-charts). An additional control chart for rare events (g-chart) was constructed to better understand the time at which the long VAP-free intervals became significant. Natural break points in the mean data were identified through special cause variation using the four conventional rules13 as opposed to investigator-defined time periods, although these were annotated on the control charts. For the purposes of this analysis, four investigator-defined time periods were set to correspond to the intervention timeline, and t tests were used to compare these defined intervals to generate the more familiar p values.
Results
In 2007, VAP became a new HAI under surveillance in the PICU, and the baseline rate was 4.2 infections per 1000 ventilator days (figure 1). During the first third of the VAP implementation phase (February 2008 to May 2008), self-reported compliance with all VAP bundle practices ranged from 48% to 59%, but in the latter third of the implementation phase (October 2008 to January 2009), average compliance ranged between 65% and 78%. During the checklist phase, the VAP rate decreased to 0.7 infections per 1000 ventilator days (p<0.059), and demonstrated special cause variation on both the u-chart (figure 1) and g-chart (figure 2). In this timeframe, there were no system-level changes identified through root cause analyses that were believed to influence VAP care, such as staffing ratios, changes in clinical management, de-emphasis of VAP at PICU QI meetings, etc.
Prospective annotated control chart of ventilator-associated pneumonia rates.
Control chart for ventilator days between ventilator-associated pneumonias.
Clinical workload peaks in the winter for the PICU, and frontline staff qualitatively reported an excessive documentation burden in January, 2009. An assessment of contemporaneous QI efforts in the PICU revealed no less than six checklists in concurrent use. Checklists for room readiness, VAP, daily goals, patient-family support, central line insertion and maintenance were either in the implementation or the sustainability phase. These checklists were utilised on a daily to weekly basis with the predominant documentation burden placed on bed-side care providers.
During this study period, the SAQ was administered annually from 2007 to 2009 (n=85–108, response rates 88–95%), and the two major domains pertaining to teamwork and safety climate were derived. Subsequent to 2007, both domains improved: teamwork climate increased from 52.8% to 71.8% agreement (p=0.003), while safety climate increased from 54.6% to 63.4% agreement (p=0.127). In the context of rising bundle compliance, improvement in safety culture, and a sustained VAP rate of zero for 8 months, the decision was made to discontinue mandatory use of the VAP checklist. Flow sheet redesign approval and reprinting resulted in an unanticipated washout phase of 4 months. Optional VAP checklist use in these 4 months dropped off precipitously to nil. This period lacking a reliable mandatory checklist and preceding flowsheet cues provided a serendipitous opportunity to observe the effects of stopping a checklist after effective implementation of a VAP bundle. After the checklist was stopped in February 2009, the VAP rates increased from 0.7 to 4.8 infections per 1000 ventilator days (p<0.042), and the control chart demonstrated special cause variation (figure 1).
Cues for VAP prevention practices were recreated in bedside flowsheets for nurses and respiratory therapists starting in June 2009. Subsequently, VAP rates dropped to 0.8 infections per 1000 ventilator days (p<0.047), and special cause variation was again detected in the control charts (figures 1 and 2).
Discussion
While the efficacy of checklists has been shown repeatedly in clinical QI work, we believe this is the first report of the converse. The data suggest that, despite the presence of strategies to maintain the gains of this QI work and overcome inattentional blindness, cessation of a clinical best-practice checklist is associated with performance degradation. The rebound in VAP rates during the checklist washout phase leads us to believe that the presence of cues to adhere to the VAP bundle practices was the critical common factor in reducing VAP rates in both the checklist and flowsheet phases—consistent with cognitive science observations.14 Re-establishing cues to perform desired practices in the flowsheet phase resulted in VAP rates of <1 per 1000 ventilator days, which was commensurate with the performance during the checklist phase.
Important clinical, pathophysiological, and definitional issues pertaining to VAP have all been well described elsewhere.15–18 Still the successes and lapses in our prevention efforts support much of the work demonstrating the preventability of such nosocomial infections. Yet a further understanding of the HFE principles that impact success and failure may prove vital in developing even more effective and sustainable solutions.
Gawande, Pronovost, and others have written extensively on the helpfulness of checklists in healthcare and other industries.19–21 Still, effective transitions between implementation and long-term sustainability can be threatened by failing to anticipate and mitigate such issues through careful project coordination. The checklist burnout experienced by staff highlights the importance of transitioning from high-intensity implementation efforts to more efficient standard workflow. While staff were enthusiastic from a previous successful checklist initiative to reduce central line infections,22 and were familiar with checklist use from the many QI projects in the PICU employing such tools, the multitude of concurrent checklists at the bedside began to increase mental workload, provoke form fatigue, create data smog, and lead to unclear prioritisation. Instead of waiting for staff overburden as the driver for change, hardwiring the intervention once the g-chart signalled special cause variation may have allowed us to make the smooth transition from implementation to sustained work.
A related example of cognitive overload was found in a study of checklists for patient-controlled analgesia pumps, where initial designs were ineffective in simulated use.23 Only a redesigned checklist with reduced cognitive workload was found to be effective. More recent work evaluating checklist design from an HFE perspective was done to assisting nurses in finding and fixing medication errors.24 Design changes in the checklist resulted in several fold improvement: from 0% to 90% detection rate in simulated applications. Finally, getting clinicians to use checklists at all was found to be very sensitive to HFE design. After redesign of a congestive heart failure standard order sheet (ie, checklist) to reduce cognitive confusion and effort, the usage rate improved from 9% at baseline (95% CI 5% to 17%) to 72% (95% CI 52% to 86%; p<0.001).25
People have generally predictable limitations to perceptual, cognitive, and physical capabilities that contribute to errors in healthcare delivery that result in harm to patients.18 26 This is true not only at the sharp end, when there is a lapse in following best practice at the point of care (eg, VAP bundle non-compliance), but also at the blunt end, when managers lose sensitivity to frontline operations (eg, new project overload in a period of peak seasonal acuity). The high-risk, high-stakes, high-stress, and high-complexity setting of the ICU readily precipitates the phenomenon of tunnelling—the narrowing of human attention.27 Focus and prioritisation can be both adaptive and maladaptive, depending on the context, and tunnelling can allow a care giver to overlook salient cues linked to routine tasks while being focused on those cues perceived as more threatening.28 Additionally, working memory capacity and the length of time information can be held in working memory decreases under stress.29 Thus, the PICU setting combined with the limitations of human attention, memory, and resilience can understandably predispose the frontline staff to form fatigue and non-compliance. The fact that staff recognised and articulated to leadership their experience of checklist burnout suggests self-insight into these phenomena.
Natural limitations to human attention can be further explained through inattentional blindness, which occurs when a person does not perceive salient objects due to a combination of low expectation and high mental workload.18 The very acts of making VAP a rare occurrence and compliance higher during the implementation phase could easily have fostered a change in the perceived state and threat. Not only can deliberate cues, such as a flowsheet documentation field, be readily overlooked in such a context, but blatant cues, like head-of-bed elevation far less than 30°, can easily become invisible to a provider. Elimination or reduction of a threat can paradoxically lead to an undervaluation of preventive strategies that eliminated the threat, and this may well have contributed to the rise in VAP during the checklist washout period.
Notably, we did not observe positive performance carryover effects during the washout period that one might relate to staff education or year-long habit formation because VAP rates rebounded to nearly the same level observed during the benchmark period. However, the educational and habitual influences created during the checklist phase may partly underpin the effectiveness of cues in the flowsheet. Each nurse and respiratory therapist in the PICU is potentially confronted with a documentation burden of between 100 and 200 items per hour in the bedside flowsheets. While relevant ‘boxes’ on bedside flowsheets compete for attention, it is likely that new cues built into the flowsheet do not garner the same attention as standalone checklists. So despite comparable rates, these two cueing strategies are probably not equivalent, but rather affected by the sequence of their implementation.
Our experience with this QI project to eradicate VAP demonstrates that, beyond clinical best practices (eg, VAP prevention bundle), specific implementation tools (eg, checklists) and multiple strategies to maintain performance (eg, education, cultural changes, visual displays of performance, and monthly VAP meetings), other human factor strategies can be exploited in the crusade against preventable healthcare-associated harm. Clearly, designing systems to take into account inherent human limitations in perception and cognition while under duress can enhance improvement and sustainability.
There are a number of limitations to this analysis. Perhaps most significant is the NHSN definition commonly used for VAP surveillance. Sensitivity, specificity, and inter-rater reliability have all been cited as confounders with this metric, and we suspect that our local infection preventionists' high standards for explicitly meeting chest radiograph criteria probably leads to under-detection of VAP at our center.14 However, such confounding issues are most pertinent in the context of inter-individual or inter-institutional comparisons, and are likely blunted in this analysis by having a consistent surveillance process and adjudicator internally. A system-wide factor that has the potential to enable and disable performance is safety culture.12 Measured over the study period, the SAQ demonstrated improvements but in spite of improving safety culture, there was still performance degradation not related to the culture of safety. Another potential confounder is the temporal association with worsening performance in what staff deemed a high acuity winter. There were no objective data indicating increased acuity relative to other winters, and there is no seasonally cyclic pattern in the VAP rate. Further, the highest VAP rates in the washout period were in the spring of 2009 rather than winter. Finally, multiple overlapping interventions were undertaken in a plan-do-study-act manner, without waiting for signal or stabilisation in all measures before further safety strategies or changes were deployed. This somewhat confounds the ability to assign discrete attributability in a simple before versus after analysis. Therefore data are presented in the format used prospectively by the improvement team—statistical process control with annotated control charts.
Summary
A significant VAP rate reduction was achieved in a PICU setting through a proven prevention bundle using a checklist during implementation. During a 4-month checklist washout phase, the VAP rate rebounded to baseline levels, suggesting little to no carryover from education or habit formation during the yearlong checklist implementation. Re-establishing key VAP checklist cues in the bedside flowsheets resulted in low VAP rates comparable to performance during checklist implementation. Beyond clinical and pathophysiological considerations of HAI prevention, attention to HFE is important—not only during implementation, but also to long-term sustainability. Healthcare quality and safety initiatives would benefit from more deliberate efforts to mitigate the perceptual, cognitive, and physical limitations inherent to our species.
References
Footnotes
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.