Article Text

Dissemination of a simulation-based mastery learning intervention reduces central line-associated bloodstream infections
1. Jeffrey H Barsuk1,
2. Elaine R Cohen1,
3. Steven Potts2,
4. Hany Demo2,
5. Shanu Gupta2,
6. Joe Feinglass1,
7. William C McGaghie3,4,
8. Diane B Wayne1
1. 1Department of Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
2. 2Department of Medicine, Mercy Hospital and Medical Center, Chicago, Illinois, USA
3. 3Center for Education in Medicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA
4. 4Institute for Medical Education, Loyola University Chicago Stritch School of Medicine, Maywood, Illinois, USA
1. Correspondence to Dr Jeffrey H Barsuk, Division of Hospital Medicine, Northwestern University Feinberg School of Medicine, 211 E. Ontario, Suite 717, Chicago, IL 60611, USA; jbarsuk{at}nmh.org

## Abstract

Background Approximately 41 000 central line-associated bloodstream infections (CLABSI) occur annually in US hospitals. We previously developed a simulation-based mastery learning (SBML) curriculum in central venous catheter (CVC) insertion that significantly reduced CLABSI rates. In this study, we evaluated the effect of dissemination of the SBML curriculum on trainee skills and CLABSI rates at a community hospital.

Methods The authors performed a cohort study of residents who rotated in the intensive care unit (ICU) at Mercy Hospital and Medical Center from September 2010 to May 2012. Residents underwent an SBML CVC insertion curriculum and were required to meet or exceed a minimum passing score on a simulated internal jugular (IJ) and subclavian (SC) CVC insertion before ICU patient care. Infection control personnel measured CLABSI rates in the ICU before and after the educational intervention.

Results Residents scored a mean IJ pretest of 35.5% (10.29/29, SD=8.30) compared with a post-test mean of 93.0% (26.96/29, SD=1.50; p<0.001). Their mean SC pretest score was 23.0% (6.68/29; SD=9.58) and increased to 96.1% (27.88/29, SD=1.41) at post-test (p<0.001). Patients experienced 3.82 infections per 1000 catheter-days (20 infections in 5235 catheter-days) in the ICU in the 23 months before the educational intervention. During the 21 months after the intervention, there were 1.29 infections per 1000 catheter-days (six infections in 4670 catheter-days (p=0.019)). The incidence rate ratio derived from the Poisson regression was 0.26 (95% CI 0.09 to 0.74) after controlling for Acute Physiology and Chronic Health Evaluation III score indicating that there was a 74% reduction in the incidence of CLABSI in the medical ICU after the intervention.

Conclusions This study demonstrates successful dissemination and implementation of a CVC SBML curriculum and shows that rigorous medical education is a powerful quality improvement tool.

View Full Text

## Statistics from Altmetric.com

In US hospitals, patients experience 30.1 million central line days annually resulting in 41 000 central line-associated bloodstream infections (CLABSI).1 Hospitals actively try to reduce CLABSI rates because of their significant morbidity and mortality. Additionally, the average cost of $16 550 per CLABSI is not reimbursed by Medicare.1 Hospitals use several interventions to reduce CLABSIs including the central line bundle (hand washing, chlorhexidine scrub, full sterile barrier, avoidance of the femoral site, daily assessment of necessity, removing unnecessary lines), checklists, antibiotic/antiseptic impregnated catheters, chlorhexidine dressings and baths, and ultrasound guided insertion.2 ,3 From 2001 to 2009, these interventions reduced CLABSI rates by 58%, saving an estimated 6000 lives and$414 million in potential excess healthcare costs. Yet, further reductions are needed to meet the US Department of Health and Human Services’ national goal for an additional 50% reduction in CLABSI by 2013.1

Health provider education has historically been considered a relatively weak intervention that is unlikely to change behaviour or improve healthcare quality.4–6 However, earlier translational science research demonstrates that rigorous simulation-based education is a powerful tool that improves patient care quality7–10 and is an effective and recommended patient safety strategy.11 Simulation allows trainees to practice complex skills such as central venous catheter (CVC) insertion in a safe environment without harm to patients.12 Northwestern University investigators previously developed a simulation-based mastery learning (SBML) curriculum to train residents in CVC insertion skills.13 SBML is a stringent form of competency-based education featuring deliberate practice, instructor feedback and rigorous assessment measures. In this model, all trainees must meet or exceed a minimum passing score (MPS) on a clinical skills examination before performing the procedure on patients.14 At Northwestern, use of the CVC SBML curriculum improved trainee skills,13 reduced mechanical complications during actual CVC insertions13 and reduced CLABSI rates by over 85% in the medical intensive care unit (MICU).8

In May 2010, the Chicago Tribune newspaper published two articles describing the burden of CLABSIs in Illinois hospitals.15 ,16 Mercy Hospital and Medical Center (MHMC), an academic community hospital in Chicago, was highlighted as a hospital with higher than expected infection rates.15 Northwestern Memorial Hospital was profiled as an institution with lower than expected rates.16 The institutions collaborated to determine if previously published principles to disseminate innovations in healthcare17 could be applied to a simulation-based educational intervention. Therefore, the aim of this study was to evaluate the effect of the Northwestern simulation curriculum on trainee skills and CLABSI rates at MHMC.

## Methods

We performed a cohort study of internal medicine (IM) and emergency medicine (EM) residents at MHMC. Starting in August 2010, all IM and EM residents underwent CVC SBML training the month before they were scheduled to rotate through the MICU. We evaluated residents’ simulated skill performance before and after training. Subsequently, we evaluated the effects of the SBML training programme on CLABSI rates in the MICU. We compared the preintervention period from October 2008 to August 2010 with the postintervention period from September 2010 to May 2012. The preintervention period started in October 2008 because that is when MHMC began reporting CLABSI rates using National Healthcare Safety Network (NHSN) criteria. The medicine chief residents directed the CVC SBML training and the 21-month postintervention period represented two academic years of chief residents. All residents rotating in the MHMC MICU were eligible to participate in the study and provided written informed consent. The Northwestern University and MHMC Institutional Review Boards approved this study.

### Dissemination and implementation

Two authors (JHB, DBW) spent three half-days at MHMC. The first half-day involved a tour of the hospital, learning about the structure of the MICU and discussing possible barriers to implementation of the curriculum. The other two half-days were spent setting up equipment and pilot testing to ensure compliance with components of the SBML curriculum including videos, lectures, feedback strategies and assessment forms. Outcome data from the Northwestern SBML CVC intervention were used to achieve ‘buy-in’ from hospital leadership who already acknowledged the need to reduce CLABSI rates.8 ,13 ,18

IM and EM residents staff the MHMC MICU and insert almost all CVCs. Surgical residents also care for a small number of patients in this unit. Based on residency procedure logs, almost all procedures are supervised by second- or third-year residents and involvement of faculty and fellows is rare. The structure and expectations for supervision of CVC insertions did not change during the study period. There was no formal method to assess resident CVC insertion competency at MHMC before the SBML intervention. Traditionally, residents learned CVC insertion during lectures or vicariously through observation of other residents inserting CVCs on patients in the MICU. The MICU was the only critical care rotation for MHMC residents.

Mercy faculty instructors and raters were two chief residents (HD, SG) who completed the SBML CVC insertion course and subsequently observed several training sessions at Northwestern. We used chief residents as programme faculty because they interact with residents daily and are directly responsible for a large component of resident education as part of their routine duties. Programme faculty were observed and given feedback as they scored pilot CVC examinations to ensure they were using the CVC checklist according to protocol. Each faculty instructor received at least 10 h of instruction, observation and supervision from volunteer Northwestern faculty before participating in the study. Northwestern faculty time provided at least monthly review of the training progress at MHMC through meetings, phone calls and email communication.

### Curriculum and evaluation

Details of the CVC SBML curriculum are described elsewhere.13 In brief, Mercy IM and EM residents took a baseline test (pretest) on internal jugular (IJ) and subclavian (SC) CVC insertion using Simulab's CentralLineMan and a previously developed CVC skills checklist.13 Subsequently, residents completed education sessions including a 1 h recorded lecture and video on CVC indications, contraindications, complications, ultrasound use, a step-by-step demonstration of IJ and SC CVC insertion technique, and guidelines for CLABSI reduction including the central line bundle. The remaining 3 h of the curriculum featured CVC simulator training with an ultrasound and the opportunity for deliberate practice with focused feedback.19 After training, residents underwent a post-test using the CVC skills checklist and were expected to meet or exceed the MPS of 79.1% previously set by an expert panel.13 ,20 Residents who did not achieve the MPS underwent more deliberate practice until the MPS was reached. Additional training and testing occurred immediately or was scheduled by the resident and instructor within 1–3 days. Participants reported their age, gender, IM or EM specialty, year of training, medical school, scores on the United States Medical Licensing Examination (USMLE) Steps 1 and 2, and number of CVCs inserted in actual clinical care prior to the intervention. We compared pretest checklist skill performance with post-test skill performance to assess the impact of the educational intervention.

CVC insertion supplies were provided by the hospital. Mercy purchased a central line simulator and replacement skins. The costs (including each resident's gown, bouffant caps, masks, sterile gloves and dressings) approximated \$2000 annually.

### Clinical outcomes

MHMC initiated chlorhexidine dressings and baths in May 2010, 4 months before our SBML intervention. There were no other hospital-wide quality improvement efforts to reduce CLABSI rates during the entire study period. The monthly number of CLABSI per 1000 catheter-days was measured routinely by MHMC infection control personnel in accordance with protocols described by the NHSN.21 These personnel, in addition to the SBML-trained residents, were unaware of the nature of this study and the timing of the simulation-based intervention. We used the Acute Physiology and Chronic Health Evaluation (APACHE) III mortality score to compare the severity of illness of all MICU patients during the study period.22 Mean monthly Apache scores were used to determine whether changes in severity of illness affected CLABSI rates.

### Statistical analysis

We used the related samples Wilcoxon signed rank test to compare differences between pretest and post-test CVC skill scores because data were skewed. We compared mean APACHE III scores before and after the intervention using an independent t test. We compared monthly CLABSI rates for the 23 months before the intervention to the 21 months after the intervention using Poisson regression.8 ,23 The number of catheter-days was included as the exposure variable and mean monthly APACHE scores were added as a covariate. We also used statistical process control (P-chart) to graphically evaluate changes in CLABSI rates per month before and after the SBML intervention. We performed statistical analyses using IBM SPSS V.21 (Chicago, Illinois, USA) and Stata V.12 (College Station, Texas, USA).

## Results

All 51 IM and EM residents who were eligible for the study provided informed consent and completed IJ pretest and post-test, while 50 residents completed the SC pretest and post-test. Demographic data can be found in the table 1.

Table 1

Demographic data

Nine residents achieved a score of 0 on the IJ pretest, while 31 residents scored 0 on the SC pretest. These residents stated they did not know how to perform the procedure and did not attempt it. The mean IJ pretest score was 35.5% (10.29/29, SD=8.30) checklist items correct while the mean SC pretest score was 23.0% (6.68/29; SD=9.58). Only 3/51 (5.9%) residents achieved skill mastery at IJ pretest, while four of 50 (8.0%) met skill mastery at SC pretest. The mean post-test IJ score was 93.0% (27.0/29, SD=1.50) and the mean SC score was 96.1% (27.9/29, SD=1.41), both p<0.001 compared with pretests. In all, 49 of 51 (96.1%) residents achieved IJ skill mastery and 47/50 (94%) achieved SC skill mastery within the standard 4-h curriculum. All other residents subsequently reached the MPS within 1 h of additional deliberate practice. A graphic portrait of residents’ pretest and post-test CVC clinical skills examination performance is shown in figure 1.

Figure 1

Variation in skills performance (checklist) of internal medicine and emergency medicine residents during internal jugular and subclavian central venous catheter insertion at pretest and post-test (p<0.001). Each circle represents an individual resident's performance. Minimum passing score (MPS)=79.1.

The mean APACHE III score during the 23-month preintervention period was 74.6 (SD=4.07) while the score was 69.1 (SD=3.75) during the 21 months after the intervention (p<0.0001). There were 3.82 infections per 1000 catheter-days (20 infections in 5235 catheter-days) in the MICU during the 23 months before the educational intervention and 1.29 infections per 1000 catheter-days (six infections in 4670 catheter-days) during the 21 months after the intervention (p=0.02). The incidence rate ratio (IRR) derived from the Poisson regression was 0.26 (95% CI 0.09 to 0.74) after controlling for APACHE III score indicating that there was a 74% reduction in the incidence of CLABSI in the MICU after the intervention.

The original research study ended in May 2012 when 21 months of follow-up data were obtained. In December 2012, Northwestern investigators discovered that SBML training had stopped during the changeover to a new (untrained) chief resident in June 2012. In a post hoc analysis that goes beyond the a priori evaluation criteria we established for our primary training implementation evaluation, we found that CLABSI rates rose during the period when MICU residents were untrained (July–December, 2012). Based on this finding, MHMC residency leadership reinstituted CVC insertion SBML for residents entering the MICU in January 2013. From July to December 2012 (6 months), there were three infections in 1010 line days (2.97 CLABSI rate). However, from January to May 2013 (5 months), infection rates remained at 0 infections in 902 line days.

Figure 2 shows a P-chart of CLABSI rates from October 2008 through May 2013. The short time period when training stopped (6 months) was not long enough to provide a statistically significant comparison. The IRR for training was not significant (p=0.21) when the initial 22 postintervention months were compared with the 6-month training hiatus period. Chlorhexidine dressings and baths were used in the MICU throughout the entire intervention and follow-up period.

Figure 2

P-chart (control chart) showing monthly central line-associated bloodstream infection rates during time periods where simulator-trained residents were present and absent from the intensive care unit. Dotted blue vertical lines=time transitions. Red lines=upper control limits (3 SDs). Green horizontal lines=means. Upper control limits vary because there were a different number of line days each month.

## Discussion

This study shows that replication of a CVC SBML curriculum from an academic medical centre to a nearby community hospital improved resident skill and patient outcomes. Mercy residents performed similar to Northwestern residents on a CVC insertion clinical skills examination, including the number of residents meeting the MPS at initial post-test and the time required to remediate residents who did not initially meet the MPS.13 The finding that only a small fraction (less than 10%) of residents met the MPS at pretest reinforces the limitations of traditional medical education on clinical skill acquisition.

SBML is a powerful tool to boost resident skill in diverse areas such as advanced cardiac life support,24 laparoscopic surgery,25–27 thoracentesis,28 paracentesis29 and lumbar puncture.30 ,31 Earlier research also demonstrates improved patient outcomes after SBML.8 ,13 ,26 ,31 The successful dissemination and implementation of the Northwestern CVC SBML curriculum at MHMC show that hospitals can use educational interventions developed at other institutions to improve patient care. Our results also advance what is known about the mastery model by documenting similar improvements in education and clinical outcomes at a second institution.

MHMC also began using chlorhexidine sponge dressings and baths 4 months before SBML began. Although chlorhexidine use may reduce CLABSI rates,3 we do not believe this was solely responsible for the decline because three central line infections occurred in the 4 months after chlorhexidine was started. After the simulation-based intervention, there were only six infections in the entire 21-month period. Additionally, in post hoc evaluation when we queried MHMC about the SBML training process, we discovered that SBML training stopped for residents entering the MICU in July 2012. Subsequently, CLABSI rates went back up, only to come down again when SBML was reinstituted. Chlorhexidine dressings and baths were used throughout this period. Although additional forces such as an improved safety culture contributed to declining CLABSI rates,32 these collateral effect data may suggest the SBML training intervention was the primary driving factor.

Our training protocol emphasised several techniques shown to reduce bloodstream infections related to CVC insertion and maintenance. These included training on the central line bundle, types of dressings, ultrasound technique, timing of line removal, avoiding changing over a guidewire and selecting the appropriate number of lumens.2 ,3 Although the intervention was highly successful overall, it is unknown which components are primarily responsible for our findings. We believe that provider education was the most likely cause for the sustained improvement in patient care shown in this study. This is consistent with earlier research showing the importance of nurse and physician education in the highly regarded study by Pronovost et al33 that reduced CLABSI rates in Michigan intensive care units (ICUs).34 A recent study by Dixon-Woods et al unpacked the Michigan programme and described components, including education, which made it successful. Each hospital involved in the ICU quality programme participated in the required educational activities including ‘immersion coaching and weekly teleconferences for six to eight weeks with hundreds of callers from the participating hospitals.’34 Our findings reinforce a growing awareness that education, combined with quality improvement methods, is needed to reach national CLABSI quality goals.35

Complex service interventions link service delivery to expected outcomes.7 These processes are rarely straightforward, and implementation is often affected by multiple social systems. Quality improvement tools, such as the central line bundle checklist, may not reach target goals if used without sufficient attention to context and culture.35 Due to the complex nature of our simulation-based curriculum, we did not believe that simply sharing our curriculum and assessment tool with Mercy would be effective. As shown in earlier research, successful interventions in one hospital do not necessarily work in other hospitals.36 We designed our intervention based on other quality improvement programmes that used external change agents performing site visits to ensure successful implementation.5 Examples include projects to reduce catheter-associated urinary tract infection and venous thromboembolism, and improve glycaemic control and care transitions.37–39

The field of implementation science explains why some interventions are successful while others fail (diffusion of innovation).17 ,40 Damschroder et al41 describe four activities that are present in organisational change models: planning, engaging, executing and evaluating. Planning was performed through site visits to both institutions. A need for change was established from the stakeholders’ interest in reducing CLABSI rates (leadership, physician, resident buy-in). The CVC SBML curriculum had already been developed, tested, and shown to improve resident skill and patient care. Rigorous monitoring and evaluation methods were employed, and team members piloted the intervention before the study began. Engagement was established by involving the appropriate individuals needed to implement change (thought leaders and champions), that is, the Mercy IM residency programme director (SP, who also serves as chair of the hospital's quality improvement committee) and the chief medical residents (HD, SG), to ensure all residents completed the curriculum. Northwestern curriculum experts (JHB, DBW) served as external change agents. Execution of the SBML curricula was performed using previously recorded videos and lectures, data collection forms, and training the trainer methods provided free of charge. Chief resident training time was part of their routine activity, an empty office was used for training space and an existing ultrasound device was used for training. Evaluation of the programme was ongoing through pretesting and post-testing of residents as well as periodic review of CLABSI rates at MHMC.

This study has several limitations. First, although successful, we have only shown dissemination and implementation of the SBML curriculum to one institution. Second, other hospital-wide interventions were attempted to reduce CLABSI rates at MHMC before our intervention. This is a known weakness of pre–post studies and may confound our results.42 However, based on CLABSI rates still present after these interventions, we believe it is less likely that the large and sustained effect on CLABSI rates at Mercy was related to the earlier interventions. Additionally, infection rates went back up when training stopped for a short time, and went back down again when training was reinitiated. We also cannot rule out the presence of unknown confounding factors in this pre–post study. For example, we did not measure CVC bundle compliance. However, bundle usage was standard practice throughout Mercy during the study period. Third, all CLABSIs in the ICU during the study period were included in our analysis. Although trained IM and EM residents performed the vast majority of CVC insertions, we did not train surgical residents who cared for a small number of patients in the ICU. We believe that if surgical residents were also trained, CLABSI rates would have been even lower. Finally, differences in patient characteristics before and after the intervention may have accounted for different risk profiles for CLABSI during the study period. Lower APACHE III scores after the intervention were potentially important because higher APACHE scores have been linked to higher CLABSI rates.22 ,43 ,44 We accounted for this finding by controlling for APACHE III scores in the Poisson regression model.

In conclusion, this translational science study demonstrates successful dissemination and implementation of a CVC SBML curriculum developed at a university residency programme to a community hospital programme. The intervention significantly improved resident CVC insertion skills and reduced CLABSI rates. The Northwestern–Mercy collaboration illustrates how clinicians and healthcare systems can use evidence-based training methods to improve clinical practice and patient care.

## Acknowledgments

We acknowledge Drs. Douglas Vaughan and Mark Williams for their support and encouragement of this work. We thank Jordan Scher for her assistance with data entry. We also thank the Mercy internal medicine and emergency medicine residents for their dedication to education and patient care.

View Abstract

## Footnotes

• Contributors JHB, ERC, DBW and JF contributed to the design of the work, analysis, interpretation of data, drafting the work, revising it critically for important intellectual content, and final approval of the version to be published. SP, HD, SG and WCM contributed to the acquisition of data, interpretation of data, revising the manuscript critically for important intellectual content and final approval of the version to be published. All authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

• Funding WCM's contribution was supported in part by the Jacob R Suker, MD, professorship in medical education and by grant UL 1 RR 025741 from the National Center for Research Resources, National Institutes of Health (NIH). The funding source had no role in the preparation, review or approval of the manuscript.

• Competing interests None.

• Ethics approval The Northwestern University and Mercy Hospital and Medical Center Institutional Review Boards approved this study.

• Provenance and peer review Not commissioned; externally peer reviewed.

## Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.