Findings of implementation studies of the WHO Surgical Safety Checklist
| Author/year | Training | Study phases and checklist fidelity | Reasons for success or failure | Opinions, knowledge and behaviour | Health outcomes |
|---|---|---|---|---|---|
| Sewell et al [2011]24 | Checklist forms placed in ORs, compulsory training video detailing correct and incorrect uses of the checklist, emphasis placed on all team members being responsible. Active discouragement of a simple tickbox approach. Checklist training was not associated with reductions in any complications or mortality | Training phase first (unreported duration). Post-training period June–October 2009 (485 operations). Correct checklist use 97%: 2 min. 20% thought it caused an unnecessary time delay | ‘The initial implementation of the checklist was met with resistance by some operating room team members as there was a belief that many of the points were already in practice’ | 77% thought it improved team communication, 68% thought it improved patient safety, 80% would want the checklist used if they were having an operation | Early complications 8.5% before checklist training and 7.6% after. Mortality 1.9% before checklist training and 1.6% after. Lower respiratory tract infections 2.1% before checklist training and 2.5% after. Surgical site infection 4.4% before checklist training and 3.5% after. Unplanned return to OR 1.0% before checklist training and 1.0% after |
| Helmio et al [2011]55 | Training involved a presentation from an outside expert and three 45 min lectures. Specific guidelines were in the OR, and short instructions on the back of the checklist | One-month implementation period in September 2009 (443 operations) | ‘Use of the checklist improved verification of patient identity, but this was still inadequate.’ ‘Our study confirms that the surgical checklist fits well into otolaryngology.’ ‘We recommend the use of this checklist in all operations’ | ‘… overall, the operating room personnel were supportive’. Anaesthesiologists’ knowledge about patients had improved compared with the pre-implementation period. Preoperative check of anaesthesia equipment increased from 71% to 84%. After implementation, staff were more likely to accurately report patient identity, procedure and operative side. After implementation, there was improvement in: knowledge of OR-teams’ names and roles ranged from 81% to 94%. Discussing risks was 38%. Postop instructions recorded 86%. Successful communication 87–96% | NR |
| Conley et al [2011]28 | NR | Duration of rollout: <2 months at three hospitals, >6 months at two hospitals | The key is whether the local champion can ‘persuasively explain why and adaptively show how to use the checklist.’ Implementation was incomplete at three hospitals: One cancelled attempts to implement the checklist due to ‘fear of insurmountable resistance and poor interdisciplinary communication’. Another cancelled attempts because they were unable to move beyond pilot testing. The third had less effective implementation because of a laissez-faire leadership style; no training; staff understood neither why nor how the checklist could be implemented | Interviews conducted, but no quantitative summary of opinions provided. Three hospitals were discussed in detail | NR |
| Bell and Pontin [2010]56, Bell57 | Training provided to prevent ‘teething problems.’ Instead of requiring paperwork, they used in each OR an A3 board (a drawing board about 14×20 inches) that was colour-coded to aid completion. Publicity campaign in both hospitals | Piloted the checklist at one of the two hospitals first | ‘To implement the checklist effectively, it was essential to engage all staff to ensure the theatre team worked together.’ ‘Working with individuals to identify any gaps or issues with implementation.’ Currently it is ‘being used as standard throughout theatres’ | ‘Communication and staff morale have definitely improved since the checklist was implemented’ | NR |
| Sparkes and Rylah [2010]58 | “Extensive educational support and training” | 3-month pilot, during which changes to the checklist were made. After the pilot, and training, the checklist was introduced to all 29 ORs in November 2009 | Even though people agreed with the checklist in theory, it was difficult to change attitudes and behaviours, particularly the senior team. The checklist was required to be signed by team members and ‘This had led to the fear that legal colleagues will apportion blame to those who have signed the checklist when complications occur’ | Before checklist introduction: ‘Although all found the checklist to be useful, many senior clinicians felt that such communication already took place informally, and that more paperwork would not add to safety.’ Audit of 250 cases in February 2010 found that team briefings occurred in 77% and time outs in 86% | NR |
| Royal Bolton [2010]36 | Drop-in educational sessions which involve 120 participants | May and June 2009 were spent getting the word out about plans to start using the checklist. Piloted first for 1 month in two of the Trust's hospitals in 62 operations. September 2009 was the trust-wide launch of the checklist. ‘Every Trust is different but implementing the checklist across the Trust rather than a prolonged pilot period.’ Within the first week 33% of operations employed the checklist. By 1 month it was at 72%. Currently all eight ORs use it | ‘The importance of communicating with and involving people beyond this core group was recognised straight away.’ ‘Essentially it is all about changing the culture, which can be a long process, but it's well worth it’ | ‘The feedback we received from staff was very positive. Most people were keen to introduce the checklist as quickly as possible’ | 1-month pilot identified nine potential incidents that were avoided as a result of the checklist |
| Vats et al [2010]26 | Limited time given to training | Checklist accelerated with use. Large variability in how the checklist was used: sometimes incompletely, hurried, dismissive replies, and without some key participants. Compliance was initially good, then fell when the research team was absent, and so the team had to re-enter ORs to encourage greater use. Compliance ranged from 42% to 80% in the 6-month period | Need a local champion as well as local organisational leadership. Importance of being able to modify to fit local needs, for example, there was no need to check pulse oximetry because it is already always used | Anaesthetists and nurses were ‘largely supportive’. Some surgeons were ‘not very enthusiastic’. Awkward self-introductions, takes time to achieve comfort, steep interpersonal hierarchy, ID the patient BEFORE draping, not after. Complaints about duplication; perhaps a revised checklist could have less duplication | ‘At our hospital, we found no significant change in overall morbidity or mortality, which were already very low, after the introduction of the checklist. However, there was a noticeable improvement in safety processes, such as timely use of prophylactic antibiotics, which rose from 57% to 77% of operations after the checklist was introduced’ |
| Kearns et al [2011]25 | Training, humorous posters provided, and ‘all staff empowered to remind the team to perform the checklist if it was forgotten.’ | Compliance with the preoperative part of the checklist was 61% after 3 months and 80% after 1 year. Compliance with the postoperative part of the checklist was 68% after 3 months and 85% after 1 year | Authors cited four contributors to success: allocation of responsibilities, local champion, sense of ownership by team members, and ongoing staff consultation | Staff attitudes 3 months after checklist introduction: 50% now ‘felt familiar’ with others in the OR; 70% felt communication had improved; 80% felt that in elective cases the checklist was useful; 30% felt that in emergency cases the checklist was inconvenient. Fifty-eight patients were asked whether they noticed the operating team performing a series of checks before the operation, and 75% said they did, and another 19% remembered it after being prompted. Of the combined 94%, they all disagreed with the idea that the checks would make them worried, and 93% said they were reassuring | NR |
| Norton and Rangel [2010]59 | 3×5 foot posters in each OR. Launch involved formal letter to staff, electronic training application, multiple in-service training sessions, and mention in hospital newsletter | December 2008 pilot test in six paediatric surgical services (general, neuro, orthopaedic, otolaryngology, plastic surgery, and urology). February 2009 pilot test on the revised procedures, and more minor edits were made. ‘Go-live’ date 1 April 2009 in all of the hospital's ORs. Surgical chiefs were local champions, and one nurse champion was paired with each surgeon champion. They divided the responsibility for leading the Time Out phase among all team members, and identified key speaking points. Compliance at ORs improved over time during this period from July 2009 to February 2010 | ‘Use of the Paediatric Surgical Safety Checklist encourages multidisciplinary teamwork and has brought increased communication to our ORs and in other areas’ | December 2008 pilot test of 30 procedures had 80–90% compliance, with ‘overwhelmingly positive’ feedback. ‘Team members have expressed satisfaction with the flow and content of the checklist’ | Checklist caught one near miss during sign in (site not marked), several near misses during time out, (antibiotics not given, problems with consent forms, site marking not visible after draping, missing equipment), and sign out (one team realised a patient needed straight catheterisation, and reviewing procedure name helped nurse documentation, one specimen was incorrectly labelled) |
| Styer et al [2011]29 | Slide presentations, educational posters in ORs, one on one sessions, frequent email updates | October 2008, 2-week trial. Day 1: checklist used by 2 surgeons; anaesthesia/nursing teams recruited to participate and provide same day feedback. Day 2: feedback incorporated, used in 4 ORs, with 8 surgeons December 2008: chiefs of nursing, surgery, anaesthesiology and surgical services asked to endorse use as hospital policy February 2008: checklist team established (leaders from surgery, anaesthesia, nursing), project manager, administrative fellow March 2009: staggered 14-week rollout to 44 ORs. Each surgical service allotted 4 dedicated weeks of attention (week 1: communication, week 2: education, week 3: go live, week 4: follow-up). During ‘go live’ period, checklist team observer assigned to each surgery to educate, provide real-time feedback, answer questions | Early endorsement by executive leadership. Each discipline equally involved in leading effort. PDSA cycle method for gradual implementation. Real-time feedback. Each discipline should lead a section of checklist. Provide data (process and outcome measures). Checklist adopted as hospital policy | NR | Allergies: RN added recent new allergy to record Antibiotics: not given (3), wrong antibiotic for procedure (2), surgeon changed mind about giving antibiotic after confirming procedure, antibiotic left in another room DVT: scheduled procedure typically would not have required compression boots, but patient found to have history of DVT Safety precautions: heparin drip had not been discontinued Plan for management of patient: chest radiograph after procedure for unsuccessful central line placement had been forgotten |
| Bittle [2011]60 | Quality division ‘coaches’ educated OR teams about checklist, and benefits | May 2010: ‘coaches’ from quality division assigned to OR to introduce checklist, first to plastics, then other specialties. Team meetings with coach, OR manager, specialty clinical nurse manager, head of surgical department and senior registrars preceded implementation. Feedback regarding checklist procedure obtained at 1 and 3 weeks | NR | Initially ‘staff were anxious and somewhat apprehensive, but it is now an established step in an operation and is carried out with confidence’ | Incorrect surgery site pointed out by patient Reported incidents fell from 12 to 11 compared with reporting period of previous year |
| Yuan et al [2012]14 | Certified registered nurse anaesthetists (CRNAs) were identified as local leaders of surgical teams. CRNAs along with surgeons, OR staff participated in 2-week training of lectures, written materials and direct guidance Large printed poster placed in ORs | Two months prior and after. All patients followed prospectively for outcomes and complications until discharge or 30 days, whichever came first | Reasons for success: checklist implementation catalysed efforts to procure equipment (ie, pulse oximeter) necessary for safety processes Reasons for failure include: lack of consistent access to crucial resources (such as antibiotics, batteries); checklist ‘did little to change the entrenched hierarchy and relationship dynamics of OR staff’; lack of sustained checklist training beyond 2 weeks | ‘… the checklist's focus on continuous improvement helped to foster a shift in mind-set among staff who were “just used to making it to the end of the day” to building a stronger culture of safety’ | Checklist associated with overall improved adherence to ≥4 (out of 6) safety processes, decreased surgical site infections (AOR 0.28, 95% CI 0.15 to 0.54), surgical complications (AOR 0.45, 95% CI 0.26 to 0.78) Stratified analysis revealed improved adherence limited to hospital 1 (AOR 4.06, 95% CI 2.2 to 7.6), decreased surgical site infections, surgical complications limited to hospital 2 No improvement in surgical outcomes |
| Kasatpibal et al [2012]34 | Circulating OR nurse participated in two meetings and 1-day data collection training session | From March 2009 to August 2009, 42.6% of operations selected for inclusion 91% of patients confirmed identity, site, procedure and gave consent. Only 19% of surgical sites marked. Anaesthesia equipment and medication checked in 90% of cases. Pulse oximeter applied in 95% of cases. Allergies, difficulty airway, aspiration risk and risk of >500 mL blood loss assessed in 100% of cases | Compliance with marking of surgical site low because: marking materials unavailable, procedure was emergent, and ‘Thai culture’ in which ‘Thais do not make marks on other people, especially on the head’ Also, ‘some surgeons assumed that wrong-site surgery would not occur because they had not experienced it themselves’ Compliance with hair removal procedures was hampered by lack of familiarity with proper procedure, lack of equipment and requests from surgeons | Surgical teams often did not introduce themselves during time out for cultural reasons. ‘In Thai culture, people usually introduce themselves only when they first meet someone and are shy about publicising their roles’ Compliance with checklist high for life-threatening issues (drug allergies, difficult airways, profuse blood loss) and confirmation of patient's name, incision and procedure. Notably, standards for these measures are already current hospital policy Compliance was low for surgical site marking and appropriate hair removal | NR |
| Bohmer et al [2012]30 | NR | Survey administered before checklist implementation, then 12 weeks after implementation | All participating specialties were involved in formulation of the questionnaire The checklist was modified for ‘local conditions’ based on feedback from staff Checklist introduced by department heads, demonstrating leadership Baseline findings and improvement after introduction of the checklist were presented to staff | OR staff felt that communication culture in OR was improved, and checklist facilitated information about intraoperative complications. The authors observed there was more discussion of critical events between surgeons/anaesthesiologists | NR |
| Fourcade et al [2012]27 | NR Training sessions, written materials and videos available from the French National Authority for Health, but use by participating centres was not reported | 11–29 January 2010. Random sample of 80 records from medical record per centre were analysed Excluded topical anaesthesia, IR, GI endoscopy and CVC placement Subsequent interviews with staff and surgeons via semi-structured interviews and email surveys | Barriers to success: 1. Many elements of checklist already exist so checklist creates duplication 2. Poor communication between surgeon/anaesthetist 3. Completing checklist took too much time, staff did perceive benefit 4. Some items confusing because they did not fit in with customary OR practices (or seemed inappropriately timed) 5. High staff turnover, new staff unfamiliar with checklist. 6. If OR staff not actively engaged during checklist, nurses felt concerned about ‘legal implications of signing the checklist as they might be held accountable for errors' 7. Some felt questions were repetitive, might frighten patients about to undergo anaesthesia 8. In 5 centres, box for checklist could be checked if safety check not performed for time constraints. Some staff worried this would make checklists fail to improve patient safety | Checklist performed in 90.2% of surgeries. However, checklist was completed in only 61% of cases | NR |
| Perez-Guisado et al [2012]62 | NR | January–December 2010 Responsibility for sections of checklist was divided between nurses, anaesthetists and surgeons | Local 10-question checklist already in place, containing 8 items from WHO checklist | Nurses achieved 99% implementation, but surgeons and anaesthetists only completed checklists 79% and 72% of time, respectively | NR |
| van Klei et al [2012]33 | Information provided in regular meetings to OR staff. Posters placed in all ORs and electronic systems | 1 January 2007–30 September 2010 Checklist implemented 1 April 2009 Monthly compliance reports provided to team managers. OR circulating nurses designated in charge of checklist completion | Checklist completion may be necessary for improved health outcomes Checklist may be less likely to be completed in patients undergoing emergency surgery who are at higher risk of mortality. This raises methodological questions of how to adjust for patient severity | Checklist fully completed in 39% of all patients. Median number of items documented was 16 | After implementation, 30-day in-house mortality decreased from 3.13% to 2.85%. Checklist associated with decreased odds of 30-day mortality (AOR 0.85, 95% CI 0.73 to 0.98) Incomplete checklist did not have a significant effect on mortality |
| Takala et al [2011]63 | ‘Brief instructions on the use of the checklist were on the checklist backside. Written guidelines on how to use the checklist were also available. Instructions were given in order to avoid variation in the use of the checklist in different hospitals and operating theatres’ | Study initiated in 2009 Nurses, anaesthetists and surgeons surveyed regarding OR practices Then, the checklist was implemented over 2–4 weeks Finally, survey of OR practices repeated 4–6 weeks after checklist implementation | NR | Nurses, anaesthetists and surgeons reported increased confirmation of patient identity and awareness of names/roles of team members Surgeons reported improvements in discussions of critical events with anaesthesiologist (34.7–46.2%, p<0.001) and gave prescriptions and instructions to post-anaesthesia care unit more often | Implementation led to discovery of systematic error in timing of prophylactic antibiotics administration |
| Truran et al [2011]64 | NR | Checklist introduced April 2009 Study evaluated compliance with NICE venous thromboembolism prophylaxis guidelines for 3-week period prior to checklist implementation, and 6 months afterwards | NR | Non-compliance with guidelines for venous thromboembolism prophylaxis decreased after checklist from 6.9% to 2.1% | NR |
| Vogts et al [2011]32 | NR | November–December 2010 Medical student observed 100 procedures, documented compliance | Authors suggest compliance with ‘sign out’ section is low because the timing is ‘not linked to a specific event in patient management’ and nurses tasked with performing this section have many competing responsibilities at the end of procedure | Compliance with ‘sign in’ and ‘time out’ sections of checklist was high. However, ‘sign out’ was only observed in 2/100 cases | NR |
| Askarian et al [2011]35 | Checklist presented to OR head Educational packages containing checklist and guidelines were distributed to surgeons, assistants, anaesthetists and nurses Checklist presented to OR teams | Included all elective general surgeries 3 months prior to checklist, followed by 3 months after implementation (144 patients) | NR | Obtaining information for time out and sign out sections of checklist improved after checklist implemented | Surgical complications (before discharge) decreased from 22.9% to 10% after checklist implementation Surgical site infections decreased |
| Levy et al [2012]31 | All OR team members except physicians viewed a computer-based training presentation one time Large poster of checklist placed in every OR | Direct observation of randomly selected non-emergent surgeries over 7-week period | Inadequate education during implementation led to confusion regarding practical execution of checklist. (Unclear if physicians received any training) Checklist poster in OR lacked practical instructions for how checklist should be executed, including which team members questions are directed towards Checklist was not adapted for paediatric patients and may have been less relevant | Although electronic medical record reported 100% compliance, only 4/172 cases completed more than 7 out of 13 checkpoints Small post-study survey of OR staff revealed confusion about proper timing of ‘time-out’ and team member responsible for ensuring checklist execution | NR |
| Helmio et al [2012]15 | OR staff heard three informative lectures before participating in WHO pilot study Specific guidelines on use of checklist were available in the OR Brief instructions appeared on the back of the checklist | Checklist implemented in September 2010. All surgeries (7148) between September 2010 and August 2011 included Survey administered October 2011 | Nurses reported ‘some senior otolaryngologists had negative attitudes towards the checklist’ ‘Active leadership, regular audits and feedback are important for successful implementation and maintenance of a checklist’ | Checklist completion rates were: sign in 62.3%, time out 61.1%, sign out 53.6% 76% of OR team agreed checklist improved OR safety, 68% agreed it improved error prevention, 93% would want checklist used during their own surgery Disregard for checklist use was revealed in the open responses: ‘answers are dismissive’, ‘it is noisy and staff is not concentrating on the checks’ … One senior otolaryngologist wrote, ‘Time out has never been performed in my operations’. In addition, there was confusion about who should lead each check section and when to do checks: ‘I have never received the information on how to use the checklist’ Positive comments included ‘the checklist is beneficial’, ‘it should always be used’ and ‘nowadays no operation should be varied out without the checklist’ | NR |
AOR, adjusted odds ratio; NICE, National Institute for Health and Clinical Excellence; NR, not reported; OR, operating room; PDSA, plan–do–study–act.