It has been more than ten years since the Institute of Medicine published the pivotal report, To Err is Human: Building a Safer Health System.1 The patient safety movement has made significant strides, and the development of the World Health Organization’s (WHO) Surgical Safety Checklist (SSC) was the first major international collaboration to improve perioperative patient safety. The SSC was piloted in eight hospitals around the world from October 2007-September 2008.2 The death rate was 1.5% before the checklist was introduced, and the rate declined to 0.8% afterward (P = 0.003). Furthermore, inpatient complications occurred in 11.0% of patients at baseline and in 7.0% of patients after the checklist was introduced (P < 0.001).2 Subsequent studies have supported the findings in this hallmark publication.3-7

The significant improvement in morbidity and mortality was shown in inpatient hospitals. During the past decade, however, there has been a 300% increase in the number of surgical/diagnostic procedures performed in ambulatory surgical centres in the United States. In ambulatory surgical centres, there is a much lower incidence in morbidity and mortality and a different set of complications.8 In a report by Kurrek and Twersky, the authors studied the literature published during 1997-2008 for incidences of mortality and morbidity in out-of-hospital facilities.9 Their findings indicated a mortality rate of 0.00008-0.05% and a complication rate of 0.7-1.3%.

Post-discharge nausea and vomiting (PDNV) is an important contributor to postoperative morbidity in the ambulatory surgical population.10 Similarly, postoperative pain is undermanaged, with as many as 86% of patients experiencing moderate to severe pain.11 The purpose of this study was to examine the effectiveness of the SSC in the context of an ambulatory surgical facility by adding ambulatory care-specific items to the checklist, assessing adherence to the safety checklist items, and examining the impact of the SSC on safety attitudes and important patient outcomes.

Methods

After Research Ethics Board approval, all operating room and postoperative anesthetic care unit personnel in an ambulatory surgical facility were asked to complete a 58-item validated Safety Attitudes Questionnaire (SAQ).12,13 Questions pertaining to communication and collaboration were assessed using a five-point scale with 1 = very low and 5 = very high, and the individual’s “experience” in the work environment was assessed using a five-point scale with 1 = disagree strongly and 5 = agree strongly.

Champions committee and checklist development

A committee was formed comprising representatives of all perioperative disciplines, including anesthesiology, surgery, and nursing. The committee consisted of three nurse administrators, a clinical nurse educator, eight registered nurses (RN), four surgeons (plastics, urology, orthopedics, and gynecology), three staff anesthesiologists, and an anesthesia resident. The mandate of the committee was to determine the specific items to add to the SSC that would reflect patient outcomes in an ambulatory surgical facility and to determine how the checklist would be implemented.

The Canadian version of the SSC (adapted from the World Health Organization’s Surgical Safety Checklist by the Canadian Patient Safety Institute) was downloaded from the Canadian Patient Safety Institute Web site and used as the preliminary template (Appendix A: reproduced with permission) (http://signup.patientsafetyinstitute.ca/English/toolsResources/sssl/Pages/SurgicalSafetyChecklist.aspx) This checklist consists of 59 items. “Items” were defined as any phrase preceded by either a checkbox or a dash. A literature review was then conducted to determine the most important postoperative outcomes in the ambulatory surgical population. Items related to these outcomes that had the potential to improve postoperative morbidity and could be added to the checklist were developed and presented to the committee for discussion. Items were added to the checklist if committee consensus was achieved. Seven additions were made to the WHO SSC. In the BRIEFING section: under “Surgeon(s) review(s)”, i.) intraoperative local anesthetic infiltration was added; under “Anesthesiologist(s) review(s)”, ii.) general anesthesia versus regional anesthesia, iii.) nerve block, and iv.) intraoperative antiemetics (as a single item) were added; and under “Special precautions”, v.) preemptive pain medication was added. In the TIME OUT section, vi.) allergies was added, and in the DEBRIEFING section, vii.) local anesthetic infiltration was added. The added items were specifically identified by using a red, italicized, and underlined font, and they were indicated with an asterisk. As a result, the BRIEFING section consisted of 41 items; the TIME OUT section consisted of ten items, and the DEBRIEFING section consisted of 15 items for a total of 65 items (Appendix B: reproduced with permission). In addition, a 17-item HAND OFF section was added; however, this section was not verbally included in the SSC and compliance data were not collected on this section. Rather, it was included as a guideline that the RN could use in handing over either during the procedure or in the postanesthetic care unit.

Data collection

A dedicated research assistant attended surgical procedures with the principal investigator and completed the SSC data form until the principal investigator was satisfied with the accuracy of data collection. University students (n = 12) were then recruited to participate as observers and data collectors over the course of the one-year study. All students received an introduction and explanation of the study protocol by the principal investigator and research assistant. The research assistant accompanied each student in the operating room until fidelity of data collection was achieved. Students were instructed to leave the answer blank if they weren’t able to determine the answer to a question. Once completed, the research assistant reviewed all data collection forms for missing data. Chart reviews were conducted to fill in missing data points where possible.

Before introduction of the adapted SSC, trained personnel attended a random sample of ambulatory surgical procedures and prospectively collected data. Operating room personnel were not informed as to the specific reason for the observer’s presence in the operating room. All ambulatory surgical procedures were eligible for inclusion in the study. Once data collection was completed on the calculated sample size and following a multidisciplinary education session, the SSC was introduced into all operating rooms and a period of three weeks was given to allow personnel to adapt to its use. A laminated copy of the adapted SSC was posted and available in every operating room, and descriptors of the individual items were located on the reverse side of the laminated SSC. The committee decided that the surgeon would conduct the BRIEFING section, the RN, anesthesiologist, or surgeon would conduct the TIME OUT section, and the circulating RN would conduct the DEBRIEFING section using the laminated SSC as a guide.

After the introductory time period, trained observers again collected the same information as prior to SSC implementation as well as data regarding adherence to the three sections of the SSC (BRIEFING, TIME OUT, and DEBRIEFING) and to the individual items within these sections. Operating room personnel were not aware of the specific data being collected by the observers. The data collector checked off those items in each section that were verbally discussed during each surgical procedure. After all data were collected, perioperative personnel were again asked to complete the SAQ.

The primary outcome was the incidence of pain and nausea/vomiting on postoperative day one, which was obtained by telephone interview. Post-discharge pain was assessed using a scale of 1-10: 1 = no pain and 10 = the worst possible pain. Nausea and vomiting was assessed using a four-point scale: 0 = no nausea/vomiting, 1 = mild, 2 = moderate, 3 = severe nausea/vomiting. Secondary outcomes included the timing of perioperative antibiotic administration (antibiotic administration to incision time), the use of preemptive nonsteroidal anti-inflammatory medications (yes/no), the use of prophylactic antiemetics (yes/no), the use of regional anesthesia (yes/no), the use of regional nerve blocks (yes/no), the length of stay in the surgical facility, and the number of patients requiring admission.

Statistical analyses

Sample size calculation

The standard deviation (SD) of postoperative pain scores following outpatient surgery was 2.4,14 and the incidence of postoperative nausea and vomiting (PONV) ranged from 20-30%.15 Based on this information and to detect a minimum difference of one point in pain score and a 15% incidence rate, the sample size was calculated as 160 per group with an alpha = 0.025 and power = 0.80.

Data analysis

The data were analyzed using descriptive statistics. For continuous variables, the Shapiro-Wilk and Kolmogorov-Smirnov tests were used to test the hypothesis of normal distribution. Non-normally distributed data were reported as median (interquartile range [IQR]) and were evaluated with a Mann-Whitney U test. Categorical variables were presented as count (percentages) and were evaluated using Fisher’s exact test. For primary outcomes, P values < 0.025 were regarded as significant, corresponding to the Bonferroni correction to control the familywise error rate at 0.05 for the two tests performed. The 97.5% confidence intervals (CI) were calculated for the differences in median and percentages. For secondary outcomes, P values < 0.007 were regarded as significant, corresponding to the Bonferroni correction for the seven tests performed. The 99.3% CI of differences were calculated.

SAQ

This is a 58-item questionnaire to assess staff safety attitudes, which includes an assessment of communication and collaboration among staff. A repeated measures analysis of variance was used to assess the results. Statistical significance was defined as P < 0.05.

Analyses were performed using SAS® version 9.2 (SAS Institute, Inc., Cary, NC, USA).

Results

Pre-implementation data collection occurred over a period of seven months and post-implementation data collection occurred over a period of five months, for a total study period of one year.

In the pre-implementation phase, SAQs were distributed to 86 perioperative personnel, 70 of which were returned. Due to the departure of a surgical program, 14 surgeons were not present in the post-implementation phase, which left 56 SAQs for distribution, 50 of which were completed. The mean (SD) value for communication and collaboration was high in both groups (mean 3.99 (0.66) and 4.09 (0.61), respectively; P = 0.36), but there was no improvement after implementation of the SSC. The assessment of the staff’s “experience” in the working environment was also good (mean 3.69 (0.29) and 3.76 (0.35), respectively; P = 0.09), but again showed no statistical difference between time periods. There was no improvement in the six items related to teamwork and safety climate that Haynes identified as most likely to respond to a checklist intervention16 (Table 1).

Table 1 Safety Attitudes Questionnaire

Data on 180 patients were obtained prior to implementation of the SSC, and data on 195 patients were obtained following implementation. There was no statistically significant difference in age, sex, or type of surgical procedure between the patient populations before and after SSC implementation (Table 2). The median difference in pain scores between groups was 0.5 (97.5% CI, 0 to 1; P = 0.13). Since the incidence of nausea/vomiting was low, comparison was made between those experiencing no nausea/vomiting and those experiencing some nausea/vomiting (mild, moderate, or severe). The difference in the incidence rate of nausea/vomiting between the pre- and post-implementation groups was −8.4% (97.5% CI, −17.9 to 1.1; P = 0.06) (Table 3).

Table 2 Demographic data
Table 3 Primary outcomes

Ninety-eight of 180 (54.4%) patients received prophylactic antibiotics pre-implementation, and 105 of 195 (53.8%) patients received antibiotics post-implementation. The antibiotics were administered after the incision to 5 of 98 (5.1%) patients pre-implementation and to 6 of 105 (5.71%) patients post-implementation (P = 0.85). The remainder of the patients in both groups received antibiotics within 60 min before the incision. The median difference in minutes in the timing of antibiotic administration was 1 (99.3% CI, −2 to 4; P = 0.45). Before and after implementation of the SSC, the difference in the use of regional anesthesia was −3.4% (99.3% CI, −9.4 to 2.5; P = 0.11), and the difference in the use of regional blocks was 1.0% (99.3% CI, −11.5 to 13.0; P = 0.89). A higher percentage of patients received PONV prophylaxis in the pre-implementation group than in the post-implementation group (86.7% vs 79.3%, respectively) with a difference of 8.0% (99.3% CI, −3.4 to 19.4; P = 0.05). There was a significant increase in the use of preemptive pain medications after implementation of the SSC (difference −23.3%; 99.3% CI, −35.5 to −1.1) (Table 4).

Table 4 Secondary outcomes

Items on the SSC

Compliance with the three sections on the checklist, BRIEFING, TIME OUT, and DEBRIEFING was 99.49%, 97.95%, and 96.92% respectively. Compliance was defined as a verbal discussion of at least one item in the section. The median [IQR] number of items verbalized was: 19 [15-22] items in the 41-item BRIEFING section, 5 [4-6] items in the ten-item TIME OUT section, and 9 [7-11] items in the 15-item DEBRIEFING section.

In the BRIEFING, TIME OUT, and DEBRIEFING sections, there were 17, two, and two items, respectively, that were verbalized < 33% of times. After consensus of the original champions’ committee members, the SSC used in this study was adapted in the following way: 15 items were deleted from the BRIEFING section; five items were deleted from the TIME OUT section, and one item was deleted from the DEBRIEFING section. This resulted in a final 45-item checklist for use in an ambulatory surgical centre (Figs. 1, 2). Items relating to the prevention of postoperative pain and nausea were highlighted on the final version by using capital letters and underlining (available from the authors upon request).

Fig. 1
figure 1

Checklist generation

Fig. 2
figure 2

Checklist details

Discussion

The purpose of this study was to adapt an already developed checklist to include items relevant to the ambulatory surgical population. Ambulatory surgery has low rates of major morbidity and mortality so outcomes, such as postoperative pain and PONV or post-discharge nausea and vomiting (PDNV), are more relevant in this population.17 There is evidence to suggest that attention to certain measures can significantly decrease the severity of postoperative pain and PONV/PDNV, including such interventions as preemptive pain medications, the use of regional anesthesia or regional nerve blocks, and the use of prophylactic antiemetics, especially in high-risk surgical procedures/patients.15 Despite adding items to the SSC that pertain to these specific interventions, verbalization of these items occurred in < 10% of observed cases. Consequently, it is not surprising that there was no improvement in post-discharge pain and nausea/vomiting on the first postoperative day before and after checklist introduction. We also did not show a difference in the use of regional anesthesia or prophylactic antiemetics. There was a significant increase in the use of preemptive analgesics after SSC introduction. This difference resulted from a change in process whereby orthopedic patients began to receive preoperative nonsteroidal medications on a routine basis during this period of time; this change was unrelated to the introduction of the SSC.

Checklist use has been a mainstay of safety practices in the aviation industry for years. The use of checklists in health care, however, has been much slower to evolve despite evidence that their use improves patient outcomes and/or improves team behaviours and communication.5,18-23 Specifically, use of the WHO SSC has shown a decrease in inpatient morbidity and mortality2,4-7,24 as well as improvement in teamwork skills, communication, and adherence to process measures.1,23,25-27 Nevertheless, Ko et al.’s recent systematic review of checklist use in medicine has shown limited evidence of their effectiveness.28 Analysis of the studies indicated that many were of poor quality with low levels of evidence and a high rate of bias. In summary, the authors determined that it was not feasible to provide an accurate summary of any trends across all studies.

Similar to the study by Vogts and Fourcade, compliance with BRIEFING and TIME OUT was high in our study, but our study also showed a high compliance with the DEBRIEFING section.29,30 The three items most commonly discussed were: Allergies, Site and procedure, and ASA Class (American Society of Anesthesiologists’ class). There is, however, a need for caution when interpreting compliance rates. Compliance is often defined as some discussion having occurred with respect to the three sections of the SSC; however, not all items may have been addressed within each section. Levy et al. recently reported that individual items were either not executed as designed or not executed at all despite the documented 100% completion of the SSC BRIEFING section.31 Therefore, some studies report on “completeness” of the SSC. The evidence suggests that rate of “completeness” of the SSC is significantly lower than the rate of compliance.5,30,32 Our study has shown that 21 items were discussed < 33% of the time.

There are a number of reports of low compliance with healthcare checklists.33,34 In this study, several factors may have contributed to the low compliance within certain areas of the SSC and the failure to use the checklist to improve the primary and secondary outcomes. For a checklist to be effective, users at the “sharp end” must appreciate its value.35 If the users are told simply that the checklist is an institutional requirement and they are not given or are not interested in the rationale for its implementation, adherence to the checklist may be suboptimal. Certainly, in the case of the SSC, its implementation was an institutional requirement and an institution’s adherence to the SSC was posted on a provincial Web site with public access. Although attempts were made to inform all stakeholders of the process and the rationale behind the use of the SSC, the lack of improvement suggests that the SSC was not implemented effectively.

This particular point is reinforced by our results from testing safety attitudes. Safety culture has been recognized as an important strategy in the improvement of the deficits identified in healthcare safety.36 Culture has been identified as a factor affecting absenteeism, employee turnover, length of stay, and readmission rates. The SAQ is a validated tool designed specifically for operating room teams and is an adaptation of the Flight Management Attitudes Questionnaire developed for aviation teams.37 Using six items from the SAQ, Haynes showed an improvement in safety attitudes before and after introduction of the SSC.38 Our study did not show an improvement in safety attitudes using these same six items, which may be accounted for by the higher values seen in our pre-intervention group (Table 1).

The checklist may have been too lengthy for use in an ambulatory surgical facility where turnover pressure is high. Limiting the length of a checklist is considered crucial for its feasibility and usefulness. In the nuclear industry, it has been stated that “…as the list of items grows, there may be a higher probability of overlooking any given item.”39 Had we pilot tested the initial adaptation of the checklist at an early stage, we may have been able to identify the checklist items that were not consistently noted, allowing for an initial modification and obviating the need for a revised checklist at the end of the study.

The Pareto principle may be employed with respect to checklist content. It assumes that 20% of adverse events will account for 80% of injuries, allowing tailoring of a checklist to focus on priority areas. If potential adverse events are not prioritized, healthcare workers may end up addressing 80% of the adverse events that account for only 20% of the injuries.35 In this study, addition of more items to an already lengthy checklist was likely compounding the lack of prioritization of SSC items. Although morbidity from post-discharge pain and nausea/vomiting occurs, major adverse events more likely to profit from a “checklist” discussion would include four items: correct patient, correct operation, correct side, and allergy identification. In fact, despite the length of the SSC used in this study, these four “big ticket” items were identified in > 90% of cases. All operating room staff may have clearly understood the rationale for discussion of these items and were therefore compliant. Other items, however, may not have been perceived as relevant by all users and therefore were not mentioned. Similar to the comments of Vats et al., some items on the SSC were simply not relevant to cases in an ambulatory facility and were therefore ignored.33

Another potential confounding factor in the lack of effective SSC implementation was the fact that the surgeons involved in the study were not all based primarily at our centre. They would therefore have been exposed to a different checklist at another institution and may not have recognized the differences between their home-based hospital SSC and our ambulatory facility SSC. Hierarchy may also have played a role in that some RNs may have been reluctant to “step in” if items were not mentioned by the attending surgeon.

In summary, the SSC was not effectively implemented in our ambulatory facility. A number of potential reasons have been cited. Specifically, items added to the SSC in an attempt to improve the primary outcomes of post-discharge pain and nausea/vomiting were ineffective, suggesting that the SSC may not be the optimal strategy to address these issues. Due to poor compliance with the adapted version (Appendix B) used in this study, a final abbreviated checklist has been developed for use in an ambulatory surgical facility and is available from the authors.