Article Text
PDF
Abstract
Objectives To determine whether the items on the Time Out and the Sign Out of the Surgical Safety Checklist are properly checked by operating room (OR) staff and to explore whether the number of checked items is influenced by the severity of the intervention and the use of the checklist as a memory tool during the Time Out and the Sign Out periods.
Methods From March to July 2010, data were collected during elective surgery at the Geneva University Hospitals, Switzerland. The main outcome was to assess whether each item of the Time Out and the Sign Out checklists have been checked, that is, ‘confirmed’ by at least one member of the team and ‘validated’ by at least one other member of the team. The secondary outcome was the number of validated items during the Time Out and the Sign Out.
Results Time Outs (N=80) and Sign Outs (N=81) were conducted quasi systematically (99%). Items were mostly confirmed during the Time Out (range 100–72%) but less often during the Sign Out (range 86–19%). Validation of the items was far from optimal: only 13% of Time Outs and 3% of Sign Outs were properly checked (all items validated). During the Time Out, the validation process was significantly improved among the highest risk interventions (29% validation vs 15% among interventions at lower risk). During the Sign Out, a similar effect was observed (19% and 8%, respectively). A small but significant benefit was observed when using a printed checklist as a memory tool during the Sign Out, the proportion of interventions with almost all validated items being higher compared with those without the memory tool (20% and 0%, respectively).
Conclusions Training on the proper completion of the checklist must be provided to OR teams. The severity of the interventions influenced the number of items properly checked.
- Checklists
- Audit and Feedback
- Communication
- Quality Improvement
- Teams
Statistics from Altmetric.com
Introduction
In June 2008, WHO introduced the Surgical Safety Checklist (hereafter, the ‘checklist’) through its initiative entitled ‘Safe Surgery Saves Lives’,1 to reduce the number of wrong site errors and the morbidity and mortality due to surgery. Since its introduction, the distribution of the checklist found relative acceptance across various countries2–5 and more than 300 professional agencies and organisations from countries around the world have endorsed the checklist.6 Self-reported compliance rates have been shown to vary between 45% and 96%,7–11 whereas audits have shown compliance rates between 66% and 100%12–18 and observational studies between 80% and 99%.19–21 Operating room (OR) teams have demonstrated favourable attitudes toward the checklist, being perceived as a tool that could improve patient safety,5 ,7 ,17 ,22 in addition to strengthening the communication between caregivers.23 It must be noted, however, that the opinion that the checklist improves the communication between the OR members is mixed.7 ,9–11 ,13 ,15 ,17 ,22 ,23 Non-medical staff tend to be more positive toward the checklist compared with medical staff.24
The efficacy of the checklist25 ,26 can be affected by a number of factors, most notably the implementation process (positive leadership, team training, interdisciplinary communication and debriefing session)13 ,26–28 and communication between the team members. Time Out and Sign Out are two critical periods where communication between members of the OR team is essential. WHO recommends28 ,29 that during these two periods, one team member, usually the scrub nurse, use the checklist to ‘confirm’ each item on the checklist by announcing the item to another team member, usually the surgeon, who ‘validates’ or acknowledges the item (ie, the scrub nurse states the identity of the patient, the surgeon validates the identity of the patient). We know little about how caregivers communicate with others around each item of the Time Out and the Sign Out.30
The objective of our observational study is to determine whether the items of the Time Out and the Sign Out are properly checked (confirmed+validated) by observing communication around each item between the OR team members. In addition, we wanted to explore whether the number of checked items is different according to the use of the checklist as a memory tool or according to the severity of the intervention.
Methods
Study design
Direct observations were conducted during the Time Out and the Sign Out periods during elective interventions. Two observers (SLD, SC) were systematically present during the Time Out and at least one observer during the Sign Out. The Sign In portion of the checklist does not involve communication between OR staff members and therefore was not a focus of this study.
Setting
In 2008, the Medical Directorate of the Geneva University Hospitals—a 2000-bed hospital with 38 ORs—mandated a policy in relation to the prevention of risks with patient's identity and surgical site in the OR. The Anaesthesiology and Surgical Department Heads launched the Surgical Safety Checklist in June 2009 with an awareness campaign consisting of posters and a formal inauguration day.
The content of the Geneva University Hospitals’ checklist was based on the WHO Surgical Safety Checklist and on the anaesthesia and surgical safety checklists already in use. The Sign In, the Time Out and the Sign Out contained 14, 7 and 6 items, respectively (table 1). The Sign In is generally done by the anaesthesia team and conducted by one caregiver (the anaesthesiologist or the nurse anaesthesiologist). The Time Out is usually conducted by the anaesthesiologist, the surgeon operator and the scrub nurse. For the Sign Out, the surgeon operator and the scrub nurse verify sponge and needle counts, specimen labels and the procedure performed. The anaesthesiologist and the surgeon operator discuss and document critical problems (eg, blood loss, equipment failure, etc), the postoperative management plan and the discharge plan for the patient (recovery room or intensive care unit (ICU)).
Elements of the Surgical Safety Checklist, Geneva University Hospitals, Switzerland, 2009
Study periods and data collection
Two periods were considered: the Time Out and the Sign Out. All domains of surgery with elective interventions were included. In the Geneva University Hospitals, OR clinical specialties were Ear, Nose and Throat (ENT) and Neurology, Heart, Thoracic, Visceral and Orthopaedic, Outpatient Surgery and Urology, Gynaecology and Obstetrics, Paediatrics, and Ophthalmology. Emergency ORs were excluded as the checklist was not implemented in this setting at the time of the study. The pretest of the observation form took place in February 2010 and the observation period from March to July 2010.
Observations took place on half days or full days, for a total of 34 observations days. The goal was to observe a diverse range of elective surgeries. Consequently, the observation days were selected to have a roughly equal distribution of the following criteria: weekday sessions and OR clinical specialties.
The planning of observation days was conducted in coordination with the head of each OR. The planned observations were announced to the OR teams in an effort to avoid the elements of surprise and suspicion due to the presence of observers in the OR. When entering the OR, observers presented themselves to the OR team and explained the objective of the study. Immediately after the Time Out (or the Sign Out), the two observers identified observed discrepancies which were then resolved by consensus.
The main outcome was to assess whether an item was properly checked or not, for each item of the checklist. The secondary outcome was the number of checked items during the Time Out and the Sign Out. Following the WHO guidelines,31 an item was defined as ‘checked’ is it has been orally ‘confirmed’ by one member of the team (one gives the content of the item, eg, ‘We are operating on Mr X’) and orally ‘validated’ by at least one other member of the team. An item was ‘unchecked’ if it had not been confirmed at all (the item was omitted), if it had been confirmed only (one person speaking and no one validating) or if one other member of the team non-verbally validated (eg, nodded by agreement).
Individual level observations
For each OR team member, the profession's specialty was noted. Individuals were also observed to determine whether they stopped all other activities or not during the Time Out and the Sign Out periods.
Intervention level observations
The severity risk score of each intervention was extracted from the hospital patient data management system. At the Geneva University Hospitals, each intervention is graded by the anaesthesiologist on a four-level scale of severity (minor I, intermediate II, intermediate-major III, major IV)32 for administrative and safety purposes. For each observed Time Out and Sign Out, we noted whether a printed copy of the checklist was used as a memory tool and whether a team member announced the beginning of the Time Out/Sign Out periods to the rest of the team. Moreover, for each Time Out, we registered the moment it was initiated (before patient was draped by sterile drapes, before incision or after incision) and assessed its duration.
Power and statistical analysis
The study was exploratory and formulated no hypothesis. Therefore, no power calculation was conducted before the data collection. A posteriori, some data were considered independent variables in describing the number of checked items: using the checklist as a memory tool (yes vs no), surgery ranking (minor, intermediary, intermediary-major, major), patient draped or not (Time Out only). These covariates were examined to determine whether or not they had an impact on the number of checked items. As the distribution of the number of checked items did not follow the normality, non-parametric Wilcoxon–Mann–Whitney rank tests were conducted to compare the groups (statistical significance at p<0.05). All covariates were considered separately (univariate analysis). Analyses were carried out using SPSS V.18.
Results
Characteristics of interventions
Observers were present during 80 periods of Time Out and 81 periods of Sign Out. Distribution of the Time Outs and Sign Outs over the OR clinical specialties was roughly the same, except for Ophthalmology (table 2). The severity of the interventions was ranked ‘minor’ among more than half of the interventions (Time Out 58%, Sign Out 61%) and ‘intermediary’ among one out of four (Time Out and Sign Out 25%; see table 2). In 2010, the distribution of minor, intermediary, intermediary-major and major interventions represented 44.8%, 37.4%, 14.7% and 3.1%, respectively of the total number of elective interventions at the Geneva University Hospitals. Consequently, our samples of 80 Time Outs and 81 Sign Outs contained higher proportions of minor interventions (p=0.023 and p=0.005, respectively) and lower proportions of intermediary interventions (p=0.022).
Number of observed Time Outs and observed Sign Outs by operating room clinical specialties and by surgery ranking, March–July 2010, Geneva University Hospitals, Switzerland
Among the 80 periods of Time Out and 81 opportunities of Sign Out, only one Time Out and one Sign Out were not conducted, corresponding to a compliance rate of 99%.
Just before the beginning of the Time Out and the Sign Out, one caregiver generally announced to the OR teams the initiation of the procedure (82% and 66%, respectively). Less than two-thirds of Time Outs and Sign Outs were conducted with reliance on the checklist as a memory tool (63% and 65%, respectively). Six out of 10 (60%) Time Outs were conducted before prepping and drafting the patient; the remainder were undertaken before skin incision.
Time out
The mean duration of the Time Outs was 36.2 s (N=58, SD 20.6 s; range 4–108 s). The surgeon operator was present for 76 Time Outs (96%). The entire team stopped activity during 1 out of 79 Time Outs. The items were confirmed mainly by the surgeon operator and by the surgical nurse. Among the Time Outs conducted with reliance on the checklist as a memory tool (N=50), the surgical nurse held the checklist the majority of the time (80%) and took control if the surgeon operator overlooked the items. When necessary, forgotten items were recalled.
During the two phases of communication—oral confirmation of the item and oral validation by someone else—we noted that among more than half (57%) of the 79 Time Outs, one item or more was not confirmed, that is, was omitted from the communication. The items were confirmed in proportions varying from 100% (‘Patient identity’) to 72% (‘Patient installed’ and ‘Equipment’) for a mean of 84% (table 3). The proportion of confirmed items did not differ according to use of the printed checklist as a memory tool or severity of the intervention, but differed according to timing of the conduct of the Time Out: after the draping of the patient, the proportion of interventions with almost all confirmed items was higher compared with prior to draping (87% and 66%, respectively); see oral confirmation in table 4.
Surgical Safety Checklist items orally ‘confirmed’ by at least one member of the team and orally ‘validated’ by at least one other member of the team, during the Time out and the Sign Out, March–July 2010, Geneva University Hospitals, Switzerland
Influence of the severity grade of the interventions, the use of a printed memory tool and the timing of patient draping on the number of items* orally confirmed and validated during the Time Out period†
Once confirmed, the items must be validated. Among the 79 Time Outs, 13% were completely checked (all seven validated items). The items were validated in proportions varying from 71% (‘Antibiotic prophylaxis’) to 37% (‘Surgical site’), for a mean of 50% (table 3).
The proportion of validated items did not differ according to use of a checklist as a memory tool or timing of the conduct of the Time Out (prior to/after draping of the patient); see oral validation in table 4. However, the number of validated items was different according to the severity of the intervention: when the severity of the intervention was intermediary and above (grades II–IV), the proportion of interventions with at least six items validated was slightly higher compared with interventions of low severity (29% and 15%, respectively).
Sign out
Among half of the Sign Outs, no professionals were stopped. During only one Sign Out, all were stopped. Considering the two phases of communication (confirm, then validate), the items were confirmed in proportions varying from 86% (‘Procedure’) to 19% (‘Recovery room or ICU’)for a mean of 58% (table 3). For more than 9 out of 10 (93%) Sing Outs, the team missed (ie, did not give) at least one item. The proportion of confirmed items did not differ according to use of the printed checklist as a memory tool or severity of the intervention; see oral confirmation in table 5.
Influence of the severity grade of the interventions and the use of a printed memory tool on the number of items* orally confirmed and validated during the Sign Out period†
Among the 80 Sign Outs, 3% were completely checked (all six validated items). The items were validated in proportions varying from 60% (‘Procedure’) to 16% (‘Recovery room or ICU’) for a mean of 41% (table 3).
When using the checklist as a memory tool, a benefit was observed for the Sign Out: the proportion of interventions with at least five validated items was 0% in the group without the checklist as a memory tool and 20% in the group with the checklist; see oral validation in table 5. The number of validated items was also different according to the severity of the intervention: when the severity of the intervention was minor (grade I), the proportion of interventions with at least give validated items was slightly lower compared with interventions of high/intermediary severity (8% and 19%, respectively).
Discussion
The checklist is well implemented in hospitals of many countries3–5 and its implementation is growing, in spite of observed heterogeneity among hospitals.24 ,28 This study showed that OR teams performed the Time Out and the Sign Out quasi systematically, a result already observed elsewhere,19 ,20 ,33 ,34 suggesting that the Time Out and the Sign Out have now entered into the OR routine. We also observed that the mean duration of Time Out completion was 36 s, a relatively short time, which contrasts with the opinion that the checklist costs time.13
In spite of these positive results, the handling of the checklist was more difficult than expected: during the Time Out, three items (‘Critical steps’, ‘Patient installed’ and ‘Equipment’) were not confirmed in almost 3 out of 10 cases. During the Sign Out, the results were even worse because all items (except ‘Procedure’) were not confirmed in at least 3 out of 10 cases. This suggests that many items were not addressed at all. This result is noteworthy considering that the presence of two observers has probably increased the checklist compliance. The use of a memory tool (eg, a printed checklist) could partially reduce this problem. Moreover, the total number of validated items was low for the Time Out and the Sign Out, suggesting insufficient communication between members of the teams: when one person in the team was confirming the items, most of the time others were simply listening without oral validation.
These results are in line with observational studies.30 ,34 ,35 The study by Vogts et al,30 which used a semi-direct observational design (ie, the OR teams were not aware of the presence of the observer), showed that item compliance varied around 69% for the Time Out and 40% for the Sign Out.30 However, Vogts’ definition of compliance (verbal communication of the items by members of the team) did not specify whether the items were validated by another member of the OR team or not. Our study went further by systematically assessing the appropriate team communication (confirm and validate) around each item. Our observations showed that communication remained predominantly inadequate, even though the OR teams were aware of the presence of two observers (which could have improved communication between the members of the OR teams due to Hawthorne effect).
Such a low level of items properly checked increases the risk of failure to detect errors. As an illustration, during the observation phase of this study, two mislabelled specimens were identified through the hospital incident reporting system. After a root-cause analysis, it was determined that the mislabelling errors might have been avoided if the Sign Out had been conducted properly. Moreover, this result may explain why some studies found no (or little) efficacy of the checklist for mortality and morbidity.36–39 Interventions are needed to improve the level of items properly checked by educating staff members on the proper conduct of the safety check (eg, Who does what, when and how? Where is the source information?) and providing thorough team training following the principle of Crew Resource Management.40
In our study, the low rates of checked items could be explained by the limited knowledge of the OR team concerning the adequate use of the checklist. This explanation is credible when one considers the body of studies pointing to the lack of training and insufficient understanding of the items on the checklist in the OR teams.13 ,14 ,17 ,27 ,41 At the Geneva University Hospitals, the OR team members were not provided with a manual or guidance on how to use the checklist at the time of its implementation. A second explanation could be the way the checklist itself is designed.42 ,43 Indeed, several surgical nurses have spontaneously expressed that some items were ambiguous and that there is a need to modify the item wording. Third, the OR is a specific multidisciplinary setting within the hospital. Due to the heterogeneity of professional cultures,44 tensions across professions can arise.45 ,46 As a result, the team communication process during the conduct of the checklist may be impaired. A fourth explanation could rely on misconception barriers: some members of the OR teams may have limited confidence of the checklist's capacity to detect an error17 while others may minimise the checklist in front of the team.47
Special attention should be brought to the Sign Out as the number of checked items was much lower compared with the Time Out. We observed that most of the OR team members were not stopped during the Sign Out period, possibly giving an explanation for the low number of checked items during the Sign Out period. The perception of when to initiate the Sign Out checklist differs between members of the surgical team, leading to conflicting teamwork processes. For example, when surgeon operators complete their surgical intervention they often want to initiate the Sign Out, although the intervention has not been completed according to the designated tasks of the other team members. This leads to potential discrepancies or incomplete safety checks prior to completion of the intervention. In fact, OR staff members have reported this barrier in one qualitative study.13 This could explain the low rate of adherence to the ‘stopping’ of all other activities, which is not limited to our hospital.17
Our study discovered significant differences in the number of checked items with the medical severity of the interventions and the use of the checklist as a memory tool (Sign Out only). During the Time Out and the Sign Out periods, the higher the severity, the higher the number of items validated. We suspect that OR staff may feel more concerned and stressed during intermediary or major interventions compared with minor interventions. As a result, teams may have taken greater care to conduct Time Out and Sign Out during intermediary or major interventions. We also found that the use of a printed copy of the checklist as a memory tool increased the number of checked items for Time Out and Sign Out, despite not being significant for the Time Out, possibly due to lack of statistical power. Whatever, the effect is relatively small. We expected an effect of the memory tool on the oral confirmation of the items but did not find significant associations for the Time Out and the Sign Out. Overall high compliance of the OR team—artificially increased by the presence of two observers—could explain this result, and staff were already used to the checklist and may have known the items by memory as its implementation occurred 1 year before this study. All said and done, these differences must be interpreted carefully as interventions were selected according to a non-probability sampling strategy and the sample size was low. Further studies are needed to confirm, or invalidate, these differences.
Strengths and limitations
This study has a number of limitations. First, as with any direct observational study, OR team members were aware of the presence of external observers. Consequently, observer bias may have influenced checklist utilisation (performing or not performing the checklist, announcing or not announcing the Time Out and the Sign Out periods). However, we think that the impact of observer bias was limited for the communication processes of item checking. An observer bias could have a strong impact on the behaviour of one individual, however we hypothesise that this impact was greatly reduced when observing a team. Indeed, the low proportions of interventions having all items checked (13% for Time Out, 3% for Sign Out) suggest that improvement bias was limited, if not null. Finally, our results were similar to a study using an indirect observational design30 (OR teams were unaware of observers).
Second, the reliability of the observations faced two types of difficulties. The accurate audibility of verbal communications among the OR team may have been impeded by face masks and, sometimes, by background noise in the OR (radio playing music). In addition, observations were sometimes hindered due to the amount of equipment in the OR, which could have interfered with the ability to adequately see each OR team member. In the case of Time Outs, having two observers may have minimised these potential biases.
Conclusion
Successful implementation of the Surgical Safety Checklist in the OR was more complex than expected. Despite performing Time Outs and Sign Outs quasi systematically, OR teams did not check the items properly. Moreover, the number of checked items was influenced by the level of severity of the interventions and the use of the checklist as a memory tool. Our results suggest that implementation of the Surgical Safety Checklist should be accompanied by workshops and training on the safety of care, the function and impact of controls, and the importance of teamwork functioning.
Acknowledgments
We express special thanks to OR staff who gave a positive welcome to the observers during the observation period. We are grateful to the Heads of Departments of Anaesthesiology (Professor François Clergue) and Surgery (Professor Pierre Hoffmeyer) for their cooperation in the conduct of this study. We would also like to thank the OR managers who supported our team throughout the observation period. We appreciated the help of Christophe Boeuf for providing institutional data related to the total number of interventions in the Geneva University Hospitals. We thank Kelly McClary for revising the manuscript and for her useful suggestions and Ida Welle for proofreading.
References
Footnotes
-
Contributors SC, SLD, ACR, MM, EK, AB, ML and PC conceived the study design. SC, ML, EK and PC coordinated the project. SC and SLD carried out data collection. SC analysed the data. All authors reviewed and discussed the results of analyses. SC drafted the paper. All authors critically revised the paper and approved the final version of this paper.
-
Competing interests None.
-
Provenance and peer review Not commissioned; externally peer reviewed.