Article Text
PDF
Abstract
Introduction More than half of in-hospital adverse events can be attributed to a surgical discipline. Checklists can effectively decrease errors and adverse events. However, the mechanisms by which checklists lead to increased safety are unclear. This study aimed to assess the number, nature and timing of incidents intercepted by use of the Surgical Patient Safety System (SURPASS) checklist, a patient-specific multidisciplinary checklist that covers the entire surgical patient pathway.
Methods The checklist was implemented in two academic hospitals and four teaching hospitals in the Netherlands. Users of the checklist had three options for each item that was checked: ‘not applicable’, ‘yes’ and ‘intercepted by checklist’. In each hospital, the first 1000 completed checklists were entered into an online central database.
Results In six participating hospitals, 6313 checklists were collected. One or more incidents were intercepted in 2562 checklists (40.6%). In total, 6312 incidents were intercepted. After correction for the number of items and the extent of adherence in each part of the checklist, the number of intercepted incidents was highest in the preoperative and postoperative stages.
Conclusions The checklist intercepts many potentially harmful incidents across all stages of the surgical patient pathway. The majority of incidents were intercepted in the preoperative and postoperative stages of the pathway. The degree to which these incidents would have been intercepted by a single checklist in the operating room only, compared with a checklist for the entire surgical pathway, remains a subject for future study.
- Checklist
- surgery
- patient safety
- incidents
- human error
- adverse events
- epidemiology and detection
Statistics from Altmetric.com
Introduction
More than half of in-hospital adverse events can be attributed to a surgical discipline.1 One of the interventions that have been proposed to improve surgical patient safety is the introduction of checklists into surgical practice.2–5 In 2009, a WHO working party showed that the use of a concise checklist in the operating room was associated with a significant decrease in postoperative complications and mortality rates in a selection of hospitals in high-income and low-income countries.3
Claims review studies and observational studies suggest that the majority of surgical errors occur in the preoperative and postoperative stages, rather than in the operating room.6–8 In addition, several studies show that many adverse events originate in the postoperative phase.9–11 This awareness has previously led to the development of a multidisciplinary checklist that covers the entire surgical patient pathway: the Surgical Patient Safety System (SURPASS) checklist.6 A recent publication showed that the implementation of this comprehensive checklist in high-income hospitals was associated with a significant reduction in postoperative complications and in-hospital mortality.2
Although the effectiveness of surgical safety checklists has been demonstrated in the studies mentioned above, the mechanism behind this effect has not been elucidated. The authors of the WHO study stated that ‘the exact mechanism of improvement (was) less clear and most likely multifactorial’.3 In the SURPASS study, a number of mechanisms were proposed, but the contributions of these mechanisms could not be assessed.2 In addition, the contribution of the preoperative, perioperative and postoperative parts of the checklist was not studied. This information might be helpful to elucidate whether the extra effort required to implement a checklist covering the entire surgical pathway is worthwhile in terms of more intercepted events. The aim of this study was to describe the number and nature of incidents intercepted by the checklist and the distribution of intercepted incidents over the different stages of the surgical pathway.
Methods
The study was reviewed by the institutional review board of the Academic Medical Center. Formal review and informed consent were not required.
Checklist
The SURPASS checklist consists of several parts for each stage of the surgical patient pathway (preoperative ward, operating room, recovery or intensive care unit, postoperative ward) and is multidisciplinary: parts of the checklist are completed by all disciplines that are involved in caring for the surgical patient. The checklist was developed based on all available literature and validated by observation of the surgical pathway.2 6 The preoperative part includes a section for the operating assistant, surgeon, anaesthesiologist, ward doctor and ward nurse; the perioperative part consists of a time out procedure; the postoperative part includes a postoperative section for the surgeon and anaesthesiologist, a section for the anaesthesiologist at transfer from recovery to ward, and a discharge section for the ward doctor and ward nurse.
The checklist is added to the patient's file on admission. As the checklists were collected during an implementation period, stopping rules (if the corresponding parts of the checklist have not been completed and signed, the patient is not allowed to continue to the next stage of the surgical pathway) were not applied stringently.
Implementation and data collection
The checklist was implemented in two academic hospitals and four regional teaching hospitals. We have previously reported on patient outcomes before and after implementation of SURPASS in these hospitals.2 In the present study, the data from the first 1000 completed checklists per hospital are reported independently from clinical outcomes. These checklists were completed and collected during the interval between pre-intervention and post-intervention measurement described in the study previously mentioned.2
Three of the hospitals implemented the checklist in the department of general surgery only; in the other three hospitals, the checklist was implemented across all surgical disciplines. Each separate section of the checklist was to be used as a ‘do-verify’ checklist; that is, the checklist was to be used and completed after all actions for the patient had been performed. Users of the checklist had three options for each item that was checked: ‘not applicable’, ‘yes’ and ‘intercepted by checklist’. This last option was to be marked when the action corresponding to this item had not been performed at the moment the checklist was completed and thus might have been forgotten had the checklist not been used.
All checklists were collected after discharge of the patient. The first 1000 completed checklists per hospital were entered into an online central database.
Analysis
Per checklist, the percentage of completion (checklist adherence) was calculated by dividing the number of completed items by the total number of items on the checklist. The number of items marked ‘intercepted by checklist’ was counted. Per item, the percentage of intercepted incidents was calculated by dividing the number of checklists for which ‘intercepted by checklist’ was marked, by the number of checklists for which ‘yes’ or ‘not applicable’ was marked. We chose to include the checklists for which ‘not applicable’ was checked in this analysis because we aimed to describe the intercepted incidents as a percentage of the number of times the checklist was completed. When the item was not completed checklists were not included in this calculation. All incidents were then described according to the stage of the surgical pathway in which they were intercepted. Because the number of perioperative items on the checklist is considerably lower than the number of preoperative and postoperative items, the number of intercepted incidents per stage of the pathway was corrected for the number of items on each part of the checklist. Differences in the rates of intercepted incidents per 1000 fully completed checklists during the preoperative, perioperative and postoperative phases were assessed based on their respective CIs. It was assumed that the rates were Poisson distributed; CIs were calculated following Rothman and Greenland.12 The unit of the denominator was expressed in fully completed checklists to account for differences in checklist adherence during the preoperative, perioperative and postoperative phases as measured by the average percentage of completed checklist items per checklist (82.5%, 82.9% and 56.1%, respectively). In addition, we assessed whether the number of intercepted incidents changed during the course of the study. All completed checklists were divided into quintiles according to the chronology of the study period per hospital. The number of intercepted incidents was compared between successive quintiles of completed checklists.
All analyses were completed using Statistical Package for the Social Sciences (SPSS) V.15.0.
Results
In total, 6313 checklists were collected, divided equally among the six participating hospitals (table 1). The majority of procedures were in general surgery (61.7%) and were performed under general anaesthesia.
Checklist characteristics (N=6313)
Per checklist, the mean percentage of items that had been completed was 72.2% (table 2). The preoperative (82.5%) and perioperative (82.9%) parts of SURPASS were completed more often than the postoperative part (56.1%).
Intercepted incidents per part of the Surgical Patient Safety System (SURPASS) checklist (N=6313 checklists) (see appendix for detailed results)
One or more incidents were intercepted in 2562 checklists (40.6%) (figure 1). In checklists with intercepted incidents, the median number of incidents was one, with a range of 1–36. Numerous incidents were intercepted by each item on the checklist; the percentage of intercepted incidents per item ranged from 0.04% to 7.4% (online appendix). An extra ‘double check’ signature item by the nurse (intercepted in 17.4%) was not included in this range as compliance was exceptionally low for this item. Incidents that were intercepted most often included the preoperative absence of instruments (4.0%), preoperative omissions in medication prescriptions (4.0%), failure by the anaesthesiologist to visit the patient preoperatively (4.1%), no preoperative removal of dental prosthesis (4.8%), lack of postoperative instructions concerning ventilation by the anaesthesiologist (7.4%) and missing medication prescriptions at discharge (6.1%).
Number of intercepted incidents by the Surgical Patient Safety System (SURPASS) checklist (N=6313 checklists).
In total, 6312 incidents were intercepted, of which 54.8% occurred preoperatively, 14.2% perioperatively and 31.0% postoperatively. The numbers of intercepted incidents during the three phases differed significantly with 17.02 (95% CI 16.47 to 17.60), 10.71 (95% CI 10.03 to 11.44) and 15.79 (95% CI 15.10 to 16.50) per 1000 fully completed checklists, respectively. In most patients with intercepted incidents, the incidents were intercepted in only one stage of the surgical pathway (figure 2). There was only a small proportion of patients (4.8%) for whom incidents were intercepted in the preoperative and the perioperative stage.
Proportion of checklists with at least one intercepted incident depicted by stage and combinations of stages (N=2562).
After an initial decrease in the first part of the study period, the number of intercepted incidents remained high during the course of the study (figure 3). The percentage of checklist completion per stage also remained constant.
Total number of intercepted incidents per quintile of checklists over time. *Total number of intercepted incidents per quintile. †Percentage of checklist completions per stage of pathway. ‡Results of 5874 of 6313 checklists are depicted here because no operation date was specified in 439 checklists.
Discussion
This study was conducted to describe the number and nature of incidents intercepted by the SURPASS checklist and the contribution of the preoperative, perioperative and postoperative parts of the checklist. The checklist intercepted one or more incidents in approximately 40% of surgical patients. Most incidents were intercepted in the preoperative and postoperative stages of the surgical patient pathway, despite the fact that adherence was far from ideal in the postoperative period. After correction for the number of completed items and the extent of adherence in each part of the checklist, the number of intercepted incidents was highest in the preoperative and postoperative stages.
There was only a small minority of patients in whom incidents were intercepted in both the preoperative and perioperative stages. Most likely, the extensive preoperative checks included in the checklist led to fewer incidents that needed to be intercepted during the time-out procedure. In accordance with James Reason's Swiss cheese model (the more layers of safety, the smaller the chance of a trajectory of error) it seems best not to perform all safety checks in one stage but to spread them out.13 By performing the necessary safety checks early in the surgical pathway, incidents are intercepted and can be corrected at an earlier stage. A checklist for the operating room only would not have intercepted these incidents until inside the operating room. While some errors may still be corrected at this late stage, this can also lead to postponements or compromised safety.14 To answer the question of whether the earlier detection of incidents translates to a larger improvement in patient outcomes, one would need to directly compare the effects of a comprehensive checklist and a checklist in the operating room. A randomised controlled trial comparing the two would be an interesting subject for further study.
In addition to providing detailed information about the number and nature of intercepted incidents by SURPASS, the registration of incidents per item enables further refinement of the checklist. Items that did not lead to intercepted incidents in over 6000 checklists would be redundant. However, every item intercepted at least one incident. Whether the number of intercepted incidents was enough to justify the inclusion of the item needs to be determined per item and depends mostly on the potential severity of the intercepted incident. For example, 0.4% is clearly enough to justify the presence of the item ‘correct patient’, but some may argue that 0.1% is too little to justify the presence of ‘pathology results discussed’.
The fact that the number of intercepted incidents remained consistently high as the study progressed indicates an ongoing effect of the checklist. Data collection started when the checklist was first introduced and ended when there was considerable experience in using it. If the effect was mainly due to the optimisation of processes triggered by checklist implementation, the number of intercepted incidents would probably have decreased once the checklist had been in use for a longer period. In contrast, even at the end of the study period, multiple incidents were still being intercepted at all stages of the surgical pathway.
Several surgical safety checklists have been described before. Generally, effectiveness of these checklists has been tested by comparing complication and mortality rates before and after implementation of the checklist,2 3 observing surgical processes and teamwork4 15 and interviewing users.16 In this study, the effect of a safety checklist was evaluated by self-registration of intercepted incidents. The finding that the majority of intercepted incidents occurred in the preoperative and postoperative stages of the surgical pathway, rather than perioperatively, is consistent with several previous publications. Both observational studies and reviews of malpractice claims found that over half of surgical errors and deficiencies in care occurred outside of the operating room.6–8
There are a number of limitations to this study. First, the data collection method by self-registration may not be entirely reliable. Caregivers might be reluctant to register their own failure to perform certain aspects of care, leading to under-registration of intercepted incidents.17 18 However, if the checklist was mistakenly used as a ‘challenge-do’ checklist—that is, the tasks represented by the items on the list are performed one by one as the checklist is being completed—instead of a ‘do-verify’ checklist, it would logically follow that ‘intercepted by checklist’ should be checked in all items, leading to an over-registration. This problem could be solved by combining the self-registration with another method of data collection, for example, observation of medical processes.19 20 This would provide detailed and objective information about intercepted incidents. However, observation is time consuming and unlikely to be performed in a large number of patients or procedures. In a previous study, it was shown in a small sample of 250 procedures that observation yielded three times as many intercepted incidents as self-registration.21
A second and more important limitation is the possibly limited relevance of the surrogate endpoint of intercepted incidents for actual improvements in patient outcomes. Any link between an intercepted incident and a potentially prevented adverse event remains hypothetical. Moreover, the demonstration of such a clear correlation is virtually impossible because of the multifactorial nature of adverse events: any adverse outcome may be dependent on a diversity of factors. While some processes have been proven to have a clear correlation with outcomes (eg, antibiotic prophylaxis and surgical site infection), most other process improvements lead to improved outcomes by ways that are less fathomable. Which patient would be more likely to have an adverse outcome: a patient for whom many incidents were intercepted or a patient for whom no incidents were intercepted? No intercepted incidents may reflect a perfectly safe process, but may also mean that imperfect processes were not corrected. At the hospital level, the correlation remains complicated as well. More intercepted incidents may reflect more room for improvement (a less optimal baseline situation) but may also reflect a more safety-oriented culture in which care giver are willing to register incidents. Despite the absence of a proven correlation between intercepted incidents and improved outcomes, we feel that the data presented in this study do add valuable information on the direct interception of safety incidents by a surgical safety checklist.
The implementation of a checklist covering the entire surgical pathway is not an easy undertaking. It requires considerable effort and resources and a hospital-wide dedication to patient safety. Continuous feedback and positive and negative reinforcement are needed to ensure a lasting improvement in patient safety.22
The SURPASS checklist has already been shown to be associated with significant improvements in patient outcomes.2 This study has provided insight into one of the effects of the checklist: the direct prevention of errors by checking specific items on the checklist. It is the first study to describe incidents intercepted by a checklist in such detail. The contribution of other mechanisms, such as improvements in processes, culture and communication, inspired by implementation of the checklist, remains to be studied. In addition, the correlation between intercepted incidents and improved outcomes remains an interesting subject for future study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Download Supplementary Data (PDF) - Manuscript file of format pdf
Footnotes
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.