Article Text
PDF
Abstract
Background Teamwork training and system standardisation have both been proposed to reduce error and harm in surgery. Since the approaches differ markedly, there is potential for synergy between them.
Methods Design: Controlled interrupted time series with a 3 month intervention and observation phases before and after. Setting: Operating theatres conducting elective orthopaedic surgery in a single hospital system (UK Hospital Trust). Intervention: Teamwork training based on crew resource management plus training and follow-up support in developing standardised operating procedures. Focus of subsequent standardisation efforts decided by theatre staff. Measures: Paired observers watched whole procedures together. We assessed non-technical skills using NOTECHS II, technical performance using glitch rate and compliance with WHO checklist using a simple quality tool. We measured complication and readmission rates and hospital stay using hospital administrative records. Before/after change was compared in the active and control groups using two-way ANOVA and regression models.
Results 1121 patients were operated on before and 1100 after intervention. 44 operations were observed before and 50 afterwards. Non-technical skills (p=0.002) and WHO compliance (p<0.001) improved significantly after the intervention in the active versus the control group. Glitch count improved in both groups and there was no significant effect on clinical outcomes.
Discussion Combined training in teamwork and system improvement causes marked improvements in team behaviour and WHO performance, but not technical performance or outcome. These findings are consistent with the synergistic hypothesis, but larger controlled studies with a strong implementation strategy are required to test potential outcome effects.
- Quality Improvement
- Surgery
- Team Training
- Patient Safety
Statistics from Altmetric.com
Introduction
The need to develop specific strategies for maximising the reliability of healthcare delivery, particularly surgery, and avoiding the risk of patient harm through unintentional deviation and error is now well recognised. One of the most common interventions used in industry to improve safety and reliability is to formalise frameworks of activity via standard operating procedures (SOPs). The primary goal of SOPs is to structure and standardise work with predefined plans, minimising errors which occur through failure of human memory or attention.1 The use of SOPs is established across all types of human activity and can be found in clinical procedures in healthcare where evidence-based ways of working are known, accepted and practiced.2
Many types of SOPs are available throughout healthcare settings such as checklists, forms, posters, protocols and manuals.3 Standardising an environment makes it more accessible to experimentation using improvement efforts such as the Deming Plan-Do-Check-Act (PDCA) cycle, since the SOP task is clearly specified and it is therefore easy to decide whether it has been performed correctly or not.4 Two main approaches have been used: SOPs may be imposed by authority with mandatory compliance audited, or staff led projects can be used to develop standardised approaches. This approach encourages team discussion and may produce more effective interventions through more intimate knowledge of potential system gaps and common deviations from correct procedures. Evidence that SOP approaches to improving surgical safety are effective has been provided by uncontrolled studies of WHO checklist,5 ,6 but also by the controlled SURPASS study.7
A quite distinct approach to minimising the risk of error through the introduction of explicit teamwork training has also been pioneered by the aviation industry, which developed the concept of crew resource management (CRM).8 CRM focuses on issues such as situational awareness, threat and error management, leadership, problem-solving, decision-making, communication and coordination, and includes all personnel involved with flight operations. The training aims to integrate technical and non-technical skills to reduce risk and maximise efficiency.9 In operating theatres, three distinct subteams (nursing, surgical and anaesthetic) work together as one unit. Research in operating theatres has revealed the importance of non-technical skills in safe and efficient performance.10 Ineffective team communication was identified in 30% of team exchanges, and a third of these episodes jeopardised patient safety by increasing cognitive load, interrupting routine and increasing tension.11 While objective evidence of improvement is difficult to obtain due to the rarity of events, operating theatre briefings significantly reduce perceived risk for wrong-site surgery and improve perceived collaboration among theatre personnel.12 Training in non-technical skills has resulted in improvements in attitudes to theatre safety, team non-technical performance and technical error rates. These findings are consistent with the MEDTEAMS study of emergency room staff, which reported a significant reduction in clinical errors after team skills training.13 A systematic review of the literature found consistent evidence of reasonable quality for an effect of CRM on team process, but less convincing evidence of effects on clinical outcomes.14 We have attempted to classify safety risks in healthcare around three ‘dimensions’ of the overall care programme: workforce culture, operational systems and technology.15 This led us to analyse the strategies used in safety intervention programmes, and we recognised that nearly all such programmes have focused primarily either on the system or the culture dimension, but not on both. This study tested the effectiveness of a combined SOP and teamwork training intervention to improve the quality, safety and reliability of surgical team performance. It forms part of a larger programme, the Safer Surgical Services programme, which attempts to answer the more general question of whether interventions based on changing workplace culture (such as CRM training) can be combined with those principally directed at rationalising and standardising work systems (such as SOPs or lean methods) to produce an additive or synergistic benefit.
Methods
Study design
The study was designed as a controlled interrupted time series, with data collection in theatre for 3 months, before and after a 3 month intervention period (for the active group only). Clinical outcome data were obtained for all relevant patients for 6 months before and 6 months after the intervention period.
Intervention
This study was designed to test a combined intervention: SOPs and team training. The SOP portion of the intervention aimed to improve standardisation and visibility of work processes, while the team training portion of the intervention aimed to inform and coach the theatre team members on the principles of effective teamwork. All theatre team members in the active group were invited to partake in out-of-theatre training sessions and intheatre coaching sessions and obtain support from the research team in facilitating and coordinating intervention tasks.
Team training
The team training portion of the intervention was led by an industry expert who provided an off-site 1-day interactive lecture-based training, using a syllabus based on aviation CRM training. This session was delivered prior to the SOP portion of the intervention with lecture material delivered in advance to participants. Example problem scenarios from theatre staff experiences were discussed, and how they affect team members, the operating process and patient safety was considered. Theatre staff received a certificate of attendance.
Standard operating procedures
This portion of the intervention was delivered by an experienced management consultant during two half-a-day workshop style sessions held off-site. The principles of improving working processes through standardisation using a quality improvement approach were explained, and teams were asked to suggest areas within the operating theatre in which they felt it would be most useful to apply standardisation techniques. The areas identified as most in need of improvement included the morning briefing, sending for a patient and handing the patient over to recovery. The teams were asked to further prioritise this list of options for areas where (a) they believed they could enact change within the organisation, (b) they could initiate and implement changes within a 3 month time period and (c) they believed that changes would improve the reliability and safety of working practices from the current state. The staff were therefore responsible, after receiving training, for deciding the focus of the intervention.
Outcome measures
We measured three observational outcome measures of team performance as well as collecting data on patient outcome and operation duration. The observational measures were: Oxford NOTECH II, a measure of theatre team non-technical skills; the glitch count, a measure of technical operative process deviation; and performance of WHO checklist ‘time out’ (T/O) and ‘sign out’ (S/O) procedures.16–18 Operations were directly observed by paired independent observers, one with clinical and one with human factors (HF) background. The clinical observers included two surgical trainees (MH, ER) and one nurse practitioner (JM); the HF specialists had a higher degree in HF and/or psychology (SPP, LM, LB). Prior to commencing the study, observers completed a 2-month training phase for familiarisation with the surgical procedures observed, harmonisation of scoring, and for defining the data collection process. Data collection booklets for each surgical procedure were developed19 to aid observers in recording observational data. Intraoperative observations began when the patient entered the operating theatre and ended when they left it. At the end of each operation, results were inputted to a secure de-identified database.
Oxford NOTECHS II
The operating team's non-technical skills were assessed through the Oxford NOTECHS II behavioural rating scale, whose properties and validation we have previously described.20 This scores each subteam (nursing, surgical and anaesthetic) on a 1–8 scale for each of four behavioural parameters: leadership and management; teamwork and cooperation; problem-solving and decision-making; and situational awareness, giving a theoretical maximum score of 96 (4×8×3).
Glitch count
This measure of the technical quality of surgical team performance records deviations from the recognised process with the potential to reduce quality or speed, including interruptions, omissions and changes, whether or not these actually affected the outcome of the procedure, and is expressed as a glitch rate per hour.17 Glitches, their details (eg, ‘equipment not available when requested’) and time of occurrence were recorded independently by each observer. Following the completion of the operation, the glitches were agreed, categorised and entered into a secure database.
WHO Surgical Safety Checklist
The compliance of teams with the T/O and S/O parts of WHO Surgical Safety Checklist was evaluated by direct observation, using a simple scoring method described previously.17 Briefly, we recorded whether T/O and S/O were attempted, and where they were, we recorded three measures of quality: whether all specified information was communicated; whether all the team members were present; and whether there was active participation. We recorded full compliance when all three measures were positive.
Clinical outcome data
Hospital episode statistics data were extracted for all patients undergoing operations in the relevant operating theatres under the involved consultants during the 6 month periods immediately before and after the intervention. This therefore represents a larger group of patients, of which those whose operations were observed represented a large convenience sample. Data were independently extracted by Trust staff and supplied to the research team in anonymised form. The information extracted for each patient was: age; sex; diagnosis; consultant; operation; operating time; length of hospital stay; complications (any) and nature; and readmission within 90 days reoperation. The parameters used in comparisons between active and control groups were: length of stay, number (%) of patients with any complication and readmissions.
Data analysis
Difference between the control and active arms was assessed using two-way analysis of variance (Group×Time), with treatment (control vs active) and time (preintervention vs postintervention) as factors. Differences between groups were assessed by the Group×Time interaction. Effectively, this allowed us to compare (Outcome measure before intervention)/(Outcome measure after intervention) in the intervention and control groups, to allow for secular trends. Preintervention and postintervention differences are reported as 95% CIs. All statistical analyses were carried out in R (V.3.0.1). For clinical outcome data, baseline demographic information was summarised using descriptive statistics. t Tests for mean age and χ2 test for gender distribution were used to compare the before and after periods. Binary clinical outcome variables before and after intervention were compared using ORs and 95% CIs from a logistic regression, and mean length of stay using linear regression, controlling for age and gender in both regression models. Given the number of before and after comparisons performed, a 1% significance level was selected. This statistical analysis was conducted in Stata V.12.
Study setting
The study was conducted with teams specialised in lower limb orthopaedic surgery, performing operations such as knee arthroscopic procedures and knee and hip primary and revision arthroplasty at two hospital sites within a university hospital trust. The active group was in one operating theatre at one hospital site, and the control group in two operating theatres at the second site. There was therefore no contamination by informal learning or staff interchange between the two groups.
Ethics
Theatre staff identified as working in the defined operating theatres and would potentially be observed were given information on the study and provided written consent before observations took place. The study was approved by Oxford A Ethics Committee (REC: 09/H0604/39).
Results
In the preintervention collection period, 567 operations were performed in the intervention theatre and 554 in the control theatres. In the postintervention period, 679 and 421 operations respectively were performed in these two settings. In all, 101 operations were observed, 47 before the intervention (30 intervention and 17 control groups) and 54 following the intervention (33 intervention and 21 control groups).
Intervention
The two areas identified by staff for improvement via SOP development were: the morning briefing and conduct of WHO Surgical Safety Checklist. The teams split into smaller discussion groups to develop plans for change, then presented their ideas to the wider group and a final implementation plan was agreed after discussion. A core group was established to lead the intervention activities, and methods for informing the wider group were agreed.
The theatre team members modified the existing briefing sheet and WHO checklist form, based on the information generated and agreed during the workshop sessions. The new versions were distributed by email to relevant team members, and edits and comments were collated from the wider group. This process was repeated iteratively until the final stable versions were agreed. Guidelines were then developed about how the work process would be redesigned around the new document, and distributed to all theatre staff who would use the new materials. Box 1 gives a summary of the changes and additions made to existing work processes in creating the SOPs. The new materials were then introduced into the operating theatre and eight intheatre coaching sessions were conducted by the CRM expert and his team over several weeks, aiming to help the team troubleshoot the process of developing and fine-tuning the new SOPs.
Changes and additions to existing practice introduced during creation of standard operating procedures
WHO checklist changes
-
Which team member is responsible for leading WHO checklist sections was specified
-
Each question was modified for clearer terminology
-
It was specified which team member each question is directed to, and who should answer
-
Directions were written for which checklist sections should be read out loud
-
Specific text was added to confirm when the checklist section was completed
-
An overall guideline was created which detailed each of the modifications and how the checklist process should be conducted
Morning briefingchanges
The morning briefing document was modified to include recording of:
-
Essential team members to be present: surgeon, anaesthetist, scrub practitioner, anaesthetic practitioner
-
Whether a copy of the theatre list is present and verified
-
A dedicated section on the briefing sheet for each patient on the list, specifying:
-
Their place in the list order for operation
-
The confirmed operation description including site and side
-
Details of specialised instruments, blood tests required, patient allergies and any other specific requirements
-
-
An area for general comments
-
An area for potential hazards
Guideline for morning briefing
A guideline for the morning briefing process was created which specified:
-
What areas of the sheet must be completed
-
Which team members MUST be present in order for the briefing to take place, including directions for whom to inform if all team members are not present
-
Directions of what information should be captured in the general comments and potential hazard areas
-
Rules for displaying the completed team brief during the operating session
-
A rule for all completed forms to be returned to the theatre office
-
A rule that the team brief does not replace WHO checklist
-
A rule that any issues identified during briefing must be escalated to the person controlling the theatre suite
-
A rule that any actions from the sign out can be recorded in the comments box on the briefing sheet
Oxford NOTECHS II
Mean Oxford NOTECHS II score was 74.83 before and 79.27 after the intervention in the active group (difference=4.44; 95% CI 1.23 to 7.65) while it was 76 before and 71.81 after in the control group (difference=−4.19; 95% CI −8.20 to −0.18) (figure 1). The difference between the change in the active and control groups was statistically significant (p=0.002; difference 7.46; 95% CI 3.00 to 11.93). Subteam analysis revealed differences in mean NOTECHS scores were non-significant for surgeons (p=0.067) and nurses (p=0.093) while statistically significant for anaesthetists (p<0.001; difference 4.24; 95% CI 2.77 to 5.68). There was no relationship between total Oxford NOTECHS II score and operation length (shown as the size of the data point).
Team Oxford NOTECHS II scores.
Glitch rate
Before the intervention, the mean glitch rate per operation was 6.84 (SD=3.05) glitches per hour in the active group, and 7.67 (SD=4.45) glitches per hour in the control group. After the intervention, mean glitch rate decreased to 5.25 (SD=3.17) glitches per hour in the active group (difference=−1.58; 95% CI −3.15 to −0.02) while in the control group it decreased to 6.52 (SD=3.06) glitches per hour (difference=−1.16; 95% CI −3.76 to 1.45). The difference between the two groups was not statistically significant (difference −1.26; 95% CI −3.01 to 0.49; p=0.153) (figure 2).
Glitch rate per hour (individual operations shown as dots; dot size representing operation duration).
The rate of planning and preparation glitches fell by almost 50% in the active theatre, but remained unchanged in the control theatre; process deviations also fell in the active theatres. Distractions fell in the control theatre, but absences and process deviations increased (figure 3).
Mean rates of the different glitch categories.
WHO compliance
Of the 101 observed operations, T/O was attempted in 94. There was no significant difference in the attempt rate between preintervention (27/30; 90%) and postintervention phases (30/33; 91%) in the active arm (difference=1%; 95% CI −15% to 16%; p=1), or between preintervention (17/17; 100%) and postintervention phases (20/21; 95%) in the control arm (difference=−5%; 95% CI −19% to 9%; p=1). S/O was attempted in only 23 operations, and there was a significant difference in the attempt rate of S/O in the active arm (preintervention (0/30; 0%), postintervention (21/33; 64%), difference=64%; 95% CI 44% to 88%; p<0.001), but not in the control arm (preintervention (2/17; 12%, postintervention (0/21; 0%), difference=−12%; 95% CI −32% to 9%; p=0.377). The difference between the change in the active and control groups was significant (difference 64%; 95% CI 43% to 84%; p<0.001).
The quality of WHO completion, in operations where it was attempted, is compared preintervention and postintervention in figure 4. All three components of T/O were completed in 3/30 (10%) cases in the preintervention active arm, which increased to 20/33 (61%) in the postintervention phase (difference=51%; 95% CI 28% to 74%). All three components of T/O were completed in 11/17 (65%) cases in the preintervention control arm, which decreased slightly to 11/21 (52%) in the postintervention phase (difference=−12%; 95% CI −49% to 24%). The difference between the change in the active and control groups was statistically significant (p<0.001).
Quality of WHO surgical safety time out and sign out checklist completion.
Table 1 provides a summary of the outcome measures.
Summary outcome measures
Discussion
Summary
This combined intervention using SOP development and CRM training together bought about considerably improved team performance, but clinical outcome benefit was not demonstrated. Team non-technical skills and compliance with WHO checklist improved both overall and in comparison with the control group, but technical performance did not, as both groups showed a similar level of improvement. In terms of causality, an improvement in patient outcomes is likely to require a measurable improvement in technical as well as non-technical performance. This was achieved, but the parallel improvement in the control group means that we cannot consider this to be due to the intervention, nor expect an easily detected difference in clinical outcome. This study was not powered to detect clinical outcome differences, and so a modest but useful effect on these could well have been missed.
Difficulties encountered
Implementation difficulties are a major barrier to the success of safety intervention schemes21–24 and this project encountered some problems which may have limited its effectiveness. The most important of these was the difficulty of achieving attendance of all theatre staff at the training sessions. Hospital management were supportive, and it was ultimately decided that maximum attendance would require closing the operating theatre for one day, and so this was eventually arranged. During the intervention, the consultant anaesthetist became the main champion for the intervention process, which may explain why the anaesthetic subteam was the only one to show a significant postintervention improvement. Staff attitudes to the intervention varied across the group from strongly supportive to sceptical and disengaged. While management were engaged and supportive, the intervention was not formally adopted as a management initiative, and this may have reduced its credibility among some staff members. Introducing safety-related changes to well-established practices in surgery is difficult, since the baseline environment is complex and pressured, professional attitudes and habits are deeply ingrained, and changes need to be introduced while activity continues at a pace near to the maximum attainable by the current system.23 ,25 ,26 This scenario results in specific difficulties around team availability, the necessity of concurrent tasks, limited formal structure, organisational/individual culture issues and communication/coordination issues. Team briefing itself is conducted at a time when the current system makes it difficult for the team to meet, and the pressure to start the operating list is intense. In other industries, the briefing is factored into the daily activities and allocated adequate time, as is quality improvement activity. The effect of a new briefing strategy may have been stronger if the implementation plan had been more radical in sweeping away existing practice, but this would have required ‘buy-in’ from both management and staff at a level which could not be achieved in a research project of limited scope.
Interpretation
While the surgical environment poses undoubted challenges, the nature and intensity of the intervention itself is another crucial factor in determining the outcome of such efforts. The resources and time available meant that our intervention was shorter than would be considered optimal by experienced practitioners in other industries, and the support available to staff was limited by the availability of expert personnel who had other duties to perform. Against this, evaluation of high-cost, high-intensity interventions which could never be implemented at scale in the National Health Service would not provide any practical benefit. The process used to develop the new SOPs was based on a theory emphasising the importance of engaging staff by allowing the ‘grassroots’ staff to define the problems and develop the solutions. This can result in staff selecting projects which did not appear likely to impact the measures of process and outcome predefined by our programme, but in this case their choices seemed appropriate. More guidance from experts, using skilled interpretation of the baseline data, might have resulted in an intervention which could improve technical performance and clinical outcome. This tension between engagement and control in quality improvement efforts requires further study. We did not use PDCA cycles to accelerate the refinement of the intervention in this study in order to ensure that the intervention was distinct in type from ‘lean’ methods, which we have used in other studies in our programme. This may have weakened the potential of the intervention to produce major changes quickly.
Relation to other evidence
There is a significant literature on the effectiveness of safety interventions based on CRM in healthcare.14 While many studies are of poor quality, this review reported two randomised controlled trials in clinical settings,27 ,28 neither of which reported any clinical outcome benefit. Improvement in non-technical skills is consistently reported, but the evidence for improvements in clinical outcome is less convincing.14 Our findings here are entirely consistent with this summary. The literature on SOP approaches to safety in surgery is also flawed by a preponderance of weak designs, but several groups have claimed major improvement in clinical outcomes. There are many papers describing major outcome improvements after implementation of WHO checklist29 which is an SOP approach by our definition. However, these nearly all suffer from major methodological flaws such as lack of a control group, lack of blinded objective assessment and short follow-up. The SURPASS study30 also had a very short follow-up, but used a strong design to evaluate a very elaborate SOP covering the whole process of surgical care, and reported patient outcome benefits; this important study merits repetition with a particular focus on longer term sustainability. At present, the literature does not provide convincing valid evidence that either the CRM approach to changing team culture or the SOP approach to rationalising work systems is consistently effective. Our current programme of work is designed to allow evaluation of the relative benefits of systems and culture-based approaches to safety improvement, and the results of this study of a combined intervention will be meta-analysed with other studies to evaluate the hypothesis that the two intervention types produce synergistic benefits.
Limitations
Our study incorporated a parallel control group, which was protected from contamination by secondary education, by being in a different hospital within the same Trust. We also used objective clinical outcome data which were collected independently by observers blind to the study groups. Our observational measures of process were carefully designed and validated, and the observers were well trained and used a clear standard protocol. Despite these strengths, our study suffered from the weakness that we were unable to blind observers as to intervention allocation, and so observer bias cannot be ruled out. The fact that our findings did not entirely bear out our expectations from our hypotheses is an important argument against this criticism. Observer bias would have been expected to result in improvements in glitch rate in the intervention group only. A Hawthorne effect is not likely to have biased our results, since both intervention and control groups should have been equally affected. However, our follow-up is relatively short and one might expect weakening of the intervention effect with time, and so it is possible that the improvements we detected may not have been sustainable. Against this, our systematic review showed that most studies which found a benefit from CRM and provided longer term follow-up found quite good sustainability.14 The numbers of events in the observational data are small, and a fairly large number of significance tests were performed. There are therefore possibilities that both type 1 and type 2 statistical errors may have occurred. However, the positive findings do appear to form a pattern consistent with an effect on team non-technical skills and specifically on WHO checklist performance, rather being random as would be expected from statistical errors.
Conclusions
The combination of the two types of quality improvement intervention, SOP+CRM-style team training, produced important improvements in non-technical skills and the conduct of WHO Surgical Safety Checklist, but did not significantly impact technical performance or clinical outcomes. Intervention studies of this type face implementation challenges and lack statistical power to prove patient benefit. They nonetheless have value in identifying strategies which impact the team processes believed to be responsible for improving clinical outcomes.
Acknowledgments
The authors thank the management, surgical, anaesthetic and theatre staff at the Trusts and sites that took part in this study for their cooperation and forbearance: Oxford University Hospitals NHS Trust, University Hospitals Coventry and Warwick NHS Trust, and Kettering General Hospital NHS Foundation Trust. The authors also thank J Matthews and L Bleakley for assistance in the intraoperative data collection. We would also like to acknowledge members of Atrainability who assisted Captain Trevor Dale in delivering the CRM training.
References
Footnotes
-
Contributors PM, SN and KC conceived and designed the intervention, with input from LM, ER, MH, SPP, DG, OR-A and GC. Data collection and floor work with frontline staff were carried out by LM, ER, SN, SPP and MH. GC and OR-A led the statistical analysis. All authors contributed throughout the writing process. LM and SPP wrote initial drafts of the article and PM completed the final one. All authors agreed the final version of the article.
-
Funding This paper presents independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research programme (Reference Number RP-PG-0108-10020). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
-
Competing interests None.
-
Ethics approval Oxford A Ethics Committee.
-
Provenance and peer review Not commissioned; externally peer reviewed.