You are here

Article Text

Article menu
Original research
Impact of multidisciplinary simulation-based training on patient safety in a paediatric emergency department
  1. Mary D Patterson1,2,
  2. Gary L Geis1,3,4,
  3. Thomas LeMaster1,
  4. Robert L Wears5,6
  1. 1The Center for Simulation and Research, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
  2. 2Akron Children's Hospital Simulation Center for Safety and Reliability, Akron, Ohio, USA
  3. 3Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
  4. 4Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
  5. 5Department of Emergency Medicine, University of Florida, Jacksonville, Florida, USA
  6. 6Clinical Safety Research Unit Imperial College, London, UK
  1. Correspondence to Dr Mary D Patterson, Akron Children's Simulation Center for Safety and Reliability Akron Children's Hospital One Perkins Square Akron, OH 44308 USA;mpatterson{at}chmca.org

Abstract

Background Cincinnati Children's Hospital is one of the busiest paediatric emergency departments (ED) in the USA; high volume, high acuity and frequent interruptions contribute to an increased risk for error.

Objective To improve patient safety in a paediatric ED by implementing a multidisciplinary, simulation-based curriculum emphasising teamwork and communication.

Methods Subjects included all healthcare providers in the ED. Multidisciplinary teams participated in simulation-based training focused on teamwork and communication behaviours in critical clinical scenarios. The Safety Attitudes Questionnaire, tests of knowledge and evaluations of critical simulations and actual performance in the ED resuscitation bay were assessed. Methods to sustain improvements included mandatory participation of all new staff in simulation-based training and the introduction of routine in situ simulations.

Results 289 participants attended the initial training. 151 participants attended the re-evaluation at a mean of 10.2 months later. Sustained improvements in knowledge and attitudes were demonstrated. Knowledge tests at baseline, postintervention and re-evaluation had scores of 86%, 96% and 93%, respectively. Friedman's test analysis of SAQ scores at baseline, postintervention and re-evaluation indicated significant attitude changes. The ED with a preintervention baseline of 2–3 patient safety events per year has now sustained more than 1000 days without a patient safety event. This improvement occurred even though the time required in initial simulation training has been condensed from 12 to 4 h.

Conclusions Simulation training is an effective tool to modify safety attitudes and teamwork behaviours in an ED. Sustaining cultural and behavioural changes requires repeated practice opportunities.

  • Emergency department
  • Simulation
  • Communication
  • Crew resource management
  • Team training

https://doi.org/10.1136/bmjqs-2012-000951

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Background and outline of problem

    The 1999 Institute of Medicine report, To Err is Human, identified the emergency department (ED) as the area of the hospital with the highest risk of adverse events.1 The simultaneous care of multiple patients with varying acuity levels, frequent interruptions, time constraints, high volume and the need to institute critical interventions with limited information contribute to an increased risk for medical error in the emergency setting. In one study of closed ED claims, teamwork failures accounted for ‘more than 40% of these cases’.2

    There is little information on the magnitude of medical error in paediatric emergency medicine.3–7 As patients, children present unique challenges including anatomical, physiological and pharmacological differences and the child's relative inability to communicate with the healthcare provider. These contribute to the risk for medical error in paediatric patients.8 It is generally conceded that the error rate in paediatrics is likely to be higher than in adult patients.

    As part of an ED needs assessment, the principal investigator conducted a small qualitative project, approved by the Institutional Review Board, in November of 2003. Indepth interviews and field observation were used to examine ED staff perceptions regarding patient safety. Experienced staff in our ED consistently identified communication, individual responsibility and inexperience as issues for patient safety. A review of patient safety events (PSEs) for the 5 years prior to this project also identified these issues. The institution has defined a PSE as an event where there was patient harm following less than ideal medical management. Twelve PSEs involving the ED were identified during this time. Communication issues were identified as a fundamental cause in nine of these and training issues were identified in 10 of these. Eight of these events involved patients in the resuscitation bays.

    Specific aims

    The overall goal of this project was to decrease the frequency and mitigate the effects of medical error in a paediatric ED by implementing standardised multidisciplinary, simulation-based training that emphasised team behaviours.

    The specific aims of this project were:

    • To implement a multidisciplinary simulation-based training that encompassed crew resource management (CRM), teamwork behaviours and critical communication skills

    • To evaluate the effectiveness of this training by assessing knowledge of and attitudes towards patient safety and by assessing teamwork behaviours among ED caregivers prior to and following this intervention.

    We also planned to track PSEs though we did not anticipate a change in the rate of PSEs during the project.

    Setting and function

    The Cincinnati Children's Hospital Emergency Department is a Level I trauma centre with an annual census of >90 000 patient visits. The volume and complexity of the patient population represent huge risks for patient safety. The hospital is located in an urban centre and provides care to patients with the entire gamut of illness and injury, for example, ear infections, complex medically fragile patients, patients with acute onset of sepsis and those suffering acute traumatic injury. Approximately 3000 patients annually are assessed as critical and immediately triaged to the resuscitation bays.

    Cincinnati Children's Hospital Medical Center (CCHMC) is a teaching institution and provides paediatric education to residents from our own and other institutions. Approximately 45–55 residents and 12 fellows rotate through our ED monthly. Supervision of all trainees is provided by board-certified paediatric emergency physicians.

    Methods

    The purpose of this project was to adapt the principles of CRM to standardised simulation-based teamwork training for paediatric ED personnel. The training focused on the development of teamwork and communication skills and the use of simulation to provide deliberate practice of these skills in a simulated high risk, time pressured critical environment.9

    The project was evaluated by the institutional review board and granted exempt status as an educational intervention. Confidentiality statements and video consents were obtained from all participants. While all ED providers were required to participate in the training, they were not required to complete the knowledge test or attitude survey.

    Participants

    Participants were ED direct care providers: faculty and staff physicians, nurses, respiratory therapists, paramedics, patient care assistants and medical residents. It was required that all disciplines be represented in each session with the exception of respiratory therapists, who are not always present in the ED. Individuals participating were frequently acquainted with one another, but intact teams from a particular shift were not trained as a group.

    Intervention

    The training was developed based on analysis of the previous 5 years of PSEs and in consultation with a patient safety expert (RW). The intervention was conducted with teams of approximately 8–10 ED personnel and was designed to replicate the teams that care for critical patients. Each group completed the intervention as a multidisciplinary team. The initial intervention took place over 2 days and required approximately 12 h to complete. Prior to the training, participants reviewed an intranet-based presentation incorporating information on the magnitude of risk from medical error, error theory and principles of CRM.

    The overall goals of the intervention were that ED personnel would demonstrate increased awareness and knowledge of patient safety. The safety culture would reflect personnel investment in patient safety and reducing medical errors. In a simulated clinical setting, ED personnel would demonstrate the ability to recognise high risk situations and use the acquired skills to prevent or decrease the impact of unexpected events and errors. The intervention included four primary participant goals and competencies which were based on Bloom's taxonomy matrix: knowledge; comprehension; application; analysis and evaluation and synthesis.10 The four primary goals of the intervention were that participants:

    1. Demonstrate knowledge and awareness of high risk situations and factors leading to medical errors in the paediatric ED as demonstrated by a score of 80% or better on a written test.

    2. Demonstrate knowledge of methods that decrease patient risk in a simulated patient environment as determined by an acceptable performance in this setting.

    3. Demonstrate critical communication and teamwork skills necessary to prevent, trap and mitigate the effects of unexpected events and errors in a simulated patient environment.

    4. Exhibit a positive attitude towards and personal investment in patient safety as measured by a Safety Attitudes Questionnaire (SAQ) assessment scale (see online supplementary appendix for complete description of curriculum constructs).

    During training, techniques to prevent medical error, improve critical communications, increase situation awareness, develop resilience, and improve sharing of mental models and closed loop communication were introduced with mini-lectures, targeted review of videotapes of simulated resuscitations and case studies in healthcare and aviation. The major elements of the curriculum and the order of presentation are described in table 1.

    View this table:
    Table 1

    Patient safety curriculum elements and order of delivery

    The major focus of training was devoted to standardised high risk medical and trauma scenarios utilising a manikin simulator (table 2). A total of five simulations, followed by immediate video assisted debriefing, were presented during the intervention. The development of skills that promote resilience was supported through the utilisation of simulations that incorporated a scripted medical error, adverse event, unexpected equipment malfunction, or failure to allow personnel to detect and mitigate the effects of errors and unexpected events. These types of simulations encouraged the team's practice of the skills needed to solve unexpected failures in time pressured environments. In particular, these simulations focused not on the prevention of error, but on the detection, trapping and mitigation of the effects of error and system failure and the stabilisation of the patient and the system. Rather than retrospectively analysing examples of resilience (or the absence of resilience), the teams participated in reproducible simulated scenarios that encouraged them to develop an approach to the management of unexpected and adverse events. Other scenarios required non-physicians or less experienced physicians to assert themselves with team leaders. Probe methodology in which only one team member is aware of a critical aspect of patient information was also used to assess information sharing, assertiveness and closed loop communication.11 Debriefing, by MP, GG or TL, included group assessment of team performance as well as a guided review of the simulation video. Participants were asked to identify and suggest solutions to the challenges encountered.

    View this table:
    Table 2

    Scenarios utilised in simulation training

    At the beginning of each training session, the SAQ Teamwork and Safety Climate version12 was used to assess attitudes toward and ownership of patient safety by each individual. A pretest assessing knowledge of patient safety issues was administered to all participants. All subjects were oriented to the simulator and to the simulation environment. Each team was then videotaped in a simulated clinical scenario to assess baseline team function with respect to teamwork and utilisation of crew resource principles during a critical situation (baseline).

    A simulated clinical scenario was also videotaped at the completion of the intervention (postintervention). The baseline simulation was in addition to the five simulations presented during the intervention. Following the training, the SAQ and a parallel knowledge assessment were utilised as post-tests to examine changes in attitude and knowledge. The post-test for knowledge and the postintervention simulation were similar in form and degree of difficulty but not identical to the preintervention assessments.

    Multidisciplinary teams of ED personnel were asked to return for re-evaluation at approximately 6 months following the intervention to repeat a parallel knowledge test, a SAQ, and to be videotaped and debriefed on several scenarios over a 2–3 h time frame. Teams returning for re-evaluation were also designed to replicate the multidisciplinary team caring for critical patients in the ED. Though the teams that trained together in the initial intervention did not return intact for the re-evaluation, all the roles were represented in the re-evaluation teams. The re-evaluation allowed assessment of the persistence of knowledge and attitude changes. Simulation ‘practice’ also served to reinforce the desired teamwork behaviours.

    Actual paediatric resuscitations in the ED resuscitation bay were video recorded and compared with preintervention and postintervention video recordings to assess performance, teamwork and incorporation of CRM principles in actual patient care. A modified University of Texas Behavioral Markers for Neonatal Resuscitation Scale (more recent teamwork scales were not published at the time) was used to assess teamwork skills in the initial (baseline) and postintervention simulations, as well as teamwork behaviours in the resuscitation bay.13

    Data sources/collection

    Measures

    The patient safety knowledge assessments were developed by the authors to evaluate participants’ understanding and retention of key concepts presented in the prework and during the intervention itself. Parallel forms of the assessment were administered at the beginning of the intervention, at the completion of the intervention and at the re-evaluation (all three versions are available in the online supplementary appendix). The post-test and re-evaluation knowledge assessments were similar in form and degree of difficulty but not identical to the preintervention assessment. The knowledge assessment was focused on concepts previously identified as challenging in our ED environment. Certain questions that were believed to be highly relevant to our ED were included in all three forms of the assessment. The authors agreed that the final knowledge assessments have high face validity for our environment.

    The SAQ Teamwork and Safety Climate version (see online supplementary appendix) was developed from the Flight Management Attitudes Questionnaire, a tool used to assess flight crew's attitudes regarding: teamwork, speaking up, communication, collaboration decision making and leadership.12 The Teamwork and Safety Climate version uses a 5-point Likert scale, and is comprised of 14 items assessing teamwork climate and 13 items assessing safety climate. Some items are reverse (or negatively) worded. The survey has been validated with more than 10 000 healthcare providers in more than 200 clinical units.12 The SAQ has been demonstrated to have test and retest reliability of 0.85–0.90 and Cronbach α values of 0.75–0.88.12 ,14 A minimum acceptable safety culture score on the SAQ has been set at 60% with a ‘goal’ of 80%–100%.15

    A modification of the Behavioral Markers for Neonatal Resuscitation Scale13 (see online supplementary appendix) was used to assess teamwork and communication in the simulated and clinical setting. This tool was based on the Line Operations Safety Audit, a tool used in aviation to assess cockpit teamwork and CRM behaviours during more than 5000 flights. The Behavioral Markers scale was developed to assess teamwork and communication skills during neonatal resuscitations. At the time this intervention began, it was one of the few existing teamwork behaviour scales and the domains addressed in the scale are not specific to neonatal resuscitations. This scale has the advantage of prior use in a number of projects and explicit guidance detailing its use.16–19 Inter-rater agreement for observation of the various behaviours has ranged from fair to good for most behaviours (κ of 0.41–0.80) with the exceptions of workload management, vigilance and leadership for which inter-rater agreement was slight (κ=0.21–0.41)16 in previous studies; teams that scored higher in vigilance and information sharing on this scale made proportionally fewer errors during actual neonatal resuscitations than those with lower scores in these areas.19 Performance on this scale has also been weakly but positively correlated with team compliance with neonatal resuscitation programme standards.16 These results seem to indicate good content and construct validity and fair to good reliability.

    Analysis

    The SAQ was analysed using the Friedman's test (for ordinal data). Knowledge assessments were analysed with repeated samples Analysis of Variance (ANOVA). The Behavioral Markers scale was used to analyse simulated clinical scenarios as well as videos recorded in the actual ED resuscitation bay. This provided us with means to compare teamwork behaviours preintervention and postintervention. These were analysed using a Friedman's test.

    Pretest and post-test results of patient safety knowledge and the SAQ were collected during the training intervention. These were again measured at the re-evaluation. This information was entered in an SPSS database.

    Five paediatricians with medical education interest and expertise were trained as behavioural raters. These paediatricians were not associated with the ED and were blinded as to whether the video was baseline or postintervention. They provided a consistent core of raters.

    All resuscitations in the EDs shock trauma bays are videotaped for performance/quality improvement. To evaluate the transfer of teamwork behaviours from the simulated setting to the actual patient care setting of our bays, we planned to analyse 15%–20% of these resuscitations using the same behavioural scale and the same raters as in our evaluations of simulation scenarios.

    Results

    A total of 289 individuals participated in the initial simulation-based training. Overall, 33 classes were conducted from March of 2005 through March of 2008. In all, 151 participants attended a re-evaluation during 36 re-evaluation classes. The significant difference in those that attended an initial intervention but not a re-evaluation was largely due to staff turnover.

    In all, 77% of the participants were female and almost 50% were nurses.

    The average number of years of experience was 5.5 with a range of 0–30 years. Approximately 50% of participants had 3 or fewer years of experience. The proportion of individuals participating in the initial intervention and the re-evaluation by role is shown in table 3.

    View this table:
    Table 3

    Percentage of participants by role in initial intervention and re-evaluation

    Knowledge

    Tests of knowledge were conducted at baseline, immediately after the intervention and at the re-evaluation (table 4). The re-evaluations occurred at a mean of 10.2 (SD 4.8) months after the initial intervention. A repeated measures ANOVA with a Greenhouse-Geisser correction determined that the mean test scores differed significantly between time points. (F (1.77, 183.99)=114.63, p< 0.001). Post hoc tests using the Bonferroni correction revealed that there were significant differences between baseline and postintervention scores as well as between baseline and re-evaluation scores (p<0.001 for both comparisons). Cohen's d was calculated to estimate the magnitude of effects. The value of 1.0 from preintervention to postintervention indicates a large effect size while the effect size from postintervention to re-evaluation is 0.42 indicating a moderate effect. There is a statistically significant difference between the postintervention and re-evaluation scores as calculated by a paired samples Student t test with p<0.001. This indicates a significant loss of knowledge at the re-evaluation compared with immediate postintervention scores. However, the level of knowledge at the re-evaluation is still significantly higher than at the baseline evaluation. This indicates a sustained improvement in knowledge scores over baseline despite a 10-month interval between the intervention and the re-evaluation.

    View this table:
    Table 4

    Results of knowledge and attitude assessment

    Attitudes

    The SAQ was also administered at baseline, postintervention and at the re-evaluation and was analysed using the Friedman's test (table 4). There was a statistical difference in attitudes over time for teamwork, climate and overall: Χ2(2)=16.09, 26.33 and 16.32, respectively (p<0.001 for all three categories). Post hoc analysis was conducted using the Wilcoxan signed ranks test with a Bonferroni correction applied for multiple comparisons, resulting in a significance level of p<0.017. The comparison of the preintervention and postintervention results indicated Z values of −4.9, −2.96 and −3.47 for climate, teamwork and overall. All three values were statistically improved following the intervention (p<0.01). Postintervention and re-evaluation scores were also compared. The Z values were −0.85, −0.46 and −0.46 for climate, teamwork and overall, respectively. This indicated that the attitude changes seen following the intervention were not significantly diminished at the time of re-evaluation (p>0.017).

    A Cohen's d was also calculated to estimate the magnitude of effect for the SAQ (table 4). The calculated values for effect size for preintervention to postintervention of the SAQ ranged from 0.30 to 0.38, indicating a small to medium effect size (d>0.2 and <0.5).20 The effect size calculations from postintervention to re-evaluation were all < 0.10 indicating a minimal effect on attitude change in that interval.

    Behaviour

    In all, 45 pairs of videos were reviewed by five reviewers to assess changes in teamwork behaviours in the simulated setting from baseline to postintervention. An additional 35 videos from the re-evaluation phase were also evaluated. Reviewers were blinded as to whether the video was recorded at baseline or postintervention. Two reviewers independently rated each video. A modified Behavioral Markers for Neonatal Resuscitation Scale was used to evaluate observability of the behavioural markers and the frequency of these behaviours. These scores were analysed using the Friedman's test to compare the baseline, postintervention and re-evaluation scores. The median observability scores at baseline, postintervention and re-evaluation scores were 3.6, 3.4 and 3.4, respectively. The Χ2(2)=2.0, p>0.05. The median frequency scores at baseline, postintervention and re-evaluation were 3.6, 3.0, and 3.0, respectively; Χ2(2)=4.0, p>0.05. To assess concordance of rating by the video reviewers the eta coefficient (nominal by ordinal data) was calculated for 14 ‘control videos’. The eta coefficient was between 0.62 and 1.0 for the sample of videos reviewed. This indicated a moderate rate of concordance between the reviewers. Only 12 videos from the ED resuscitation bay were reviewed. This was due to technical difficulties associated with a change in the video recording system in the ED during enrolment.

    Participant experience

    CCHMC participants ranked the value of this training highly. Day 1 evaluations had an overall score of 4.6 on a 5-point Likert scale, with a median score of 4.7. The day 2 evaluations had an overall score of 4.8 on a 5-point Likert scale, with a median score of 5.

    Unexpected results

    Reduction of PSEs in the ED

    At the time of the implementation of this training the ED had experienced 12 PSEs during the previous 5 years. The entire institution had embarked on a strategic effort to reduce PSEs. Multiple interventions were employed throughout the institution in an effort to meet this goal, including mandatory 3 h error prevention classroom training and yearly online safety modules for all healthcare providers. At this time, the ED was the only clinical unit utilising multidisciplinary simulation-based training. During the 2555 days since the beginning of this project, only two PSEs have occurred in the ED with the longest interval between PSEs being 1425 days. In the two PSEs that occurred during this time frame, each patient was cared for by several services and each involved a higher than usual number of handoffs of care. Though these factors are known risks for patient safety, this again speaks to the reality that safety is a dynamic state and not one that is ever fully achieved. During the initial 3 years of this project, the institution as whole did not experience a decrease in the rate of PSEs despite other interventions aimed at reducing PSEs.

    Discussion

    Summary

    Our goal was to demonstrate the effectiveness of simulation-based training in improving patient safety. The findings demonstrate this training resulted in improvements in knowledge, safety attitudes and changes in behaviour in the simulated setting. We were able to demonstrate significant and sustained improvements in knowledge of and attitudes toward communication and teamwork behaviours. Challenging aspects of this project include the different educational levels and functions of the learners as well as the ‘built-in’ medical hierarchy. Though paediatrics is perhaps less hierarchical than some other medical specialties, there is still discomfort or reluctance on the part of less influential personnel to assert their concerns. We addressed this in several ways. First, in the simulation lab we oriented all participants to the ‘rules of engagement.’ These rules mandate respect for all team members. We intentionally flattened the hierarchy during training by using first names for all participants. We used video recordings to illustrate certain concepts and to provide examples of positive and negative teamwork behaviours. In this setting, observable communication and teamwork behaviours were the primary goal and evaluation focus. Second, the intervention included training in respectful assertion and voicing of concerns (deference to expertise, advocacy and inquiry) and closed loop communication as well as opportunities for less powerful team members to practice these skills in a simulated setting. Burke et al have stated,‘...it does no good if team members cross check and monitor one another's actions if when something out of the ordinary is recognized no one speaks up due to fear of reprimand.’21

    The video reviews did not demonstrate a significant change in behaviour as measured by the Behavioral Markers of Neonatal Resuscitation Scale. It is feasible that the behavioural markers scale used was not sensitive to the changes in behaviour that occurred. Since the initiation of this study, additional scales of teamwork behaviour have been validated and published including the ANTS and Mayo scales.22–24 These scales include assessment of behaviours that are not evaluated in the Behavioral Markers of Neonatal Resuscitation Scale. For example, the Mayo scale includes elements specifically assessing cross monitoring by team members, involvement of all team members, role clarity, closed loop communication, assertive statements and situation awareness.24 While communication and vigilance are included in a general way in the Behavioral Markers of Neonatal Resuscitation Scale, the description of these elements is much less specific than the description in the Mayo scale. Given the emphasis of the current intervention on behaviours such as closed loop communication, cross monitoring, assertive statements, maintaining situation awareness and the utilisation of all team resources, it is possible that had an alternative scale (specifically including these elements) been used as the assessment tool, a change in behaviour would have been detected in the video reviews.

    Pronovost et al reported that change in attitude was most predictive of intensive care units personnel's ability to decrease bloodstream infections.25 There has been a substantial and sustained change in the number of PSEs in the clinical environment of our ED and ED personnel report anecdotally that there is a change in communication and behaviour. This suggests that the sustained improvement in the rate of PSEs is likely associated with the change in attitudes.

    The decrease in PSEs during the initial period of this project was unexpected. We believed that changing the culture in a large urban ED would require a number of years. Though it is difficult to definitely ascribe the decrease in PSEs to this intervention alone, it is important to note that the ED was the only clinical unit employing simulation-based training at a time of significant focus and effort by the institution on reducing PSEs across the organisation. During this time, the hospital initiated a significant number of programmes employing other types of safety training and tools and did not realise the desired decrease in PSEs. Organisational initiatives that were initiated or ongoing during the simulation-based training included:

    • Mandatory attendance of a formal 3 h didactic error prevention training course by all clinical personnel (physicians, nurses, respiratory therapists, technicians, etc)

    • Institution of a ‘safety coaches’ programme associated with a behavioural observation tool and focused feedback

    • Surgical checklist

    • Institution of paediatric early warning system on inpatient units (classifying patients who are potentially unstable)

    • Participation in Ohio Children's Hospital Solutions for Patient Safety Collaborative

    • Initiative to reduce surgical site infections

    • Initiative to reduce adverse drug events

    • Initiative to reduce catheter related blood stream infections

    • Barcoding of medications

    • Global trigger tool adoption

    • Daily organisational safety briefings.

    We have now converted the entire didactic portion of this training to an interactive course based in the institutional intranet. This provides a precourse assessment of knowledge and incorporates videos and other tools to deliver the concepts of communication and teamwork. This intranet-based course must be completed prior to attending the training and is accredited for continuing education credits. The simulation-based portion of the training has been condensed to 4 h. Since the completion of the initial project an additional 19 courses have been offered and 152 additional ED staff have completed the condensed 4 h simulation-based training.

    Relationship of teamwork training, simulation and safety

    Vincent and colleagues describe four team factors that ultimately affect the risk of medical error: ‘verbal communication, written communication, supervision and seeking help, and structure of the team’.26 In emergency medicine, Risser et al have described a teamwork behaviour structure that delineates five major goals for improving teamwork and incorporates principles of aviation CRM. These goals were developed through analysis of medical errors and determining how teamwork failure contributed to these errors. The five major areas of interest are: ‘maintain team structure and climate, apply problem-solving strategies, communicate with the team, execute plans and manage workload and improve team skills’.27 Both of these frameworks describe elements frequently incorporated in teamwork training.

    Weaver et al reviewed the state of teamwork training in a 2010 article and identified 48 publications describing the use of teamwork training.28 The 2010 systematic review of Rabol et al on classroom-based teamwork training identified generally high staff satisfaction, but similar to the review of Weaver et al, a shortage of work that relates teamwork training to clinical outcomes and improvements in patient safety.29 This may be related to the absence of standardisation relative to this training and therefore an inability to define the components of teamwork training most likely to have a clinical effect.30 ,31

    We identified several factors that differentiate this project from many others using ‘teamwork training’. First, all ED personnel were required to participate in the initial training, as well as any individual hired into the ED since the project's completion. This ensures that all personnel have a similar foundation concerning behavioural expectations and attitudes related to communication and team behaviours. Second, the ED leadership has committed to, and simulation centre staff continue to provide, ongoing in situ simulations on all shifts. Though the primary purpose of these in situ simulations is related to identification of latent threats, they also provide an opportunity to reinforce teamwork and communication behaviours. We theorise that the continued lower rate (though not the elimination) of PSEs in the ED is due in large part to the requirement for all new staff to participate in simulation-based training and the ongoing programme of in situ simulations. We believe that the ongoing emphasis on continuous training has allowed the ED to sustain these gains. The other critical distinction of the teamwork training described in this intervention was the significant proportion of the training that was simulation-based.

    Simulation-based teamwork training has become more common over the last several years especially in critical care areas like EDs, intensive care units, labour and delivery and even frontline battlefield hospitals.32–38 Simulation-based training as described by Riley et al and Siassakos et al has demonstrated positive clinical impact in perinatal patient outcomes, as have Capella et al in trauma patients.32 ,39 ,40 The relationship of teamwork training, or more specifically the relationship of simulation-based teamwork training, to patient safety becomes fundamental. As described above, our institution had implemented a number of interventions directed at improving safety during the time this project was conducted. A substantial number of these interventions were directed at improving communication between healthcare providers, overcoming authority gradients and creating expectations of speaking up. However, none of these resulted in the desired change in clinical outcomes.

    In examining why simulation-based teamwork training succeeded in reducing the number of PSEs in this environment, which already placed a heavy emphasis on patient safety, several possibilities emerge. First, simulation-based training results in higher levels of physiological stress in the participants (as measured by heart rate and salivary cortisol) compared with didactic educational methods.41–43 Though it is not clear if these measures are comparable with those seen in actual clinical situations, clearly the participants are ‘activated’ and this experiential learning is more likely to be retained by the participant.

    Second, simulation-based learning (especially when used with video assisted debriefing) allows healthcare providers to participate in high risk situations, the execution of errors and (subsequently) the detection, mitigation and recovery from those errors. The participant's experience of participation and reflection, reinforced by the video debriefing, makes the potential to harm patients very real to the participants. Simulation-based training requires the participation of all providers in an active, experiential fashion. Scenarios are scripted to provoke certain kinds of responses, including personal experiences of loss of situation awareness, recognition that the individual team member is not likely to possess all necessary information, the importance of sharing information, and the need for the entire team to work together to overcome unanticipated difficulties and malfunctions. The intellectual and emotional experience of these scripted simulations contradicts the strongly held belief system of many healthcare providers that virtue and vigilance will prevent medical errors. Providers come to realise that every healthcare provider will make mistakes; it is a question of when not if. The importance of team functioning and cross checking is brought to the fore in these types of simulation scenarios in a way that didactic training cannot match. It is likely that simulation-based training, which includes equipment malfunctions, incomplete information and infrequent, high risk events, also contributes to the attitudes necessary to pursue ‘high reliability’. High reliability organisations are described as those that operate in unforgiving high risk settings, but have markedly fewer than expected failures. These organisations place a high priority on (among other things) heedful interrelating, sensitivity to operations, deference to expertise and preoccupation with failure and cultivation of resilience.44 Simulation-based training that incorporates infrequent and unexpected events, equipment malfunctions, and provider errors reinforces the importance of these concepts as well as providing the healthcare team the means to practice resilient behaviour.

    Finally, simulation scenarios that are ‘real enough’ in terms of the contextual cues provoke the participants to reflect and recognise clinical situations that are similar to the simulations and identify previously unrecognised conditions as threats to patient safety. During the debriefings, a substantial number of latent hazards related to the clinical environment were identified by participants. Some of these latent hazards were well known to one group (eg, nursing) but unknown to physicians. While this has previously been reported with respect to in situ simulation, it has not been reported with respect to simulation occurring in a simulation centre. This experience and the reflection associated with it promoted identification and recall of latent threats in the actual clinical environment.

    Overall, the emotional and behavioural engagement engendered by participation in simulation-based teamwork has the potential to change the participant's beliefs about the individual's and team's roles in patient safety and thereby lead to culture change within the microsystem.

    Limitations

    This intervention was associated with a decrease, but not the elimination of PSEs, in the target ED. Though it is tempting to attribute the decrease in the number of PSEs to the intervention described, it is impossible to do so definitively due to the number of potential confounders. These results should be interpreted with caution. Moving forward it becomes important to understand how or if the PSEs that did occur during the intervention could have been prevented by modifications to the intervention or alternative techniques. Replication of these results with a similar type of simulation-based teamwork training programme in other high risk clinical units would substantiate the behavioural and clinical effects of this type of training.

    The knowledge assessment used in this intervention was focused specifically on the local environment and not independently validated. Though this measure was believed to demonstrate face validity, formal psychometric evaluation was not performed. Test/retest reliability was not believed to be appropriate due to the intervention between baseline and postintervention measures and the interval between the postintervention and re-evaluation measures.

    The results did not demonstrate a measured change in behaviour in the simulated or clinical settings. While we theorise that alternative scales may have allowed us to detect a change, this is speculative. Replication of this intervention employing alternative scales would help us understand the strength of the behavioural changes.

    Finally, one of our initial outcomes was the assessment of change in behaviour during clinical patient care in our trauma bays. We were limited in this assessment due to changes in our ED's video recording system. Resuscitations of actual patients were not recorded for a portion of enrolment due to this change and after this change the recordings were unavailable for review for a period. This affected our capability of assessing behavioural changes in the clinical setting.

    Significance and implications

    This project contributes to our knowledge of the value of simulation-based multidisciplinary training to improve safety for our patients. Our institution has come to value this type of training for teams operating in high risk units. Simulation training was expanded to accommodate a substantial increase in this type of training in all critical care areas.

    As simulation-based training continues to develop, it is crucial that the factors that are most likely to result in sustained behaviour changes and reliable ways to measure these changes are identified. It is also critical that the optimum interval and method of reinforcing this training be clarified. We have used in situ simulation as a means to reinforce teamwork training, but there is little evidence as to the best method and optimal interval for retraining. Our experience also suggests that team training can be used to train for resilience, but research specifically addressing this is absent in healthcare simulation and remains one of the many unexplored areas of potential benefit in healthcare simulation.

    Acknowledgments

    The authors would like to thank our contributors Jerome Bauer, Jennifer Manos, Michael Moyer, Tiffany Pendergrass, Brian Pio and Regina Taylor and David Gothard for their work on this project. Additionally, we would like to thank the support staff at the Center for Simulation, leadership of Cincinnati Children's Hospital Medical Center, and the emergency department physicians, nurses, respiratory therapists, paramedics and patient care assistants for their dedication to providing safer care and for their assistance in the completion of this project.

    References

    1. ↵
      To err is human: building a safer health system. (Eds) Linda T Kohn, Janet M. Corrigan and Molla S. Donaldson, Washington, DC: National Academy Press, 2000.
    2. ↵
      1. Croskerry P,
      2. Wears RL,
      3. Binder LS
      . Setting the educational agenda and curriculum for error prevention in emergency medicine. Acad Emerg Med 2000;7:1194–200.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Chamberlain JM,
      2. Slonim A,
      3. Joseph JG
      . Reducing errors and promoting safety in pediatric emergency care. Ambul Pediatr 2004;4:55–63.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Miller MR,
      2. Pronovost PJ,
      3. Burstin HR
      . Pediatric patient safety in the ambulatory setting. Ambul Pediatr 2004;4:47–54.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Ghaleb MA,
      2. Barber N,
      3. Franklin BD,
      4. et al
      . The incidence and nature of prescribing and medication administration errors in paediatric inpatients. Arch Dis Child 2010;95:113–18.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Hostetler MA,
      2. Mace S,
      3. Brown K,
      4. et al
      . Emergency department overcrowding and children. Pediatr Emerg Care 2007;23:507–15.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Rinke ML,
      2. Moon M,
      3. Clark JS,
      4. et al
      . Prescribing errors in a pediatric emergency department. Pediatr Emerg Care 2008;24:1–8.
      OpenUrlPubMedWeb of Science
    8. ↵
      1. Fernandez CV,
      2. Gillis-Ring J
      . Strategies for the prevention of medical error in pediatrics. J Pediatr 2003;143: 155–62.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Kern T
      . Controlling PIlot Error: Culture, Environment and CRM. New York: McGraw-Hill, 2001.
    10. ↵
      1. Palomba CABT
      . Assessment essentials. first edn. San Francisco: Jossey-Bass, 1999.
    11. ↵
      1. Blum RH,
      2. Raemer DB,
      3. Carroll JS,
      4. et al
      . A method for measuring the effectiveness of simulation-based team training for improving communication skills. Anesth Analg 2005;100:1375–80.
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Sexton JB,
      2. Helmreich RL,
      3. Neilands TB,
      4. et al
      . The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res 2006;6:44–54.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Thomas EJ,
      2. Sexton JB,
      3. Helmreich RL
      . Translating teamwork behaviours from aviation to healthcare: development of behavioural markers for neonatal resuscitation. Qual Saf Health Care 2004;13(Suppl 1):i57–64.
      OpenUrlAbstract/FREE Full Text
    14. ↵
      1. Pronovost P,
      2. Sexton B
      . Assessing safety culture: guidelines and recommendations. Qual Saf Health Care 2005;14:231–3.
      OpenUrlFREE Full Text
    15. ↵
      1. Paine LA,
      2. Rosenstein BJ,
      3. Sexton JB,
      4. et al
      . Assessing and improving safety culture throughout an academic medical centre: a prospective cohort study. Qual Saf Health Care 2010;19:547–54.
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Thomas EJ,
      2. Sexton JB,
      3. Lasky RE,
      4. et al
      . Teamwork and quality during neonatal care in the delivery room. J Perinatol 2006;26:163–9.
      OpenUrlCrossRefPubMed
    17. ↵
      1. Thomas EJ,
      2. Taggart B,
      3. Crandell S,
      4. et al
      . Teaching teamwork during the Neonatal Resuscitation Program: a randomized trial. J Perinatol 2007;27:409–14.
      OpenUrlCrossRefPubMedWeb of Science
    18. ↵
      1. Thomas EJ,
      2. Williams AL,
      3. Reichman EF,
      4. et al
      . Team training in the neonatal resuscitation program for interns: teamwork and quality of resuscitations. Pediatrics 2010;125:539–46.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Williams AL,
      2. Lasky RE,
      3. Dannemiller JL,
      4. et al
      . Teamwork behaviours and errors during neonatal resuscitation. Qual Saf Health Care 2010;19:60–4.
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Fritz CO,
      2. Morris PE,
      3. Richler JJ
      . Effect size estimates: current use, calculations, and interpretation. J Exp Psychol Gen 2012;141:2–18.
      OpenUrlCrossRef
    21. ↵
      1. Burke CS,
      2. Salas E,
      3. Wilson-Donnelly K,
      4. et al
      . How to turn a team of experts into an expert medical team: guidance from the aviation and military communities. Qual Saf Health Care 2004;13(Suppl 1):i96–104.
      OpenUrlAbstract/FREE Full Text
    22. ↵
      1. Flin R,
      2. Maran N
      . Identifying and training non-technical skills for teams in acute medicine. Qual Saf Health Care 2004;13(Suppl 1):i80–4.
      OpenUrlAbstract/FREE Full Text
    23. ↵
      1. Flin R,
      2. Patey R
      . Non-technical skills for anaesthetists: developing and applying ANTS. Best Pract Res Clin Anaesthesiol 2011;25:215–27.
      OpenUrlCrossRefPubMed
    24. ↵
      1. Malec JF,
      2. Torsher LC,
      3. Dunn WF,
      4. et al
      . The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc 2007;2:4–10.
      OpenUrlPubMed
    25. ↵
      1. Pronovost P,
      2. Needham D,
      3. Berenholtz S,
      4. et al
      . An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med 2006;355:2725–32.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Vincent C,
      2. Taylor-Adams S,
      3. Stanhope N
      . Framework for analysing risk and safety in clinical medicine. BMJ 1998;316:1154–7.
      OpenUrlFREE Full Text
    27. ↵
      1. Risser DT,
      2. Rice MM,
      3. Salisbury ML,
      4. et al
      . The potential for improved teamwork to reduce medical errors in the emergency department. The MedTeams Research Consortium. Ann Emerg Med 1999;34:373–83.
      OpenUrlCrossRefPubMedWeb of Science
    28. ↵
      1. Weaver SJ,
      2. Lyons R,
      3. DiazGranados D,
      4. et al
      . The anatomy of health care team training and the state of practice: a critical review. Acad Med 2010;85:1746–60.
      OpenUrlCrossRefPubMedWeb of Science
    29. ↵
      1. Rabol LI,
      2. Ostergaard D,
      3. Mogensen T
      . Outcomes of classroom-based team training interventions for multiprofessional hospital staff. A systematic review. Qual Saf Health Care 2010;19:e27.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. Weaver SJ,
      2. Salas E,
      3. Lyons R,
      4. et al
      . Simulation-based team training at the sharp end: A qualitative study of simulation-based team training design, implementation, and evaluation in healthcare. J Emerg Trauma Shock 2010;3:369–77.
      OpenUrlCrossRefPubMed
    31. ↵
      1. Rosen MA,
      2. Weaver SJ,
      3. Lazzara EH,
      4. et al
      . Tools for evaluating team performance in simulation-based training. J Emerg Trauma Shock 2010;3:353–9.
      OpenUrlCrossRefPubMed
    32. ↵
      1. Capella J,
      2. Smith S,
      3. Philp A,
      4. et al
      . Teamwork training improves the clinical care of trauma patients. J Surg Educ 2010;67:439–43.
      OpenUrlCrossRefPubMedWeb of Science
    33. ↵
      1. Deering S,
      2. Rosen MA,
      3. Ludi V,
      4. et al
      . On the front lines of patient safety: implementation and evaluation of team training in Iraq. Jt Comm J Qual Patient Saf 2011;37:350–6.
      OpenUrlPubMed
    34. ↵
      1. Fernandez R,
      2. Vozenilek JA,
      3. Hegarty CB,
      4. et al
      . Developing expert medical teams: toward an evidence-based approach. Acad Emerg Med 2008;15:1025–36.
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      1. Frengley RW,
      2. Weller JM,
      3. Torrie J,
      4. et al
      . The effect of a simulation-based training intervention on the performance of established critical care unit teams. Crit Care Med 2011;39:2605–11.
      OpenUrlPubMedWeb of Science
    36. ↵
      1. Guise JM,
      2. Lowe NK,
      3. Deering S,
      4. et al
      . Mobile in situ obstetric emergency simulation and teamwork training to improve maternal-fetal safety in hospitals. Jt Comm J Qual Patient Saf 2010;36:443–53.
      OpenUrlPubMed
    37. ↵
      1. Rosen MA,
      2. Salas E,
      3. Wu TS,
      4. et al
      . Promoting teamwork: an event-based approach to simulation-based teamwork training for emergency medicine residents. Acad Emerg Med 2008;15:1190–8.
      OpenUrlCrossRefPubMedWeb of Science
    38. ↵
      1. Siassakos D,
      2. Fox R,
      3. Crofts JF,
      4. et al
      . The management of a simulated emergency: better teamwork, better performance. Resuscitation 2011;82:203–6.
      OpenUrlCrossRefPubMedWeb of Science
    39. ↵
      1. Riley W,
      2. Davis S,
      3. Miller K,
      4. et al
      . Didactic and simulation nontechnical skills team training to improve perinatal patient outcomes in a community hospital. Jt Comm J Qual Patient Saf 2011;37:357–64.
      OpenUrlPubMed
    40. ↵
      1. Siassakos D,
      2. Hasafa Z,
      3. Sibanda T,
      4. et al
      . Retrospective cohort study of diagnosis-delivery interval with umbilical cord prolapse: the effect of team training. BJOG 2009;116:1089–96.
      OpenUrlCrossRefPubMedWeb of Science
    41. ↵
      1. Wetzel CM,
      2. Black SA,
      3. Hanna GB,
      4. et al
      . The effects of stress and coping on surgical performance during simulations. Ann Surg 2010;251:171–6.
      OpenUrlCrossRefPubMed
    42. ↵
      1. Harvey A,
      2. Nathens AB,
      3. Bandiera G,
      4. et al
      . Threat and challenge: cognitive appraisal and stress responses in simulated trauma resuscitations. Med Educ 2010;44: 587–94.
      OpenUrlCrossRefPubMed
    43. ↵
      1. Bong CL,
      2. Lightdale JR,
      3. Fredette ME,
      4. et al
      . Effects of simulation versus traditional tutorial-based training on physiologic stress levels among clinicians: a pilot study. Simul Healthc 2010;5:272–8.
      OpenUrlCrossRefPubMed
    44. ↵
      1. Sutcliffe KM
      . High reliability organizations (HROs). Best Pract Res Clin Anaesthesiol 2011;25:133–44.
      OpenUrlCrossRefPubMed

    Supplementary materials

    Footnotes

    • Contributors MP, TL and RW made substantial contributions to conception and design of the current project. MP, GG, TL and RW made substantial contributions to acquisition of data and/or the interpretation and analysis of data. MP, GG, TL and RW were involved in the drafting and/or revision of the article and approved the final version.

    • Funding The principal investigator and the research team gratefully acknowledge the support of this project from the Agency for Healthcare Research and Quality. Grant No. 5U18HS015841-02.

    • Competing interests None.

    • Ethics approval Cincinnati Children's Institutional Review Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.