Article Text

Download PDFPDF

Impact of multidisciplinary simulation-based training on patient safety in a paediatric emergency department
  1. Mary D Patterson1,2,
  2. Gary L Geis1,3,4,
  3. Thomas LeMaster1,
  4. Robert L Wears5,6
  1. 1The Center for Simulation and Research, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
  2. 2Akron Children's Hospital Simulation Center for Safety and Reliability, Akron, Ohio, USA
  3. 3Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
  4. 4Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
  5. 5Department of Emergency Medicine, University of Florida, Jacksonville, Florida, USA
  6. 6Clinical Safety Research Unit Imperial College, London, UK
  1. Correspondence to Dr Mary D Patterson, Akron Children's Simulation Center for Safety and Reliability Akron Children's Hospital One Perkins Square Akron, OH 44308 USA;mpatterson{at}


Background Cincinnati Children's Hospital is one of the busiest paediatric emergency departments (ED) in the USA; high volume, high acuity and frequent interruptions contribute to an increased risk for error.

Objective To improve patient safety in a paediatric ED by implementing a multidisciplinary, simulation-based curriculum emphasising teamwork and communication.

Methods Subjects included all healthcare providers in the ED. Multidisciplinary teams participated in simulation-based training focused on teamwork and communication behaviours in critical clinical scenarios. The Safety Attitudes Questionnaire, tests of knowledge and evaluations of critical simulations and actual performance in the ED resuscitation bay were assessed. Methods to sustain improvements included mandatory participation of all new staff in simulation-based training and the introduction of routine in situ simulations.

Results 289 participants attended the initial training. 151 participants attended the re-evaluation at a mean of 10.2 months later. Sustained improvements in knowledge and attitudes were demonstrated. Knowledge tests at baseline, postintervention and re-evaluation had scores of 86%, 96% and 93%, respectively. Friedman's test analysis of SAQ scores at baseline, postintervention and re-evaluation indicated significant attitude changes. The ED with a preintervention baseline of 2–3 patient safety events per year has now sustained more than 1000 days without a patient safety event. This improvement occurred even though the time required in initial simulation training has been condensed from 12 to 4 h.

Conclusions Simulation training is an effective tool to modify safety attitudes and teamwork behaviours in an ED. Sustaining cultural and behavioural changes requires repeated practice opportunities.

  • Emergency department
  • Simulation
  • Communication
  • Crew resource management
  • Team training
View Full Text

Statistics from

Background and outline of problem

The 1999 Institute of Medicine report, To Err is Human, identified the emergency department (ED) as the area of the hospital with the highest risk of adverse events.1 The simultaneous care of multiple patients with varying acuity levels, frequent interruptions, time constraints, high volume and the need to institute critical interventions with limited information contribute to an increased risk for medical error in the emergency setting. In one study of closed ED claims, teamwork failures accounted for ‘more than 40% of these cases’.2

There is little information on the magnitude of medical error in paediatric emergency medicine.3–7 As patients, children present unique challenges including anatomical, physiological and pharmacological differences and the child's relative inability to communicate with the healthcare provider. These contribute to the risk for medical error in paediatric patients.8 It is generally conceded that the error rate in paediatrics is likely to be higher than in adult patients.

As part of an ED needs assessment, the principal investigator conducted a small qualitative project, approved by the Institutional Review Board, in November of 2003. Indepth interviews and field observation were used to examine ED staff perceptions regarding patient safety. Experienced staff in our ED consistently identified communication, individual responsibility and inexperience as issues for patient safety. A review of patient safety events (PSEs) for the 5 years prior to this project also identified these issues. The institution has defined a PSE as an event where there was patient harm following less than ideal medical management. Twelve PSEs involving the ED were identified during this time. Communication issues were identified as a fundamental cause in nine of these and training issues were identified in 10 of these. Eight of these events involved patients in the resuscitation bays.

Specific aims

The overall goal of this project was to decrease the frequency and mitigate the effects of medical error in a paediatric ED by implementing standardised multidisciplinary, simulation-based training that emphasised team behaviours.

The specific aims of this project were:

  • To implement a multidisciplinary simulation-based training that encompassed crew resource management (CRM), teamwork behaviours and critical communication skills

  • To evaluate the effectiveness of this training by assessing knowledge of and attitudes towards patient safety and by assessing teamwork behaviours among ED caregivers prior to and following this intervention.

We also planned to track PSEs though we did not anticipate a change in the rate of PSEs during the project.

Setting and function

The Cincinnati Children's Hospital Emergency Department is a Level I trauma centre with an annual census of >90 000 patient visits. The volume and complexity of the patient population represent huge risks for patient safety. The hospital is located in an urban centre and provides care to patients with the entire gamut of illness and injury, for example, ear infections, complex medically fragile patients, patients with acute onset of sepsis and those suffering acute traumatic injury. Approximately 3000 patients annually are assessed as critical and immediately triaged to the resuscitation bays.

Cincinnati Children's Hospital Medical Center (CCHMC) is a teaching institution and provides paediatric education to residents from our own and other institutions. Approximately 45–55 residents and 12 fellows rotate through our ED monthly. Supervision of all trainees is provided by board-certified paediatric emergency physicians.


The purpose of this project was to adapt the principles of CRM to standardised simulation-based teamwork training for paediatric ED personnel. The training focused on the development of teamwork and communication skills and the use of simulation to provide deliberate practice of these skills in a simulated high risk, time pressured critical environment.9

The project was evaluated by the institutional review board and granted exempt status as an educational intervention. Confidentiality statements and video consents were obtained from all participants. While all ED providers were required to participate in the training, they were not required to complete the knowledge test or attitude survey.


Participants were ED direct care providers: faculty and staff physicians, nurses, respiratory therapists, paramedics, patient care assistants and medical residents. It was required that all disciplines be represented in each session with the exception of respiratory therapists, who are not always present in the ED. Individuals participating were frequently acquainted with one another, but intact teams from a particular shift were not trained as a group.


The training was developed based on analysis of the previous 5 years of PSEs and in consultation with a patient safety expert (RW). The intervention was conducted with teams of approximately 8–10 ED personnel and was designed to replicate the teams that care for critical patients. Each group completed the intervention as a multidisciplinary team. The initial intervention took place over 2 days and required approximately 12 h to complete. Prior to the training, participants reviewed an intranet-based presentation incorporating information on the magnitude of risk from medical error, error theory and principles of CRM.

The overall goals of the intervention were that ED personnel would demonstrate increased awareness and knowledge of patient safety. The safety culture would reflect personnel investment in patient safety and reducing medical errors. In a simulated clinical setting, ED personnel would demonstrate the ability to recognise high risk situations and use the acquired skills to prevent or decrease the impact of unexpected events and errors. The intervention included four primary participant goals and competencies which were based on Bloom's taxonomy matrix: knowledge; comprehension; application; analysis and evaluation and synthesis.10 The four primary goals of the intervention were that participants:

  1. Demonstrate knowledge and awareness of high risk situations and factors leading to medical errors in the paediatric ED as demonstrated by a score of 80% or better on a written test.

  2. Demonstrate knowledge of methods that decrease patient risk in a simulated patient environment as determined by an acceptable performance in this setting.

  3. Demonstrate critical communication and teamwork skills necessary to prevent, trap and mitigate the effects of unexpected events and errors in a simulated patient environment.

  4. Exhibit a positive attitude towards and personal investment in patient safety as measured by a Safety Attitudes Questionnaire (SAQ) assessment scale (see online supplementary appendix for complete description of curriculum constructs).

During training, techniques to prevent medical error, improve critical communications, increase situation awareness, develop resilience, and improve sharing of mental models and closed loop communication were introduced with mini-lectures, targeted review of videotapes of simulated resuscitations and case studies in healthcare and aviation. The major elements of the curriculum and the order of presentation are described in table 1.

Table 1

Patient safety curriculum elements and order of delivery

The major focus of training was devoted to standardised high risk medical and trauma scenarios utilising a manikin simulator (table 2). A total of five simulations, followed by immediate video assisted debriefing, were presented during the intervention. The development of skills that promote resilience was supported through the utilisation of simulations that incorporated a scripted medical error, adverse event, unexpected equipment malfunction, or failure to allow personnel to detect and mitigate the effects of errors and unexpected events. These types of simulations encouraged the team's practice of the skills needed to solve unexpected failures in time pressured environments. In particular, these simulations focused not on the prevention of error, but on the detection, trapping and mitigation of the effects of error and system failure and the stabilisation of the patient and the system. Rather than retrospectively analysing examples of resilience (or the absence of resilience), the teams participated in reproducible simulated scenarios that encouraged them to develop an approach to the management of unexpected and adverse events. Other scenarios required non-physicians or less experienced physicians to assert themselves with team leaders. Probe methodology in which only one team member is aware of a critical aspect of patient information was also used to assess information sharing, assertiveness and closed loop communication.11 Debriefing, by MP, GG or TL, included group assessment of team performance as well as a guided review of the simulation video. Participants were asked to identify and suggest solutions to the challenges encountered.

Table 2

Scenarios utilised in simulation training

At the beginning of each training session, the SAQ Teamwork and Safety Climate version12 was used to assess attitudes toward and ownership of patient safety by each individual. A pretest assessing knowledge of patient safety issues was administered to all participants. All subjects were oriented to the simulator and to the simulation environment. Each team was then videotaped in a simulated clinical scenario to assess baseline team function with respect to teamwork and utilisation of crew resource principles during a critical situation (baseline).

A simulated clinical scenario was also videotaped at the completion of the intervention (postintervention). The baseline simulation was in addition to the five simulations presented during the intervention. Following the training, the SAQ and a parallel knowledge assessment were utilised as post-tests to examine changes in attitude and knowledge. The post-test for knowledge and the postintervention simulation were similar in form and degree of difficulty but not identical to the preintervention assessments.

Multidisciplinary teams of ED personnel were asked to return for re-evaluation at approximately 6 months following the intervention to repeat a parallel knowledge test, a SAQ, and to be videotaped and debriefed on several scenarios over a 2–3 h time frame. Teams returning for re-evaluation were also designed to replicate the multidisciplinary team caring for critical patients in the ED. Though the teams that trained together in the initial intervention did not return intact for the re-evaluation, all the roles were represented in the re-evaluation teams. The re-evaluation allowed assessment of the persistence of knowledge and attitude changes. Simulation ‘practice’ also served to reinforce the desired teamwork behaviours.

Actual paediatric resuscitations in the ED resuscitation bay were video recorded and compared with preintervention and postintervention video recordings to assess performance, teamwork and incorporation of CRM principles in actual patient care. A modified University of Texas Behavioral Markers for Neonatal Resuscitation Scale (more recent teamwork scales were not published at the time) was used to assess teamwork skills in the initial (baseline) and postintervention simulations, as well as teamwork behaviours in the resuscitation bay.13

Data sources/collection


The patient safety knowledge assessments were developed by the authors to evaluate participants’ understanding and retention of key concepts presented in the prework and during the intervention itself. Parallel forms of the assessment were administered at the beginning of the intervention, at the completion of the intervention and at the re-evaluation (all three versions are available in the online supplementary appendix). The post-test and re-evaluation knowledge assessments were similar in form and degree of difficulty but not identical to the preintervention assessment. The knowledge assessment was focused on concepts previously identified as challenging in our ED environment. Certain questions that were believed to be highly relevant to our ED were included in all three forms of the assessment. The authors agreed that the final knowledge assessments have high face validity for our environment.

The SAQ Teamwork and Safety Climate version (see online supplementary appendix) was developed from the Flight Management Attitudes Questionnaire, a tool used to assess flight crew's attitudes regarding: teamwork, speaking up, communication, collaboration decision making and leadership.12 The Teamwork and Safety Climate version uses a 5-point Likert scale, and is comprised of 14 items assessing teamwork climate and 13 items assessing safety climate. Some items are reverse (or negatively) worded. The survey has been validated with more than 10 000 healthcare providers in more than 200 clinical units.12 The SAQ has been demonstrated to have test and retest reliability of 0.85–0.90 and Cronbach α values of 0.75–0.88.12 ,14 A minimum acceptable safety culture score on the SAQ has been set at 60% with a ‘goal’ of 80%–100%.15

A modification of the Behavioral Markers for Neonatal Resuscitation Scale13 (see online supplementary appendix) was used to assess teamwork and communication in the simulated and clinical setting. This tool was based on the Line Operations Safety Audit, a tool used in aviation to assess cockpit teamwork and CRM behaviours during more than 5000 flights. The Behavioral Markers scale was developed to assess teamwork and communication skills during neonatal resuscitations. At the time this intervention began, it was one of the few existing teamwork behaviour scales and the domains addressed in the scale are not specific to neonatal resuscitations. This scale has the advantage of prior use in a number of projects and explicit guidance detailing its use.16–19 Inter-rater agreement for observation of the various behaviours has ranged from fair to good for most behaviours (κ of 0.41–0.80) with the exceptions of workload management, vigilance and leadership for which inter-rater agreement was slight (κ=0.21–0.41)16 in previous studies; teams that scored higher in vigilance and information sharing on this scale made proportionally fewer errors during actual neonatal resuscitations than those with lower scores in these areas.19 Performance on this scale has also been weakly but positively correlated with team compliance with neonatal resuscitation programme standards.16 These results seem to indicate good content and construct validity and fair to good reliability.


The SAQ was analysed using the Friedman's test (for ordinal data). Knowledge assessments were analysed with repeated samples Analysis of Variance (ANOVA). The Behavioral Markers scale was used to analyse simulated clinical scenarios as well as videos recorded in the actual ED resuscitation bay. This provided us with means to compare teamwork behaviours preintervention and postintervention. These were analysed using a Friedman's test.

Pretest and post-test results of patient safety knowledge and the SAQ were collected during the training intervention. These were again measured at the re-evaluation. This information was entered in an SPSS database.

Five paediatricians with medical education interest and expertise were trained as behavioural raters. These paediatricians were not associated with the ED and were blinded as to whether the video was baseline or postintervention. They provided a consistent core of raters.

All resuscitations in the EDs shock trauma bays are videotaped for performance/quality improvement. To evaluate the transfer of teamwork behaviours from the simulated setting to the actual patient care setting of our bays, we planned to analyse 15%–20% of these resuscitations using the same behavioural scale and the same raters as in our evaluations of simulation scenarios.


A total of 289 individuals participated in the initial simulation-based training. Overall, 33 classes were conducted from March of 2005 through March of 2008. In all, 151 participants attended a re-evaluation during 36 re-evaluation classes. The significant difference in those that attended an initial intervention but not a re-evaluation was largely due to staff turnover.

In all, 77% of the participants were female and almost 50% were nurses.

The average number of years of experience was 5.5 with a range of 0–30 years. Approximately 50% of participants had 3 or fewer years of experience. The proportion of individuals participating in the initial intervention and the re-evaluation by role is shown in table 3.

Table 3

Percentage of participants by role in initial intervention and re-evaluation


Tests of knowledge were conducted at baseline, immediately after the intervention and at the re-evaluation (table 4). The re-evaluations occurred at a mean of 10.2 (SD 4.8) months after the initial intervention. A repeated measures ANOVA with a Greenhouse-Geisser correction determined that the mean test scores differed significantly between time points. (F (1.77, 183.99)=114.63, p< 0.001). Post hoc tests using the Bonferroni correction revealed that there were significant differences between baseline and postintervention scores as well as between baseline and re-evaluation scores (p<0.001 for both comparisons). Cohen's d was calculated to estimate the magnitude of effects. The value of 1.0 from preintervention to postintervention indicates a large effect size while the effect size from postintervention to re-evaluation is 0.42 indicating a moderate effect. There is a statistically significant difference between the postintervention and re-evaluation scores as calculated by a paired samples Student t test with p<0.001. This indicates a significant loss of knowledge at the re-evaluation compared with immediate postintervention scores. However, the level of knowledge at the re-evaluation is still significantly higher than at the baseline evaluation. This indicates a sustained improvement in knowledge scores over baseline despite a 10-month interval between the intervention and the re-evaluation.

Table 4

Results of knowledge and attitude assessment


The SAQ was also administered at baseline, postintervention and at the re-evaluation and was analysed using the Friedman's test (table 4). There was a statistical difference in attitudes over time for teamwork, climate and overall: Χ2(2)=16.09, 26.33 and 16.32, respectively (p<0.001 for all three categories). Post hoc analysis was conducted using the Wilcoxan signed ranks test with a Bonferroni correction applied for multiple comparisons, resulting in a significance level of p<0.017. The comparison of the preintervention and postintervention results indicated Z values of −4.9, −2.96 and −3.47 for climate, teamwork and overall. All three values were statistically improved following the intervention (p<0.01). Postintervention and re-evaluation scores were also compared. The Z values were −0.85, −0.46 and −0.46 for climate, teamwork and overall, respectively. This indicated that the attitude changes seen following the intervention were not significantly diminished at the time of re-evaluation (p>0.017).

A Cohen's d was also calculated to estimate the magnitude of effect for the SAQ (table 4). The calculated values for effect size for preintervention to postintervention of the SAQ ranged from 0.30 to 0.38, indicating a small to medium effect size (d>0.2 and <0.5).20 The effect size calculations from postintervention to re-evaluation were all < 0.10 indicating a minimal effect on attitude change in that interval.


In all, 45 pairs of videos were reviewed by five reviewers to assess changes in teamwork behaviours in the simulated setting from baseline to postintervention. An additional 35 videos from the re-evaluation phase were also evaluated. Reviewers were blinded as to whether the video was recorded at baseline or postintervention. Two reviewers independently rated each video. A modified Behavioral Markers for Neonatal Resuscitation Scale was used to evaluate observability of the behavioural markers and the frequency of these behaviours. These scores were analysed using the Friedman's test to compare the baseline, postintervention and re-evaluation scores. The median observability scores at baseline, postintervention and re-evaluation scores were 3.6, 3.4 and 3.4, respectively. The Χ2(2)=2.0, p>0.05. The median frequency scores at baseline, postintervention and re-evaluation were 3.6, 3.0, and 3.0, respectively; Χ2(2)=4.0, p>0.05. To assess concordance of rating by the video reviewers the eta coefficient (nominal by ordinal data) was calculated for 14 ‘control videos’. The eta coefficient was between 0.62 and 1.0 for the sample of videos reviewed. This indicated a moderate rate of concordance between the reviewers. Only 12 videos from the ED resuscitation bay were reviewed. This was due to technical difficulties associated with a change in the video recording system in the ED during enrolment.

Participant experience

CCHMC participants ranked the value of this training highly. Day 1 evaluations had an overall score of 4.6 on a 5-point Likert scale, with a median score of 4.7. The day 2 evaluations had an overall score of 4.8 on a 5-point Likert scale, with a median score of 5.

Unexpected results

Reduction of PSEs in the ED

At the time of the implementation of this training the ED had experienced 12 PSEs during the previous 5 years. The entire institution had embarked on a strategic effort to reduce PSEs. Multiple interventions were employed throughout the institution in an effort to meet this goal, including mandatory 3 h error prevention classroom training and yearly online safety modules for all healthcare providers. At this time, the ED was the only clinical unit utilising multidisciplinary simulation-based training. During the 2555 days since the beginning of this project, only two PSEs have occurred in the ED with the longest interval between PSEs being 1425 days. In the two PSEs that occurred during this time frame, each patient was cared for by several services and each involved a higher than usual number of handoffs of care. Though these factors are known risks for patient safety, this again speaks to the reality that safety is a dynamic state and not one that is ever fully achieved. During the initial 3 years of this project, the institution as whole did not experience a decrease in the rate of PSEs despite other interventions aimed at reducing PSEs.



Our goal was to demonstrate the effectiveness of simulation-based training in improving patient safety. The findings demonstrate this training resulted in improvements in knowledge, safety attitudes and changes in behaviour in the simulated setting. We were able to demonstrate significant and sustained improvements in knowledge of and attitudes toward communication and teamwork behaviours. Challenging aspects of this project include the different educational levels and functions of the learners as well as the ‘built-in’ medical hierarchy. Though paediatrics is perhaps less hierarchical than some other medical specialties, there is still discomfort or reluctance on the part of less influential personnel to assert their concerns. We addressed this in several ways. First, in the simulation lab we oriented all participants to the ‘rules of engagement.’ These rules mandate respect for all team members. We intentionally flattened the hierarchy during training by using first names for all participants. We used video recordings to illustrate certain concepts and to provide examples of positive and negative teamwork behaviours. In this setting, observable communication and teamwork behaviours were the primary goal and evaluation focus. Second, the intervention included training in respectful assertion and voicing of concerns (deference to expertise, advocacy and inquiry) and closed loop communication as well as opportunities for less powerful team members to practice these skills in a simulated setting. Burke et al have stated,‘ does no good if team members cross check and monitor one another's actions if when something out of the ordinary is recognized no one speaks up due to fear of reprimand.’21

The video reviews did not demonstrate a significant change in behaviour as measured by the Behavioral Markers of Neonatal Resuscitation Scale. It is feasible that the behavioural markers scale used was not sensitive to the changes in behaviour that occurred. Since the initiation of this study, additional scales of teamwork behaviour have been validated and published including the ANTS and Mayo scales.22–24 These scales include assessment of behaviours that are not evaluated in the Behavioral Markers of Neonatal Resuscitation Scale. For example, the Mayo scale includes elements specifically assessing cross monitoring by team members, involvement of all team members, role clarity, closed loop communication, assertive statements and situation awareness.24 While communication and vigilance are included in a general way in the Behavioral Markers of Neonatal Resuscitation Scale, the description of these elements is much less specific than the description in the Mayo scale. Given the emphasis of the current intervention on behaviours such as closed loop communication, cross monitoring, assertive statements, maintaining situation awareness and the utilisation of all team resources, it is possible that had an alternative scale (specifically including these elements) been used as the assessment tool, a change in behaviour would have been detected in the video reviews.

Pronovost et al reported that change in attitude was most predictive of intensive care units personnel's ability to decrease bloodstream infections.25 There has been a substantial and sustained change in the number of PSEs in the clinical environment of our ED and ED personnel report anecdotally that there is a change in communication and behaviour. This suggests that the sustained improvement in the rate of PSEs is likely associated with the change in attitudes.

The decrease in PSEs during the initial period of this project was unexpected. We believed that changing the culture in a large urban ED would require a number of years. Though it is difficult to definitely ascribe the decrease in PSEs to this intervention alone, it is important to note that the ED was the only clinical unit employing simulation-based training at a time of significant focus and effort by the institution on reducing PSEs across the organisation. During this time, the hospital initiated a significant number of programmes employing other types of safety training and tools and did not realise the desired decrease in PSEs. Organisational initiatives that were initiated or ongoing during the simulation-based training included:

  • Mandatory attendance of a formal 3 h didactic error prevention training course by all clinical personnel (physicians, nurses, respiratory therapists, technicians, etc)

  • Institution of a ‘safety coaches’ programme associated with a behavioural observation tool and focused feedback

  • Surgical checklist

  • Institution of paediatric early warning system on inpatient units (classifying patients who are potentially unstable)

  • Participation in Ohio Children's Hospital Solutions for Patient Safety Collaborative

  • Initiative to reduce surgical site infections

  • Initiative to reduce adverse drug events

  • Initiative to reduce catheter related blood stream infections

  • Barcoding of medications

  • Global trigger tool adoption

  • Daily organisational safety briefings.

We have now converted the entire didactic portion of this training to an interactive course based in the institutional intranet. This provides a precourse assessment of knowledge and incorporates videos and other tools to deliver the concepts of communication and teamwork. This intranet-based course must be completed prior to attending the training and is accredited for continuing education credits. The simulation-based portion of the training has been condensed to 4 h. Since the completion of the initial project an additional 19 courses have been offered and 152 additional ED staff have completed the condensed 4 h simulation-based training.

Relationship of teamwork training, simulation and safety

Vincent and colleagues describe four team factors that ultimately affect the risk of medical error: ‘verbal communication, written communication, supervision and seeking help, and structure of the team’.26 In emergency medicine, Risser et al have described a teamwork behaviour structure that delineates five major goals for improving teamwork and incorporates principles of aviation CRM. These goals were developed through analysis of medical errors and determining how teamwork failure contributed to these errors. The five major areas of interest are: ‘maintain team structure and climate, apply problem-solving strategies, communicate with the team, execute plans and manage workload and improve team skills’.27 Both of these frameworks describe elements frequently incorporated in teamwork training.

Weaver et al reviewed the state of teamwork training in a 2010 article and identified 48 publications describing the use of teamwork training.28 The 2010 systematic review of Rabol et al on classroom-based teamwork training identified generally high staff satisfaction, but similar to the review of Weaver et al, a shortage of work that relates teamwork training to clinical outcomes and improvements in patient safety.29 This may be related to the absence of standardisation relative to this training and therefore an inability to define the components of teamwork training most likely to have a clinical effect.30 ,31

We identified several factors that differentiate this project from many others using ‘teamwork training’. First, all ED personnel were required to participate in the initial training, as well as any individual hired into the ED since the project's completion. This ensures that all personnel have a similar foundation concerning behavioural expectations and attitudes related to communication and team behaviours. Second, the ED leadership has committed to, and simulation centre staff continue to provide, ongoing in situ simulations on all shifts. Though the primary purpose of these in situ simulations is related to identification of latent threats, they also provide an opportunity to reinforce teamwork and communication behaviours. We theorise that the continued lower rate (though not the elimination) of PSEs in the ED is due in large part to the requirement for all new staff to participate in simulation-based training and the ongoing programme of in situ simulations. We believe that the ongoing emphasis on continuous training has allowed the ED to sustain these gains. The other critical distinction of the teamwork training described in this intervention was the significant proportion of the training that was simulation-based.

Simulation-based teamwork training has become more common over the last several years especially in critical care areas like EDs, intensive care units, labour and delivery and even frontline battlefield hospitals.32–38 Simulation-based training as described by Riley et al and Siassakos et al has demonstrated positive clinical impact in perinatal patient outcomes, as have Capella et al in trauma patients.32 ,39 ,40 The relationship of teamwork training, or more specifically the relationship of simulation-based teamwork training, to patient safety becomes fundamental. As described above, our institution had implemented a number of interventions directed at improving safety during the time this project was conducted. A substantial number of these interventions were directed at improving communication between healthcare providers, overcoming authority gradients and creating expectations of speaking up. However, none of these resulted in the desired change in clinical outcomes.

In examining why simulation-based teamwork training succeeded in reducing the number of PSEs in this environment, which already placed a heavy emphasis on patient safety, several possibilities emerge. First, simulation-based training results in higher levels of physiological stress in the participants (as measured by heart rate and salivary cortisol) compared with didactic educational methods.41–43 Though it is not clear if these measures are comparable with those seen in actual clinical situations, clearly the participants are ‘activated’ and this experiential learning is more likely to be retained by the participant.

Second, simulation-based learning (especially when used with video assisted debriefing) allows healthcare providers to participate in high risk situations, the execution of errors and (subsequently) the detection, mitigation and recovery from those errors. The participant's experience of participation and reflection, reinforced by the video debriefing, makes the potential to harm patients very real to the participants. Simulation-based training requires the participation of all providers in an active, experiential fashion. Scenarios are scripted to provoke certain kinds of responses, including personal experiences of loss of situation awareness, recognition that the individual team member is not likely to possess all necessary information, the importance of sharing information, and the need for the entire team to work together to overcome unanticipated difficulties and malfunctions. The intellectual and emotional experience of these scripted simulations contradicts the strongly held belief system of many healthcare providers that virtue and vigilance will prevent medical errors. Providers come to realise that every healthcare provider will make mistakes; it is a question of when not if. The importance of team functioning and cross checking is brought to the fore in these types of simulation scenarios in a way that didactic training cannot match. It is likely that simulation-based training, which includes equipment malfunctions, incomplete information and infrequent, high risk events, also contributes to the attitudes necessary to pursue ‘high reliability’. High reliability organisations are described as those that operate in unforgiving high risk settings, but have markedly fewer than expected failures. These organisations place a high priority on (among other things) heedful interrelating, sensitivity to operations, deference to expertise and preoccupation with failure and cultivation of resilience.44 Simulation-based training that incorporates infrequent and unexpected events, equipment malfunctions, and provider errors reinforces the importance of these concepts as well as providing the healthcare team the means to practice resilient behaviour.

Finally, simulation scenarios that are ‘real enough’ in terms of the contextual cues provoke the participants to reflect and recognise clinical situations that are similar to the simulations and identify previously unrecognised conditions as threats to patient safety. During the debriefings, a substantial number of latent hazards related to the clinical environment were identified by participants. Some of these latent hazards were well known to one group (eg, nursing) but unknown to physicians. While this has previously been reported with respect to in situ simulation, it has not been reported with respect to simulation occurring in a simulation centre. This experience and the reflection associated with it promoted identification and recall of latent threats in the actual clinical environment.

Overall, the emotional and behavioural engagement engendered by participation in simulation-based teamwork has the potential to change the participant's beliefs about the individual's and team's roles in patient safety and thereby lead to culture change within the microsystem.


This intervention was associated with a decrease, but not the elimination of PSEs, in the target ED. Though it is tempting to attribute the decrease in the number of PSEs to the intervention described, it is impossible to do so definitively due to the number of potential confounders. These results should be interpreted with caution. Moving forward it becomes important to understand how or if the PSEs that did occur during the intervention could have been prevented by modifications to the intervention or alternative techniques. Replication of these results with a similar type of simulation-based teamwork training programme in other high risk clinical units would substantiate the behavioural and clinical effects of this type of training.

The knowledge assessment used in this intervention was focused specifically on the local environment and not independently validated. Though this measure was believed to demonstrate face validity, formal psychometric evaluation was not performed. Test/retest reliability was not believed to be appropriate due to the intervention between baseline and postintervention measures and the interval between the postintervention and re-evaluation measures.

The results did not demonstrate a measured change in behaviour in the simulated or clinical settings. While we theorise that alternative scales may have allowed us to detect a change, this is speculative. Replication of this intervention employing alternative scales would help us understand the strength of the behavioural changes.

Finally, one of our initial outcomes was the assessment of change in behaviour during clinical patient care in our trauma bays. We were limited in this assessment due to changes in our ED's video recording system. Resuscitations of actual patients were not recorded for a portion of enrolment due to this change and after this change the recordings were unavailable for review for a period. This affected our capability of assessing behavioural changes in the clinical setting.

Significance and implications

This project contributes to our knowledge of the value of simulation-based multidisciplinary training to improve safety for our patients. Our institution has come to value this type of training for teams operating in high risk units. Simulation training was expanded to accommodate a substantial increase in this type of training in all critical care areas.

As simulation-based training continues to develop, it is crucial that the factors that are most likely to result in sustained behaviour changes and reliable ways to measure these changes are identified. It is also critical that the optimum interval and method of reinforcing this training be clarified. We have used in situ simulation as a means to reinforce teamwork training, but there is little evidence as to the best method and optimal interval for retraining. Our experience also suggests that team training can be used to train for resilience, but research specifically addressing this is absent in healthcare simulation and remains one of the many unexplored areas of potential benefit in healthcare simulation.


The authors would like to thank our contributors Jerome Bauer, Jennifer Manos, Michael Moyer, Tiffany Pendergrass, Brian Pio and Regina Taylor and David Gothard for their work on this project. Additionally, we would like to thank the support staff at the Center for Simulation, leadership of Cincinnati Children's Hospital Medical Center, and the emergency department physicians, nurses, respiratory therapists, paramedics and patient care assistants for their dedication to providing safer care and for their assistance in the completion of this project.


View Abstract

Supplementary materials


  • Contributors MP, TL and RW made substantial contributions to conception and design of the current project. MP, GG, TL and RW made substantial contributions to acquisition of data and/or the interpretation and analysis of data. MP, GG, TL and RW were involved in the drafting and/or revision of the article and approved the final version.

  • Funding The principal investigator and the research team gratefully acknowledge the support of this project from the Agency for Healthcare Research and Quality. Grant No. 5U18HS015841-02.

  • Competing interests None.

  • Ethics approval Cincinnati Children's Institutional Review Board.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.