You are here

Article Text

Article menu
Original research
In situ simulation: detection of safety threats and teamwork training in a high risk emergency department
Free
  1. Mary D Patterson1,2,
  2. Gary Lee Geis1,3,4,
  3. Richard A Falcone5,
  4. Thomas LeMaster1,
  5. Robert L Wears6,7
  1. 1The Center for Simulation and Research, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
  2. 2Akron Children's Hospital Simulation Center for Safety and Reliability, Akron Children's Hospital, Akron, Ohio, USA
  3. 3Department of Pediatrics, University of Cincinnati College of Medicine, Cincinnati, Ohio, USA
  4. 4Division of Emergency Medicine, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
  5. 5Division of Pediatric General and Thoracic Surgery, , Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio, USA
  6. 6Department of Emergency Medicine, University of Florida, Jacksonville, Florida, USA
  7. 7Clinical Safety Research Unit, Imperial College, London, UK, UK
  1. Correspondence to Dr Mary D Patterson, Akron Children's Simulation Center for Safety and Reliability, Akron Children's Hospital, One Perkins Square, Akron, OH 44308, USA; mpatterson{at}chmca.org

Abstract

Objective Implement and demonstrate feasibility of in situ simulations to identify latent safety threats (LSTs) at a higher rate than lab-based training, and reinforce teamwork training in a paediatric emergency department (ED).

Methods Multidisciplinary healthcare providers responded to critical simulated patients in an urban ED during all shifts. Unannounced in situ simulations were limited to 10 min of simulation and 10 min of debriefing, and were video recorded. A standardised debriefing template was used to assess LSTs. The primary outcome measure was the number and type of LSTs identified during the simulations. Secondary measures included: participants’ assessment of impact on patient care and value to participants. Blinded video review using a modified Anaesthetists Non-Technical Skills scale was used to assess team behaviours.

Results 218 healthcare providers responded to 90 in situ simulations conducted over 1 year. A total of 73 LSTs were identified; a rate of one every 1.2 simulations performed. In situ simulations were cancelled at a rate of 28% initially, but the cancellation rate decreased as training matured. Examples of threats identified include malfunctioning equipment and knowledge gaps concerning role responsibilities. 78% of participants rated the simulations as extremely valuable or valuable, while only 5% rated the simulation as having little or no value. Of those responding to a postsimulation survey, 77% reported little or no clinical impact. Video recordings did not indicate changes in non-technical skills during this time.

Conclusions In situ simulation is a practical method for the detection of LSTs and to reinforce team training behaviours. Embedding in situ simulation as a routine expectation positively affected operations and the safety climate in a high risk clinical setting.

  • Emergency department
  • Simulation
  • Teamwork
  • Safety culture
  • Qualitative research

https://doi.org/10.1136/bmjqs-2012-000942

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Background and context

    In situ simulation has been described as ‘crash testing the dummy’.1 More formally, it is a team-based training technique conducted in actual patient care units using equipment and resources from that unit and involving actual members of the healthcare team.2–5 While simulation has often been used as a strategy to train individuals in both technical and non-technical (communication and teamwork) skills, in situ simulation can be used to evaluate system competence and identify latent conditions that predispose to medical error.

    James Reason has described ‘two approaches… to the problem of error: a person approach and a system approach.’ The person approach ‘focuses on the errors of individuals, blaming them for forgetfulness, inattention or moral weakness’.6 This is echoed by Leonard et al who state: ‘A large and ever present cultural barrier is the deeply embedded belief that quality of care and error free clinical performance are the result of being well trained and trying hard. In this paradigm, inevitable mistakes are viewed as episodes of personal failure with the predictable result that these events are minimized and not openly discussed’.7

    The system approach examines ‘the conditions under which individuals work, and tries to build defenses to avert or mitigate errors’.6 Reason's ‘Swiss Cheese model’ of error is often cited as an example of the system approach. Despite multiple barriers and safeguards which are ‘built in’ to the system, when the ‘holes line up’, or redundancies fail, an error can occur.8 Unfortunately, in analysis of medical errors, the holes line up more often than one would anticipate. In these situations, it is often latent failures resulting in conditions such as time pressure, understaffing, fatigue, inadequate equipment and inexperience, inadequate supervision, and miscommunication ‘that precipitate errors and violations’.8–11

    Latent failures, originally defined in the aviation safety industry, are conditions or threats that result from ‘decisions made or by positions taken by organizations as whole, where the damaging consequence may lie dormant for some time, only becoming evident when local triggering factors overcome the organisations’ defense’.6 In medicine, latent safety threats (LSTs) have been defined as system-based threats to patient safety that can materialise at any time and are previously unrecognised by healthcare providers, unit directors or hospital administration’.12 Operationally, we use this definition and classify LSTs identified in this project into medication, equipment, resources and miscellaneous categories (figure 1). These errors in design, organisation, training or maintenance may have a significant impact on patient safety and, if not mitigated, could have a negative impact on patient care.6

    Figure 1
    Figure 1

    Standardised debriefing template.

    Simulation-based training provides an opportunity to formally debrief participants, something that rarely occurs after actual patient encounters. Multidisciplinary training and debriefing encourages sharing of information that while widely recognised in one ‘silo’ is unrecognised in other ‘silos’. In previous lab-based simulation training with emergency department (ED) providers, we recognised that multidisciplinary training provided a unique method of identifying LSTs. Many of the identified LSTs verbalised by nursing staff during debriefings were previously unrecognised by physician staff and prompted multidisciplinary problem solving. It appeared that simulation training which realistically recreates the clinical environment can inspire reflection on clinical experiences and allow the individual(s) to express concerns that relate to the actual clinical environment.13 In situ simulation training, a potentially more realistic method of training, may provide a better evaluation of the patient care units in relation to hidden or LSTs.

    Our hypothesis was that the implementation of in situ simulation-based training in an actual clinical environment would promote the identification of LSTs and systems issues at a higher rate than seen in the simulation lab setting. The purpose of the described project was twofold. First, we planned to accelerate the identification and remediation of LSTs and system issues in this high risk setting. Second, we intended to embed simulation training as part of the routine work in our environment in order to reinforce and maintain the gains in teamwork behaviours demonstrated in our previous lab-based simulation teamwork training. Ultimately, our aim was to improve the safety of the system of care for our patients.

    Methods

    Setting

    This project was conducted in the paediatric ED of Cincinnati Children's Hospital Medical Center. This ED is an urban Level I trauma centre and one of the busiest paediatric EDs in this country with >90 000 patient visits annually. This translates to an average of 246 unscheduled visits per day. On some days the daily census may only be 190; in busy times the census is over 300 and has reached 400 or more.

    In this system, critical and unstable patients are immediately triaged to the resuscitation bay. Emergency response teams are specific to illness (medical team) or injury (trauma team) and are activated approximately 3000 times per year. The normal response to a medical team during this intervention included a faculty physician, a resident physician, a nursing team leader, a bedside nurse, a medication nurse, a respiratory therapist, a paramedic (or patient care assistant) and a child life specialist (or chaplain), all of whom are ED personnel. For trauma teams, additional personnel included a surgery resident, an intensive care unit nurse and an operating room nurse. For the highest level of trauma, these teams were augmented by a surgery Fellow (or faculty), a critical care Fellow and an anaesthesiologist. Thus, the normal care teams included between 8 and 14 personnel. The inner circle of providers, those at the bedside providing direct care, was limited to 5–6 providers. There were 3–8 providers deployed in the outer circle performing tasks such as medication and equipment preparation, phone calls and consultation. About half of these activations are subsequently determined to be stable and minimal immediate action is required. The remaining patients require immediate interventions to stabilise their physiological status. Historically, the majority of adverse events in our ED have occurred in critical patients presenting to our resuscitation bay. The volume, acuity and complexity of our patient population, in addition to the many different disciplines involved in the care, represent huge risk factors for medical error. These factors highlight the importance of teamwork training within the ED and the pursuit of a shared mental model during the care of critical patients in the resuscitation bay.

    Despite direct supervision by board certified paediatric emergency physicians and surgeons, the inexperience and transient nature of the resident physicians also represent a substantial risk to our patients. As a teaching institution, we provide paediatric education to medical students, paediatric residents from our own and other institutions, family medicine residents, emergency medicine residents as well as residents from various surgical specialties. In any one month, 45–55 residents and 12 Fellows rotate through our ED.

    While, this ED is busier than most paediatric EDs, it is not unique in either the kinds of patients that are cared for or in the variety and inexperience of the residents that are trained. It is logical to believe that in order to safeguard our patients we must standardise the abilities of the ‘permanent staff’ to function in as safe a mode as possible and that this culture of safety is clearly expected of all non-permanent staff as well.

    Participants

    We implemented the safety intervention (in situ simulation training) targeting all personnel who respond to medical or trauma team activations in the ED's resuscitation bay. The intervention was conducted with a team of frontline healthcare providers. Each group completed the intervention as a multidisciplinary team. The number of providers who participated in each simulation was determined by whether it was a medical or trauma team and the level of trauma activation.

    This project was approved by the institutional review board of Cincinnati Children's Hospital Medical Center. Initially, informed consent was obtained from all participants. However, approximately half way through the project, the ED leadership felt the training had become so valuable as to require mandatory participation of all care providers in the ED. Though ED staff was not required to respond to electronic inquiries, they were required to participate in the in situ simulations. At that time, the institutional review board waived the requirement for informed consent. Video consents continued to be obtained from all participants.

    Approach

    We used previously developed critical scenarios to pilot the process in the resuscitation bay and for the initial portion of the training. These scenarios were supplemented during the course of the project by scenarios related to near misses, adverse events or situations. These simulations occurred on all shifts and were presented in an unannounced fashion; the inhouse paging system was used and providers responded believing it was a ‘real’ resuscitation. No supplemental staffing was provided during the training. Simulations occurred at a frequency of 1–2 times a week with a goal of 90 in situ simulations in the first year.

    Simulations included trauma and medical simulations and were based on high-risk clinical cases, either identified by one of the investigators (MP, GG, and RF) or referred to us by ED staff, divisional or institutional safety leadership. A number of these were based on near misses or cases that did not progress smoothly. In addition, cases that were seasonally appropriate (near drowning during summer months, bronchiolitis and hypothermia in the winter months) were used to screen for potential LSTs. Each simulation had specific goals with triggers embedded to stimulate appropriate technical and non-technical behaviours. Scenarios that incorporated deliberate medical error and equipment malfunctions were used to allow personnel to ‘trap and/or mitigate’ error. Other scenarios required non-physicians or less experienced physicians to assert themselves to team leaders. An example of one scenario used is presented in online supplementary appendix A. This template demonstrates our process for developing scenarios, all which were developed by either MP or GG.

    As these simulations were performed on the clinical unit during work hours, we balanced the need for the training with the current status of the ED. Parameters to cancel the simulations were developed with ED leadership and included the unit census, number of high acuity patients, shift change, presence of a critical patient in a resuscitation bay and knowledge of an incoming critical patient. We also discussed the simulation with the ED charge nurse immediately prior to its initiation in order to ensure there were no other factors, such as limited staffing, of which we were unaware.

    Since the simulations were conducted in a clinical setting, the simulations and debriefings were limited to 10 min each. The simulation and debriefing were digitally recorded. Debriefing occurred immediately following the simulation. A standardised debriefing checklist was used (figure 1). GG, MP, RF and/or TL served as the facilitator for each simulation and debriefing. One to two simulation specialists attended each simulation, operated the simulator, recorded the simulation and provided technical support. Debriefing included self-assessment and group assessment of performance. Participants were asked to identify, evaluate and offer solutions to the challenges identified. A key component of the debriefing was the identification of any LSTs by the facilitator and/or team members.

    Outcome measures, data collection and analysis

    Process and outcome measures were selected in order to evaluate the effectiveness of the project's aims. The primary outcome measure was the number and types of LSTs identified during the in situ simulations. In order to facilitate the identification of LSTs, a standardised checklist to assess threats and systems issues identified during the simulation debriefing was used (figure 1). Facilitators emphasised participant identification of threats, but the facilitator also documented his/her observations and used these to prompt discussion during debriefings. Video review of the simulations was not routinely used to identify further threats; however, video was used to refine and classify these threats when needed. In addition to the identification of LSTs, we tracked solutions suggested by ED team and those that were actually implemented.

    Participants were asked to complete an electronic survey following the simulation. Signed participation lists were used to generate a list of participants to whom an online, anonymous survey instrument was electronically mailed. Reminder emails were sent to encourage completion. A typical five-level Likert scale was used to assess secondary measures such as participants’ assessment of impact on clinical care, value of the training and timing of the simulations (online supplementary appendix B).

    Finally, all ED in situ simulations were digitally recorded. Approximately a third of these were reviewed over the course of a year using a modified version of the Anaesthetists Non-Technical Skills (ANTS) tool as a means to evaluate the teamwork behaviours of the participating clinicians (online supplementary appendix C).14 This particular scale is a behavioural marker system developed by Flin and colleagues, industrial psychologists and anaesthetists.15 This scale is based on observation of four skill categories and 15 skill elements. The ANTS scale has been shown to have a high level of validity and a reasonable level of reliability when used by anaesthetists to rate non-technical skills, such as teamwork and situation awareness.16 It has been adapted for use in intensive care units and the operating room.17–22 Flin and Maran have also described the parallel non-technical skills required for team function in an ED setting and the use of a ‘second generation’ training course and ANTS tool that has been piloted with emergency providers.17 Given that the ED environment requires similar levels of complex procedures, multidisciplinary teams and critical care to these environments, we believed the ANTS is an appropriate tool to use in this project.

    Ratings were performed by one trained reviewer, who was blinded to debriefing results and feedback from participants. The rater was trained through the review of the ANTS online manual as well as several hours of deliberate practice of video review.14 ,15 As only one reviewer was used, no inter-rater reliability analysis was performed.

    Analysis

    LSTs, knowledge deficits and system issues identified during the in situ sessions were described and categorised qualitatively; therefore, no formal statistical analysis for the primary outcome was performed. Data collected during the simulations were classified by the source of information and type of identified threat. Survey responses were collected electronically and results were presented as descriptive frequencies. Behaviour ratings using the ANTS scale were plotted against time and observed for trends.

    Results

    A total of 218 individuals from the ED and trauma services participated in at least one of 90 in situ simulations over a 12-month period (table 1). In all, 65 of these scenarios were critical medical patients and 25 were trauma patients; this approximates the actual proportion of critical patients presenting in each category. Approximately 10% included scripted malfunctions/omissions and 5% included scripted behavioural errors. For example, in one simulation, the faculty physician deliberately performed ‘stacked’ shocks, which at the time were not (and is still not) recommended on a patient in ventricular fibrillation. During debriefing, multiple team members, including a senior resident, acknowledged that they knew this was wrong, but did not intervene ‘because he was the attending’. Overall, 35 other simulations were cancelled before initiation due to a critical patient in the resuscitation bay, high patient census or high overall ED patient acuity. Though this represents an overall cancellation rate of 28%, as ED personnel became accustomed to in situ simulations, the need to cancel in situ simulations decreased. In the last two quarters of the year, only 18% of in situ simulations were cancelled.

    View this table:
    Table 1

    Individuals participating in in situ simulation training (N=218)

    A total of 73 LSTs were identified: 22 medication threats, 26 equipment threats and 25 systems/resource threats (table 2). This resulted in an identification rate of one latent threat for every 1.2 in situ simulations performed. This was in contrast to our previous simulation lab results, where 24 LSTs were identified during 33 courses for a rate of one latent threat for approximately every seven simulations performed.13 Examples of threats identified during in situ simulations included missing or malfunctioning equipment, knowledge gaps concerning availability of dilution and infusion kits of critical medications and delayed or absent response of vital team members. (A comprehensive list of LSTs identified is presented in online supplementary appendix D.) Though many of these threats were directly observed or identified by the facilitators, approximately a third were identified by simulation participants. Nurses identified half of these.

    View this table:
    Table 2

    Examples of LSTs and source role for identification of threat

    As demonstrated on the standardised debriefing template, facilitators also captured the teamwork concepts that were discussed by the participants. During the 90 in situ simulations, participants initiated discussion of teamwork concepts in 80 of the 104 instances that teamwork was discussed (table 3). While debriefing the ‘stacked shocks’ example above, one of the bedside nurses described feeling an authority gradient between herself and the faculty physician, a concept she had learned in the simulation lab. This was identified as a nurse initiated discussion of an authority gradient.

    View this table:
    Table 3

    Teamwork concepts discussed and source of information

    In all, 118 participants (54%) responded to the postsimulation survey. Of those responding, 92 of 118 (78%) rated the in situ simulation as extremely valuable or valuable, while only six of 118 (5%) rated the simulation as having little or no value. Of those responding to the question on clinical impact, 50 of 65 (77%) reported little or no clinical impact. We defined clinical impact as whether the participants felt spending 20 min training in the resuscitation bay during their shift affected the care of actual patients in the ED that day. Six participants reported a positive clinical impact and four reported the simulation was disruptive or affected the participant in a negative fashion. In all, 61 of 65 (94%) participants rated the length of the simulation as ‘about right’ while five rated the simulation as ‘too short’. None reported it was too long. Free text feedback included 52 positive comments on the value of the simulations, eight negative comments and 19 comments that provided information or were neutral.

    All in situ simulations were recorded and 33 (37%) of these distributed over the 12-month intervention were formally reviewed by a trained and blinded reviewer using the ANTS behavioural Scale.15 The ANTS scale has four categories: Task Management, Teamwork, Situation Awareness and Decision Making. Behaviours are rated on a four point scale and those not observed in a particular simulation are not rated. Figure 2 demonstrates the ratings of the participating teams over the intervention period. A total of 121 behaviours were scored during the blinded review. Though we hoped for an improvement in the ANTS scores over time, no particular trend is observed. However, the vast majority of simulations reviewed were scored on the ‘high end’ of behaviours, with 35 of the scored behaviours rating a ‘4’, 46 rating a ‘3’, 25 rating a ‘2’ and only 15 rating a ‘1’. Described another way, the majority of the teams scored a 3 or 4 (out of 4 points) (table 4).

    View this table:
    Table 4

    ANTS scores: descriptive statistics

    Figure 2
    Figure 2

    Team Behaviours as assessed by the Anaesthetists Non-Technical Skills scale (ANTS) (Note: behaviours not observed in a particular simulation are not rated: 33 of 90 simulations were reviewed and scored using the ANTS scale).

    This project also produced tangible effects on ED culture and operations. The nursing role that had previously been described as the nurse documenter became and is now described as the nursing team leader. The ED participants recognised the importance of a shared mental model. This concept accounted for almost a third of the teamwork concepts discussed during debriefings (table 3). As the physician team leader may become involved in certain critical procedures, the nursing team leader took on the crucial responsibility of maintaining a shared mental model for the team. The concept of a shared mental model was viewed as so crucial that frontline nurses insisted on adding it to the resuscitation flow sheet as an element that must be communicated with the team within the first 3–5 min of caring for a critical patient.

    Discussion

    This project was unique in promoting recurring in situ multidisciplinary simulation-based training as a method to improve clinical care as well as a way to discover safety threats and system issues in a high risk environment. The experiential learning afforded by this process provided the best opportunity to transfer these skills into the real life setting of a paediatric ED. Based on our previous work we expected to demonstrate that multidisciplinary simulation-based training is a valuable method of identifying safety risks in the clinical environment. In situ simulations allowed identification of a latent threat in almost every simulation performed. This is in contrast to the rate of identification we have observed in the lab setting. In our previous lab-based ED training, we conducted approximately seven simulations to identify one latent threat. Also, in lab-based training of extracorporeal membrane oxygenation providers, another high-risk environment, we identified 30 care environment LSTs during 96 simulations, or one for every 3.2 simulations performed.23 We believe the in situ environment provides significant advantages over lab-based simulation for the identification of systems issues. These include a more time-pressured environment and the ability to test the actual clinical care system, including equipment, processes and staff response. If debriefing occurred immediately following clinical cases, it is likely that a number of our identified threats could be identified, albeit at the risk of harm to an actual patient. However, in our ED (and other critical care settings), immediate debriefing after actual resuscitations almost never occurs as some of the team departs with the patient, others return to care for waiting ED patients, and the remainder clean the trauma bay and restock. Thus, in situ simulation is an important strategy in identifying threats to patient safety.

    This project also demonstrates the feasibility of using in situ simulation on a large scale to reinforce and sustain teamwork and communication skills. Although the ANTS results did not show any obvious improvements over time, most of the teams were scored on the high end of the scale (figure 2). These high ratings hint at maintenance of behaviours that were learned, practiced and improved upon during the simulation lab training.

    In situ simulation provides a means to identify workarounds, knowledge gaps and ‘accidents waiting to happen’ in a way that few other methods are capable of matching. Combining system evaluation and teamwork training synchronously is an efficient improvement strategy for the healthcare organisation. In addition, a significant proportion of the LSTs were identified by the providers involved in the simulation. This speaks to the necessity of including frontline care providers in the evaluation of the systems in which they work.

    Relation to other evidence: teamwork training, simulation and safety

    In situ simulation has become more common particularly in its use to evaluate new facilities and systems.5 ,24–26 Especially related to critical conditions in obstetrics and cardiopulmonary arrest, there are deliberate efforts to use in situ simulation to detect LSTs.27 ,28 This developing body of work speaks to the value of in situ simulation as a means to identify threats to patient safety within the system before patient harm occurs. This project is unique in that it is part of an ongoing venture to embed in situ simulation as part of the daily work of the ED and combines a deliberate effort to identify LSTs with ongoing teamwork and communication training. It is also distinctive in recognising the contribution of the frontline care providers in the identification of these LSTs. This represents another way in which ‘deference to expertise’ is a requirement in high reliability organisations.29

    In our setting, in situ simulation also provides a means to continuously reinforce communication and teamwork skills. The use of in situ simulation as we have described it accomplishes the dual goals of identifying and remedying LSTs as well as providing continuous opportunities to deliberately practice technical and non-technical skills. This provides immediate benefit to the patient, the individual healthcare provider and the healthcare team. However, on a strategic level it also contributes to changing the safety culture of the system. High reliability organisations are described as those that operate in unforgiving high risk settings, but have markedly fewer than expected failures. These organisations place a high priority on (among other things) heedful interrelating, sensitivity to operations, deference to expertise and preoccupation with failure and cultivation of resilience.30 In situ simulations, as part of the daily work of the ED team, address and reinforce these values. The realism and actual clinical environment of in situ simulation engage participants and drive home the need to incorporate a high reliability mind-set to improve the system and thereby patient safety as few other training methods can.

    Challenges and limitations

    The implementation of in situ simulation with working clinical teams presents challenges in terms of time pressure, acuity and patient census in a busy ED or other critical care units. Despite active participation of ED staff in training at our simulation centre for more than a year, the implementation of in situ simulation proved challenging. Performance anxiety of healthcare providers posed a significant challenge initially. Particularly in the early phase of the project, during high census winter months, there was reluctance on the part of staff to participate in the care of simulated patients. Over time, the providers did come to understand that our goal was to understand the team process and identify LSTs as opposed to identifying individual shortcomings. The addition of the mental model to the resuscitation flow sheet and the ongoing in situ simulations in the ED speak to the value now ascribed to this process by ED staff.

    ED staff also raised concerns regarding the impact of in situ simulations on patient care. As noted previously, prior to implementation ‘no go’ guidelines had been developed and agreed to by the project team and the ED leadership. In addition, GG met with the Patient and Family Advocacy Board to seek their input. This group, composed largely of family members of patients with complex medical problems who often seek care in the ED, offered that they were willing to spend additional time in the ED if care providers were ‘practicing’ to deliver safer care.

    As individuals gained experience with simulation this was less of a concern. While an ED census of 35 or 40 patients would have forced cancellation in the early months of the project, by that last quarter of the intervention, we were able to conduct simulations with an ED census of 70 patients. This is important as the most likely time for LSTs to cause a problem is during high-volume, high-intensity care or when all the ‘holes line up’. Thus, a limitation of, and potential bias within, our findings is that we may have missed significant LSTs by not running simulations at times when the department was very busy or by limiting a debriefing session when a critical patient arrived in the bay. Limiting cancellations is important as the simulation team invests significant time and resources in setting up the simulation which takes them away from other responsibilities. Refining our ‘no go’ guidelines supplemented by ongoing communication with the charge nurse prior to setting up the simulation has significantly removed the frustrations we initially experienced. The simulation team also came to recognise that approximately 20% of in situ simulations would be cancelled and that the engagement of the healthcare providers was substantially decreased when the ED census was high. There will always be a tension between patient care and in situ training and the reality is that a significant percentage of in situ simulations will be cancelled due to acuity and/or census. However, in comparison with similar work in our critical care and operating room settings, where cancellation rates are as high as 43% often due to lack of ‘an open bed’, in situ simulation in the ED is more feasible. As we learned, it is necessary to overschedule simulations by approximately 20% in order to achieve the target number and distribution on various shifts.

    Given the time constraints of the in situ simulations, it was not possible to use video debriefing. Digital recordings of all the in situ simulations were used to clarify issues raised during the simulations as well as for the ANTS scale analysis. We were concerned that the 10 min limit of the debriefing was a significant limitation. We partially overcame this by using a standardised electronic follow-up with in situ participants. This allowed us to seek additional (and anonymous feedback) concerning other LSTs, the perceived value of the process and impact on the participant and patients.

    The postsimulation survey (online supplementary appendix B) was an internally developed instrument, so generalisation of results should be tempered. We were primarily interested in the perceived value of performing the simulations during the workday, any negative impact on actual patient care and the identification of any additional issues that participants might not have felt comfortable voicing during the debriefing. The majority of our population had previously participated in an intense lab-based simulation training course and may have been more likely to adapt to in situ simulation training and perceive its value, compared with those without previous simulation experience. This could have biased our results towards ‘the positive’. Second, the response rate to the electronic surveys was 54%. Ideally, the rate would have been higher, but as response to the survey was voluntary we did not follow-up with individually targeted correspondence. The response rate may reflect the fact that participants believed that they had already provided their input during debriefing. Of note, for email-based surveys a response rate of 40% is average, 50% is considered ‘good’ and 60% ‘very good’.31

    Response to findings and sustainability

    The identification of LSTs alone is not sufficient to promote patient safety. All but two LSTs identified during this intervention were remedied by ED staff and leadership. (The two not remedied were related to nurses’ identification of issues related to a general resuscitation flow sheet and the need for a dedicated cardiac arrest flow sheet.) To address identified LSTs on an ongoing basis, the ED has partnered with the Center for Simulation to work on solutions for the identified threats and to sustain ongoing in situ simulations at a rate of 6–8 times per month on all shifts. The ED's in situ training has become a joint effort between the Center for Simulation and the ED's Medical Resuscitation Committee (MRC). This committee, developed and chaired by one of this project's investigators (GG), includes physicians, nurses and an equipment specialist who have been formally trained in simulation-based facilitation and debriefing. This group developed a reporting system for provider concerns in the resuscitation bays to augment findings from simulations. These concerns and identified threats have been combined to form ‘action items’ for the committee and are addressed through committee discussions, formal presentations to the division, changes to the resuscitation bays equipment, organisation and processes, and reincorporation of these changes into in situ simulation training. As an extension of this project, a formal curriculum was developed by the MRC and the Center for Simulation involving scenarios run in monthly blocks around high-risk, low frequency medical and traumatic conditions. After completion of our study, 52 simulations were conducted in 2010 on hypertensive emergency, hypothermic arrest, smoke inhalation, foreign body airway obstruction and respiratory failure due to severe status asthmaticus. As part of this curriculum, simulations not only identified LSTs and team-level knowledge deficits, but allowed the MRC to merge published literature and local practice capabilities to refine management algorithms for these five disease states. Currently, this Committee and the Center for Simulation are using findings from 2011 in situ simulations on adult cardiac disease and rapid sequence intubation to develop similar algorithms, as well as implement a new Critical Airway Team within the ED.32

    In conclusion, this project demonstrates that in situ simulation is feasible in a busy ED and provides a strategy that simultaneously allows for the identification of LSTs, deliberate practice of teamwork and communication skills and provides multiple opportunities to improve patient safety. Future efforts should focus on the implementation of ongoing in situ efforts in other critical care environments and the adaptations that may be required to create a similarly efficient use of simulation in these environments.

    Acknowledgments

    The authors would like to thank our contributors Jerome Bauer, Jennifer Manos, Michael Moyer, Tiffany Pendergrass, Brian Pio, Shobha Iyer and Regina Taylor for their work on this project. Additionally, we would like to thank the support staff at the Center for Simulation, leadership of Cincinnati Children's Hospital Medical Center, and the emergency department physicians, nurses, respiratory therapists, paramedics and patient care assistants for their dedication to providing safer care and for their assistance in the completion of this project.

    References

    1. ↵
      1. Blike GT
      , Personal communication. 2007.
    2. ↵
      1. Miller K,
      2. Riley W,
      3. Davis S
      . Identifying key nursing and team behaviours to achieve high reliability. J Nurs Manag 2009;17:247–55.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. Nunnink L,
      2. Welsh AM,
      3. Abbey M,
      4. et al
      . In situ simulation-based team training for post-cardiac surgical emergency chest reopen in the intensive care unit. Anaesth Intensive Care 2009;37:74–8.
      OpenUrlPubMedWeb of Science
    4. ↵
      1. Weinstock PH,
      2. Kappus LJ,
      3. Garden A,
      4. et al
      . Simulation at the point of care: Reduced-cost, in situ training via a mobile cart. Pediatr Crit Care Med 2009;10:176–81.
      OpenUrlCrossRefPubMed
    5. ↵
      1. Kobayashi L,
      2. Patterson MD,
      3. Overly FL,
      4. et al
      . Educational and research implications of portable human patient simulation in acute care medicine. Acad Emerg Med 2008;15:1166–74.
      OpenUrlCrossRefPubMedWeb of Science
    6. ↵
      1. Reason J
      . Human error: models and management. BMJ 2000;320:768–70.
      OpenUrlFREE Full Text
    7. ↵
      1. Leonard M,
      2. Graham S,
      3. Bonacum D
      . The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care 2004;13(Suppl 1 SPECIAL ISSUE ON SIMULATION):i85–90.
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Feldman JA
      . Medical errors and emergency medicine: will the difficult questions be asked, and answered? Acad Emerg Med 2003;10:910–11.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Reason J
      . Understanding adverse events: human factors. Qual Health Care 1995;4:80–9.
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Vincent C,
      2. Taylor-Adams S,
      3. Stanhope N
      . Framework for analysing risk and safety in clinical medicine. BMJ 1998;316:1154–7.
      OpenUrlFREE Full Text
    11. ↵
      1. Wears RL
      . Beyond error. Acad Emerg Med 2000;7:1175–6.
      OpenUrlPubMedWeb of Science
    12. ↵
      1. Alfredsdottir H,
      2. Bjornsdottir K
      . Nursing and patient safety in the operating room. J Adv Nurs 2008;61:29–37.
      OpenUrlPubMedWeb of Science
    13. ↵
      1. Patterson MD,
      2. Geis GL,
      3. Wears R
      . Unexpected benefits of simulation-based teamwork training: detection of latent safety threats. 2006 Pediatric Academic Societies Meeting. May 1, 2006, San Francisco, CA. 2006.
    14. ↵
      1. Flin R,
      2. Glavin R,
      3. Maran R,
      4. et al
      . ANTS System-Observation and Rating Sheet 2004. http://www.abdn.ac.uk/iprc/papers%20reports/Ants/ANTS%20System%20Observation%20Rating%20Sheet.doc (accessed May 2006).
    15. ↵
      1. Flin R,
      2. Glavin R,
      3. Maran R,
      4. et al
      . Framework for Observing and Rating Anaesthetists’ Non-Technical Skills 2004. http://www.abdn.ac.uk/iprc/papers%20reports/Ants/ANTS_handbook_v1.0_electronic_access_version.pdf (acessed May 2006).
    16. ↵
      1. Fletcher G,
      2. Flin R,
      3. McGeorge P
      . Preliminary Evaluation of the Prototype Behavioural Marker System for Anaesthetists’ Non-Technical Skills (ANTS) 2003. http://www.abdn.ac.uk/iprc/ants_papers.shtml (accessed May 2006).
    17. ↵
      1. Flin R,
      2. Maran N
      . Identifying and training non-technical skills for teams in acute medicine. Qual Saf Health Care 2004;13(Suppl 1 SPECIAL ISSUE ON SIMULATION):i80–4.
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Mitchell L,
      2. Flin R
      . Non-technical skills of the operating theatre scrub nurse: literature review. J Adv Nurs 2008;63:15–24.
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Reader T,
      2. et al
      . Flin R, Lauche K Non-technical skills in the intensive care unit. Br J Anaesth. 2006;96:551–9.
      OpenUrlAbstract/FREE Full Text
    20. ↵
      1. Yule S,
      2. Flin R,
      3. Maran N,
      4. et al
      . Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg 2008;32:548–56.
      OpenUrlCrossRefPubMedWeb of Science
    21. ↵
      1. Yule S,
      2. Flin R,
      3. Paterson-Brown S,
      4. et al
      . Non-technical skills for surgeons in the operating room: a review of the literature. Surgery 2006;139:140–9.
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      1. Yule S,
      2. Flin R,
      3. Paterson-Brown S,
      4. et al
      . Development of a rating system for surgeons’ non-technical skills. Med Educ 2006;40:1098–104.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Burton KS,
      2. Pendergrass TL,
      3. Byczkowski T,
      4. et al
      . Impact of simulation-based extracorporeal membrane oxygenation training in the simulation laboratory and clinical environment. Simul Healthc 2011;6:284–91.
      OpenUrlCrossRefPubMed
    24. ↵
      1. Geis GL,
      2. Pio B,
      3. Pendergrass TL,
      4. et al
      . Simulation to assess the safety of new healthcare teams and new facilities. Simul Healthc 2011;6:125–33.
      OpenUrlCrossRefPubMed
    25. ↵
      1. Kobayashi L,
      2. Overly FL,
      3. Fairbanks RJ,
      4. et al
      . Advanced medical simulation applications for emergency medicine microsystems evaluation and training. Acad Emerg Med 2008;15:1058–70.
      OpenUrlCrossRefPubMedWeb of Science
    26. ↵
      1. Kobayashi L,
      2. Shapiro MJ,
      3. Sucov A,
      4. et al
      . Portable advanced medical simulation for new emergency department testing and orientation. Acad Emerg Med 2006;13:691–5.
      OpenUrlCrossRefPubMedWeb of Science
    27. ↵
      1. Lighthall GK,
      2. Poon T,
      3. Harrison TK
      . Using in situ simulation to improve in-hospital cardiopulmonary resuscitation. Jt Comm J Qual Patient Saf 2010;36:209–16.
      OpenUrlPubMed
    28. ↵
      1. Riley W,
      2. Davis S,
      3. Miller KM,
      4. et al
      . Detecting breaches in defensive barriers using in situ simulation for obstetric emergenciesQual Saf Health Care 2010;19(Suppl 3 SPECIAL ISSUE ON SIMULATION):i53–6.
      OpenUrlAbstract/FREE Full Text
    29. ↵
      1. Weick KE,
      2. Sutcliffe KM
      . Managing the unexpected : assuring high performance in an age of complexity. 1st edn. San Francisco: Jossey-Bass; 2001. xvi, 200 p.
    30. ↵
      1. Sutcliffe KM
      . High reliability organizations (HROs). Best Pract Res Clin Anaesthesiol 2011;25:133–44.
      OpenUrlCrossRefPubMed
    31. ↵
      Instructional Assessment Resources. University of Texas: University of Texas; 2007 (updated Sept 21, 2011. http://www.utexas.edu/academic/ctl/assessment/iar/teaching/gather/method/survey-Response.php (accessed Aug 2012).
    32. ↵
      1. Johnson K GG,
      2. Oehler J,
      3. Houlton J,
      4. et al
      . High Fidelity Simulation to Design a Novel System of Care for Pediatric Critical Airway Obstruction. The American Society of Pediatric Otolaryngology April 2012; San Diego, CA 2012.

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors MP, GG, TL, RF and RW made substantial contributions to conception and design of the current project and to acquisition of data and/or the interpretation and analysis of data. MP, GG, TL, RF and RW were involved in the drafting and/or revision of the article and approved the final version.

    • Funding The PI and team members gratefully acknowledge the support of the Agency for Healthcare Research and Quality. The support enabled the development and completion of this project: In situ Teamwork Training and Detection of Safety Threats in High Risk Settings, Grant number: 1 U18 HS016615-01.

    • Competing interests None.

    • Ethics approval Cincinnati Children's Institutional Review Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement All data are included in the manuscript.