Teamwork training has made a fundamental impact on error reduction and human performance improvement in a number of commercial areas such as aviation^1,2 and other major industries. Aviation provides a good example of how simulation experts and human factors psychologists have collaborated to produce flight simulators that are intended to train and test both crew technical and human interaction skills. Medicine has had a long history of training and testing caregiver clinical skills and performance that is primarily individually oriented. As a result of traditional training and norms, physicians in particular tend to function autonomously. Some clinical tasks are easily simulated and are measurable in environments such as those used in Advanced Cardiac Life Support (ACLS) and Advanced Trauma Life Support (ATLS) courses. Less importance has been assigned to training and assessing teamwork skills. Despite the prominent role that teams play in delivering health care, opportunities to formally practise teamwork skills and receive expert feedback do not exist. A recent Institute of Medicine report³ reminds us that teamwork plays a role in avoiding error that has not been adequately explored.

Patient safety has been focused heavily on system redesign with an emphasis on technology. However, there also needs to be an appropriate focus on human performance as a system.⁴ Organizational models of error in complex industries have been adapted for medicine to help understand and investigate causality, but they also emphasize the importance of defence barriers to prevent error. At present it is the medical professionals who help create safety by holding the technology mosaic together.⁵

MedTeams™ was a multi-year Department of Defense research project that introduced formal teamwork training—based on aviation crew resource management training concepts—into health care. MedTeams began with a retrospective closed claim review of emergency department (ED) risk management cases and found that 43% of errors were due to problems with team coordination. In these cases an effective team structure in the ED and caregivers trained in team behavior would either have mitigated or prevented 79% of the identified failures. The projected liability cost savings, based on the number of US emergency department patient visits each year, was $3.45 per patient visit.⁶ In the prospective phase of the MedTeams project, nine EDs from civilian and military institutions were involved in curriculum development and staff education. The result was the Emergency Team Coordination Course (ETCC®). This 8 hour didactic course was taught by a physician and nurse pair for mixed classes of approximately 16 physicians, nurses, technicians, and support personnel. The curriculum was organized into five team dimensions or topic areas:

• maintain team structure and climate,

• apply problem solving strategies,

• communicate with the team,

• execute plans and manage workload,

• improve team skills.

Examples of teamwork behavior presented from each of these five team dimensions are:

• assign roles and responsibilities,

• engage team members in the planning process,

• provide situational awareness updates,

• cross-monitor actions of team members, and

• conduct event reviews of teamwork.

The training day also provided for (1) behavioral modeling through review of professionally produced videotaped segments of actors illustrating good and poor teamwork, (2) practical table top exercises to engage students in practising components of teamwork such as task prioritization and case review from a teamwork perspective, and (3) analysis and discussion of clinical vignettes (developed by an expert panel) conveying features of good and poor teamwork. This training phase lasted approximately 1–3 months depending on the number of ED staff trained at each institution and did not incorporate high fidelity simulation. The ETCC is described in detail elsewhere.⁷

Research revealed that EDs using the ETCC had a 67% increase in error averting behavior as measured by previously validated behavioral anchored rating scales (BARS) and a 58% reduction in observable errors.^2,7 The project verified the applicability of the behaviourally oriented training based on the aviation model to providers in emergency care. High fidelity medical teamwork simulation with video feedback debriefing was suggested as one way to reinforce and sustain teamwork behavior after ETCC training.⁸

Historically, high medical fidelity simulation training in health care has been most widely developed and implemented in anesthesiology crisis management training.⁹ The centerpiece of the anesthesia simulator is a computer driven mannequin resting in a fully equipped mock-up of a real operating room. Typically, a single doctor/single patient training model has been used. In the case of trauma, simulators show promise for evaluating individual and team performance in a single patient scenario^10,11 but, unlike previous high fidelity simulation studies, the present study involves medical simulation using an ED team (composed of physicians, nurses and technicians from the same ED) involved in the care of a number of patients. This is a departure from the anesthesia simulator single trainee/single patient model with role playing support staff.⁹

The research question addressed in this study is whether high fidelity simulation team training can enhance didactic training to achieve improved teamwork behavior in the ED. In contrast to technological solutions which improve specific patient safety problems, improved teamwork will probably increase resilience across a broader spectrum of the healthcare delivery system. However, it must be recognized that teamwork measures represent an intermediate safety outcome because patient level outcomes have not been demonstrated and are likely to continue to be a challenge for future studies.¹² Most studies to date have demonstrated the realism of simulation and participant acceptance of this modality but have not illustrated the transference of learning to the clinical environment.

METHODS

Study design

This was a single crossover, prospective, blinded, and controlled observational study.

Setting

ED observations occurred in a 700 bed, level 1 trauma center which is the major teaching hospital for Brown Medical School. The ED has an active emergency medicine residency training program and an annual adult census of more than 70 000 visits. The study intervention was performed at the Center for Medical Simulation (CMS), a Harvard Medical School affiliated training site which was instrumental in the early development of Anesthesia Crisis Resource Management (ACRM).

Study subjects

Study subjects were random teams of ED physicians and nurses from one of the experimental sites in the MedTeams research project who were already trained with the ETCC.¹³ The four teams were each composed of one attending physician, one resident physician, and three nurses. One author (RS) randomized the teams into two experimental and two comparison groups without knowledge or regard to the strengths and weaknesses of the individual team members. These four ETCC trained teams of ED caregivers were identified and followed prospectively in their normal ED rotations from 1 April 1999 to 1 June 1999 to collect baseline data on team performance using the BARS metric. Staff schedules were periodically manipulated to create the four study teams for the purposes of observing teamwork behavior. Study participants were unaware of this manipulation. Otherwise, staff participating in the study worked in regularly scheduled team assignments that did not involve study participants. There was no crossover of study subjects between experimental teams.

Teamwork observations, conducted in a manner to which staff were already accustomed during the MedTeams project,⁷ were performed at baseline (pre-training phase) and post intervention (post-training phase) for both groups of teams. Each team was observed twice in each phase. The two experimental group teams received the simulator based intervention at the Center for Medical Simulation before the second observation period. There were no other simulator based interventions in the ED at the time of this study, and only the experimental group received the intervention. Individuals within the comparison group were assigned to work together in the ED for a shift without an intervention. The four teams were observed and scored using the BARS metric in the ED by a pair of observers—an organizational psychologist and a MedTeams study nurse. Both observers were blind to the identity of the experimental and comparison groups. The post intervention observations occurred within 2 weeks of the training. Observations of both experimental and comparison groups were performed as synchronously as possible.

The study was reviewed and approved by the Institutional Review Board at Rhode Island Hospital. The protocol was approved as an addendum to the MedTeams project and did not require informed consent.

Simulator environment

The Center for Medical Simulation occupies an office suite that surrounds a large central room equipped with video cameras. This central room serves as a realistic mock-up for any desired patient care location. It has been successfully used as an operating room, intensive care unit, and part of an ED.

Simulator based intervention

A daylong session was held at the simulator center for the two teams in the experimental group. The day began with an overview of crew resource management by showing a video reconstruction of a well known commercial aviation accident.⁹ Subsequently, three patient care scenarios of increasing complexity were conducted. Scenario 1 presented part of the team with one patient in distress who was then supported by the remainder of the team. Scenarios 2 and 3 presented the team with three patients of increasing complexity. In each scenario one team was involved and the other observed through a partition of one-way glass. Each scenario was videotaped. The scenarios were designed to last 30 minutes and were followed by a comprehensive debriefing of both teams by simulation and teamwork experts using the video to identify specific teamwork actions and failures. The debriefing was structured to minimize discussions of task work and instead to focus on the team dimensions and behavior.^6,7

Outcome measures

The performance measurement instrument was the Team Dimensions Rating Form consisting of five seven-point behaviorally anchored rating scales. These scales have been previously validated in aviation safety studies and the MedTeams project.^2,7 Participants completed a survey regarding features of the simulation experience, what impact it would have on their teamwork skills, and whether they were able to suspend disbelief in the simulated environment. The survey consisted of seven items (table 1), five of which were answered with a 5 point Likert scale. The survey was administered at the end of the simulation training day.

Data analysis

The primary hypothesis was that an increase in the BARS scores would be observed in the experimental group but not in the comparison group. The five teamwork rating scales were evaluated with two planned comparisons using one way multivariate analyses of variance. The first planned comparison examined the differences between the first and second observations (pre-training to post-training) for the experimental group, and the second examined the differences between the first and second observations for the comparison group. A two factor mixed model multivariate analysis of variance evaluated the group by training phase interaction. Intraclass correlations were used to evaluate inter-rater reliability for the five teamwork rating scales.

The survey results in the experimental group after training were tabulated by response category and evaluated using χ² goodness of fit tests. The feasibility of conducting multi-provider and multi-patient scenarios was predicated on uniformly distributed positive responses within the adequate to excellent range of the survey response scale.

Statistical analyses were completed using SPSS Windows 12.0. Because of the small sample size, the significance levels for the χ² tests were evaluated using SPSS exact tests. The statistical significance level for all tests was set at p = 0.05.

RESULTS

Participants

Four attending physicians, four resident physicians, and 12 ED nurses participated in the study. They were selected at random from 32 residents and 120 nurses. All participants were full time employees in the ED where clinical observations were performed. The attending physicians were certified by the American Board of Emergency Medicine and ranged in age from 34 to 40 years. The resident physicians were postgraduate year 2 (n = 2) and year 3 (n = 3) trainees. The nurses ranged in age from 29 to 43 years. Each of the attending physicians and nurses had been employed in the study ED for at least 3 years.

Changes in team behavior

The median inter-rater reliability of the BARS scores was 0.67 (range 0.41–0.86) across the five team dimensions, indicating moderate agreement between the two raters. Mean team dimension rating scores for the experimental and comparison groups are shown in table 2. There was no statistically significant difference between the experimental and comparison group ratings at baseline (Wilkes’ lambda = 0.44, F (5, 10) = 2.56, p = 0.10). The experimental group planned comparison showed a trend towards improvement in teamwork behavior ratings after the simulator based intervention (Wilkes’ lambda  =  0.62, F (5, 20) = 2.43, p = 0.07) while the comparison group showed no change between the two observation periods (Wilkes’ lambda = 0.83, F (5, 20) = 0.82, p = 0.55). The group by training phase interaction was not statistically significant (Wilkes’ lambda = 0.66, F (5, 20) = 2.03, p = 0.12). Table 2 shows that ratings for three of the five team dimensions in the experimental group were less than those in the comparison group before training. However, after training the four team dimension ratings in the experimental group were greater than those in the comparison group. The experimental and comparison groups had identical mean ratings for team dimension 4 in the pre-training phase and the same level of improvement in the post-training phase.

Acceptability of multi-provider and multi-patient simulation

Table 1 shows the responses of the 10 respondents to the survey instrument. χ² tests of the response pattern in the first five questions evaluated the a priori hypothesis that responses would be limited to—and uniformly distributed across—the adequate to excellent range. Table 1 shows that no responses were outside this range for any of the five questions.

Statistically significant χ² test results were associated with responses exclusively in the very good to excellent range for two questions. The 10 study participants who experienced the simulator rated their overall experience as either excellent (n = 9) or very good (n = 1). In addition, the quality of the debriefings was judged either excellent (n = 7) or very good (n = 3). The participants were about equally divided in their opinion as to whether the dissimilarity of the simulated environment to the home hospital’s actual ED was a barrier to practising teamwork skills. Overall, participants in the experimental group were very satisfied with the experience and suggested in written comments that (1) more time was needed for orientation to the simulator environment and equipment, and (2) there should be a shorter time lapse between scenarios. Only one participant felt that the scenarios were too challenging for refining teamwork skills.

DISCUSSION

High fidelity medical simulation was pioneered by anesthesia researchers and until recently has focused on single patient and trainee scenarios with supporting personnel. This study extends simulation activities to include multi-patient scenarios and hence broadens the potential use of realistic simulations to many specialties. The feasibility of conducting these multi-patient simulations is shown by the positive survey results obtained from the experimental group. Use of this strategy in a teamwork training effort was also novel.

High fidelity medical simulation certainly has face validity based on sophisticated simulation technologies used in other industries.¹⁴ There was a positive impact on teamwork behavior in a clinical environment following our simulation based intervention. Despite the lack of statistical significance, this result is still important. An additive effect produced by simulation based training compounding the previous didactic teamwork training curriculum (ETCC) appears possible. These results mirror the responses of anesthesiology residents undergoing crisis resource management training in a simulated operating room environment with extensive video based debriefing.¹⁵ Most papers which advocate simulation training are based on survey responses from trainees, but there is a lack of data connecting simulation training to improvement in clinician behavior which has the potential to improve patient outcomes. Patient safety experts believe that study design should focus on process improvement measures rather than patient level outcomes, so intermediate measures such as those used in this study are sufficient.¹² Our results suggest that simulation based training does have an impact on clinical performance. With a small study such as ours it is likely that there was an impact from the simulation and that the lack of statistically significant differences is related to the small sample size affecting the statistical power of the multivariate tests. Also, it is crucial to remember that all of the study participants had already participated in a teamwork course and this will make it harder to identify the impact of simulation training, especially with a single intervention.

Emergency caregivers receiving additional teamwork training embedded in a high fidelity medical simulation reacted positively to the experience in both subjective and objective ways. Efforts to retain learned teamwork skills have taken different forms such as on the job training and additional classroom didactic sessions. We believe that teamwork training supported by high fidelity medical simulation is important to the continued growth of teamwork skills among emergency care providers. It also appeared to the participants that focusing on teamwork skills within a medical simulation offered an opportunity to sustain the lessons learned during the initial ETCC which the participants had received some months earlier.

Simulation based teamwork training is feasible, but challenges remain to better represent actual clinical practice. Although attempts were made to recreate the common supplies and equipment from the participating hospital in the simulated environment, differences still existed.

Unfamiliarity with the host environment distracted some participants from the goal of practising teamwork skills given that scenarios were compressed, had periods of high workload, and presented unanticipated problems. Good scenario design for teamwork training can tolerate such issues as long as participants experience a naturalistic flow of events.

Multiple patient scenarios create a challenge for audiovisual design in the simulator because noise and activity levels overwhelmed the capabilities of the in house system designed for single patient anaesthesia care. Exchanges that would demonstrate knowledge and use of appropriate teamwork skills were difficult to decipher for use during debriefings. Further development and consultation with recording professionals is indicated to identify the exact balance of equipment needed to have useful interactions depicted on the video for educational and performance assessment purposes.

Limitations

Caregivers in the experimental group were not blinded to the intervention and were aware of its potential significance. However, they were not aware that workplace observations of teamwork were related to the simulator experience. The periods of workplace observation for the caregivers as a team occurred within 2 weeks of simulation exposure. This short retention interval leaves unanswered the key question of when sustainment training—that is, further refresher training on the simulator—would be needed. Videotaping of teams in the workplace would present significant challenges except perhaps in focused settings such as surgical or major medical resuscitation rooms. Technical challenges not withstanding, videotaping also faces significant administrative and ethical hurdles. In addition, future efforts which seek to quantify subjective responses from participants should use a larger number of survey questions with sufficient response sensitivity to determine the strengths and weaknesses of the simulation experience.

CONCLUSION

Medicine is one of the few high risk, high stakes industries that has not yet embraced the importance of simulation in primary and continuing education of healthcare providers. Teamwork training for emergency physicians and nurses supported by high fidelity simulation training and video playback shows promise in helping providers to exercise these skills. Multi-patient simulator based training offers the opportunity to integrate task and teamwork skills in an environment that more closely represents clinical care. Teamwork training conducted within the simulated environment may offer an additive benefit to the traditional didactic instruction, enhance performance, and possibly reduce medical error

The next experimental step for teamwork training enhanced with high fidelity simulation is its deployment within a cohort of postgraduate emergency training programs. It is envisaged that failure and error patterns commonly encountered in ED morbidity and mortality cases will be identified and vignettes constructed, followed by mandatory simulation of these vignettes for all staff. This educational process would serve several purposes—not only to heighten awareness about particular recurrent failure and error patterns but also to build a safety management culture. Issues of cost, feasibility, matching degree of fidelity to learning and assessment objectives, and acceptance must be addressed. Widespread exposure to simulation tools and procedures and refinement of simulation environments needs to occur, as well as creating the infrastructure and processes for training the trainers. Finally, a standardized error management curriculum needs to be created with input from multiple professional societies who are stakeholders in ED care. Many questions remain to be answered—such as the ideal frequency and mode of training and refresher courses, the impact of training on actual care, and whether these innovations in training would improve learning and retention.