Concern about levels of patient safety in the UK and beyond has directed attention to the techniques used by other industries to reduce human error and enhance risk management. The US Institute of Medicine¹ advised,

“The experiences of other industries provide valuable insight about how to begin the process of improving safety of health care by learning how to prevent, detect, recover and learn from accidents.” (p 137)

Drawing on research from aviation safety, this paper outlines a system developed for rating pilots’ cognitive and social (non-technical) skills for working in a flight deck team. It then explains how a similar method was designed for rating the non-technical skills of members of operating theatre teams, and describes a training course used to develop these skills, based in an operating theatre simulator.

NON-TECHNICAL SKILLS FOR PILOTS

In the aviation industry, accident analyses, simulator research, and cockpit voice recordings revealed that unsafe flight conditions were frequently related to failures in pilots’ non-technical (cognitive and social) skills, rather than a lack of technical knowledge, flying ability, or aircraft malfunction. The relationship between team members’ skills, group processes, and performance (quality and safety outcomes) is illustrated in fig 1.

Specialist training programmes called crew resource management (CRM) were designed to increase the use of non-technical skills to improve safety critical behaviours on the flight deck.² These CRM programmes are extensively used in aviation, mandated by some regulators, and applied in other high risk industries.^3,4 Their effectiveness is difficult to measure because of low accident rates, but there is evidence that they have positively influenced pilots’ attitudes and behaviour.^5–7 A review of UK air operators showed that although training captains appreciated the importance of evaluating their CRM programmes, in practice, assessment methods were not always implemented.⁸ This is beginning to change, as regulators^9,10 are starting to require formal assessment of pilots’ CRM skills, as well as their technical proficiency. In turn, this should enable airlines to evaluate the impact of their CRM programmes on pilots’ CRM performance.

Commercial pilots fly as ad hoc rather than constituted crews, so they need to have good non-technical skills for working as an effective team with unfamiliar team members. In contrast, industrial and military operations tend to rely on established teams that train together to work as a unit over a period of months or years. Therefore, the aviation domain provides the best analogy for the “temporary” teams that are found in acute medical specialities (for example, operating theatre and resuscitation teams). The CRM training, which emphasises portable skills for whatever crew a pilot is rostered into on a given flight, was soon recognised to have potential for application in medicine, especially for anaesthetists and surgeons.^11,12 The next section describes the development of a system for rating pilots’ non-technical skills.

Rating pilots’ non-technical skills

By the mid-1990s, CRM training for pilots was widely established in airlines across Europe and North America. As with any other training programme, the crucial test is whether the skills being taught actually transfer from the classroom to the workplace.¹³ For CRM, this required a method of measuring observed behaviour on the flight deck. With a view to encourage the evaluation of CRM skills across Europe, a research project was initiated to develop a method of evaluating pilots’ CRM skills that could be used on a pan-European basis. This type of assessment requires a behaviour rating system based on a defined set of skills, with their component elements and associated examples of desirable and undesirable behaviours—these are called behavioural marker systems.¹⁴ The resulting system for measuring pilots’ non-technical skills was called NOTECHS.^15–17 This has four skill categories with 15 component elements and behavioural markers. Commercial pilots work in a team setting (with the other pilot, air traffic controllers, and despatch) and shared cognition is crucial to maintain a safe operation. Therefore, communication is inherent in almost all elements and so was not listed as a separate category. Given the temporary nature of teams on a flight deck, NOTECHS measures the CRM skills of the individual pilots working in the team, rather than the crew as a single entity. NOTECHS has been adopted by several airlines,¹⁶ and has also been used in research studies to measure pilots’ non-technical skills, before and after CRM training.¹⁸

IDENTIFYING ANAESTHETISTS’ NON-TECHNICAL SKILLS

Anaesthetists’ workload profiles show similarities with pilots (high intensity at task initiation and completion, monitoring, and rapid response to critical events). Although the technical aspects of their performance are covered and assessed comprehensively in anaesthetic training, the associated non-technical skills have not been clearly defined.¹⁹ With the aim of developing a taxonomy (non-technical skills list) for structured observations of anaesthetists, which might be used to develop and support training both in the clinical and simulator environments, a research project was set up involving consultant anaesthetists from the Scottish Clinical Simulation Centre and industrial psychologists from the University of Aberdeen (who had taken part in the development and testing of the NOTECHS system for pilots). Using a design methodology that included an attitude survey,²⁰ a review of existing behavioural marker systems for anaesthetists, incident analysis, observations in theatre, and critical incident interviews with consultants, a prototype system, Anaesthetists Non-Technical Skills (ANTS), was developed.²¹ This was then tested experimentally based on a method (previously used for NOTECHS¹⁷) using standardised videotapes of teams created in the simulator. A sample of 50 consultant anaesthetists was trained and then they were asked to rate the non-technical skills of the anaesthetists acting in eight simulated cases and their scores were compared with ratings obtained from subject matter experts. The results indicated that, even with minimal training of the raters, there were acceptable levels of validity, reliability, and usability to allow further testing for usability in the real environment.²² Following subsequent trials with consultants (trained to use ANTS) rating trainees during operations,²³ a first version of the system was released in 2004 (details available on the ANTS website; http://www.abdn.ac.uk/iprc/ants.shtml). The hierarchical structure of the ANTS system is illustrated in fig 2. There are four categories of behaviour, subdivided into 15 elements, with examples of good and poor behaviours provided for each element. The associated rating scale has four points; good, acceptable, marginal, and poor, plus not observed.

Behavioural rating systems, such as ANTS, designed for evaluating individual team members or a whole team, need to be implemented with considerable care, particularly in relation to rater training.²⁴ For advice on the development and use of behavioural marker systems, based on experience in aviation, the nuclear industry, and hospitals, see Klampfer et al.²⁵ There are now a number of clinical simulation centres (Germany, Canada, Australia, and UK) testing the ANTS system for research and/or training purposes. (A new project is underway to develop a similar taxonomy of non-technical skills for surgeons (NOTSS).²⁶)

TRAINING NON-TECHNICAL SKILLS IN ANAESTHESIA AND EMERGENCY MEDICINE

To train the anaesthetists’ non-technical skills, it was decided to adopt a CRM approach using both classroom and simulator sessions. The development of high fidelity human patient simulators—which feature a lifesize, lifelike, computer driven mannekin set within a realistic clinical environment—enables the creation of clinical scenarios to examine behavioural aspects of performance. Such simulators have been used successfully to integrate the theory with practice of CRM skills in a number of courses, particularly within anaesthesia.²⁷ Some of these courses use the simulator to create scenarios that involve management of anaesthetic emergencies and thus focus on the use of non-technical skills to manage crises.¹¹ Unlike aviation, where the assessment instruments used for training and feedback have been extensively validated, the same is not true of the assessment instruments used in anaesthetic simulator courses.²⁸ The development of the ANTS system thus allowed the development of a course focusing on the skills identified as directly relevant to anaesthetic practice. The course, which was developed by anaesthetists and psychologists, used the aviation model and emphasised the role of non-technical skills in preventing adverse events as well as in managing crises—giving rise to the name Crisis Avoidance and Resource Management for Anaesthetists (CARMA). Underlying theory was delivered as formal presentations (see Box 1) and the themes explored through the use of case based discussion and small group exercises.

Simulated scenarios involving operating theatre based cases were used to allow participants to put non-technical skills into practice and debriefing was conducted using the ANTS framework to discuss and provide feedback on behavioural aspects of performance. Debriefing was also used to investigate underlying cognitive processes. Further courses held throughout specialist registrar training gave further opportunity to rehearse CRM skills and to review and learn from performance. Over 100 anaesthetic specialist registrars attended the CARMA course with very positive evaluation and appeared to offer skills training suited to the anaesthetic team. Only one course participant had undergone any previous human factors training (he was a pilot). All participants said they would value further training and 100% indicated that the course would make them change some aspect of their practice. The most common areas identified for change were better communication, reviewing aloud, and improved team working. Ongoing follow up will look at implementation of perceived changes in clinical practice.

Recurrent comments from the courses suggested that many of the situations that trainee anaesthetists found most challenging were those involving the interaction with other specialties, in particular, working with surgeons in the operating theatre and working as part of a team involved in resuscitation of the acutely ill or severely injured patient in the emergency room. However, in all scenarios, anaesthetic faculty members played surgical roles. Although some simulation centres have developed courses for surgeons to participate in in-theatre training,²⁹ the functional fidelity of currently available surgical simulators limits the extent to which such joint operating theatre team training could occur in the foreseeable future.

It was felt strongly by the anaesthetists attending the original CARMA courses that the course (and indeed clinical practice) would benefit from the presence of the other disciplines with whom they would work in these challenging situations. Therefore, development of “second generation” CARMa (Crisis Avoidance and Resource Management) focused on the emergency room and involved the disciplines most likely to be involved in such areas, namely anaesthesia, emergency medicine, and surgery.

Second generation CARMa

Management of the acutely ill or injured patient in the emergency room presents many clinical challenges, especially as this often involves a rapidly forming (and changing) team with staff from multiple medical and surgical disciplines. This environment shares many of the attributes of other high risk domains³⁰ in that:

• problems are ill structured

• information may be incomplete or conflicting

• situation is rapidly changing or evolving

• there may be multiple conflicting goals

• time pressure may be intense

• consequences of error are grave.

These teams bring with them their own specialty specific knowledge and technical skills and may convey differing cultural approaches.³¹ Courses such as Advanced Life Support (ALS) and Advanced Trauma Life Support (ATLS) have been developed in an attempt to ensure that all members of these “acute” teams are working to the same guidelines or framework. They rely heavily on teaching of knowledge and clinical skills. While aspects of teamwork and leadership are acknowledged, specific non-technical skills are not addressed on such courses. Observations made during resuscitations in the clinical environment demonstrate that ALS training does not enhance leadership performance.³² Similarly, recent work by Marsch has demonstrated that, despite having sufficient knowledge and training, teams managing cardiac arrest in a simulator were unable to follow guidelines successfully with the major obstacles being those of poor leadership and lack of explicit task distribution.³³ The presence of clear leadership in the emergency room has been shown to lead to improved adherence to ATLS frameworks and more rapid formulation of definitive plans.³⁴

Second generation CARMa retains the generic elements of the original course in presentations but utilises the high fidelity simulator to recreate scenarios typical of the emergency room. The scenarios develop in real time and involve participants in their own specialty role. The course adopts the construct that team situation awareness is critical to good teamworking³⁵ and uses the scenarios to explore shared mental models.³⁶ This is done using the Situation Awareness Global Assessment Technique method³⁷ by “freezing” the scenarios before resolution of the clinical situation and using probe questions to establish the individual participants’ level of situation awareness. Measures of the patients’ presenting history and illness severity score, change in physiology throughout the scenario, perceived roles and responsibilities of team members, and clinical priorities at the end of the scenario are made and used to compare mental models across members of the team. Examples of good sharing or divergence of mental models are used as the basis for debriefing using video to explore the underlying team working processes. Trained simulator faculty makes global ratings of performance.

To date only 20 teams have been observed. Preliminary observations demonstrate that approximately 30% of teams are very competent but, as Hoff et al³⁴ found, the emergence of a good emergency room leader appears to be pivotal. The more competent teams demonstrate closer sharing of mental models. Where teams do not work well together, there is always a breakdown in some aspect of shared mental models. Most problems occur with declarative models³⁶—either in what is happening to the patient (disparity of stories about presentation can be extreme) or in what was expected of other specialists when they arrived. This relates to perception of roles and is influenced by the fact that departments will “do things differently” in terms of organisation in different hospitals and the trainees have seldom worked together before. (Having said that, the most marked example of complete separation of mental models came from a team where team members did know one another and significant false assumptions were made.) Where procedural models break down, it is most commonly as a result of previous failure of declarative models.

Repeated problems in team coordination are related to leadership issues, most commonly conflict between the emergency physician and anaesthetist, who may both consider themselves to be leading the team. Where the focus of the first generation CARMA course was on individual feedback and individual behaviours are assessed for feedback on the current course, it is clear that team skills are critical. Team situation awareness in the resuscitation room is more than the sum of the situation awareness of the individual team members.

DISCUSSION

For CRM training to transfer effectively from aviation to other work settings, such as health care, it has to be carefully designed to address the non-technical skills required in that domain. It is not sufficient to take aviation training materials and simply delete “pilot” and replace it with “nurse” or “anaesthetist”. The flight deck is an extremely proceduralised work environment compared with an operating theatre or emergency room, and there are marked differences in professional as well as national cultures, which exert powerful influences on accepted behaviours.^38,39 As Helmreich has noted,⁴⁰ these warnings do not seem to be being heeded by healthcare providers now purchasing (sometimes at considerable expense) CRM and other team training packages ill-suited for hospital staff or other medical teams:

“One of the risks to the effectiveness of patient safety programs was learned in aviation. After CRM had gone through the process of being recognized, acknowledged, and formally mandated, a number of consultants emerged from ‘under rocks’ offering packaged programs that promised to cure all an organization’s ills.”(p130)

Nevertheless, when designed with sufficient care, CRM training has been successfully introduced for a wide range of work teams and there are significant benefits in exchanging information across disciplines. Aberdeen University has been running an annual CRM users group for 6 years, which is attended by organisations running CRM programmes (for example aviation, healthcare, nuclear and conventional power, fire service, prison service, and air traffic control). Delegates have attended each others’ CRM courses and they share training materials, case studies, and, more recently, their non-technical skills rating systems.

Well grounded development of behavioural markers for use in anaesthesia and surgery have formed the basis for development of a course addressing team working skills in anaesthesia and the emergency room. While ANTS emphasises the individual skills of the anaesthetist within a team, initial experiences of the emergency room course suggest that good team performance goes beyond the contributions of the individuals and further work, especially around team situation awareness, and shared mental models in these challenging situations is warranted.