REQUIREMENTS FOR EFFECTIVE MEDICAL DECISION MAKING

Competency in professional endeavors may require much more than finding a single “correct” response to some particular challenge.^1,2 There are, of course, task situations where a single correct action or where multiple correct actions will solve the problems at hand, but not all challenges fit that pattern. As complexity theory (see below) has shown, complex tasks—including medical decision making tasks—can generate unpredictable dynamics that defy treatment with standard content knowledge approaches.^3–5 Factual (content) knowledge can be gained from reading books or from memorizing lecture notes, and from repeating successful prior actions in response to repetitive challenges that led to success.

What is needed where task challenges do not fit a memorized or documented pattern? To handle highly challenging tasks of this nature adequately, an additional set of skills is required; sufficient competency in information processing is essential.⁶ Just as factual knowledge must be learned, information processing skills are also subject to learning and training—albeit a training that cannot be transmitted via books or lectures.⁷ In fact, modern learning theorists distinguish between the acquisition and use of specific content knowledge and the acquisition and use of intellectual processing skills that are free of specific knowledge content.⁸ The latter skills represent cognitive strategies that a professional decision maker uses to regulate his or her own processes of attending, learning, remembering and thinking,⁹ involving both external (incoming) information as well as remembered information and concepts.¹⁰ These “information processing strategies” are not fixed; they must adjust to changes in task challenges—for example, different patients with different sets of morbidities and conditions—and they must adjust to gains in knowledge over time.¹¹ Learning to apply such processing strategies requires guided personal experience.

Health care in the 21st century has become complex, and treating patients often requires more than factual knowledge. A simple infection might often be treated with a “textbook case” of a particular antibiotic. Some “complicated” cases, where a patient presents with multiple problems, might respond to separate “correct” treatments of each component morbidity. Yet, other more “complex” cases can generate dynamics that require information processing which must adjust to the less predictable dynamics of the patient's illness. An example of a “complicated” case may be a patient with multiple organ injuries. An example of a “complex” case may be a patient with several known medical problems as well as a new additional problem whose nature is uncertain but requires major surgical treatment. The complexity of the second patient is quickly realized in the postoperative period when different morbidities and treatment outcomes interact. One small change in the patient's condition can generate a downhill spiral that takes on its own dynamic. As an additional complication, pharmaceutical agents can generate unexpected drug interactions. The physician or medical team is thus often challenged by VUCAD (volatility, uncertainty, complexity, ambiguity and by problems with feedback such as test results)⁶ that are delayed beyond the time when decisions have to be made. In some cases the medical decision maker may not possess the information processing skills that provide the needed mental model to perceive, understand, and respond optimally to such highly complex challenges.¹² In medicine (and many other professional fields), processing skills such as in diagnosis and treatment are not learned by reading textbooks or listening to lectures.^13,14 The consequence for the patient can be unfortunate. Once post hoc insights are attained, the inadequate response to the patient's illness may be classified as a “medical error”.

How can we reduce medical errors, especially when VUCAD strikes? How can we make sure that physicians will effectively manage a network of interrelated problems that involve ambiguity, inconsistency, novelty and surprise?¹⁵ We have known for some time that learning, transfer of knowledge, and ability are impacted by both task structure and task complexity, and by the structural information processing competence of the individual (physician) involved.¹⁶ We need to ensure that medical personnel have the factual content knowledge needed to respond to the task at hand, but we also need to make sure that they can respond to complex challenges by processing information optimally. Simulations, if used as part of an appropriate training system, provide an optimal opportunity to acquire both.

USE OF SIMULATIONS

In medicine unintended deleterious outcomes of treatment are often described as medical “errors”. Such errors may or may not reflect faulty decision making in simple or even complicated task settings. They may also be generated by decision makers who lack adequate cognitive information processing competence.¹⁷ Avoiding errors involves a number of factors including experience, cognitive skills, attention, and motivation. Experience can make an excellent teacher because it not only inculcates facts but also generates knowledge within a meaningful context. Of course, experience can be gained in the real world where “errors” may have deleterious impacts. On the other hand, experience can be gained via simulations and associated training procedures. Just as task settings in medicine differ, simulations can vary along a number of characteristics.^18–21 When decisions are made in a simulated setting, “errors” can be used toward insights via training procedures. At worst, poor medical decision making in simulated settings may generate discomfort in the individual or in the team that failed to function optimally.

Experience—whether generated on the job or via simulation—makes the best teacher only where the reality of task setting is given. Reality of simulations is often called “fidelity”. Using simulations as training devices that avoid injuring the patient requires simulated task settings that optimally represent the challenges faced by decision makers, including tasks that are simple, those that are complicated, and those that are complex. The kind of fidelity needed depends on the purpose of the simulation. Where we assess and train factual knowledge, fidelity implies props, procedures, and challenges that match the medical task environment. Where we assess and train information processing, fidelity means task demands and settings that call for relevant information processing competencies. Simulations with adequate relevant fidelity allow the participant decision maker to engage in the same (relevant) actions or cognitive processes that he or she would use in his or her day to day work environment. An analysis of those actions allows us to assess the competency of an individual or of a team, and it allows us to develop training procedures to generate more adequate decision making in the future.

Many professional fields such as nuclear energy, aeronautics, business, and military decision making have used simulations for years to assess professional competency and to reduce error rates.²² Both content knowledge (what to do when a specific problem occurs) and information processing competency (how to deal with complex, volatile and uncertain situations that may be ambiguous and may not immediately respond to corrective action) have been trained.²¹ The use of simulations in medicine is more recent. Early simulations in the medical field have tended to focus on particular task content settings—for example, anesthesia. The application of simulation to training medical decision making across specialties is even more recent. This paper presents the strategic management simulation (SMS) system which has been used worldwide and in a number of professional fields including the military, executive effectiveness, and pharmacology, and has more recently been applied to medical decision making.

STRATEGIC MANAGEMENT (SMS) SIMULATIONS

Complexity theory and SMS simulations

As suggested above, the SMS simulation was developed on the basis of complexity theory. Interactive complexity theory considers the interplay between environmental task conditions—for example, stress in the medical environment—with response competencies in the realm of thought and action process—that is, “how” a person thinks and approaches problems, less so “what” specific knowledge he or she possesses or applies.²³ In early versions of the theory, process variables were limited to “differentiation” (the breadth of approach to tasks—that is, how many different action options were considered and applied) and to “integration” (the degree to which strategic interrelationships among events and/or action options were considered and used in decision making).²⁴ Subsequent developments of the theory expanded theoretical predictions to include additional process based aspects of functioning in complex task settings such as information search and utilization,¹⁷ emergency management,²⁵ initiative, planning, and more.²⁶

Complexity theory driven aspects of SMS simulations use environmental (task) demands that vary along predetermined parameters. For example, all SMS scenarios include an emergency which generates considerable specific task stress contrasted with prior and subsequent task periods where stress levels are more normal (moderate). Furthermore, all SMS scenarios provide participants with the opportunity to engage in information searches and to use that information to develop initiative, to plan, and to develop strategies.⁶ Computer calculated scoring of functioning in both numerical⁷ and graphical²⁷ format is also based on the theory.²⁸

Comparability across participants and criteria of excellence

The SMS simulation is a “quasi experimental” simulation technique.¹⁸ The majority of events which occur during simulation are preprogrammed (see below) to assure that all participants have equivalent experiences, allowing comparison of participant competencies and of their performance with established criteria of excellence. Excellence, for this simulation, implies competency in processing information. In contrast to many other simulation formats, the SMS does not assess available knowledge in some specific medical field; rather, it focuses on the competency to deal with complex problems across various fields in healthcare systems—that is, the underlying capacity of individuals or teams to make decisions that use their knowledge and their information more effectively. It provides the opportunity to test these competencies under conditions of uncertainty, ambiguity, and situational change—that is, the simulation can measure whether healthcare personnel can deal with complex real world challenges that go beyond ordinary technical knowledge. A wide range of responses to performance demands that vary from routine task requirements to highly complex challenges are measured and trained. SMS simulations provide computer generated real world task settings that extend over a period of several hours. Four equivalent simulation scenarios with high interscenario reliability are used for these purposes. Participants face challenges, solve problems, and make decisions. Based on a participant's performance in the simulation, the computer calculates 25 critical parameters of decision making. Interactions between theses various components of decision making are also evaluated.²⁸

Procedure

Simulation participants are provided with printed and video information about their task setting. For example, in the “Woodline County” scenario participants deal with the consequences of a dam break and its effects on the population in the affected area. Simulation participants from medical fields often state that experiencing the Woodline County scenario is “just like dealing with multiple serious problems in the emergency room”. Participants, either as individuals or as teams, work in a room that is equipped with all props required to perform the simulated task. Once the simulation begins, participants receive computer generated (visual and verbal) information about events that are relevant to their task setting. They are free to request information and to make decisions (within the framework of available resources) at any time they choose. Decisions might be responsive to information received or can represent self-generated initiative. An assistant enters their decisions into a computer system.

Participants receive meaningful responses to some of the decisions they make. Experienced events (and timing of those events) are provided to all participants at the same point in simulated time and in identical visual/verbal format. The fixed (non-responsive) message information components generate highly similar experiences for all participants, allowing performance comparisons (based on computer calculated scores) among participants and comparison of any participant's (or participant team's) scores with established criteria of optimal performance. Performance is computer scored, based on the actions taken by the participants, their stated future plans (action sequences), their responses to incoming information, and their use of prior actions and outcomes. The absence of expert evaluations of performance avoids potential bias. Debriefing is based on computer generated performance scores and the relationship of these scores to established standards of excellence (success and failure rates of thousands of previous participants worldwide).

Assessment of functioning

Computer generated measures focus on “how” the participant or the participating team processes information. Measures vary from simple counts (number of decisions made) via formula based calculations of intermediate performance characteristics (such as use and application of initiative) to calculations of highly complex competencies (such as strategic sequencing of multiple broad approaches that focus on interrelated problem areas). More than 80 scores are calculated; the 25 measures presented in box 1 are based on statistically independent performance attributes obtained via varimax factor analysis procedures across national and cultural boundaries.

Pedigree of the SMS simulation system: reliability and validity

The SMS has been used to assess and train decision makers worldwide in diverse fields including medical and healthcare systems. High levels of predictive validity (r >0.60 with real word success as judged by peers and as demonstrated by income, job level, promotions, level in organizations, corrected for geographical area and job) as well as high levels of test-retest reliability across the four simulation scenarios (r =0.72–0.94) have repeatedly been reported.^1,29 The simulations have been extensively used in research on the effect of information load and other stressors, of drugs and pharmaceutical agents, and in measuring and predicting the consequences of morbidity experience on functioning. Additional validity is demonstrated by the deterioration of various performance indicators with 0.05 alcohol intoxication and seriously diminished functioning with intoxication at the 0.10 level.

Questions addressed by the system

Data obtained from participating in an SMS simulation can answer a number of questions about concurrent competency including:

• Will an individual or a team consider multiple potential antecedents to a problem?

• Will they discover and explore multiple options to problem solving?

• Will decisions be timely?

• Will earlier antecedents be considered, or is the focus only on the present?

• Will responses to a sudden unanticipated emergency be optimal?

• What about obtaining and utilizing available information optimally?

• Are initiative and creative strategic thought used to discover unusual or even previously unknown sources of problems?

• How are the potential interactions among multiple symptoms and morbidities evaluated?

• How are the potential consequences of single and multiple treatments and their respective or interactive outcomes viewed?

Questions of this nature, if favorably scored by the computer system, tend to produce greater excellence and fewer errors in healthcare delivery. The SMS simulation helps us to discover whether or not individuals and teams that comprise the healthcare system are able to deal adequately with complex problems—that is, it provides information about the processing competency of participants. The simulation does not answer questions about participants' content knowledge—for example, we will not know whether their medical decisions would be technically correct. Answers about the technical “correctness” of decisions are investigated by other methods including simulations that focus on decision accuracy.

SMS and training

Participation in a simulation can be instructive, but may or may not benefit learning or training.²¹ Merely participating in a simulation (without follow on training) would limit “take home” learning even more when the simulation involves the use and assessment of cognitive processing rather than content based skills. As Streufert et al³⁰ have shown, participation in the SMS in the absence of associated focused training programs does not generate performance improvement. Consequently, specific SMS based training programs were developed. As a precursor to the training itself, a trainee's specific shortcomings are determined via his or her effectiveness in an initial SMS scenario. Once the computer calculates 25 SMS performance scores, the performance of an individual or of a team can be compared with established criteria of excellence. In many cases several scores will be in the good to excellent range while others scores may show less than optimal performance. The fact that simulation based score values are independent of each other provides information about specific limitations of the participant(s). Training (or re-training) as a result need not be generic; it can specifically focus on one or more areas where there is evidence of a lack of effectiveness. Selection of these specific areas points towards training/learning objectives.

SMS based training towards improved performance follows principles enumerated by several authors in the field of training.^21,31 The methodology initially provides a specific challenging stimulus during simulation participation (similar to event based approach to training (EBAT)).³² For example, each SMS scenario generates a specific emergency event that should be managed. Responses to that event are measured both in terms of “elapsed time between event and response to the event” and in terms of the “characteristics of the response” (both are computer scored). Feedback information about any lacking competency is communicated to trainees. Feedback includes information about both the characteristic of the cognitive processing skill needed and suggestions on how information processing towards performance could have been more adequate. The trainee is then provided with the opportunity to practise the lacking or deficient skill by (1) considering action options that he or she might have used during the prior simulation experience and (2) as responses to specifically presented textual or verbal challenges which make similar relevant cognitive demands. The difficulty level of these challenges increases as performance improves. Feedback about appropriateness is provided. Examples from the day to day challenges of trainees at their job (in this case, medical practice) are frequently used.

Another SMS simulation scenario is then administered to reassess competency after training to see whether improvements are evident. Earlier research has shown that SMS based training generates more effective functioning in decision makers³⁰ as well as in patients.³³ Improvements after a single training session can be as high as 53%³²; with more extensive training improvements of functioning can exceed 76%.^25,26,29

Individuals and teams as simulation participants

The SMS simulations are used to assess and train individual and team performances. Data for individuals reflect his/her performance and allow comparison of processing competence with established criteria of excellence (based on prior performance of successful individuals as well as on 360° ratings of these individuals). Team participation can generate two sets of data. One set reflects the functioning of the extant team, permitting subsequent training of the team as a whole. Based on additionally obtained ratings or observations, team interaction processes (leadership, cooperation, etc) can be added to simulation based training. A second analysis considers the contributions of any one individual team member to other members of the team and to the team as a whole. As in the case of individual participants, specific shortcomings of that individual may be selected for simulation based training. Where a team does not function adequately, any individual who fails to contribute a needed competency might be retrained toward team effectiveness, or might be replaced by a more competent individual.

In medicine many decisions are made by teams. Optimal contributions to the team's effectiveness are of great importance. The capacity to assess and train competency of the individual and his/her information processing contributions to the team as a whole can be of considerable importance.

Applications in medical settings

Use of the SMS simulation in medicine has not been as extensive as in a number of other professional fields. Two examples of its application within the medical field are provided here (others are presently in an operational phase). The first example will deal with assessment, the second with training.

Assessment

The SMS was used to predict the competency in decision making of surgical residents with at least 2 years experience.³⁴ Assessment of skills required for integrative surgical decision making—including critical thinking, crisis management, flexibility, factual knowledge, and team building—were assessed. Attending faculty members who were familiar with the residents' work evaluated them on a standard comprehensive rating scale. Simulation performance with the “Woodline County” scenario of the SMS was obtained over a 3 hour period. Faculty ratings were based on familiarity with the performance of the residents over a period of 2 years.

Simulation scores on multiple measures were compared with faculty ratings. Measures obtained on simulation performance (activity level, response speed, initiative, adequate use of information, and appropriate search for relevant information) generated high correlations with comprehensive faculty assessment (measures of crisis management, team interactions, flexibility of approach). Residents who interacted well with their team members communicated better with patients, were more task focused, and more capable of generating multiple alternatives to problems. A display of greater initiative levels during SMS participation reflected a superior capacity to act quickly and decisively in day to day work situations. The SMS initiative component correlated with the ability to learn from mistakes made by others as well as their own mistakes, and with the ability to consider several potential causes of and solutions to problems that they experienced. SMS values for “breadth of approach” correlated significantly with parameters such as knowledge of alternative approaches and non-surgical options. An important parameter measured by the SMS is the ability to seek and use information optimally. Data indicate that residents who are able to attain an optimal balance between searching for information and effective use of this information are better able to act decisively and will demonstrate speed and accuracy of judgment in clinical situations. The ability to cope with stressors in the simulation correlated with optimal emergency responsiveness. In sum, the simulation was able rapidly to identify shortcomings in information processing and “on the job” performance that faculty members did not recognize until considerable time had passed. Identification of each resident's specific limitations provided an opportunity to generate focused training to improve performance in the clinical setting.

Training

Medical personnel (nurses, physicians) have been trained with the SMS as part of the National Forum on Quality Improvement in Health Care operated by the Institute for Healthcare Improvement (Boston). Simulation participation was followed by feedback and discussions about alternative information processing in areas of moderate or serious deficiencies. Residents at three medical schools have been provided with more extensive feedback and training. Attending physicians have indicated that performance improvements are clearly visible. Information about upgraded performance after training has been presented at various workshops at Penn State College of Medicine³⁵ and other medical facilities.

CONCLUSIONS

Training of factual knowledge alone is valuable but not sufficient in professional fields, especially in medicine where complex problems which often involve uncertainty, ambiguity, and possible rapid change are not uncommon. Learning from books and lectures is useful but should be amplified by direct experience. However, developing experience only by treating actual patients can be risky. Errors made in a simulated setting provide excellent learning opportunities to identify limitations and participate in training. Such methods do not injure the patient. Most of all, healthcare personnel not only learn “what” to do but, with the SMS simulation based training, they also improve their thought and action processes—that is, they learn to deal more effectively with dynamic task settings. A well developed simulation can focus on a number of independent (but potentially interactive) competencies and can identify specific decision making problems of a care provider, allowing focused training that is cost and time effective.