Psychometric properties of an integrated assessment of technical and communication skills

doi:10.1016/j.amjsurg.2008.08.011

The American Journal of Surgery

Volume 197, Issue 1, January 2009, Pages 96-101

https://doi.org/10.1016/j.amjsurg.2008.08.011 Get rights and content

Abstract

Purpose

The Integrated Procedural Performance Instrument (IPPI) consists of clinical scenarios in which bench-top models are positioned to simulated patients. Trainees are required to perform technical skills while engaging with the patient. The purpose of this study was to determine whether an IPPI format examination could discriminate between different levels of trainees.

Methods

Sixteen fourth-year medical students and 16 first-year surgery residents participated in 4 IPPI scenarios. Videotaped performances were scored by 2 blinded independent clinician raters on previously validated instruments: checklist of technical skills, Global Rating Scale of technical skills, and communication scale. We conducted separate mixed design analyses of variance (level × cases) on the 3 scales.

Results

Residents performed better than medical students on the checklist (74% vs 60%, P < .05), the Global Rating Scale of technical skills (75% vs 56%, P < .01), and the coherence communication subscale (79% vs 69%, P < .05).

Conclusions

An IPPI examination discriminated between students' and residents' technical skills and coherence in communication skills. It also highlighted a potential gap in the training of residents' communication skills.

Section snippets

Participants

Participants consisted of 16 fourth-year undergraduate medical students and 16 surgical residents at the end of their first postgraduate year in a large Canadian medical school. All participants took part in the study at the end of the academic year (May). Participants received a modest monetary honorarium for the time required to participate in this study. Research ethics approval was conducted by the relevant academic health sciences center and university research ethics boards.

Experimental design

Each

Global rating scales

The GRS scores on the 4-scenario IPPI examination yielded an IRR of .76 (intraclass correlation) and an α of .75.

The results of the mixed-design analysis of variable revealed a main effect of group (F1,29, 42.44; MSE, .025; P = .000; eta² = .59), a main effect of scenario (F3,87, 19.92; MSE, .015; P = .000; eta² = .41), but no significant scenario by group interaction (F3,87, 1.03; MSE, .015; P = .38; eta² = .03). As can be observed in Fig. 1, the residents obtained higher GRS scores on the

Comments

The results of this study provide some preliminary psychometric properties of an IPPI examination in surgical skills. The results show that a 4-scenario IPPI examination showed acceptable levels of interrater reliability on assessments of the technical and the communication components of performance. The IRR was lower for the communication scale, suggesting that communication skills are more difficult to assess and may be more susceptible to individual differences in rater definitions of good

Acknowledgments

This study was graciously funded by a research grant from the Medical Council of Canada.

References (12)

J.A. Martin et al.
Objective structured assessment of technical skill (OSATS) for surgical residents
Br J Surg
(1997)
G. Regehr et al.
Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination
Acad Med
(1998)
B. Hodges et al.
Evaluating communication skills in the OSCE format: reliability and generalizability
Med Educ
(1996)
B. Hodges et al.
Analytic global OSCE ratings are sensitive to level of training
Med Educ
(2003)

There are more references available in the full text version of this article.

Cited by (58)

Evaluation of mountain rescuers’ (non-)technical skills during simulated resuscitation
2022, Studies in Educational Evaluation
Mountain rescuers work voluntarily but must provide high-quality cardiopulmonary resuscitations (CPR). As demands for their services rise, the factors influencing CPR skills are investigated. Research has shown that technical skills (TS), non-technical skills (NTS), experience, experienced patient deaths, and monitoring impact resuscitation performance. This study explores the influence of these factors for this kind of voluntary medical personnel. A total of 103 mountain rescuers participated. A specially created NTS observation form, questionnaires and simulation manikin were used to measure NTS, TS, experience, and monitoring skills. Results indicate that NTS and TS did not correlate significantly. Only the number of real resuscitations differed significantly between the TS of experienced and inexperienced mountain rescuers. Experienced patient deaths were not significantly related to CPR performance. Furthermore, mountain rescuers could not fully correctly assess their colleagues during simulation. The study shows that previous research results might not hold true in a different medical domain.
Simulation-based training in cardiology: State-of-the-art review from the French Commission of Simulation Teaching (Commission d'enseignement par simulation–COMSI) of the French Society of Cardiology
2021, Archives of Cardiovascular Diseases
Citation Excerpt :
Beyond technical skills, simulation in interventional cardiology tackles non-technical and human factors. Hybrid simulation is one way to mix modalities to enhance the training experience, such as performing radial puncture on an arm simulator with a standardized patient who mimics anxiety or aggression [36]. Simulation is relevant for learning cardiovascular imaging with echocardiography, both transthoracic (TTE) and transoesophageal (TEE).
In our healthcare system, mindful of patient safety and the reduction of medical errors, simulation-based training has emerged as the cornerstone of medical education, allowing quality training in complete safety for patients. Initiated by anaesthesiologists, this teaching mode effectively allows a gradual transfer of learning, and has become an essential tool in cardiology teaching. Cardiologists are embracing simulation to master complex techniques in interventional cardiology, to manage crisis situations and unusual complications and to develop medical teamwork. Simulation methods in cardiology include high-fidelity simulators, clinical scenarios, serious games, hybrid simulation and virtual reality. Simulation involves all fields of cardiology: transoesophageal echocardiography, cardiac catheterization, coronary angioplasty and electrophysiology. Beyond purely technical issues, simulation can also enhance communication skills, by using standardized patients, and can improve the management of situations related to the announcement of serious diseases. In this review of recent literature, we present existing simulation modalities, their applications in different fields of cardiology and their advantages and limitations. Finally, we detail the growing role for simulation in the teaching of medical students following the recent legal obligation to use simulation to evaluate medical students in France.
Dans notre système de santé soucieux de la sécurité des patients et de la réduction des erreurs médicales, l’enseignement par simulation s’impose depuis plusieurs années comme la pierre angulaire de l’enseignement médical et paramédical, permettant une formation de qualité en toute sécurité pour les patients. Initié par les médecins anesthésistes, ce mode d’enseignement permet en effet un transfert progressif des compétences et devient un outil essentiel dans la formation en cardiologie. Les cardiologues adoptent la simulation pour maîtriser les techniques complexes de la cardiologie interventionnelle, pour gérer les situations de crise et les complications inhabituelles, ainsi que pour développer le travail d’équipe. Les méthodes de simulation en cardiologie comprennent les simulateurs haute-fidélité, scénarios cliniques, serious games, l’utilisation de la simulation hybride ou encore la réalité virtuelle. La simulation concerne tous les domaines de la cardiologie: échocardiographie transoesophagienne, cathétérisme cardiaque, angioplastie coronaire et électrophysiologie. Au-delà des questions purement techniques, la simulation permet aussi d’améliorer les compétences en communication à travers des exercices sur patients standardisés, permettant notamment d’améliorer la gestion de situations complexes comme celles liées à l’annonce d’une maladie grave. Dans cette revue de la littérature récente, nous présentons les modalités de simulation existantes, leurs applications dans différents domaines de la cardiologie, ainsi que leurs avantages et leurs limites. Enfin, nous détaillons le rôle croissant de la simulation dans l’enseignement des étudiants en médecine suite à la récente obligation légale d’utiliser la simulation pour évaluer les étudiants en médecine en France.
Developing a validity argument for a simulation-based model of entrustment in dispensing skills assessment framework
2020, Currents in Pharmacy Teaching and Learning
Integrated assessment of multiple competencies at once, including entrustable professional activity (EPA) based assessment, is emerging as an effective approach to competency-based evaluation of health professionals. However, there is an absence of validated assessment frameworks in entry level pharmacy education. We aimed to develop an assessment framework and establish a validity argument, containing multiple sources of evidence, for use in the integrated assessment of pharmacy student's competency in all aspects of the supply of prescribed medicine(s).
A two-phase prospective study was conducted. Phase 1 involved development and content validation of the Model of Entrustment in Dispensing Skills (MEDS) assessment framework using a literature review, a think-aloud study, and expert consultation. In phase 2, a pilot study was conducted with faculty and expert assessors to test the framework. Subsequent analysis involved psychometric evaluation of rating scales and usability testing.
Validity evidence was collected and organised across the two study phases. The MEDS framework had good evidence of content validity supported by the rigorous development and consultation process, as well as case sampling, with 88% of national practice-based competencies represented across the two simulations. Reliability coefficients were high and acceptable, supporting strong agreement across domains, students, and simulations as well as a strong correlation between the EPA and total score (spearman correlation rho 0.725, P < .001).
This study describes a valid and rigorous approach for the implementation and interpretation of an integrated simulation-based assessment tool for determining pharmacy student's progress towards entrustment for independent medication supply practice.
An observational study of technical and non-technical skills in advanced life support in the clinical setting
2020, Resuscitation
Citation Excerpt :
Previous studies have documented a positive correlation between NTS and TS in simulated ALS scenarios by anaesthesiologist7,11,12 as well as in both simulated13 and in real-life14,15 situations in surgery and obstetric emergencies.16 Even though it is known that a significant number of avoidable incidents occurring during operative procedures are linked to failures in NTS,15 there are some studies in which the correlation between these skills is only considered low,8–10 none at all5,6 or even negative.17 Even less is known if a certain part of NTS impacts on the overall ALS performance, or further, if excellent NTS improve TS in ALS.
Technical skills (TS) and non-technical skills (NTS) are the primary elements ensuring patient safety during advanced life support (ALS) and effective crisis resource management (CRM). Both skills are needed to perform high-quality ALS, though they are traditionally practiced separately. The evidence of the association between NTS and TS in high-quality ALS performance is insufficient. Hence, we aimed to evaluate the association between the skills in real-life in-hospital ALS situations.
We video recorded real-life in-hospital ALS situations, analyzed TS and NTS demonstrated in them with an instrument measuring TS and NTS, and tested the linear association between NTS and TS using a linear mixed model.
Among 50 real-life in-hospital ALS situations that we recorded, 20 had adequate data for analysis. NTS and TS total scores were associated with one another (slope 0.48, P < 0.001). All NTS subcategories were associated with the TS total score (slopes ranging from 0.29 to 0.39, P < 0.001). The NTS total score and TS subcategories (chest compression quality, ventilation quality, rhythm control and defibrillation quality) were associated with one another (slopes ranging from 0.37 to 0.56, P < 0.01).
The resuscitation teams who demonstrated good NTS also performed the technical aspects of ALS better. The results suggest that NTS and TS have an association with one another in real-life in-hospital ALS situations. NTS performance had the most evident association with chest compression quality and rhythm control and defibrillation quality; these are considered the most crucial elements affecting outcomes of ALS. The findings of the study present novel information of what and why to emphasize in ALS training.
Clinical trial registration. ClinicalTrials.gov, NCT03017144.
How simulation teaching is revolutionizing our relationship with cardiology
2020, Archives of Cardiovascular Diseases
Simulated and standardized patients
2019, Clinical Simulation: Education, Operations and Engineering
The terms simulated patient and standardized patient (SP) are used to describe a person who has been trained to portray a patient with a healthcare problem or concern. They have been used for learning and assessment of simple and complex procedures, including medical procedures, interpersonal skills, and professionalism. SPs can take on a large spectrum of different roles; they can play a patient with simple or complex medical problems or a member of the family. They can also participate in rating the learners and providing them with feedback. SPs have also been involved in newer types of patient interactions, such as telephone encounters and computer-based encounters. Unannounced SPs can be sent to actual clinical practices or community settings to assess the skills of graduate trainees or practicing healthcare professionals, and the local procedures and work methods.
Preparing for an encounter with SPs requires clearly articulated objectives, careful planning, and notable attention to detail; it involves SP recruitment, training, and rehearsal. Templates for the development of training materials can help in designing SP scenarios. Quality assurance of SP activities allows verification of appropriateness of the portrayal (both content and affect), enhances the skills of the patients through postportrayal feedback, and adds to the credibility of the program.

View all citing articles on Scopus

View full text

Association for Surgical EducationPsychometric properties of an integrated assessment of technical and communication skills

Abstract

Purpose

Methods

Results

Conclusions

Section snippets

Participants

Experimental design

Global rating scales

Comments

Acknowledgments

Objective structured assessment of technical skill (OSATS) for surgical residents

Br J Surg

Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination

Acad Med

Evaluating communication skills in the OSCE format: reliability and generalizability

Med Educ

Analytic global OSCE ratings are sensitive to level of training

Med Educ

Association for Surgical Education
Psychometric properties of an integrated assessment of technical and communication skills