A comparison of anesthesiology resident and faculty electronic evaluations before and after implementation of automated electronic reminders

doi:10.1016/j.jclinane.2005.10.008

Journal of Clinical Anesthesia

Volume 18, Issue 4, June 2006, Pages 264-267

https://doi.org/10.1016/j.jclinane.2005.10.008 Get rights and content

Abstract

Study Objective

To develop an automated e-mail reminder system to contact residents and faculty regarding incomplete evaluations.

Design, Setting, Intervention, and Measurements

In the retrospective study, two 9-month periods were evaluated representing pre- and post-introduction of the automated e-mail reminder system. Data collected contained the number of evaluations completed and the rating of residents and faculty at the University of Michigan Health System in 5 different categories on a 5-point scale.

Main Results

The use of electronic reminders resulted in a nearly 4-fold increase in the number of resident evaluations by faculty from 1050 to 3761. Faculty completing evaluations increased from 40 (61%) to 66 (100%). The mean evaluation scores showed statistically significant but clinically negligible change for resident judgement, interpersonal skills, and intraoperative management. Resident preoperative evaluation and knowledge did not show statistically significant changes. Residents completing evaluations of faculty increased from 244 to 1798, and the number of residents completing evaluations rose from 40 (56%) to 87 (100%). There were no statistically significant changes in the evaluation of faculty except in the category of feedback.

Conclusion

An automated e-mail reminder system implemented to contact residents and faculty regarding incomplete evaluations for residents and faculty dramatically enhanced participation in the evaluation process.

Introduction

Regular performance evaluations of resident and faculty physicians in anesthesiology is a requirement of the Accreditation Council of Graduate Medical Education [1]. Feedback from these evaluations is intended to improve resident performance and enable faculty to hone teaching skills [2]. Most evaluation systems use a numerical scale and additional optional narrative statements. Although most evaluation systems started with paper-based data collection, more recently, electronic evaluation has become available. Whether paper or electronic in structure, high compliance with routine evaluation is often difficult to achieve. It has been shown that less than 75% of faculty are involved in evaluation at greater than 55% of institutions [3]. Some institutions have had to resort to punitive measures for lack of faculty participation including extra weekend call and reduced bonuses [3]. In addition, residents may be concerned that less than positive evaluations of faculty may in some way lead to repercussions. One of the reasons for low participation could be forgetfulness by the faculty and residents in the clinical environment.

The University of Michigan's anesthesiology residency has used an electronic evaluation system that is integrated into the electronic perioperative information system since March of 2001. This system provides for daily evaluation of both residents and faculty. In November of 2002, an automated reminder mechanism was implemented to electronically contact residents and faculty via e-mail regarding incomplete evaluations. If forgetfulness was the principle reason for low participation, then this intervention would increase participation in the evaluation process by both faculty and residents. Furthermore, it was unknown if increasing the participation would change the distribution of the grades given to faculty and residents. This study was designed to evaluate the effectiveness of this intervention.

Section snippets

Materials and methods

The University of Michigan began to use a perioperative medical information system (Centricity, GE Medical Systems, Milwaukee, WI) to collect resident and faculty performance data in March of 2001. Automated reminders to complete daily evaluations were implemented in November of 2002. The reminders were sent by e-mail to faculty and residents who had worked together on an electively scheduled operating room day, but had not completed an evaluation within 24 hours. Identical 9-month periods from

Evaluations of residents by faculty

The use of electronic reminders resulted in a profound increase in the number of evaluations completed from 1050 to 3761 (Fig. 1). The number of faculty completing evaluations also was distinctly increased from 40 (61%) to 66 (100%) (Fig. 2). The mean score of resident preoperative evaluation increased from 3.36 to 3.40 (P = 0.069). Resident judgment declined from 3.52 to 3.47 (P = 0.046). Interpersonal skill evaluation declined from 3.75 to 3.61 (P < 0.001). Intraoperative management

Discussion

The intention of the electronic reminder system was to encourage increased participation by both residents and faculty in the evaluation process (Fig. 3). There was a remarkable response to the electronic reminder system, greatly enhancing the number of evaluations completed by both residents and faculty and the number of residents and faculty involved in those evaluations. Overall, there was a 358% and 737% increase in the number of resident and faculty evaluations, respectively.

The electronic

References (3)

M.A. Rosenblatt et al.
Evaluation, feedback, and remediation in anesthesiology residency training: a survey of 124 United States programs
J Clin Anesth
(1999)

There are more references available in the full text version of this article.

Cited by (10)

Impact of faculty-specific electronic reminders on faculty compliance with daily resident evaluations: a retrospective study
2009, Journal of Clinical Anesthesia
Citation Excerpt :
Triola et al [7] reported that faculty compliance improved from 35% to 85% when the Department of Internal Medicine at NYU School of Medicine replaced their paper-based system with a web-based resident evaluation system equipped with automated electronic reminders. In an 18-month study by Blum et al [8], the overall number of faculty evaluations increased 358% after implementing an automated electronic reminder system in the University of Michigan Anesthesiology Training Program. Although “compliance” or “participation” is not clearly defined in either of these studies, it appears to refer to completing a resident evaluation at any point in the study period.
To assess the effectiveness of faculty-specific electronic reminders on increasing faculty compliance with completing daily anesthesia resident evaluations.
Retrospective study.
Anesthesiology Residency Program at Oregon Health & Science University.
41 faculty members on staff from February 2004 through January 2006 were studied for compliance with submitting daily anesthesia resident evaluations 12 months before and 12 months after implementation of a faculty-specific electronic reminder system. Evaluations were entered into a customized, web-based Daily Resident Evaluation database. Data for the reminders were obtained from the Surgical Schedule database. At the end of the study period, data on compliance with our daily resident evaluation system was aggregated by month and compared before versus after system implementation.
A total of 1,222 of 3,893 possible evaluations (31.4%) were submitted before the faculty-specific electronic reminder system was in place, with 38 faculty submitting at least one evaluation (93% participation). After the intervention, 1,824 of 4,263 possible evaluations (42.8%) were submitted, with all 41 faculty submitting at least one evaluation (100% participation). The percentage of resident evaluations submitted by the faculty increased from 29.3% to 42.9% (P < 0.0001) after introducing the faculty-specific reminders. Faculty in the lowest quartile of compliance prior to the intervention showed the largest improvement (446% increase).
A faculty-specific electronic reminder system improved faculty compliance with submitting resident evaluations. The faculty members with the lowest compliance prior to implementation of this system benefited the most.
Technology in anesthesia education and evaluation
2009, Journal of Clinical Anesthesia
Obstacles Affecting the Implementation of the O-SCORE for Assessment of Orthopedic Surgical Skills Competency
2019, Journal of Surgical Education
Citation Excerpt :
Furthermore, since most residents worked with multiple faculty members during a given week, it is unlikely that faculty consistently received requests for greater than 2 forms per week and thus they may not have felt the burden of excessive forms each week. Previous literature has demonstrated a positive effect using electronic reminders to improve the rate of return for evaluations,11,12 however the frequency of these reminders ranged from once at the end of a 9 or 12-month rotation11 to once a day for a daily evaluation program. Therefore, in the context of this study, it is difficult to quantify the magnitude of the effect of the reminder e-mail intervals on faculty perception of the evaluation requests.
There is a need for meaningful and reliable measures of surgical competency in residency education. The goal of the current study is to incorporate the Ottawa Surgical Competency Operating Room Evaluation (O‐SCORE) into the process of resident evaluation at our institution and to assess the feasibility and effectiveness of its use through a web‐based platform.
This is a feasibility study that prospectively assesses the implementation of a web-based O-SCORE at our institution. Over a 16‐week period, 19 orthopedic surgery residents (PGY2‐PGY5) participated in a quality improvement study, which involved collecting 2 feedback forms per week. Each form consisted of a resident form and a linked attending form. At the conclusion of the 16‐week trial period, residents and faculty members were asked to complete a survey about their perceptions of the O‐SCORE program.
An academic medical center.
The study included only residents in postgraduate training years (PGY) 2 through 5 (n = 20) and attendings (n = 37).
During the 16‐week study period, 608 resident surveys were requested for the 19 participating residents, of which 404 surveys (66.5%) were completed. Faculty completed 207 of 326 surveys for an overall compliance rate of 63.5%. The O‐SCORE was able to significantly differentiate between all training years (p < 0.0001) with the exception of PGY3 residents when compared to PGY4 residents. Overall, residents and faculty found the program valuable and feasible. Resident and faculty perception of the value of the O‐SCORE correlated with compliance rate of the O‐SCORE surveys.
This study demonstrates that implementation of an immediate feedback program utilizing an electronic platform is achievable and offers reproducible construct validity. However, issues affecting compliance among both residents and faculty physicians must temper optimism for the program and should be systematically addressed to allow for successful implementation.
Anesthesiology: From patient safety to population outcomes: The 49th annual rovenstine lecture
2011, Anesthesiology
Citation Excerpt :
This will allow us to manage patients more specifically by using data from current physiologic monitors and the anesthesia machine and from the patient's medical history and laboratory data to develop the “multifunctional display” and “primary flight display”; in medicine, this is called decision support. There are many examples of automatic decision support being used, such as reminders for antibiotic timing (either pop-up displays on an information system or α-numeric pages), alerts for abnormal laboratory values, and alerts for the potential of awareness during anesthesia.23–25 The integration of these multiple sources of data provides us with the opportunity to move into a new era of perioperative care.
Electronic reminders improve procedure documentation compliance and professional fee reimbursement
2007, Anesthesia and Analgesia
Medicolegal, clinical, and reimbursement needs warrant complete and accurate documentation. We sought to identify and improve our compliance rate for the documentation of arterial catheterization in the perioperative setting.
We first reviewed 12 mo of electronic anesthesia records to establish a baseline compliance rate for arterial catheter documentation. Residents and Certified Registered Nurse Anesthetists were randomly assigned to a control group and experimental group. When surgical incision and anesthesia end were documented in the electronic record keeper, a reminder routine checked for an invasive arterial blood pressure tracing. If a case used an arterial catheter, but no procedure note was observed, the resident or Certified Registered Nurse Anesthetist assigned to the case was sent an automated alphanumeric pager and e-mail reminder. Providers in the control group received no pager or e-mail message. After 2 mo, all staff received the reminders.
A baseline compliance rate of 80% was observed (1963 of 2459 catheters documented). During the 2-mo study period, providers in the control group documented 152 of 202 (75%) arterial catheters, and the experimental group documented 177 of 201 (88%) arterial lines (P < 0.001). After all staff began receiving reminders, 309 of 314 arterial lines were documented in a subsequent 2 mo period (98%). Extrapolating this compliance rate to 12 mo of expected arterial catheter placement would result in an annual incremental $40,500 of professional fee reimbursement.
The complexity of the tertiary care process results in documentation deficiencies. Inexpensive automated reminders can drastically improve compliance without the need for complicated negative or positive feedback.
Safety in Healthcare: From the Flight Deck to the Operating Room
2023, Turkish Journal of Anaesthesiology and Reanimation

View all citing articles on Scopus

View full text

Original contributionA comparison of anesthesiology resident and faculty electronic evaluations before and after implementation of automated electronic reminders