Article Text

PDF

Effectiveness of a graduate medical education program for improving medical event reporting attitude and behavior
  1. Y M Coyle1,
  2. S Q Mercer1,
  3. C L Murphy-Cullen2,
  4. G W Schneider2,
  5. L S Hynan3
  1. 1Internal Medicine, The University of Texas Southwestern Medical Center at Dallas, TX, USA
  2. 2Family Practice and Community Medicine, The University of Texas Southwestern Medical Center at Dallas, TX, USA
  3. 3Psychiatry, Center for Biostatistics and Clinical Sciences, The University of Texas Southwestern Medical Center at Dallas, TX, USA
  1. Correspondence to:
 Dr Y M Coyle
 Internal Medicine, The University of Texas Southwestern Medical Center at Dallas, 5323 Harry Hines Boulevard, Dallas, Texas 75390-9103, USA; yvonne.coyleutsouthwestern.edu

Abstract

Objectives: To evaluate the effectiveness of an educational program for improving medical event reporting attitude and behavior in the ambulatory care setting among graduate medical trainees.

Design: One group pre- and post-test study.

Setting: The University of Texas Southwestern Medical Center at Dallas Family Medicine Residency Program.

Participants: All family practice residents (n = 30).

Intervention: Patient safety educational program implemented through an introductory lecture and 6 monthly conferences, June to December 2002, involving medical events that occurred in the ambulatory care setting.

Outcome measures: Medical event reporting attitude and behavior at baseline and at 6 month follow up, and barriers to medical event reporting at the 6 month follow up.

Results: Program attendance was significantly correlated with medical event reporting attitude and behavior change (rho = 0.525, p = 0.003). The median change in medical event reporting attitude and behavior was zero and not statistically significant (p = 0.566). Major barriers to medical event reporting were lack of time, extra paper work, and concern about career and personal reputation.

Conclusions: Attending the patient safety educational program was key for promoting a positive medical event reporting attitude and behavior change among graduate trainees. Major barriers to medical event reporting were lack of time, extra paper work, and concern about career and personal reputation. Future research will need to focus on reducing these barriers and to evaluate the effectiveness of such a program over longer periods of time, since making a positive change in medical event reporting attitude and behavior must be made at the individual and organizational levels.

Statistics from Altmetric.com

A report from the Institute of Medicine in 1999, To Err is Human, emphasizes that incorporating patient safety education into clinical training programs is a key mechanism for improving patient safety.1 Furthermore, it is recommended that the initial exposure to patient safety should occur early in undergraduate and graduate medical education programs and be ongoing through continuing medical education.2 Multiple activities can be directed towards understanding the causes of medical events and ultimately their potential role in promoting patient safety, which include team training programs, information technology improvements, and medical event reporting programs.3 In tandem with these activities, individual and organizational attitudes must promote a supportive patient safety culture.1,4 This study focuses on one dimension of patient safety culture—namely, the attitude and behavior towards the reporting of medical events by graduate medical trainees.

Past case studies have shown that medical events involving graduate trainees (known in the US as interns, residents, and fellows) have the potential for causing harm to recipients of health care.5 Fortunately, most medical events do not progress to cause patient harm due to the timely intervention by the graduate trainees themselves, other members of the healthcare team, or because the outcome of the medical event was benign in nature.

Graduate medical education programs have historically relied on the assessment of individual behaviors, through morbidity and mortality conferences and faculty peer review, to analyze adverse medical outcomes including medical events. This approach has become outdated since current evidence indicates that a substantial proportion of medical events are attributable to our processes of patient care rather than being the sole result of poor individual clinical performance.6

Although there have been reports of incorporating medical event education into the medical curriculum, no educational programs have been evaluated for their effectiveness in improving the patient safety culture of the healthcare system or its participants.7–9 The purpose of this study was to evaluate a patient safety educational program for its effectiveness in improving attitude and behavior related to medical event reporting in the ambulatory care setting among graduate medical trainees. The primary outcome for the study—attitude and behavior related to medical event reporting—was assessed by a self-administered questionnaire at baseline and at a 6 month follow up. A secondary outcome of the study was the barriers to medical event reporting which were assessed by a self-administered questionnaire at the 6 month follow up.

The educational program of the study consisted of an introductory lecture followed by six structured, faculty facilitated, monthly conferences. The introductory lecture covered the rationale for the study of medical events, particularly near misses. A near miss is a medical event that does not progress to an adverse outcome as the result of a planned or unplanned identification in association with a change in the circumstances that led to the medical event.5 These types of medical events are frequent and readily lend themselves to group discussion due to the absence of high emotion that occurs with medical events that result in adverse outcomes.6 During the duration of the program, the residents had the option to report medical events using a paper based anonymous reporting system.

METHODS

Study design

This one group pre- and post-test study10 was conducted over 6 months to determine the effectiveness of an educational program for improving attitude and behaviour of graduate trainees to medical event reporting in the ambulatory care setting. The program was integrated into The University of Texas Southwestern Medical Center at Dallas (UT Southwestern) Family Medicine Program longitudinal curriculum. The UT Southwestern institutional review board approved the study protocol.

Study site

The UT Southwestern Family Medicine Residency Program is housed in two clinic sites. One is located on the UT Southwestern campus and the other is a community based clinic located in a Dallas suburb. The campus clinic is jointly operated by UT Southwestern and Parkland Health and Hospital System (Parkland), a public hospital system. The patient population using this clinic is primarily referred from Parkland’s hospital facility for follow up care. This clinic serves an ethnically diverse patient population comprised of 60% Hispanics, 20% African-Americans, 18% white, and 2% other, of which more than 85% live at or below 100% poverty. Approximately 50% of this population has health insurance coverage, primarily Medicare and Medicaid. This clinic has over 12 000 patient visits per year and is served by 18 residents (six in each of three years), two registered nurses, and four medical assistants. The community based clinic serves a patient population that, for the most part, has managed care or commercial health insurance coverage. The ethnic distribution within the clinic is 55% white, 25% Hispanic, 16% African-American, and 4% other. This clinic has 9000 patient visits per year and is served by 12 residents, two registered nurses and 1.5 medical assistants.

Study participants

US family practice postgraduate residencies are of 3 years duration. This study included all 30 graduate trainees in the UT Southwestern Family Medicine Residency Program, with 10 from each of the three postgraduate residency years.

Patient safety educational program

We developed the program using the human error in medicine teaching approach of Gosbee and Stahlhut.11 The program consisted of six 1 hour conferences on patient safety and near misses held monthly from July to December 2002, preceded by a 1 hour introductory lecture on medical error given by two of the study investigators (Coyle and Mercer) in July 2002.

The content of the program’s lecture included a brief overview of the impact of medical error on patient safety and had five educational objectives: (1) to define medical error; (2) to define medical event; (3) to describe the conditions that promote medical events; (4) to describe the process (root cause analysis) used to identify causes of medical events; and (5) to state the purpose of a medical event reporting system.

We used a variant of root cause analysis known as “modeling” to identify the causes of the medical events in this project during the conferences on near miss and patient safety. Modeling uses the collective experiences of the participants to describe how a particular type of medical event can occur and how it might be prevented in the future.12

The introductory lecture used a video dramatization of a medical event taken from a current television show “ER”, coupled with a study investigator led large group discussion of the causal analysis for this medical event. Each of the program’s subsequent six monthly conferences had an educational theme and featured the case discussion of one ambulatory care event and the associated analysis. The case study was derived from actual medical events that had occurred within one of the study’s clinic sites during the 6 months before the initiation of the study. The description and circumstances related to these medical events were obtained through faculty interviews. A written case package that identified the moderators, the theme for the conference, and a case presentation divided into two sections (“What happened?” and “Discovery and recovery”) was mailed to the study participants 1 week before the conference. Table 1 provides a summary of the cases presented at the six conferences.

Table 1

 Summary of cases presented at the six conferences on patient safety and near misses

The format of the conferences consisted of the following, in order of occurrence: (1) opening remarks (5 minutes) by the faculty moderators on the educational theme for the case to be discussed; (2) reading of the case aloud by a moderator; (3) small group discussions (15 minutes) by residents facilitated by a family practice faculty member at which time the group collectively completed a structured worksheet to identify the type of medical event, potential causes, and its prevention strategies; (4) large group discussion (30 minutes) on the analysis of the medical event that included interventions for prevention, if possible; and (5) case summary (10 minutes) by the faculty moderators supplemented by written materials distributed to the participants.

Study variables

Six predictor variables were chosen for this study: age, sex, US versus non-US medical school training, type of graduate training degree, postgraduate year, and frequency of conference attendance. We included US versus non-US medical school training as a predictor variable since differences in culture and attitude have been noted in individuals receiving training in different regions of the world.13 Similarly, we included the type of graduate degree as the investigators wanted to examine the differences in medical event reporting behavior between individuals with an MD degree and those with a DO degree, as there are differences in their undergraduate medical education curricula (http://www.aacom.org/om.html, accessed 4 April 2005). The main outcome variable for the study was change in attitude and behaviour to medical event reporting. A secondary outcome variable category included barriers to medical event reporting.

Medical event reporting attitude and behavior questionnaire

The 5 item questionnaire was based on the work of Gosbee and Stahlhut11 related to medical event reporting attitude and an adaptation of the Prochaska and DiClemente’s stages of change model for measuring behavior change.14 Prochaska and DiClemente developed this model to measure behavior change in smoking cessation research, which has since been extended to cessation of other problem behaviors as well as the acquisition of new behaviors (such as mammography screening and exercise).14 Research shows that this model predicts behavior change related to smoking cessation. Applications of this model measure behavior change categorically.

The categorical measures for the Prochaska and DiClemente model include pre-contemplation, contemplation, preparation, and action. Pre-contemplation includes individuals not currently reporting medical events and not planning on doing so in the next 6 months. Individuals classified as contemplators are not currently reporting medical events but are considering doing so in the next 6 months. Those in the preparation stage are not currently reporting medical events but are planning to start reporting within the next 30 days and have attempted to report a medical event within the past year. Action stage participants have begun reporting events within the last 6 months.

The medical event reporting attitude and behavior questionnaire (table 2) included five levels of possible attitude and behavior related to the reporting of medical events (negative attitude, pre-contemplation, contemplation, preparation, and action).

Table 2

 Medical event reporting attitude and behavior questionnaire

Barriers to medical event reporting survey

Since it is well known that there is substantial under-reporting of medical events due to a number of barriers inherent in medical culture, we developed a survey to identify these barriers based on the reported experience of others who have implemented medical event reporting sytems.3,15 These barriers were categorized into eight categories, each as a single item. A ninth category, designated as “other”, was included to enable respondents to record barriers not included on the list. At the 6 month follow up the study participants were asked to complete the 9 item survey anonymously, identifying all of the factors that prevented them from reporting medical events.

Data collection

Data for the demographic variables were obtained by review of the UT Southwestern Family Medicine Residency Program records. A study investigator (Murphy-Cullen) recorded attendance at each conference. The outcome variables, medical event reporting attitude and behavior change, and barriers to medical event reporting were derived from self-administered questionnaire data. The medical event reporting attitude and behavior questionnaire was self-administered and completed by the program’s participants before the introductory lecture in June 2002 and after the last conference on near misses and patient safety in December 2002. The data for these variables were entered into a Microsoft Excel spreadsheet.

Statistical analysis

Using the scoring protocol developed for the stages of change model, the answers to the medical event reporting attitude and behavior questionnaires (table 2) were used to calculate the pre-program and post-program scores for attitude and behavior related to medical event reporting.14 No response to question 1 in the medical event reporting attitude and behavior questionnaire (table 2) indicated a negative attitude to medical event reporting. Positive responses for one or more of the first four questions of the medical event reporting attitude and behavior questionnaire (table 2) determined the five behavioral levels for medical event reporting as follows: (1) pre-contemplation (question 1), (2) contemplation (questions 1 and 2), (3) preparation (questions 1, 2, and 3), and (4) action (questions 1 and 4). If assigned to the action category, question 5 specified the amount of time for medical event reporting. There were five levels to the scoring protocol that corresponded to the attitude and behavioral levels for medical event reporting. At each of these levels, beginning with the negative attitude category and followed by the four behavioral level categories (pre-contemplation, contemplation, preparation, and action), scores were assigned in increments of 0.2 ranging from 0 (negative attitude) to 0.8 (action). Changes in the attitude and behavior level categories were calculated by subtracting the post-program scores from the pre-program scores.

Study participants characteristics for the continuous measures are presented as medians and range, and for the dichotomous measures as numbers (frequency of event) and proportions. The Wilcoxon signed rank test was used to compare the changes in rank for the pre-program and post-program medical event reporting attitude and behavior scores. Spearman rank order correlations (rho) were used to describe the association between the characteristics of the study participants and the change in medical event reporting attitude and behavior score.

SPSS Version 11.5 was used to analyze the data. The statistical tests used were two tailed. Statistical significance was set at a p level of 0.05.

RESULTS

Characteristics of study participants

Thirty graduate trainees participated in the study, representing 100% of the residency program. The median age of the participants was 30.5 years, 33.3% were male, 76.7% received their medical training in the US, and 76.7% had MD degrees (table 3). For the three resident groups, age, sex, conference attendance, US versus foreign medical school training, graduate training degree, and postgraduate year for residency training were found to be similar.

Table 3

 Characteristics of study participants (n = 30)

Change in attitude and behavior to medical event reporting

The response rate for the medical event reporting attitude and behavior questionnaire was 100% (n = 30). The median change in attitude and behaviour to medical event reporting was zero, and was not significantly different 6 months after implementation of the patient safety educational program compared with baseline (p = 0.566, Wilcoxon sign rank test). Figure 1 shows the attitude and behaviour to medical event reporting at baseline and 6 months after implementation of the program.

Figure 1

 Change in attitude and behaviour to medical event reporting from baseline to follow up at 6 months.

On average, 65% of the study participants attended each of the patient safety educational program conferences (range 55–76%), with none of the study participants having attended significantly more or less of these sessions than the others. Of the study participant characteristics, only the number of patient safety educational program conferences attended by the study participants (median 4, range 2–6) were significantly correlated with the change in the medical event reporting attitude and behavior score (rho = 0.525, p = 0.003) at the 6 month follow up (fig 2).

Figure 2

 Change in attitude and behavior to medical event reporting in relation to number of conferences attended. The number of study participants for each of the medical event reporting attitude and behavior scores and the number of conferences attended (1–6) are included in parentheses if the number is greater than 1.

Barriers to medical event reporting

All of the medical events reported into the paper based event reporting program were from the family medicine program clinics. Clinic staff initiated 73% of the medical event reports, the remaining reports were initiated by the graduate trainees. The response rate for barriers to the medical event reporting survey was 73.3% (n = 22). The most frequently cited barrier was a lack of time to report due to other clinic duties (13 responses). An opportunity for respondents to suggest a barrier not included in the instrument was provided. Major barriers to medical event reporting were lack of time, extra paper work, and concern about career and personal reputation (table 4).

Table 4

 Barriers to positive medical event reporting change by study participants (n = 22)

DISCUSSION

Although the medical event educational program related to the ambulatory care practice setting implemented in our study was not associated with a positive change in attitude and behaviour to medical event reporting at the 6 month follow up compared with baseline (fig 1), it was noted that participation in this program was key for promoting a positive change in medical event reporting attitude and behavior among graduate trainees (fig 2). Major barriers to medical event reporting were lack of time, extra paper work, and concern about career and personal reputation.

The patient safety program implemented in this study to promote a positive attitude and behaviour to medical event reporting among graduate trainees in the ambulatory care practice setting used methods for promoting this change that are consistent with recommendations for teaching medical students and residents about error in health care.11 These methods included an introductory lecture followed by a video dramatization of a blood transfusion event taken from an episode of the currently running television show “ER”, and medical event cases derived from ambulatory care that were relevant to the learner’s training experience. These medical event cases were discussed using an interactive and problem based format that included a discussion of countermeasures to reduce the probability of their recurrence (table 1). In addition, based on the study of social change,16 it is likely that a key factor for promoting attendance in this program is the active participation of faculty and other educational role models for patient safety involved in its curriculum.

In addition, because of the competitive environment in which medical graduates are trained17 and the negative impact that a medical error can have on a physician’s emotional state,18 we conducted a survey of the study participants to identify the barriers to anonymous medical event reporting at the 6 month follow up. In this survey, 59% of the survey respondents indicated that the most important barrier to medical event reporting was the lack of time due to clinical duties (table 4); 27% of the respondents indicated that the other important barriers were the excess paper work involved in reporting a medical event and the risk of jeopardizing one’s career and personal reputation due to reporting these events. Furthermore, 27% of the program participants reported that they did not recognize any medical events to report, indicating that they may have had a lack of awareness as to what constituted a medical event. Two of the program participants reported that they were unfamiliar with the process for reporting medical events, which could have been because they did not attend the introductory lecture to the program. These findings are consistent with past research, which indicates that the major factors contributing to under-reporting of medical events are understanding what constitutes a medical event, how to use the medical event reporting system, and whether the reporting is anonymous or mandatory.3,15

The results of this survey highlight two major areas that need to be addressed to increase medical event reporting by graduate trainees: (1) the medical event reporting process needs to be streamlined to save time and eliminate unnecessary paper work; and (2) some of the survey respondents indicated that they did not note any medical events to report, as has been reported previously with graduate trainees,9 emphasizing that graduate medical training programs need to educate their trainees as to what constitutes a medical event and provide examples of medical events throughout their training experience. Faculty need to be role models for their graduate trainees by reporting medical events themselves, encouraging their trainees to report medical events, and to provide emotional counseling and support as needed to those touched by medical events or participating in the medical event disclosure and analysis process.19,20 The concern about one’s career and/or reputation being at stake due to disclosure of medical events is to be expected, but should be less threatening if the medical events are reported anonymously.

A potential limitation of the study was that the adapted Prochaska and DiClemente stages of change model14 used in our study to assess the acquisition of medical event reporting behavior was not specifically validated for this purpose. However, it is important to note that Willey and colleagues showed, in a recent study,21 that the predictive validity of the stages of change model14 was supported by a significant association between the stages of change for medication adherence and electronically monitored medication taking behavior (p<0.03). Another recent study by Donovan and colleagues22 determined that the test-retest reliability of the stages of change model,14 as applied to acquiring more exercise activity, was also moderately encouraging (κ = 0.52). The most important limitation of the study is that a longer follow up period is needed to determine whether this study’s educational program is effective for promoting a positive change in attitude and behaviour to medical event reporting since such a change must be made at both the individual and organizational levels.23 It would also be worthwhile to test the effectiveness of this program in a variety of academic settings to further assess its effectiveness for promoting a positive change in medical event attitude and behavior. Future studies will need to evaluate the effectiveness of educational programs for improving patient safety culture over periods of time longer than 6 months and, if possible, to include a control group in the study design.

To date, applications of the Prochaska and DeClemente stages of change model14 have been successful in changing health behaviors within a 6 month period of time—for example, cigarette smoking. Greater success for changing health behaviors may be primarily related to the individual having more control over the associated environmental influences. Changing behavior related to medical event reporting would probably involve factors within the culture of the organization or profession over which the individual may have little control. It is therefore likely that successful adoption of medical event reporting for graduate trainees will require changes within the organization that promote patient safety. Westrum23 indicated that a major step in moving an organization towards a generative safety culture, or one in which the focus is on the members of an organization as a whole moving forward to improve patient outcomes based on medical error information, was to engage all members of the organization in detecting and reporting medical events. Successfully creating a generative patient safety culture for a graduate medical training program is likely to require integrating patient safety activities such as medical error disclosure and participation in the analysis of medical events and documenting competence for these activities. The key to creating this type of environment is to integrate patient safety learning activities throughout the residency curriculum, which will require the residency program faculty to know how to use medical errors as educational tools since much of medical education is apprenticeship.11 Further progress towards successfully institutionalizing patient safety as a value in the culture of medicine is to make it an integral part of the academic curriculum, beginning with the first year of medical school6 and extending throughout residency training and later on in the curriculum of continuing medical education.24

Key messages

  • Past studies have shown that medical events involving graduate medical trainees have the potential for causing harm to healthcare recipients.

  • It is believed that incorporating patient safety education into graduate medical training programs offers the opportunity to improve patient safety.

  • The study indicated that attending a patient safety educational program was key for promoting a positive change in the attitude and behaviour to graduate medical trainees to medical event reporting at the 6 month follow up.

  • The study indicated that major barriers to medical event reporting were lack of time, extra paper work, and concerns about career and personal reputation

  • Faculty need to be role models for their graduate medical trainees by reporting medical events themselves, encouraging their trainees to report medical events, and providing emotional counseling and support to those trainees touched by medical events or participating in the medical event disclosure analysis process.

  • Future research will need to focus on reducing barriers to medical event reporting and to evaluate the effectiveness of patient safety educational programs over a longer period of time, since making a positive change in medical event reporting attitude and behavior must be made at both the individual and organizational levels

Acknowledgments

The authors thank Julia Holly Reza for interviewing faculty in The University of Texas Southwestern Medical Center at Dallas Family Medicine and Community Medicine Department to obtain medical event case materials for the study.

REFERENCES

View Abstract

Footnotes

  • This study was funded by the 2001 GlaxoSmithKline Pharmacy Research Award. The contents of the article reflect the view of the authors, not the official position or policy of GlaxoSmithKline.

  • Competing interests: none declared.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles

  • Quality Lines
    BMJ Publishing Group Ltd