Article Text

Download PDFPDF

How an educational improvement project improved the summative evaluation of medical students
  1. K G Hoffman,
  2. R Margaret A Brown,
  3. J W Gay,
  4. L A Headrick
  1. University of Missouri School of Medicine, Columbia, Missouri, USA
  1. Dr K Hoffman, Offices of Medical Education, MA213 Medical Science Building, Columbia, MO 65212, USA; hoffmank{at}health.missouri.edu

Abstract

Background: At the University of Missouri-Columbia School of Medicine (USA) “commitment to improving quality and safety in healthcare” is one of eight key characteristics set as goals for our graduates. As educators, commitment to continuous improvement in the educational experience has been modelled through improvement of the Medical Student Performance Evaluation (MSPE) letter (formerly the Dean’s letter).

Discussion: This educational improvement project decreased waste, increased collaboration and developed locally useful knowledge. By applying continuous improvement principles to the construction of the MSPE the overall efficiency of the process could be enhanced, and the MSPE committee was able to spend less cognitive energy on structure and format and focus more on the content of the letters. Four MSPE cycles have been completed using a new Web-based system; after each cycle, additional enhancements were identified and implemented. This work adds to the literature, as it describes the application of continuous improvement principles to an educational system.

View Full Text

Statistics from Altmetric.com

Educators who hope their students will finish their training able to improve quality and safety in healthcare can reinforce that goal by demonstrating continuous improvement in the educational experience, not only modelling the behaviours they hope graduates will integrate into patient care, but also creating benefits for the education programme with improved quality and better use of limited resources. At the University of Missouri-Columbia School of Medicine (MUSOM), “commitment to improving quality and safety in healthcare” is one of eight key characteristics we set as goals for our graduates.1 We present here an example of continuous improvement applied to an educational process, the improvement of the Medical Student Performance Evaluation (MSPE) letter (previously known as the “Dean’s Letter”).

Background

For physicians, evidence of their ability to deliver quality care begins with performance assessment in medical school. In the USA, the “Dean’s Letter” began as a letter of evaluation detailing levels of accomplishment students had achieved during medical school.

In 2002, the Dean’s Letter was renamed the Medical Student Performance Evaluation (MSPE), and the letter format and content were further standardised. The MSPE is divided into six standard sections: identifying information, unique characteristics, academic history, academic progress, summary and appendices.2

Problem

Accurately describing a student’s performance and readiness for the next level of training is a significant challenge for any educational institution. Finding ways to enhance efficiency and effectiveness in such a process is of benefit across disciplines and educational settings.

At MUSOM, as with most medical schools, the creation of the MSPE required significant faculty and administrative resources. Prior to the intervention, creating MSPEs was a manual, paper-based process. A seven-member, faculty committee met face-to-face to review available data, then created, edited and finalised the evaluation for each of 96 graduating medical students. The MSPE committee began its work in late August and met two or three times each week until November. The committee was supported by two administrative staff. The paper process is illustrated in fig 1.

Figure 1

Paper process for creating Medical Student Performance Evaluation (MSPE). ERAS, Electronic Residency Application Service.

In the manual, paper process, committee members accessed student files and draft letters only during committee meetings. The primary reviewer accessed the student file prior to committee review. If a faculty member missed a meeting, it was difficult to follow the progression of the committee’s decisions. For the student, it was difficult to link the committee comments with performance data, and students had no knowledge of what data were available for the committee’s consideration.

We addressed this process as an educational improvement project in two distinct stages. We first conceptualised an improved work-flow design. Then, we looked for opportunities to use technology to enhance the new process.

Purpose of the change

We had three aims for this educational improvement project: (1) to improve efficiency and decrease waste; (2) to increase process transparency for faculty and students; and (3) to integrate the process into our established student performance information systems.

METHODS

Setting

The project was centred in the Offices of Medical Education (OME) of the MUSOM. The programme leading to the MD degree consists of a preclerkship problem-based learning curriculum, seven core clerkships, and required fourth-year clinical and advanced biomedical courses. The MUSOM is located on the main campus of the University of Missouri, a public research university supported by the state and designated as Doctoral Research University-Extensive by the Carnegie Foundation for the Advancement of Teaching. State funds and tuition make up approximately 16% of the MUSOM budget, and the School is ranked 69th of 76 in state funding per student among publicly supported medical schools in the US. Thus, efficient use of scarce resources is vital.

Function

Over the past 10 years, the OME utilised local talent to develop an integrated information system linking applicant data and student performance data across the medical school. Multiple Web-based programmes gather faculty comment on student applicants, student performance data and student comment on the quality of their educational experiences. Data are stored in a Microsoft SQL Server 2000 database that supports locally developed queries on outcomes of our educational programmes and longitudinal assessments of individual student performance. Student performance data are collected, managed and distributed through this integrated system. Students have continuous access to their performance data through a Web-based student portfolio. The MSPE draws heavily from the performance data available in the integrated information system.

Interventions

Improving the process

The process improvement team (table 1) began the educational improvement project in May 2005, with a desired implementation date of September 2005. The Institute for Healthcare Improvement’s (IHI) Eight Knowledge Domains for the Improvement of Health Care3 and Langley, Nolan and Nolan’s Model for Improvement4 guided our thinking. We created process flow diagrams (using Microsoft Office Visio) of the existing manual, paper process.

Table 1 Process improvement work team

To identify waste, the MSPE committee assessed the importance of each step in the paper process. We then engaged fourth-year medical students to better understand their needs. Synthesising these data, we used Visio diagrams to describe an ideal workflow. We then considered two questions: (1) In what area(s) can automation maximise efficiency in the process? (2) How can we leverage our current information systems to eliminate redundancy?

Leveraging technology in support of an improved process

We created another Visio diagram to identify: (1) areas for automation and (2) areas where existing data could be used to create the MSPE.

Recognising the need for collaboration between the people who will use the system and the persons who build the system, we created a process improvement team (table 1).

The team: (1) developed scenarios for how the educational improvement would be used; (2) created draft designs for faculty and students who would interact with the educational improvement; (3) described the integration of databases to support the MSPE; and (4) developed example reports. Recommendations were vetted with members of the MSPE committee and administrative staff.

Once we were clear on how the educational innovation should be used, we moved from illustrations of how the improved MSPE might work to a functioning prototype. In July 2005, the prototype provided an opportunity to work through the submission of a “demonstration MSPE” and to identify gaps between initial design and actual use. Faculty, staff and students provided feedback, recommending changes in both the content and processes. This feedback directed small cycles of change.

MSPE committee members completed training, and students were introduced to the educational improvement in August 2005; a month later, the redesigned Web-based MSPE system was launched.

The improved MSPE process supported seven major functions: (1) data sharing from established systems; (2) student contributions of data not previously available; (3) student-generated summary of performance; (4) assignment of faculty committee members to review student input; (5) creation of a draft summary by faculty committee members; (6) modification of draft documents by committee members; and (7) verification of the final MSPE by electronic signature. Table 2 illustrates the sources of data.

Table 2 Sources of data

In the improved process, each student drafted a summative performance statement. Administrative staff assigned the student letter to an MSPE member who simultaneously reviewed the original data and the student-generated summary, making edits to the letter as appropriate. All members of the MSPE faculty committee could electronically review the faculty-generated draft prior to the committee meeting. Furthermore, they submitted electronic edits prior to arriving at the committee meeting. Changes were easily tracked using the new process. Once agreement was reached on each letter, MSPE committee members verified the letter through electronic signatures. Because the MSPE letters were already in electronic format, they were easily transmitted to Electronic Residency Application Service, completely eliminating the second part of the old paper-based process. Table 3 illustrates the differences between the new and old process.

Table 3 Original and education innovation process

Measures

To assess the outcomes of the educational improvement project, we conducted pre-/post implementation comparisons of materials, administrative support and faculty time. Administrative staff tabulated the number of revisions, letters produced, photocopies made and letter length to calculate the total number of pages. Staff identified the time associated with each step in the original process and frequency with which that work was repeated to calculate administrative time. Faculty time was calculated from the MSPE committee meeting schedule.

To gauge student perceptions of transparency, we tracked the number of student requests for changes to their finalised letters in both the manual and improved processes. We solicited committee member feedback on the new improved process through weekly 1 h (May to June) interviews with committee members and a 2 h debriefing session at the conclusion of the first MSPE cycle. Because each graduate constructs only one MSPE, no pre-/post student-impression data are available.

RESULTS

Situation analysis

Prior to participation in the educational improvement project, the MSPE committee had conducted its work using the same work flow for 15 years. Models for process improvement were not well known to faculty, students, OME staff or the information technology group. Students and staff were familiar with Web-based applications, but faculty had varying levels of computer literacy. The situational analysis brought into focus the importance of: (1) a user-friendly interface, (2) online examples, (3) one-on-one training and (4) assigning a technical person to attend and support the MSPE meeting.

Outcomes

Table 4 illustrates the outcomes of the educational improvement project.

Table 4 Outcome data for the redesigned Medical Student Performance Evaluation educational improvement project

Only one student in the graduating Class of 2007 and one in the Class of 2008 requested a change to the final MSPE letter.

The MSPE committee debriefing identified the following improvements in the efficiency and effectiveness of their work.

First is remote access to materials. Previously, academic records were maintained within the OME and accessed only when committee members were physically present.

Second is the ability to complete online edits to the MSPE. Previously, the chair made pen-and-pencil changes to draft MSPEs. After the meeting, staff retyped the letter. Each committee member completed a specific function: scribing, checking grammar, verifying data. Committee members reported that these “clerical” roles made it more difficult to focus on the overall content of the letter. The educational improvement leveraged technology for real-time edits, eliminating re-typing.

Third is asynchronous collaboration. In the manual process, all work occurred within the committee meeting, so the MSPE represented the contributions of only those committee members physically present. With the busy schedules of clinicians, committee members arrived late, left early, or stepped out to attend to patient care issues. Given these interruptions, the committee devoted significant time to ensuring that all edits were captured, and members had current information. In the improved process, committee members worked asynchronously and from locations outside the OME, reviewing edits even if they were unable to attend the committee meeting.

Fourth is more robust conversations within the committee. Committee members reported that the increased efficiency of the improved process allowed them to have more robust conversations that positively influenced the quality of the MSPE. Prior to redesign, the committee reviewed three letters in a 2 h session. After redesign, letters were completed in 20–30 min.

In the manual process, committee members met 6 h each week for 10 weeks. Face-to-face meetings of the MSPE committee for the Class of 2006 decreased to 20 h and to 15 h for the Class of 2007. MSPE committee members were better able to integrate the committee’s work into the flow of their other duties, thus increasing the efficiency of the process. We have used the new process to complete MSPEs for the graduating Classes of 2006–2008. After the completion of each cycle, we collected feedback and used this to enhance the process for the next cycle.

DISCUSSION

The changes described here demonstrate improvements in efficiency, quality and costs (human and materials). The educational improvement project was part of our school’s commitment to model the characteristics we expect of our graduates—a commitment to quality and safety in their work.

Summary

This project is congruent with several of the IHI’s eight knowledge domains for the improvement of healthcare,3 as these domains relate to improvements in medical education: customer/beneficiary knowledge, thinking about education as a process or system, making changes and collaboration.

Customer/beneficiary knowledge

This project assessed the needs and preferences of persons that would use the improved process, then used this information to frame the educational improvement project. Students needed increased transparency and assurance that the data were both complete and accurate. Faculty needed more efficient methods to manage information, and a work flow to accommodate their busy schedules.

Education as process/system

A system is an interdependent group of people, procedures, activities and technologies with a common purpose or aim.3 The MSPE improvement project integrated several existing data systems to: (1) enhance data integrity, as multiple sources of data were integrated, minor inconsistencies were brought to light; (2) remove waste by leveraging existing processes; and (3) reduce rework, as the same data were used to support multiple school functions. We felt it was critical to map out the desired system first and only then look for ways that technology could help. We believe the same results would not have occurred if we had simply looked for a technological “fix.”

Leading, following and making changes

Lessons regarding making changes in complex organisations include: (1) forming relationships with stakeholders; (2) drawing strong connections between automation and continuous quality improvement principles; (3) collecting feedback from stakeholders and use of this feedback in small cycles of change to continuously improve.

Collaboration

The MSPE project fostered skill in collaboration by bringing together persons with diverse skill sets and perspectives. We believe that the work was successful in part because participants developed a common language to advance the project goals and gained experience in an effective team with clearly defined responsibilities.

Developing new, locally useful knowledge

Educational leaders and staff analysed current processes, looked for improvement opportunities, solicited actionable feedback and used this feedback to make further improvements. Our experience with the MSPE improvement project informed new educational improvement projects (creation of standardised assessment tools; supplemental admissions application redesign). As was done in the MSPE project, in each of these initiatives we: (1) engaged stakeholders to better understand their needs and preferences; (2) looked for opportunities to integrate projects with ongoing work; (3) analysed the projects through several different lenses to understand how the projects “fit” into the larger educational system; and (4) collected feedback during multiple small cycles which we used to further improve the process. We continue to build on the collaborative relationships developed in the MSPE.

Context

There are two literatures that inform this work: the MSPE and continuous quality improvement applied to medical education. The literature on the MSPE is limited. We conducted a Medline search of English-language articles published between 1993 and 2007 using search terms “Dean’s Letter,” “medical student performance,” “clinical competence,” “educational measurement,” “summative assessment,” “achievement, leadership, internship and residency,” “readiness to practise” and “correspondence.” The search resulted in 81 articles, 19 were relevant to the topic, and five were published after 2001.

Extant research found improvements in the Dean’s letters but that 35% of schools still produce unacceptable letters.5 Letters lacked a systematic study of students’ non-academic records,6 and a substantial number of letters did not provide comparative information.7

Discussions about the application of quality improvement to health professions education have occurred since the early days of quality improvement in healthcare,812 but there are few examples in the literature. Hollander et al13 described how a quality improvement approach improved a medical school curriculum reform, including an evaluation point before the new curriculum was launched.13 Malik et al14 reported using continuous quality improvement in a programme to support community-based clinical teaching faculty.

A nursing school reported continuous improvement as the theoretical underpinning of its governance structure, including the creation of an annual all-school “Report Card.”15 Another described how a multitrait, multimethod model for programme evaluation was integrated into the college’s overall approach to improvement.16 These efforts are similar to the Baldrige national quality award programme.17

Armstrong et al18 suggested a systems approach to improving medical education: (1) defining an organisation’s objectives by characteristics (volume, variety, features) of the services provided; (2) designating who should do what specific work in what sequence for what other person to achieve organisational objectives; (3) designating characteristics for services one person provides, given the capabilities and needs of processes that precede and follow; and (4) determining how the individual work assignments should be done, given their relationship to the system overall.18

Currently, the Schools of Medicine, Nursing, Pharmacy and programs in Health Administration participating in the Institute for Health Care Improvement’s Health Professions Education Collaborative have made a commitment to “model improvement principles in our own work as education leaders.”19 The work described in this paper illustrates how the MUSOM, a founding member of the Collaborative, is seeking to fulfil that commitment.

Limitations

The current study is limited in that it reports one medical school’s experience. There was no opportunity to create a control group, as we believed it unethical to continue the baseline system for our students. However, we are unable to identify other factors that would have led to the changes in resource use (including faculty/staff time) and student appeals if the original system had been continued for the Class of 2006.

Conclusions

By applying continuous improvement principles to the MSPE, we were able to enhance the efficiency of the process. The MSPE committee was able to spend less cognitive energy on structure and format issues and more on the content of the letters. The intervention enhanced transparency for faculty and students. Integrating the MSPE with established information systems enhanced the data integrity of existing systems. We have completed four MSPE cycles using the new process. After each cycle, we have identified and implemented additional enhancements.

REFERENCES

View Abstract

Footnotes

  • Competing interests: None.

  • Ethics approval: Ethics approval was provided by University of Missouri Health Sciences Institutional Review Board (project no 1053839).

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Linked Articles