Article Text

Download PDFPDF

Can first-year medical students acquire quality improvement knowledge prior to substantial clinical exposure? A mixed-methods evaluation of a pre-clerkship curriculum that uses education as the context for learning
  1. Allison Brown1,2,
  2. Aditya Nidumolu2,
  3. Alexandra Stanhope3,
  4. Justin Koh2,
  5. Matthew Greenway2,4,
  6. Lawrence Grierson2,4,5
  1. 1 Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada
  2. 2 Undergraduate MD Program, DeGroote School of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ontario, Canada
  3. 3 Hamilton Health Sciences, Hamilton, Ontario, Canada
  4. 4 Department of Family Medicine, McMaster University, Hamilton, Ontario, Canada
  5. 5 McMaster Education, Research, Innovation and Theory, McMaster University, Hamilton, Ontario, Canada
  1. Correspondence to Dr. Lawrence Grierson, McMaster University - David Braley Health Sciences Centre, 100 Main Street West, Hamilton ON, L8P 1H6, Canada; griersle{at}mcmaster.ca

Abstract

Background Quality Improvement (QI) training for health professionals is essential to strengthen health systems. However, QI training during medical school is constrained by students’ lack of contextual understanding of the health system and an already saturated medical curriculum. The Program for Improvement in Medical Education (PRIME), an extracurricular offered at the Michael G. DeGroote School of Medicineat McMaster University (Hamilton, Canada), addresses these obstacles by having first-year medical students engage in QI by identifying opportunities for improvement within their own education.

Methods A sequential explanatory mixed-methods approach, which combines insights derived from quantitative instruments and qualitative interview methods, was used to examine the impact of PRIME on first-year medical students and the use of QI in the context of education.

Results The study reveals that participation in PRIME increases both knowledge of, and comfort with, fundamental QI concepts, even when applied to clinical scenarios. Participants felt that education provided a meaningful context to learn QI at this stage of their training, and were motivated to participate in future QI projects to drive real-world improvements in the health system.

Conclusions Early exposure to QI principles that uses medical education as the context may be an effective intervention to foster QI competencies at an early stage and ultimately promote engagement in clinical QI. Moreover, PRIME also provides a mechanism to drive improvements in medical education. Future research is warranted to better understand the impact of education as a context for later engagement in clinical QI applications as well as the potential for QI methods to be translated directly into education.

  • graduate medical education
  • health professions education
  • medical education
  • quality improvement

Statistics from Altmetric.com

Introduction

Quality Improvement (QI) is essential for patient safety and an ever-strengthening healthcare system. Accordingly, both the Canadian Medical Education Directions for Specialists Competency Framework and the American Accreditation Council for Graduate Medical Education have identified QI in medical education as an area for strategic focus.1 2 Consequently, training strategies are required to ensure all future physicians have the foundational QI knowledge and skills to be able to implement changes to identified needs.3 4 This is typically achieved through applied clinical projects throughout medical training. The majority of QI curricula described in the literature occur at the clerkship and residency level, involving a mix of didactic and experiential learning opportunities, wherein learners complete QI projects directly within the health system in which they train.3 5–7 However, this literature consistently indicates that these approaches to training are constrained by the infrequent availability of opportunities and restrictive limits on resources and participant (ie, faculty and learners) time and energy. As a consequence, many of these projects are not seen through to completion, such that the intended learning is not always achieved.4 8 Fittingly, educators have started to consider the possibility of delivering QI curricula to learners at an earlier, pre-clinical stage of their medical training.9

Of particular note in this regard is the Program for Improvement in Medical Education (PRIME), a 12-week extracurricular activity that is offered to first-year medical students at McMaster University (Hamilton, Canada) to facilitate their acquisition of fundamental QI skills and knowledge. The most distinguishing feature of PRIME is that it operationalises QI training through experiential approaches that challenge the students to acquire QI skills and knowledge through applied projects that are situated directly within the context of their pre-clerkship educational experiences (see online supplementary appendix A for curriculum overview). Specifically, enrolled students participate in a facilitated workshop that introduces the fundamentals of quality improvement, before forming small groups (~3–4 people) that are challenged to identify a ‘quality gap’ in their own medical education and to use the Model for Improvement to propose a small-scale intervention to address these identified gaps.10 Early in the process, each group submits an abstract that describes their quality gap according to the Institute of Medicine’s Six Domains of Quality, a potential intervention, and an aim statement.11 This abstract is reviewed and feedback is provided to refine the project development and to ensure that true deficits in educational quality are identified rather than issues that may be resolved in due course (eg, the identification of insufficient anatomy exposure when the musculoskeletal unit has yet to occur). Student groups then develop full project charters using a standard clinical QI project format that include descriptions of implementation plans for small scale changes and Plan-Do-Study-Act (PDSA) cycles, which are assessed and form the basis for further feedback.

Supplemental material

The objectives of PRIME are to facilitate the early acquisition of basic QI knowledge and to highlight the importance of quality improvement in medical practice so that participants are encouraged to pursue future QI training. In this regard, recent evaluations of PRIME indicate that those that complete the program are able to apply QI concepts effectively to the development of an implementation plan for an educational improvement intervention, report increased comfort with the application of QI principles, and are motivated to participate in future QI training in clerkship.9 Most importantly, the program is designed to promote the acquisition of QI skills and knowledge that are applicable in the clinical context. With this in mind, the instructional strategy of conducting the training in the context of familiar pre-clerkship education is based on the idea that the fundamental concepts of successful QI implementation are not tied to the specific context of application. Rather, exposure to the structure of the concepts that comprise the foundation of the QI process provides learners a framework, or scaffold, that can be used to transfer acquired skills and knowledge across contexts that are characterised by a variety of surface features.12 The influence of learning foundational structural concepts to cross-contextual learning has been demonstrated in studies concerned with the relevance of basic biomedical sciences training to the performance of diagnostic skills, but not yet with respect to the transfer of QI skills and knowledge.12–15 This study takes a first step in addressing this gap.

To do so, medical students enrolled in two successive offerings of PRIME (2016; 2017) completed the Quality Improvement Knowledge Application Tool (QIKAT-R) prior to and following completion of the program. The QIKAT-R is designed specifically to appraise the application QI knowledge in the clinical context and is the current standard for assessments of QI curricula.12 In particular, the tool presents three clinical scenarios wherein participants are expected to apply the core principles of the Model for Improvement and answer questions pertaining to the identification of an appropriate intervention, the aim of the improvement and how one would measure the impact within the system. Our primary hypothesis is that if engagement in the PRIME program has relevance to clinical applications of QI knowledge, then the resulting scores on this clinically situated assessment tool will reveal improvements in the participants’ performances at the end of the program. A lack of improvement on these assessments would then be indicative that the learning provided by PRIME does not have clinical relevance.

In recognising the potential of educationally situated QI training for pre-clerkship students and for institutions that view the resulting projects as meaningful markers for improvements within medical school curricula, we are also interested in understanding more about the student experience during this type of training. As such, this study also includes the collection of secondary data, which informs on participants’ comfort with various QI techniques and concepts and their perceptions of the value of the training they have received. In particular, participants will also complete pre-PRIME and post-PRIME iterations of the Quality Assessment and Improvement Curriculum (QAIC) toolkit—a self-assessment survey that measures comfort with various QI skills—and participate in focused interviews that are designed to yield participant testimonials that will support the further the development of QI training programs for pre-clerkship medical students.

Methods

Design

This sequential explanatory mixed-methods study involved a repeated measures quantitative design with both the QIKAT-R and QAIC instruments, and a qualitative component that involved semi-structured interviews. The sequential explanatory mixed-methods design was selected because it would provide a richer understanding of a phenomenon that is more comprehensive than each individual method on its own.16 17 The follow-up qualitative component specifically aimed to develop an understanding of the student experience during PRIME, including how and why the use of education may be an effective context for learning quality improvement.18

Participants

First-year medical students from the Michael G. DeGroote School of Medicine at McMaster University were eligible to participate in PRIME across program iterations run in 2016 and 2017. Eighty-three of 203 medical students in the class of 2018, and 117 of 203 medical students in the class of 2019 completed the program. Program participants were invited to participate in a mixed-methods study evaluating the impact of PRIME on medical student knowledge, skills, and attitudes towards QI. All participants provided written consent at the outset of this study.

A convenience sample, including 77% (64/83) of participants from 2016 and 50% (59/117) participants from 2017, completed the pre-PRIME and post-PRIME instruments. In selecting participants for the interviews, purposive, stratified, and maximum variation sampling was used to recruit participants from the 2016 iteration of PRIME. Maximum variation sampling was used to identify participants who had varying experiences during the program (ie, not all in the top scoring groups or most vocal about their enthusiasm for QI during the program). Participants were also stratified by campus in order to examine perspectives from those at each of McMaster’s three campuses, as PRIME was largely supported by the Niagara Regional Campus, to ensure that non-Niagara students were represented in the sample and examine differences in student experience across all sites.

Data collection

Two instruments were administered prior to (pre) and following completion (post) of PRIME. The first instrument, QIKAT-R, is the current standard for evaluations of QI curricula and was the primary outcome of interest for this study.19 The QIKAT-R presents three clinical scenarios, wherein participants are expected to apply their knowledge of QI. Responses are assessed through three questions1: (1) What would be the aim?2 (2) What would you measure to assess the situation? and,3 (3) Identify one change that might be worth testing. Ultimately, the QIKAT-R assesses participants' ability to apply the core principles of the Model for Improvement and primarily allow us to examine the impact of this education on clinical QI knowledge. QIKAT-R scores were independently assessed by two blinded raters using the standardised scoring rubric.19 The second instrument, QAIC, was administered to participants as a self-assessment survey which involves 12 items that assess comfort with various QI skills, and include response options on a 4-point adjectival scale (1=not comfortable at all to 4=extremely comfortable).20 This secondary outcome measures the learner’s perceptions of their QI knowledge and skills, which we explored further in the qualitative arm.

Independent semi-structured interviews were conducted with 11 participants during the month following program completion (July 2016) by an investigator with interviewing experience that was not involved with the program. An interview guide, consisting of a series of open-ended questions, was used initially but revised after the first three interviews to modify wording (see online supplementary appendix B for interview guide). Interview questions explored their overall experience during PRIME, the various components of the program, as well as their perceptions and attitudes towards QI.

Each interview ranged between 30 and 75 minutes in duration, and was audio-recorded and transcribed verbatim. Transcripts were cleaned manually to remove any personal identifiers and to ensure accuracy of the transcripts prior to analysis. No further interviews were arranged following the interviews with the 11 participants as two investigators felt that interview responses became redundant, no new themes had emerged, and that saturation was reached.

Data analysis

Inter-rater reliability of the QIKAT-R was assessed using an Intraclass Correlation Coefficient (ICC) for each scenario and overall total scores. QIKAT-R measures were determined by way of the mean rater scores for each scenario (out of 9) and for the total test (out of 27). These variables were then analysed in a paired t-test (pre; post) in order to determine the influence of the program on the students’ QI knowledge.

Item means and scale totals were calculated for the QAIC instrument. These were analysed using a paired t-test (pre; post) to highlight the influence of the program on students’ comfort with QI concepts. Internal consistency of the QAIC was determined by way of Cronbach’s method.

Interview transcripts were analysed in NVivo V.11 using qualitative analysis techniques adapted from constructivist methodological approaches.21 Open, focused, and axial coding were conducted by two investigators independently. Open coding involved line-by-line reading of the data, whereby each investigator independently defined what was occurring by coding all events in the transcripts inductively. Then, the investigators compiled their initial codes and met to consolidate them into a list of common themes emerging from the data. These themes were used to re-analyse and organise data during focused coding. Constant comparative analysis allowed for the investigators to compare data across transcripts to analyse how themes were emerging between participants and the data. The themes were also further explored by each investigator independently, which confirmed thematic saturation, and allowed for a deeper understanding of nuances that were present in the data. Finally, during axial coding, focused codes were classified into broader categories. These were analysed by the investigators to describe the overall phenomenon present throughout the data, examining relationships between categories and the outcomes of these interactions.

Following each phase of analysis (open, focused, and axial), the two investigators met to review codes to ensure consistency and agreement on coding. Contradictory observations were addressed and resolved. These meetings allowed for ongoing investigator triangulation, and promoted trustworthiness of results. An audit trail was kept by an investigator to track decision making throughout the qualitative analysis process.

Results

Impact on clinical QI knowledge

ICC between the total QIKAT-R scores between the two raters in cycle 4 was 0.725 and 0.759 in cycle 5, suggesting high consistency between raters and inter-rater reliability. Increases in knowledge scores on the QIKAT-R were observed for all three scenarios following PRIME (P<0.05) (table 1). Objective QI knowledge on the QIKAT-R had a mean increase of 6.0496±0.5477 (95% CI 7.6206 to 9.7893, P<0.05).

Table 1

Paired t-test of QIKAT-R by Scenario and Total Score (n=123)

Impact on student comfort and experience

Comfort with all 12 QI principles increased on the QAIC following PRIME (P<0.05) (table 2). Overall, QAIC total scores increased a mean of 13.365±7.712 (95% CI 11.98925 to 14.74246, P<0.05). Cronbach’s alpha was 0.907 for the combined pre-PRIME and post-PRIME scores. When calculated separately, the pre-PRIME Cronbach’s alpha was 0.927 and post-PRIME alpha was 0.901, suggesting that the QAIC assessment of comfort has high internal consistency among subjects.

Table 2

Item descriptive statistics and paired t-tests of QAIC (n=123)

Analyses of the interviews revealed key themes and shared experiences across PRIME participants. In particular, program components were identified as catalysts to this learning process, including: continuous support throughout the program, genuine interest in improving medical education, team-based learning and problem solving, regular project feedback, and access to QI education resources. These combined aspects provided first-year medical students with a positive environment for early learning of QI.

Specifically, using education as a context for learning QI allowed medical students to identify a personally meaningful area for improvement within the existing curriculum:

I felt a lot more tied to the project, I guess, because it was about medical education so you feel a lot more connected to it versus research in other areas …

Participant #2

A consistent report from participants was that the program nurtured new attitudes and perceptions about QI, which resulted in a desire to participate in future ‘real-world’ improvement efforts.

I guess also just on the whole, I feel like I have a better understanding of the scope of quality improvement and how significant of a thing it is when you decide that you’re going to do this. I guess I just kind of had this idea that when you’re out there in healthcare and you see an area for improvement, then you’ll just implement whatever you think it needs and see what happens, but really there’s a systematic way to do things… I feel like I have a much more realistic and systematic idea now of what it means to make positive changes in the area you work.

Participant #10

­

I think moving forward as a clerk and as a resident, using these QI skills to make operations better in the hospital is going to be very important. And you need to have a basic understanding of QI in order to actually be able to implement a project that is going to be effective in making small scale changes towards a better outcome.

Participant #7

Participants noted a number of challenges that were informative to the way the program will be conceptualised in the future. A common limitation of the program was the lack of implementation of their ideas given that PRIME ended at the project charter stage, as participants described wanting to carry out their project ideas as well as pursue further opportunities in QI. A second challenge described by participants was the perceived value of QI by academics and medical leaders. Participants shared concerns that this type of scholarly activity would not be as valued as traditional research when applying for competitive residency spots in the future. Altogether, the use of education as the context for learning enhanced participant comfort with QI, allowing them to feel competent in the basic tools and techniques and encouraging applications in the real world that could ultimately generate tangible improvements.

Discussion

QI training is essential to prepare future physicians for a role in improving the health system. As a result, medical schools are increasingly driven to teach students QI, but are limited in resources, expertise, and space in an already busy curriculum. While a combination of didactic and experiential QI curricula is recommended, the literature has shown mixed results in their effectiveness at the undergraduate level.3 Medical students struggle to understand the importance of QI during the pre-clerkship period, because their focus is directed towards learning clinical content and they lack knowledge of the health system. Arguably, pre-clerkship is the ideal time to begin QI training.22

In this study, we investigated whether a pre-clerkship QI training offering that situates experiential learning activities within the context of pre-clerkship education (rather than a clinical context) leads to the acquisition of QI knowledge that can be effectively applied to the clinical context. This was accomplished by administering the QIKAT-R, an assessment tool for QI knowledge that is clinically contextualised, to learners prior to and following their involvement in PRIME, with the hypothesis that transferable learning from the educational context should yield meaningful improvements on the assessment tool with the clinical relevance. In this regard, overall the data point to PRIME as an effective training experience, which promotes the development of basic clinical QI knowledge. This is supported by the increases in scores following PRIME on the QIKAT-R, which assess the ability of a participant to apply QI concepts to three hypothetical clinical scenarios. Interestingly enough, the increases observed on the QIKAT-R are comparable to QI curricula evaluations at the residency level.23 24 As such, the present results indicate that this foundational QI knowledge is potentially transferable for application into real-world clinical environments.25–27

This may, in part, be because medical education represents a context that students can understand more directly than the clinical environments with which they have limited exposure.4 This finding resonates with the idea that training that focuses on the fundamental concepts that underpin a skilled process can be effective in promoting transfer of that learning into new contexts, which potentially differ in their outfacing characteristics.12–15 28 29 In this study, medical students were able to transfer their knowledge to solve problems in the unfamiliar context of the health system, suggesting that a program that uses education as the system by which students learn QI can prepare students for future learning. To our knowledge, this is the first QI curricula described in the literature with participants in their first year of medical school learn QI in the context of education and demonstrate a transfer of knowledge to the clinical context.

In addition to acquiring QI knowledge, the secondary outcomes of interest suggest that PRIME positively influences medical students’ comfort with and perceptions of QI. Several of the 12 items on the QAIC assess similar concepts to the QIKAT-R, including the participants’ ability to develop an aim (item A), define measures (item C) and make a change within the system (item E). Increases in comfort with these items following PRIME also remains consistent with their increases in QIKAT-R scores, which objectively assessed their ability to perform those aspects of QI. The largest increases following PRIME were two questions assessing participants’ comfort with using small cycles of change (item G) and the PDSA framework (item J), which can be considered methodological hallmarks of QI.

Furthermore, emergent themes from the interviews were congruent with the results from the quantitative components of the study, as students consistently reported feeling more competent in their QI abilities following the program. The qualitative portion highlights that teamwork, autonomy in selecting an area for improvement,5 and access to program supports6 7 were key factors that underpinned student learning throughout the program. Although this was an education-focused curriculum, a common theme that emerged in the analysis of the interviews were that medical students were already making the connection to QI in the clinical environment and wanted to further their QI training through clinical applications that lead to real-world improvements. In previous cycles of PRIME, 100% of medical students who completed the program further participated in applied clinical QI projects during clerkship.9 This may be a strategy of engaging medical students early on in their training and foster the development of QI champions in the future.

Recognising that medical education is a system with improvement needs, QI may be an appropriate methodology to enhance the quality of medical education programming itself. Throughout PRIME, first-year medical students were able use new QI knowledge to identify perceived gaps in their own medical curriculum, and design interventions for the improvement of education quality that will directly impact their classmates and future cohorts of medical students. PRIME’s influence may eventually extend beyond student learning, and serve as a catalyst for critical systems thinking that encourages students to take on active roles in improving their own educational experience. Medical students are eager to engage in the improvement of medical education and to bring unique perspectives to educators. In this regard, they may be ‘untapped change agents’.30 As such, students should be involved in activities that seek to improve the quality of medical education, similar to the manner that patients should be engaged in healthcare improvements.

A limitation of this study is that both the QIKAT-R and QAIC assessment tools are restricted in their ability to comprehensively evaluate QI knowledge and skills to determine if one is truly competent in QI. However, the observed increases in QIKAT-R scores were comparable to prior studies examining residency-level QI training.23 24 Ideally, to assess clinical QI competence we would observe student performance during real-world applications, longitudinal performance in QI activities and clinical and system-level improvements as the ultimate end point. It remains unknown how participation in PRIME enhances clinical QI applications, potentially making them more efficient and ultimately more effective in leading to sustained improvements. Future research will seek to examine how the early engagement in QI using education as a context influences future performance in the clinical realm.

Conclusion

QI training using a clinical context can seem abstract for medical students who, at an early stage of their training, have little experience with health systems. The promising results of this study suggest that medical education may be an appropriate context to teach pre-clerkship medical students about quality improvement, leading to an increase in knowledge of and comfort with clinical QI principles. Our position is that this is likely to foster positive attitudes and behaviours towards QI that may promote later participation in clinical QI activities. Moreover, utilisation of quality improvement in the context of medical education may simultaneously act as a catalyst for improvements to educational programming. Further research is warranted to better understand the potential for this methodology in medical education and the impact of PRIME on student involvement in clinical QI activities.

Acknowledgments

The authors wish thank the members of the PRIME Working Group and the medical education community at McMaster University who made this program possible.

References

Footnotes

  • Contributors Each author contributed to the conception of the study, the study design, writing and critical review of the manuscript. JK conducted the interviews and coding of the qualitative data with AB. AS and DN scored the QIKAT instruments independently. Each author approved the final version of the manuscript.

  • Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent Obtained.

  • Ethics approval This study was granted approval from the Hamilton Integrated Research Ethics Board (HIREB File #0930).

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.