Article Text
PDF
Abstract
Background With the integration of quality improvement (QI) into competency-based models of physician training, there is an increasing requirement for medical students and residents to demonstrate competence in QI. There may be factors that commonly facilitate or inhibit the desired outcomes of QI curricula in undergraduate and postgraduate medical education. The purpose of this review was to synthesise attributes of QI curricula in undergraduate and postgraduate medical education associated with curricular outcomes.
Methods A realist synthesis of peer-reviewed and grey literature was conducted to identify the common contexts, mechanisms, and outcomes of QI curricula in undergraduate and postgraduate medical education in order to develop a programme theory to articulate what works, for whom, and in what contexts.
Results 18854 records underwent title and abstract screening, full texts of 609 records were appraised for eligibility, data were extracted from 358 studies, and 218 studies were included in the development and refinement of the final programme theory. Contexts included curricular strategies, levels of training, clinical settings, and organisational culture. Mechanisms were identified within the overall QI curricula itself (eg, clear expectations and deliverables, and protected time), in the didactic components (ie, content delivery strategies), and within the experiential components (eg, topic selection strategies, working with others, and mentorship). Mechanisms were often associated with certain contexts to promote educational and clinical outcomes.
Conclusion This research describes the various pedagogical strategies for teaching QI to medical learners and highlights the contexts and mechanisms that could potentially account for differences in educational and clinical outcomes of QI curricula. Educators may benefit from considering these contexts and mechanisms in the design and implementation of QI curricula to optimise the outcomes of training in this competency area.
- graduate medical education
- healthcare quality improvement
- health professions education
- medical education
Statistics from Altmetric.com
- graduate medical education
- healthcare quality improvement
- health professions education
- medical education
Background
Over the last 20 years, quality improvement (QI) has increasingly become integrated into numerous competency frameworks for physician training.1–4 This formal integration of QI as a core competency reflects the importance of physician engagement with system-level efforts to improve the quality and safety of care. As a result, QI training is becoming integrated into the core medical curriculum. Previous systematic reviews on QI in medical education have reported that the majority of QI training occurs at the postgraduate (ie, residency) level5 6; didactic and experiential learning is one of the most effective strategies6; evaluations typically report educational outcomes rather than clinical outcomes7; clinical outcomes are more likely to be achieved through projects, interprofessional teams and with coaching8; and evaluations of QI curricula are generally of poor methodological quality.9 A 2015 realist review that examined QI curricula suggested that clear communication about the learner’s time requirements, available data for performance indicators, autonomy in project selection and faculty guidance were common factors that contribute to successful QI curricula.10 This review synthesised findings from 39 studies published in the literature until March 2013 and excluded grey literature (eg, curriculum reports and abstracts), which can be included in realist syntheses and could be potentially valuable in answering realist research questions.11 Since 2013, QI has become further embedded into competency frameworks and additional evaluations of QI curricula have been published.
With the adoption of outcomes-oriented, competency-based models of training, medical schools and residency programmes may be increasingly incorporating QI training into their core curriculum to meet the formal requirements that all trainees demonstrate competence in this domain. There is a need to identify optimal strategies for teaching by delineating the key factors that may be associated with educational and clinical outcomes of successful QI curricula.
The objective of this study was to synthesise the existing literature on QI in medical education and to identify the contexts and mechanisms that promote optimal education and clinical outcomes. The focus was on physician trainees at the undergraduate medical education (UGME) level (ie, medical students) and postgraduate medical education (PGME) level (ie, resident physicians, interns, house staff, and fellows). A realist synthesis was conducted to examine interactions between contexts, mechanisms, and outcomes to answer the overarching question of ‘what works, for whom, in what contexts?’. To answer the overarching research question, the following subquestions were posed:
In what contexts does QI education for physician trainees occur?
What are the key mechanisms that can facilitate or inhibit learning QI for physician trainees, and in what contexts are they triggered?
What are the characteristics that facilitate learning QI for physician trainees, and what outcomes do they have?
Methods
Realist synthesis (also called a realist review) is an approach to knowledge synthesis that focuses on understanding how and why an intervention may work through the exploration of the relationships between the contexts and mechanisms that may be associated with various outcomes of an intervention (see Pawson et al for an overview of realist review methods, and Wong et al regarding the value of the methods in medical education).12 13 Contexts refer to the features that can influence the implementation and uptake of a programme, including macro-level factors (eg, organisational culture nd political climate) and micro-level factors (eg, curricula and participants). Mechanisms are underlying processes that are triggered in a context to generate a specific outcome.11 14 In contrast to other approaches (eg, systematic reviews), realist reviews can be inclusive of a wide range of methodological approaches to generating knowledge (ie, does not only focus on experimental studies) and diverse forms of evidence (eg, grey literature). A primary goal of realist synthesis is the development of an explanatory model, called a programme theory, to visualise the relationships between contexts, mechanisms, and outcomes.12 The use of realist methods to synthesise evidence and develop explanatory programme theories have been increasingly used in health services research.10 15–17
This study followed the Realist and Meta-narrative Evidence Syntheses – Evolving Standards (RAMESES) guidelines and associated training materials.11 18 A summary of the approach to this synthesis is outlined in online supplemental appendix A.
Supplemental material
Search strategy, selection and appraisal
A search of the terms ‘Quality Improvement’ and ‘medical education’ was conducted in the peer-reviewed and grey literature. No methodological filter was used to select articles (ie, screening for certain study designs); rather, the relevance to the realist research question was used.11 18 References were imported into Covidence software (Veritas Health Innovation, Melbourne, Australia) for screening and data management. The search was not restricted by date and included records published before 23 August 2018.
Two independent reviewers screened titles, abstracts and full-text records. One author (AB) screened all records, and the remaining authors were involved as second reviewers during each stage. This was important in providing multiple perspectives throughout the research as the primary author has previously been involved in developing QI curricula in medical education. All disagreements were resolved through discussion, revisiting the original record, or involvement of a third reviewer.
Data extraction
An instrument was designed to extract information regarding the features of the QI curriculum, including contexts, mechanisms, and outcomes (online supplemental appendix B). Outcomes were classified based on the Kirkpatrick Four-Level Model19 given that this model is commonly used to evaluate QI curricula in the literature that examines reaction (level 1, ie, learner satisfaction), learning (level 2, ie, knowledge and skill acquisition), behaviour (level 3, transfer of new knowledge and skills to the workplace or future behaviours relating to QI), and results (level 4, ie, system-level improvements to processes or patient care).6 20 21 Data from each record were extracted by two individuals (AB and a second investigator) using the standardised form and then consolidated into one record. Discrepancies or disagreements were resolved through revisiting the original record and discussion.
Supplemental material
In accordance with the RAMESES materials, records were scored on a five-point scale for their relevance to the research question, as well as their rigour.11 A decision was made to not use established critical appraisal tools in the assessment of rigour due to the volume of records and the focus on identifying contexts, mechanisms, and outcomes rather than the methodological quality of the literature for QI in medical education, which has been previously examined.9 11
Data analysis
Mean relevance and rigour scores were calculated between reviewers and used to sort the data. Records were sorted by highest relevance to the research question and analysed until the programme theory became saturated. Previous reviews were excluded from the analysis in order to develop the programme theory inductively.
Patterns and relationships between contexts, mechanisms, and outcomes among the remaining records were explored to develop the refined programme theory.22 23 The frequency of each construct as well as the intensity was analysed. For instance, one aspect of the data extraction instrument required members of the research team to extract data regarding barriers and facilitators to QI education explicitly described by the authors of each record. Factors explicitly highlighted in a record as barriers and facilitators to the success of the curriculum were considered to have more ‘intensity’ than those extracted from the record without being clearly specified as such. Records that did not contribute to the development or refinement of the programme theory were excluded from the synthesis. To summarise the distribution of the data, a data visualisation strategy was emulated from a previous realist review to present the frequency of records affiliated with contexts, mechanisms, and outcomes.15
Results
A total of 18 854 unique records were initially screened, of which 609 records underwent full-text assessment and 358 underwent data extraction (figure 1). One hundred and forty of these records did not contribute to the development or refinement of the programme theory and were excluded. Two hundred and eighteen records were included in the final synthesis.
PRISMA diagram. PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.
Table 1 summarises the characteristics of the included records. Of these records, 87% were from the USA and 75% described curricula at the postgraduate level of training. A hybrid curriculum with experiential learning was most often described, and the Model for Improvement was the most common QI approach taught.
Characteristics of records (n=218) included in the programme theory
The final programme theory (figure 2) provides a visual summary of the relationships between the various contextual factors and mechanisms associated with educational and clinical outcomes of QI curricula. A high-level summary of these factors are summarised in table 2 (see online supplemental appendix C for the frequency of records for each variable and the associated references).24–241
Supplemental material
Refined programme theory. This programme theory provides a simplified visualisation of the relationships between contextual factors (left), mechanisms (centre), and outcomes (right) for QI curricula. Aspects that have an overall stronger association with successful QI curricula have been darkened to reinforce their importance as a successful component of QI curricula based on the frequency of records associated with each category and the association between these variables and the desired outcomes of QI curricula. CME, continuing medical education; PGME, postgraduate medical education; QI, quality improvement; UGME, undergraduate medical education.
Summary of contexts and mechanisms associated with successful outcomes of QI curricula
Contexts
Contexts included the level of training, the primary curriculum strategy, the setting, and factors relating to the local environments. Fewer curricula were described at the UGME level, but effective strategies included: longitudinal scholarly concentrations in QI that spanned the entire duration of UGME, with experiential activities typically during clerkship. Medical student involvement in QI projects was most successful when they worked on preidentified or ongoing projects aligned with the priorities of the healthcare system, in heavily mentored teams, with clear expectations and protected time. The majority of QI curricula were at the PGME level, almost half of which were in Internal Medicine programmes. Longitudinal curricula spanning multiple years were commonly described that combined didactic and experiential learning (ie, resident QI projects) in the ambulatory or outpatient clinic. Additional strategies included fellowships or certificate programmes, short-term curricula (eg, rotations), the involvement of residents in ongoing, short-term projects during inpatient rotations, and monthly conferences (eg, QI-focused morbidity and mortality rounds), but these strategies were less predicated on educational and project outcomes. A ‘chief resident for quality and safety’ or ‘resident champion’ was described where senior residents coordinated QI teaching, mentored learners, led projects, and were members of hospital QI and patient safety committees—all while being simultaneously mentored themselves. At an organisational level, the presence or perception of a pre-existing quality and safety culture, explicit support for the curriculum, and alignment between the curriculum and the local health system were noted by authors as a factor that contributed to the success of the curriculum.
Mechanisms
Mechanisms were identified that generally supported the curriculum, learning QI, and projects. General mechanisms included: clear expectations and deliverables, protected time, mentorship, feedback, curriculum leadership, and incentives. First, highly structured curricula with clear expectations, concrete timelines, and deliverables were explicitly described by the authors as a facilitator. Second, protected time was associated with increased learning and project success. Holding didactic sessions during protected academic time ensured that all learners were present to learn QI concepts and revisit their QI projects on a recurrent basis to sustain momentum. Third, mentorship was one of the most commonly noted facilitators of successful QI curricula. Mentors were typically physicians or senior residents with formal QI training and a familiarity with the project contexts (ie, clinical area), who liaised with the healthcare system to engage stakeholders, meet with learners to review project progress, navigate challenges, and oversee team performance to ensure all learners are contributing. Several curricula highlighted the importance of assigning mentors early rather than requiring learners to find their own mentors. In addition, the provision of feedback concurrent with project activities reinforced QI principles and strengthened project efforts. Fourth, explicit roles for curriculum leadership (eg, QI directors) supported the overall curriculum. Finally, academic and financial incentives were noted as a mechanism.
Learning mechanisms were associated with both educational and clinical outcomes. First, content delivery strategies were associated with improved knowledge as well as project outcomes. This included the ‘flipped classroom’ technique (ie, self-directed content acquisition ahead of in-person sessions) and ‘just-in-time’ learning (ie, delivery of QI content over time to allow learners to apply content to the stage of their project). The majority of successful curricula were hybrid models that taught the Model for Improvement, typically in the form of lectures or workshops, with experiential learning. Online modules (ie, Institute for Healthcare Improvement Open School modules) or resources (eg, custom workbooks) were used in combination with didactic sessions. The Open School modules were perceived as ineffective and were later eliminated from several curricula.42 45 69 72 108 188 197
Project mechanisms supported the conceptualisation, implementation, and sustainability of projects and were associated with health system outcomes. First, project screening mechanisms ensured appropriate scope and alignment with healthcare system priorities. While providing learners with the autonomy to choose project topics was associated with increased satisfaction, learner-identified topics that did not undergo screening were less likely to be completed. Preidentification of topics was also described. Hybrid approaches offering autonomy in selecting from preidentified projects were associated with learner engagement, satisfaction, and project success. Shared projects where learners participated in ongoing projects were associated with clinical improvements but less associated with educational outcomes. Second, frequent meetings with team members and the mentor were associated with both learning and project outcomes. Third, access to existing data or assistance with data retrieval and analysis allowed learners to measure the impact of their projects and was associated with project success. Fourth, project handover mechanisms helped sustain project momentum for short-term or shared projects. Fifth, working in teams was a prominent mechanism, including interprofessional teams with a wide range of practising health professionals and learners. Working in interprofessional teams was described with challenges in arranging meetings, professional boundaries, and perceived hierarchies. Uniprofessional teams (ie, medical learners only) were most common, often forming around project topics or shared settings (eg, ambulatory clinic). Vertical, uniprofessional teams involving multiple levels of learners (eg, senior and junior residents and medical students) were also described. Challenges for teams (interprofessional and uniprofessional) included scheduling meetings, power dynamics, and maintaining engagement.84 86 166 171 172 The most effective group sizes were between two to five learners. Finally, recognition or incentives for top-performing groups was a final mechanism noted.
Outcomes
The majority of records reported level 1 (reaction; n=78) and level 2 (learning) outcomes. Fewer records (n=36) discussed improvements to learner attitudes (level 2A). The most commonly reported outcome were improvements to learner knowledge and skills of QI concepts and methods (level 2B; n=101). Only 24 of these records reported increases to objective QI knowledge using an established instrument, whereas the remainder used self-report or de novo assessment tools.
Thirty-six records reported changes in behaviours (level 3), including increased motivation to participate in future QI initiatives, mentor junior learners in QI and the pursuit of further training or leadership roles in QI. Eighty-seven records reported level 4 outcomes for improvements to clinical processes (level 4A; n=68) and/or to patient care (level 4B; n=35). Less than half (n=26) of level 4A outcomes provided numeric evidence of improvements to a process but included improvements to documentation, process efficiency, and screening. At level 4B, 15 records providing numeric evidence of improvements to patient care and 22 records provided anecdotal evidence.
Derived from the resulting refined programme theory and previous pedagogical arguments surrounding how best to teach QI to medical trainees, figure 3 proposes potential strategies for teaching and integrating QI across the continuum of medical education. While these suggested strategies do not represent any global consensus or gold standards for teaching QI, these may be useful to provide a preliminary roadmap of pedagogical strategies for teaching QI, particularly for educators designing and implementing new curricula.
A road map of pedagogical strategies for teaching QI in medical education. Note: potential key mechanisms listed above are those that may be particularly beneficial and do not exclude others outlined in the programme theory. PGME, postgraduate medical education; PS, patient safety; QI, quality improvement; UGME, undergraduate medical education.
Discussion
This realist synthesis has identified the contexts and mechanisms that are commonly associated with educational and clinical outcomes for QI curricula in UGME and PGME and visualises the relationships between these factors through a realist programme theory. The results of this research may be useful to inform the design and implementation of QI curricula by educators in close consideration with how these strategies might work in their own programmes based on various contextual factors. With the increasing adoption of competency-based, outcomes-oriented models at both the undergraduate and postgraduate levels of medical training, educators may be looking to modify the core curriculum in order to meet the defined outcomes of training articulated in these models.242–245
The findings from this research can be used to inform recommendations for educators when teaching QI to undergraduate and postgraduate medical learners, particularly surrounding how experiential learning could be optimised through the strategic integration of contexts and mechanisms identified throughout this research. Previous arguments in the literature suggest that learners require ‘legitimate peripheral participation’ in quality and safety initiatives, beginning with initial involvement with simple, yet meaningful, tasks followed by more immersive and embedded activities.246 The acquisition of QI competence could follow a trajectory similar to the acquisition of clinical knowledge and skills throughout medical education. A longitudinal curriculum at the undergraduate level could provide medical students with a foundational skill set in QI that can be further developed throughout postgraduate training. During preclerkship, didactic instruction regarding basic QI principles and its importance to the health system could be provided.247 This early exposure may be particularly beneficial to convey the value of QI before learners become desensitised to any dysfunctional aspects of the health system.248 249 Applications to non-clinical contexts (ie, education) may be another strategy to introduce novice learners to QI and establish a baseline understanding of QI concepts and methods before later applications within clinical systems.30 38 250 251 During clerkship, learners could become more involved in supervised, time-limited, experiential clinical QI activities (ie, predefined projects in a mentored team).247 The integration of QI throughout undergraduate training could then, theoretically, graduate cohorts of physicians who possess a baseline set of QI knowledge and skills that can later be built on through applied improvement activities during postgraduate training.252 This ideal is reflected in both the Association of Faculties of Medicine of Canada as well as the Association of American Medical College in their core Entrustable Professional Activities for entering residency, providing further support for the explicit emphasis on QI in the undergraduate curriculum.245 253 254
At the postgraduate level, educators could similarly consider their local training contexts and conscientiously design the QI curriculum with various mechanisms in mind. In general, longitudinal curricula involving increasing autonomy in applied QI projects appears to be an effective approach to not only promote knowledge, skills, and attitudes towards QI, but in reinforcing the desired behaviours relating to QI that might be an expected outcome of training. These curricula often spanned a year or longer, including the entire duration of residency; however, engagement of first-year learners was sometimes challenging. Experiential QI training can allow residents the opportunity to take on increasingly autonomous roles and make real-world improvements to patient care, but the specifics of the experiential curriculum may be heavily dependent on the context. For instance, programmes may not have continuity or ambulatory clinics (eg, surgical programmes), thus could consider alternative strategies (eg, short-term, team-based, shared projects). Towards the senior years of training, more advanced strategies could further develop and support advanced expertise and leadership in QI, such as the chief resident of quality and safety. A recent evaluation of fellowship programmes in QI and patient safety highlight their positive influence on the local culture and career trajectory of graduates, 96% of whom secured formal academic positions with a heavy emphasis on QI and patient safety leadership and education on completion of the fellowship.255
Several mechanisms identified in this synthesis could be integrated to enhance experiential activities, particularly regarding team structure, and project conceptualisation. To balance learning and project impact, two to five learners may be an ideal team size to maintain engagement and accountability, with an assigned team leader (eg, chief or senior resident) and mentor.68 172 The positioning of senior residents as team leaders or mentors may be beneficial in developing advanced leadership skills to increase faculty capacity for QI—a commonly cited barrier for QI curricula and a factor that could positively influence the organisational quality and safety culture.70 222 223 256 Frequent meetings and protected time may also benefit these teams, promote project progression and increase the likelihood that these projects will improve outcomes. Despite the project mechanisms identified in this synthesis, the factors that underpin successful project-based learning warrant future examination from a theoretical lens given the value of experiential QI curricula in promoting both educational and clinical outcomes257 258
A fundamental barrier to the development of QI competence among medical learners is the disintegration of QI and patient safety education with clinical care delivery.256 The alignment between the education and clinical contexts may be a precursor to the development of training environments where these topics are fully embedded. Project selection mechanisms may promote this alignment while maintaining trainee engagement. While medical students are often less experienced within the health system, they may struggle to identify project topics and may benefit from involvement in preidentified or ongoing projects. However, residents can be successful in identifying their own project topics with proper guidance and support.259 Allowing residents to reflect on their clinical experiences and identify topics of direct interest may foster motivation and sustain engagement throughout the longitudinal, experiential QI curricula.7 175 259 Ultimately, the combination of ‘top-down’ (eg, institutional QI projects and priorities) and ‘bottom-up’ (eg, learner-identified QI projects) approaches may be most effective in order to ensure institutional support and alignment from the health system while simultaneously maintaining trainee engagement and motivation.260 261 Academic and health institutions should work closely to function synergistically in supporting QI curricula—a concept referred to as ‘bidirectional alignment’.262 263 Regardless of the level of training, the mechanisms identified in this research may promote greater alignment between these two systems and potentially enhance the integration of QI into the clinical environments and augment curricular outcomes.256
Strengths and limitations
This synthesis is first limited by a potential publication bias, as unsuccessful curricula may be under-reported. Second, the scope of this synthesis could have been narrowed to focus on certain learners (eg, medical students), settings (eg, ambulatory setting), specialties (eg, surgical programmes), or other variables. Third, the outcomes reported in the literature to provide evidence of success vary widely, as the majority of evaluations reported short-term educational outcomes rather than long-term behavioural and patient care outcomes.
The quantity of evidence to inform the development of QI curricula at the undergraduate level of medical training warrants further research, particularly concerning how early strategies might promote QI engagement along the continuum of training. Evidence is needed to understand the impact of early QI training for medical students on later behaviours and QI capabilities throughout residency and professional practice, including the achievement of competencies in outcomes-based models.264
This synthesis did not examine the methodological quality of the records based on existing appraisal instruments, which would be expected in a systematic review, given the diversity of records and methods. The inclusion of grey literature was beneficial in providing details about curricula, the facilitators and barriers of curricular outcomes and, ultimately, in refining the programme theory.
This research has several strengths. It provides evidence to corroborate previous arguments surrounding how best to teach QI. While the previous 2015 realist review10 reported that communication about time requirements, available data, project choice, accounting for competing clinical demands, and locally QI-trained faculty were elements of QI curricula, the results of this synthesis delineate additional contexts and mechanisms associated with successful QI curricula and the relationships between these factors. To our knowledge, this is the most extensive review of QI curricula in medical education to date. The use of realist methods to synthesise diverse sources of evidence to develop a refined programme theory may assist in the development and improvement of QI curricula in medical education.
Conclusion
As QI has become recently emphasised within various competency frameworks for physician training and outcomes-oriented models of training are increasingly adopted, educators may be looking to develop and refine QI curricula. This research has illuminated the mechanisms that may enhance various components of the QI curriculum, the contexts in which they are triggered, and the educational and clinical outcomes that result. Educators may benefit from considering how the strategic alignment and integration of mechanisms with the local contextual factors could optimise the curricular outcomes to ensure that all future physicians become competent in this area throughout their core training.
Acknowledgments
The research team wishes to thank the Office of Health and Medical Education Scholarship for their financial support of this study as well as Drs Nishan Sharma and Deirdre McCaughey for their support as doctoral committee members for AB. The research team would also like to thank Dana Hunter for her early support on this project.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @allison_rants, @academialiya
Contributors AB conceptualised the study as part of their doctoral research, which was supervised by AK and KH. AB, KL, DF and AN screened titles, abstracts and full-text articles. All authors extracted data from the included articles. Each author reviewed and approved the final manuscript.
Funding This research was supported by a grant through the Office of Health and Medical Education Scholarship at the University of Calgary.
Competing interests None declared.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available on reasonable request. Our protocol and data are available on request.