You are here

Article Text

Article menu
Systematic review
Key characteristics of successful quality improvement curricula in physician education: a realist review
Free
  1. Anne C Jones1,2,3,4,
  2. Scott A Shipman2,3,5,
  3. Greg Ogrinc1,2
  1. 1Veterans Affairs Medical Center, White River Junction, Vermont, USA
  2. 2Geisel School of Medicine at Dartmouth, Hanover, New Hampshire, USA
  3. 3The Dartmouth Institute for Health Policy and Clinical Practice, Lebanon, New Hampshire, USA
  4. 4Gannett Health Services, Cornell University, Ithaca, New York, USA
  5. 5Association of American Medical Colleges, Washington, DC, Washington,USA
  1. Correspondence to Dr Anne C Jones, Assistant Medical Director, Gannett Health Services, 110 Ho Plaza, Cornell University, Ithaca, NY 14853, USA; acj22@cornell.edu

Abstract

Purpose Quality improvement (QI) is a common competency that must be taught in all physician training programmes, yet, there is no clear best approach to teach this content in clinical settings. We conducted a realist systematic review of the existing literature in QI curricula within the clinical setting, highlighting examples of trainees learning QI by doing QI.

Method Candidate theories describing successful QI curricula were articulated a priori. We searched MEDLINE (1 January 2000 to 12 March 2013), the Cochrane Library (2013) and Web of Science (15 March 2013) and reviewed references of prior systematic reviews. Inclusion criteria included study design, setting, population, interventions, clinical and educational outcomes. The data abstraction tool included categories for setting, population, intervention, outcomes and qualitative comments. Themes were iteratively developed and synthesised using realist review methodology. A methodological quality tool assessed the biases, confounders, secular trends, reporting and study quality.

Results Among 39 studies, most were before–after design with resident physicians as the primary population. Twenty-one described clinical interventions and 18 described educational interventions with a mean intervention length of 6.58 (SD=9.16) months. Twenty-eight reported successful clinical improvements; no studies reported clinical outcomes that worsened. Characteristics of successful clinical QI curricula include attention to the interface of educational and clinical systems, careful choice of QI work for the trainees and appropriately trained local faculty.

Conclusions This realist review identified success characteristics to guide training programmes, medical schools, faculty, trainees, accrediting organisations and funders to further develop educational and improvement resources in QI educational programmes.

  • Quality improvement
  • Medical education
  • Continuous quality improvement

https://doi.org/10.1136/bmjqs-2014-002846

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Background

    Stemming from the Institute of Medicine's reports, To Err Is Human in 2000 and Crossing the Quality Chasm in 2001, improvement in patient outcomes and reduction in medical errors are foci for healthcare institutions around the world.1 In 2003, the Accreditation Council for Graduate Medical Education (ACGME) and American Osteopathic Association responded by integrating systems-based practice (SBP) and practice-based learning and improvement (PBLI) as two of the six core competencies of medical education.2 Quality improvement (QI) teaching encompasses content of both SPB and PBLI. However, uncertainty remains about which methods are the most effective, and in what circumstances, for improving educational and clinical outcomes.3–5

    Reviews of QI teaching in undergraduate and graduate medical education have found some improvement in educational outcomes but little effect on patient outcomes.6–8 Wong et al9 ,10 identified three categories of QI education, most of which fall into the first category: (1) formal curricula that teach concepts or methods intended to facilitate trainees’ participation in QI activities; (2) educational activities that impart specific related skills and (3) QI initiatives that involve trainees as active or passive participants. Many others worldwide have developed clinical teaching of QI, aiming to engage physician trainees to improve the care of the patients they serve and the function of the system in which they practice.10

    Although helpful in summarising the novel approaches to QI education, prior systematic reviews have been limited. They appropriately sought to answer the question of whether or not QI educational interventions had an impact on physician trainees’ ability to gain knowledge and sought to identify themes associated with successful QI curricula, but did not assess the specific mechanistic and contextual factors that predicted success, especially for improvement of patient care and system performance outcomes. The realist review approach offers one approach to deconstruct such complex interventions to assess key success characteristics and develop recommendations.11 ,12

    In this study, we define key characteristics of successful QI curricula in medical education. This realist review determines how the teaching of QI in the clinical setting enhances patient care and system performance while increasing trainee knowledge and skills.

    Methods

    Review framework

    A realist review is based on the premise that complex interventions are successful when certain characteristics facilitate the optimal functioning of a system to produce a particular outcome when a complex intervention is applied.11 In a realist review, an iterative approach is used to identify the characteristics of complex interventions in the following categories: ‘what works,’ ‘for whom,’ ‘under what circumstances’ and ‘to achieve what outcomes’.11 The realist review begins with the articulation of candidate theories that may explain the characteristics required for interventions to be successful.11 Next, identification and selection of studies is achieved through standard systematic review approach.11 Once relevant studies are chosen for inclusion, data are systematically abstracted from the studies and the studies are read and reread to identify themes.11 An iterative approach is used to identify data, quotations, tables and figures that either support or refute the candidate theories articulated at the outset. Theories are refined as more data are gathered from the articles.11

    We began by searching the literature for existing theories which explained teaching of QI in the clinical setting. We evaluated the prior systematic reviews on the topic of QI medical education,7–9 ,13 spoken with experts in the field,7 ,9 and prepared a candidate conceptual framework (see online supplement 1) and accompanying theory for review.7 ,14 Our candidate theory hypothesised that the process of educating physicians begins with a curriculum and is impacted by characteristics of the learner, teacher, community and others, all encompassed within the educational context. From within the educational context emerges engaged learners and teachers, who produce improved educational and clinical outcomes. The combination of these successes produces physicians who are capable of and believe that it is their job to do their work and improve their work.

    The second step of the realist review is development of inclusion criteria, search strategies, a data abstraction tool and methodological quality assessment for review of the literature and analysis of included studies. Throughout the above process, the candidate theories were tested and refined and new theories added. As the studies were evaluated, themes emerged that were based on the predetermined theories. Each theme was assigned a code and linked to a quotation in the study. As a new theme emerged it was assigned a new code; we then searched for this theme in all the included studies in the review. New and revised candidate theories were synthesised into the set of candidate theories to test in our realist review.

    Study eligibility criteria

    Included studies had the following criteria:

    • Study design—original journal articles (no commentaries, letters to the editor, editorials or position pieces).

    • Setting—medical schools, residency and fellowship programmes worldwide.

    • Population—physician trainees (medical students, residents and/or fellows).

    • Interventions—whether clinical or educational—that engage trainees in QI work, where they are involved in changes to the delivery of care to patients within the clinical setting.

    • Reporting of clinical outcomes (patient care outcomes and system performance improvements) as the primary outcome measure.

    Search methods

    In collaboration with a professional librarian, one reviewer (ACJ) developed search strategies for the following databases: MEDLINE (2000 to 12 March 2013), Cochrane Library (2013) and Web of Science (15 March 2013). To locate potentially relevant studies in MEDLINE, we used exploded Medical Subject Headings terms and key words to generate sets for the themes of QI and medical education. We then used the Boolean term ‘AND’ to find their intersection. This basic approach was modified as necessary to search each electronic database. No language restriction was applied. Time limit was applied to obtain articles published after 2000, which corresponded with the publication of the Institute of Medicine reports To Err Is Human and Crossing the Quality Chasm. We excluded commentaries, editorials and letters. The full search strategy is available upon request. Reference reviews of the four earlier systematic reviews7–9 ,13 were performed by obtaining all references cited and searching forward using Web of Science to find all papers which cited these reviews and including them in title and abstract review.

    Study selection

    One reviewer (ACJ) independently screened each title and abstract for eligibility. Then, two non-blinded reviewers (ACJ and GO) independently assessed the eligibility of each full text record. Discrepancies were resolved by consensus between both reviewers after full text review.

    Data collection

    One reviewer (ACJ) abstracted data from the full text articles. A standardised data collection tool was used to capture identifying information, intervention summaries, details of study protocol, all primary and secondary outcome data and a section to extract quotations from the articles for the realist review (see online supplement 2).

    Analysis

    Analysis of interventions and outcomes

    Anticipating that the QI interventions and outcomes would be complex and different depending on the training programme, we used an iterative approach to categorise the different types of interventions and outcomes described in each study. We focused on any qualitative or quantitative reports of change in clinical outcomes. If outcomes were reported quantitatively, we determined whether statistical analysis was performed, either in the form of enumerative statistics or analytical statistics using statistical process control, a method of time-ordered analysis for QI.15 For the secondary outcome, we noted the findings of the educational outcomes such as knowledge, skills and attitudes.

    Assessment of methodological quality

    The Medical Education Research Quality Instrument16 is a validated instrument for methodological quality assessment of the medical education literature, but does not allow assessment of the methodological quality of the QI education literature specifically. Thus, we developed a set of criteria based upon the Cochrane Risk of Bias Tool and the Standards for Quality Improvement Reporting Excellence publication guidelines.17 ,18 Criteria were based on attention to quality factors in three major categories: the population, intervention and outcome reporting (see online supplements 3 and 4). Each study was assessed on all the criteria for quality, taking into account bias, confounding and study quality.

    A final methodological quality score was given to each study, based on a scale ranging from ‘fair,’ if almost none of the criteria were met, to ‘good,’ when minor flaws were found, to ‘very good,’ when high-quality reporting was achieved in at least one criterion each for population, intervention and outcome reporting, and ‘excellent,’ when high quality was achieved for all criteria (see online supplement 5).

    The realist review

    After completing the methodology of a systematic review, we used the realist review methodology to rigorously test the conceptual framework. We iteratively identified relevant themes, and through continuous data collection and rereading the articles, we tested the conceptual framework. Specifically, we looked for examples of when the curriculum was developed at the outset of the intervention to support the educational context of the candidate theory. We also looked for examples of successful completion of QI curricula and examples of physicians who are capable of and believe that it is their job to do their work and improve it.

    Results

    Results of search

    We included 39 studies for our final review (figure 1; excluded studies after full text review is available upon request), most of which were before–after studies or case reports in internal medicine or family medicine residency programmes (table 1). Less common were controlled trials, studies with medical students or reports from subspecialty resident programmes. Of the 29 studies that reported a sample size, the mean sample size was 5.6 trainees (SD=102), the median sample size was 24 trainees and the range was from 3 to 510 trainees.

    View this table:
    Table 1

    Baseline characteristics of included studies

    Figure 1
    Figure 1

    Study selection flow diagram.

    Among the interventions, 21 were primarily clinical interventions in which the goal was clearly to improve patient care or system performance in the clinical setting with education of the trainee not the primary focus of the intervention; 18 studies primarily had educational interventions, in which the goal was to deliver a curriculum to trainees focused on learning about improvement of which a component was to improve patient care and system performance in the clinical setting. The mean intervention length was 6.6 months (SD=9.2).

    Twenty studies reported system performance outcomes (e.g., improved documentation) while three reported only patient care outcomes (e.g., haemoglobin a1c, blood pressure) and 16 reported both patient care and system performance outcomes. Among the clinical outcomes, 28 studies reported successful improvements, two of which were not sustained. Ten studies demonstrated improvement in some measures and two clinical outcome reports were equivocal. No studies reported clinical outcomes that worsened.

    Nineteen studies reported educational outcomes, using various measures of knowledge, skills and attitudes such as the Quality Improvement Knowledge Application Tool (QIKAT),19 satisfaction surveys or objective structured clinical examinations.

    Description of studies

    The most common types of interventions were team projects and involvement in an existing clinical QI team (table 2). Among the team project interventions, trainees worked together to make improvements in the clinical setting; these studies had varied methodological quality scores. Seven of these studies demonstrated significant improvement in documentation, critical care measures and medication adherence20–26; six studies showed no significant clinical outcomes.4 ,27–31

    View this table:
    Table 2

    Summary of interventions, outcomes and methodological quality

    Of the interventions that involved trainees taking a role within an existing clinical interprofessional QI team, most were of ‘good’32–41 methodological quality. Most showed no significant clinical outcomes,33 ,35 ,37 ,40–43 but some showed statistically significant improvement in diabetes measures, vaccination rates, chronic care measures and critical care measures.32 ,34 ,36 ,38 ,39

    The third most common type of intervention used a chart audit. Most of these had ‘good’ methodological quality. About half these studies showed statistically significant outcomes, specifically in improved diabetes care, preventive care measures, documentation and critical care measures44–48; and the other half did not show statistically significant outcomes.49–52

    The least common intervention used individual projects in which the trainee worked independently. Most of these studies were rated ‘good,’53–56 and one received a ‘fair’.57 Most of these studies showed no significant clinical outcomes,53 ,55–57 while one study showed significant improvement in care for heart failure patients.54

    Methodological quality

    A majority of studies (30) had ‘good’ methodological quality. Six were ‘fair’ studies and 3 were ‘very good’. There were no ‘excellent’ studies.

    Among the included studies, all but three studies described the intervention in sufficient detail so that it could be replicated. Although all the studies had, by our inclusion criteria, trainees participating in QI work within the clinical setting, only 14 studies articulated educational objectives for trainees. There were five studies which described a clinical intervention rather than an educational intervention and reported no educational outcomes (table 2).

    It is notable that three studies took steps to minimise bias and confounding. Holmboe et al45 matched each second year resident in the intervention track to a third year control. Dysinger and Pappas21 enrolled the entire fourth year medical school class in a required month-long clinical QI rotation over 3 years, resulting in 510 students completing the curriculum, and allowing for comparison over 3 years of data collection to observe and account for secular trends. Asao et al,49 in addition to enrolling only second year residents in the chart audit curriculum intervention, completed a multivariate analysis to account for the trainee's experience as an auditor, the duration of exposure to the curriculum independent of training level and number of comorbidities in the resident's patient sample.

    Synthesis of results and realist review

    We identified several major themes through realist review. These themes are organised by ‘what works,’ ‘for whom,’ ‘under what circumstances’ and ‘to achieve what outcomes’ (table 3). After synthesising the range of interventions, clinical and educational outcomes, the methodological quality and realist review of the 39 studies, we tested the candidate theory and conceptual framework by iteratively analysing the major themes which emerged from the data. Specifically, we looked for evidence of a predetermined curriculum and educational context, as we hypothesised at the outset. However, we failed to find evidence of these mechanistic and contextual factors as important determinants in producing physicians who are lifelong learners and improvers. Thus, we revised the candidate theories and developed a conceptual framework (figure 2).

    View this table:
    Table 3

    Relevant themes from realist review

    Figure 2
    Figure 2

    A conceptual framework describing the relationships between the contexts, mechanisms and outcomes for quality improvement (QI) in medical education.

    Many different types of curricula are described in the included studies, some of which are distinct educational interventions and some of which involve trainees in existing clinical QI in their practices. Among the included studies, these interventions fell into four categories, and we found examples of statistically significant improvements in clinical outcomes in each category. Those studies with ‘fair’ methodological quality scores did not report significant results or failed to report numerical results with statistical analysis. Therefore, the higher-quality studies suggest that, with certain contexts and mechanisms, significant improvement in clinical outcomes is achievable when trainees are exposed to QI within the clinical setting.

    Several success characteristics were common to different contexts of clinical QI education (table 3; for illustrative quotes see online supplement 6). As noted by Sockalingham et al56 ‘residents identified workload as a major barrier to (doing QI work).’ Successful QI teaching programmes were consistently clear about the time required for trainee work-hour rules, competing demands and for faculty involvement. Success of specific programmes will also depend on whether it makes more sense to train all faculty members in QI principles or to have a dedicated select faculty group in charge of the QI curriculum. The availability of data through information systems is also a facilitator to trainee satisfaction and engagement. The sustained improvement reported by Halverson et al36 was achieved through the use of timely regular data feedback to all providers, including trainees, about the care of patients with diabetes in the practice. Two studies also highlighted the challenges that occur when trainees must abstract their own data, such as is the case in a practice which has not implemented an electronic medical record, or when data feedback is not timely enough for continuous QI.34 ,44 Choice of the project topic is also important for trainees. QI educators need to consider the needs of the clinical setting as well as level of trainee; however, no consensus emerged as to the best approach.

    Discussion

    How can our findings inform a comprehensive model of teaching, learning and doing QI in medical education? Our realist analysis and conceptual framework (figure 2) suggest that clinical education, whether providing care to an individual patient or doing QI work, does not begin with the curriculum. It begins with the trainee and patient at the centre of a healthcare system that encompasses many institutional levels. The trainee exists within two overlapping worlds: educational and clinical. The educational world is comprised of teaching faculty and together they exist within the training programme. The clinical world is made up of the patient and family at the centre of an interprofessional care team, of which the trainee is one part. The outcome of the educational world is improved trainee knowledge, skills and attitudes. The outcome of the clinical world is improved patient care and system performance. While these are often seen as separate, QI is one key area that exposes the inter-related complexities of the educational and care-delivery systems. In order to produce physicians who are capable of and believe that their duty is to do their work and improve their work, we should rethink the conventional wisdom around education and acquisition of improvement knowledge and skills. As Batalden and Davidoff58 wrote, ‘Learning how to do quality improvement and actually carrying out quality improvement are essentially one and the same; both are special forms of experiential learning.’ In fact, medical education is currently embracing this culture in teaching clinical skills using Adult Learning Theory, which reminds us that professionals learn best when they see a need to acquire the knowledge and skills for fulfilment of goals.59 If trainees see that faculty are asking questions of the process and needing to learn more to improve the system, then they have the opportunity to engage with them.60 Interestingly, Asch et al61 have demonstrated that within obstetrical residencies, it is possible and, perhaps beneficial, to rank obstetrical programmes based on overall performance on clinical rather than educational outcomes, representing yet another innovation for the future of the medical profession. The ACGME has acknowledged the importance of the clinical learning environment as an essential component of resident education and, accordingly, adopted the Clinical Learning Environment Review, ‘to generate national data on programme and institutional attributes that have a salutary effect on quality and safety in settings where residents learn and on the quality of care rendered after graduation.’62 Through the knowledge, skills and attitudes that trainees achieve during their educational programme while taking part in the improvement of the clinical care, we expect to nurture lifelong learners and improvers who will advance clinical improvements in patient care and system performance.

    This realist review has limitations that begin with the known publication bias in this field. This bias was corroborated in our review, as none of the published studies described a clinical process that was worsened. Not sharing failed pilots and curriculum limits the learning that can occur across programmes. Also, although our search strategy allowed us to analyse any circumstance of trainees being involved in QI in the clinical setting, this may have caused us to falsely criticise studies with primarily clinical interventions, because they did not aim to prove that involving trainees made a difference in their clinical outcomes. The realist review process, however, helps to differentiate these studies and their important qualitative information about ‘what works,’ ‘for whom,’ ‘under what circumstances’ and ‘to achieve what outcomes’.

    Because a methodological quality tool does not exist for assessment of the QI education literature, we created a tool by combining elements of existing validated tools. Although we did not use a validated instrument, the tool we developed contains the specificity for the QI education literature and thus made the quality assessment more rigorous. We did not, however, identify any ‘excellent’ quality studies (see online supplement 5). Although we found that all the lowest-quality studies did not demonstrate significant results, there were no strong studies to show improvement in both clinical and educational outcomes. We identified many studies with minor weaknesses, and the realist review process helps to glean the notable characteristics from these data. More high-quality studies would take steps to minimise bias in the study population, clearly describe the intervention and educational objectives, minimise other exposures or secular trends that could have accounted for the results, analyse results with enumerative or analytic statistics, explain all biases and confounders and report funding sources. Among the studies in this review, many would have been improved by particular attention to minimising bias and confounding in the study population, and a clear articulation of educational objectives.

    Conclusion

    The studies in this review reported on many more clinical outcomes than had been described in previous reviews, in large part due to the development of clinically oriented QI programmes since the prior reviews. Using the realist approach allowed us the advantage of synthesising these data to not just update, but reconceptualise (figure 2) the current landscape of QI teaching. Advances in teaching and doing QI has made tremendous strides in the past decade, but further work is needed to determine the factors that reliably facilitate the development of physicians who will believe and are capable of doing their work and improving their work—ultimately, physicians who are lifelong learners and improvers.

    Acknowledgments

    The authors thank Tom Mead, MLS, Reference Librarian, Biomedical Libraries, Dartmouth College, for assistance with development of the search strategy; Aurora Leute Matzkin, PhD and Martha Reagan-Smith, MD, EdD (Professor Emerita, Geisel School of Medicine) for their assistance in reading drafts of the work and preparing the conceptual framework.

    References

    1. ↵
      1. Kohn LT,
      2. Corrigan JM,
      3. Donaldson MS
      , eds. To err is human: building a safer health system. Washington, DC: National Academy Press, 2000.
    2. ↵
      ACGME Common Program Requirements. 2012. http://www.acgme.org/acgmeweb/ProgramandInstitutionalGuidelines.aspx (accessed 27 Dec 2012).
    3. ↵
      1. Patterson BR,
      2. Kimball KJ,
      3. Walsh-Covarrubias JB, et al
      . Effecting the sixth core competency: a project-based curriculum. Am J Obstet Gynecol 2008;199:561.e561–566.
      OpenUrlPubMed
    4. ↵
      1. Ogrinc G,
      2. Nierenberg DW,
      3. Batalden PB
      . Building experiential learning about quality improvement into a medical school curriculum: the Dartmouth experience. Health Aff 2011;30:716–22.
      OpenUrlAbstract/FREE Full Text
    5. ↵
      1. Josephson SA,
      2. Engstrom JW
      . Residency Training: developing a program of quality and safety to train resident neurologists for the future. Neurology 2012;78:602–5.
      OpenUrlCrossRef
    6. ↵
      1. Boonyasai R
      . Effectiveness of teaching quality improvement to clinicians: a systematic review. JAMA 2007;298:1023–37.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Ogrinc G,
      2. Headrick LA,
      3. Mutha S, et al
      . A framework for teaching medical students and residents about practice-based learning and improvement, synthesized from a literature review. Acad Med 2003;78:748–56.
      OpenUrlCrossRefPubMedWeb of Science
    8. ↵
      1. Patow CA,
      2. Karpovich K,
      3. Riesenberg LA, et al
      . Residents’ engagement in quality improvement: a systematic review of the literature. Acad Med 2009;84:1757–64.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Wong BM,
      2. Etchells EE,
      3. Kuper A, et al
      . Teaching quality improvement and patient safety to trainees: a systematic review. Acad Med 2010;85:1425–39.
      OpenUrlCrossRefPubMedWeb of Science
    10. ↵
      1. Wong BM,
      2. Levinson W,
      3. Shojania KG
      . Quality improvement in medical education: current state and future directions. Med Educ 2012;46:107–19.
      OpenUrlCrossRefPubMed
    11. ↵
      1. Pawson R,
      2. Greenhalgh T,
      3. Harvey G, et al
      . Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy 2005;10:21–34.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Shepperd S,
      2. Lewin L,
      3. Straus S, et al
      . Can we systematically review studies that evaluate complex interventions? PLoS Med 2009;6:e1000086.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Boonyasai RT,
      2. Windish DM,
      3. Chakraborti C, et al
      . Effectiveness of teaching quality improvement to clinicians—a systematic review. JAMA 2007;298:1023–U1039.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      Scott Shipman M, MPH; Course Director, Capstone Series at The Dartmouth Institute for Health Policy and Clinical Practice; Director for Primary Care Affairs and Workforce Analysis, American Association for Medical Colleges.
    15. ↵
      1. Benneyan JC,
      2. Lloyd RC,
      3. Plsek PE
      . Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003;12:458–64.
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Reed DA,
      2. Cook DA,
      3. Beckman TJ, et al
      . Association between funding and quality of published medical education research. JAMA 2007;298:1002–9.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Higgins JPT,
      2. Altman DG
      . Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions: John Wiley & Sons, Ltd, 2008:187–241.
    18. ↵
      1. Ogrinc G,
      2. Mooney SE,
      3. Estrada C, et al
      . The SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Qual Saf Health Care 2008;17:13–32.
      OpenUrlCrossRefWeb of Science
    19. ↵
      1. Singh MK,
      2. Ogrinc G,
      3. Cox KR, et al
      . The Quality Improvement Knowledge Application Tool—Revised (QIKAT-R). Acad Med 2014. In press.
    20. ↵
      1. Diaz VA,
      2. Carek PJ,
      3. Dickerson LM, et al
      . Teaching quality improvement in a primary care residency. Jt Comm J Qual Patient Saf 2010;36:454–60.
      OpenUrlPubMed
    21. ↵
      1. Dysinger WS,
      2. Pappas JM
      . A fourth-year medical school rotation in quality, patient safety, and population medicine. Am J Prev Med 2011;41(4 Suppl 3):S200–5.
      OpenUrlCrossRefPubMed
    22. ↵
      1. Laiteerapong N,
      2. Keh CE,
      3. Naylor KB, et al
      . A resident-led quality improvement initiative to improve obesity screening. Am J Med Qual 2011;26:315–22.
      OpenUrlAbstract/FREE Full Text
    23. ↵
      1. Oyler J,
      2. Vinci L,
      3. Arora V, et al
      . Teaching Internal Medicine residents quality improvement techniques using the ABIM's practice improvement modules. J Gen Intern Med 2008;23:927–30.
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Oyler J,
      2. Vinci L,
      3. Johnson JK, et al
      . Teaching internal medicine residents to sustain their improvement through the quality assessment and improvement curriculum. J Gen Intern Med 2011;26:221–5.
      OpenUrlCrossRefPubMed
    25. ↵
      1. Shiner B,
      2. Green RL,
      3. Homa K, et al
      . Improving depression care in a psychiatry resident psychopharmacology clinic: measurement, monitoring, feedback and education. Qual Saf Health Care 2010;19:234–8.
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Varkey P,
      2. Karlapudi SP
      . Lessons learned from a 5-year experience with a 4-week experiential quality improvement curriculum in a preventive medicine fellowship. J Grad Med Educ 2009;1:93–9.
      OpenUrlCrossRefPubMed
    27. ↵
      1. Arbuckle MR,
      2. Weinberg M,
      3. Cabaniss DL, et al
      . Training psychiatry residents in quality improvement: an integrated, year-long curriculum. Acad Psych 2013;37:42–5.
      OpenUrlCrossRefWeb of Science
    28. ↵
      1. Clark CJ,
      2. Sindell SL,
      3. Koehler RP
      . Template for success: using a resident-designed sign-out template in the handover of patient care. J Surg Educ 2011;68:52–7.
      OpenUrlCrossRefPubMed
    29. ↵
      1. Tomolo AM,
      2. Lawrence RH,
      3. Aron DC
      . A case study of translating ACGME practice-based learning and improvement requirements into reality: systems quality improvement projects as the key component to a comprehensive curriculum. Qual Saf Health Care 2009;18:217–24.
      OpenUrlAbstract/FREE Full Text
    30. ↵
      1. Varkey P,
      2. Karlapudi SP,
      3. Bennet KE
      . Teaching quality improvement: a collaboration project between medicine and engineering. Am J Med Qual 2008;23:296–301.
      OpenUrlAbstract/FREE Full Text
    31. ↵
      1. Varkey P,
      2. Reller MK,
      3. Smith A, et al
      . An experiential interdisciplinary quality improvement education initiative. Am J Med Qual 2006;21:317–22.
      OpenUrlAbstract/FREE Full Text
    32. ↵
      1. Buckley JD,
      2. Joyce B,
      3. Garcia AJ, et al
      . Linking residency training effectiveness to clinical outcomes: a quality improvement approach. Jt Comm J Qual Patient Saf 2010;36:203–8.
      OpenUrlPubMed
    33. ↵
      1. Carey WA,
      2. Colby CE
      . Educating fellows in practice-based learning and improvement and systems-based practice: the value of quality improvement in clinical practice. J Crit Care 2013;28:112.e1–5.
      OpenUrlCrossRefWeb of Science
    34. ↵
      1. Coleman MT,
      2. Nasraty S,
      3. Ostapchuk M, et al
      . Introducing practice-based learning and improvement ACGME core competencies into a family medicine residency curriculum. Jt Comm J Qual Saf 2003;29:238–47.
      OpenUrlPubMed
    35. ↵
      1. Fischman D
      . Applying Lean Six Sigma methodologies to improve efficiency, timeliness of care, and quality of care in an internal medicine residency clinic. Qual Manag Health Care 2010;19:201–10.
      OpenUrlCrossRefPubMed
    36. ↵
      1. Halverson LW,
      2. Sontheimer D,
      3. Duvall S
      . A residency clinic chronic condition management quality improvement project. Fam Med 2007;39:103–11.
      OpenUrlPubMed
    37. ↵
      1. Landis SE,
      2. Schwarz M,
      3. Curran DR
      . North Carolina family medicine residency programs’ diabetes learning collaborative. Fam Med 2006;38:190–5.
      OpenUrlPubMedWeb of Science
    38. ↵
      1. Mohr JJ,
      2. Randolph GD,
      3. Laughon MM, et al
      . Integrating improvement competencies into residency education: a pilot project from a pediatric continuity clinic. Ambul Pediatr 2003;3:131–6.
      OpenUrlCrossRefPubMedWeb of Science
    39. ↵
      1. Stevens DP,
      2. Bowen JL,
      3. Johnson JK, et al
      . A multi-institutional quality improvement initiative to transform education for chronic illness care in resident continuity practices. J Gen Intern Med 2010;25(Suppl 4):S574–80.
      OpenUrlCrossRefPubMed
    40. ↵
      1. Stueven J,
      2. Sklar DP,
      3. Kaloostian P, et al
      . A resident-led institutional patient safety and quality improvement process. Am J Med Qual 2012;27:369–76.
      OpenUrlAbstract/FREE Full Text
    41. ↵
      1. Yu GC,
      2. Beresford R
      . Implementation of a chronic illness model for diabetes care in a family medicine residency program. J Gen Intern Med 2010;25(Suppl 4):S615–19.
      OpenUrlCrossRefPubMed
    42. ↵
      1. Stapleton FB,
      2. Hendricks J,
      3. Hagan P, et al
      . Modifying the Toyota Production System for continuous performance improvement in an academic children's hospital. Pediatr Clin North Am 2009;56:799–813.
      OpenUrlCrossRefPubMed
    43. ↵
      1. Vidyarthi AR,
      2. Baron RB
      . Financial incentives for residents and fellows: a disruptive innovation to drive quality improvement. Acad Med 2011;86:1338.
      OpenUrlCrossRefPubMedWeb of Science
    44. ↵
      1. Gould BE,
      2. Grey MR,
      3. Huntington CG, et al
      . Improving patient care outcomes by teaching quality improvement to medical students in community-based practices. Acad Med 2002;77: 1011–18.
      OpenUrlCrossRefPubMedWeb of Science
    45. ↵
      1. Holmboe ES,
      2. Prince L,
      3. Green M
      . Teaching and improving quality of care in a primary care internal medicine residency clinic. Acad Med 2005;80:571–7.
      OpenUrlCrossRefPubMedWeb of Science
    46. ↵
      1. Kaddan W,
      2. Poznansky O,
      3. Amir L, et al
      . Medical education and quality of care in the pediatric emergency department setting: a combined model. Eur J Emerg Med 2006;13:139–43.
      OpenUrlCrossRefPubMed
    47. ↵
      1. Kirschenbaum L,
      2. Kurtz S,
      3. Astiz M
      . Improved clinical outcomes combining house staff self-assessment with an audit-based quality improvement program. J Gen Intern Med 2010;25:1078–82.
      OpenUrlCrossRefPubMed
    48. ↵
      1. Paukert JL,
      2. Chumley-Jones HS,
      3. Littlefield JH
      . Do peer chart audits improve residents’ performance in providing preventive care? Acad Med 2003;78(Suppl 10):S39–41.
      OpenUrlCrossRefPubMedWeb of Science
    49. ↵
      1. Asao K,
      2. Mansi IA,
      3. Banks D
      . Improving quality in an internal medicine residency program through a peer medical record audit. Acad Med 2009;84:1796–802.
      OpenUrlCrossRefPubMedWeb of Science
    50. ↵
      1. Carek PJ,
      2. Dickerson LM,
      3. Boggan H, et al
      . A limited effect on performance indicators from resident-initiated chart audits and clinical guideline education. Fam Med 2009;41:249–54.
      OpenUrlPubMed
    51. ↵
      1. Krajewski K,
      2. Siewert B,
      3. Yam S, et al
      . A quality assurance elective for radiology residents. Acad Radiol 2007;14:239–45.
      OpenUrlCrossRefPubMed
    52. ↵
      1. Smith KL,
      2. Ashburn S,
      3. Rule E, et al
      . Residents contributing to inpatient quality: blending learning and improvement. J Hosp Med 2012;7:148–53.
      OpenUrlCrossRefPubMed
    53. ↵
      1. Canal DF,
      2. Torbeck L,
      3. Djuricich AM
      . Practice-based learning and improvement: a curriculum in continuous quality improvement for surgery residents. Arch Surg 2007;142:479–82; discussion 482–473.
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      1. Oujiri J,
      2. Hakeem A,
      3. Pack Q, et al
      . Resident-initiated interventions to improve inpatient heart-failure management. [Reprint in Postgrad Med J 2011;87:700–5; PMID: 21954033]. BMJ Qual Saf 2011;20:181–6.
      OpenUrlAbstract/FREE Full Text
    55. ↵
      1. Reardon CL,
      2. Ogrinc G,
      3. Walaszek A
      . A didactic and experiential quality improvement curriculum for psychiatry residents. J Grad Med Educ 2011;3:562–5.
      OpenUrlCrossRefPubMed
    56. ↵
      1. Sockalingham S,
      2. Stergiopoulos V,
      3. Maggi J, et al
      . Quality education: a pilot quality improvement curriculum for psychiatry residents. Med Teach 2010;32:e221–6.
      OpenUrlCrossRefPubMed
    57. ↵
      1. Weingart SN,
      2. Tess A,
      3. Driver J, et al
      . Creating a quality improvement elective for medical house officers. J Gen Intern Med 2004;19:861–7.
      OpenUrlCrossRefPubMedWeb of Science
    58. ↵
      1. Batalden P,
      2. Davidoff F
      . Teaching quality improvement—the devil is in the details. JAMA 2007;298:1059–61.
      OpenUrlCrossRefPubMedWeb of Science
    59. ↵
    60. ↵
      1. Cooke M,
      2. Ironside PM,
      3. Ogrinc GS
      . Mainstreaming quality and safety: a reformulation of quality and safety education for health professions students. BMJ Qual Saf 2011;20(Suppl 1):i79–82.
      OpenUrlAbstract/FREE Full Text
    61. ↵
      1. Asch DA,
      2. Nicholson S,
      3. Srinivas S, et al
      . EValuating obstetrical residency programs using patient outcomes. JAMA 2009;302: 1277–83.
      OpenUrlCrossRefPubMedWeb of Science
    62. ↵

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors The conception or design of the work and interpretation of data was performed by ACJ, SAS and GO. The acquisition and analysis of data was performed by ACJ and GO. The manuscript was drafted by ACJ, and revised critically for important intellectual content by ACJ, SAS and GO. All authors approved the final version of the manuscript. ACJ, SAS and GO agree to be accountable for all aspects of this work and will ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

    • Funding This work was supported by the VA National Quality Scholars Fellowship Program, the VA Office of Academic Affiliations, and the Geisel School of Medicine at Dartmouth Office of Health Systems and Clinical Improvement with the use of facilities and materials from the White River Junction VA in White River Junction, VT.

    • Competing interests ACJ and SAS have no competing interests to report. GSO is an Associate Editor of BMJ Quality and Safety but otherwise has no competing interests to report.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Data sharing statement ACJ had full access to all the data in the study, and takes responsibility for the integrity of the data and the accuracy of the data analysis.