Skip to main content
SpringerLink
Log in

Validation of a Method for Assessing Resident Physicians’ Quality Improvement Proposals

  • Original Article
  • Published:
Journal of General Internal Medicine Aims and scope Submit manuscript

Abstract

BACKGROUND

Residency programs involve trainees in quality improvement (QI) projects to evaluate competency in systems-based practice and practice-based learning and improvement. Valid approaches to assess QI proposals are lacking.

OBJECTIVE

We developed an instrument for assessing resident QI proposals—the Quality Improvement Proposal Assessment Tool (QIPAT-7)—and determined its validity and reliability.

DESIGN

QIPAT-7 content was initially obtained from a national panel of QI experts. Through an iterative process, the instrument was refined, pilot-tested, and revised.

PARTICIPANTS

Seven raters used the instrument to assess 45 resident QI proposals.

MEASUREMENTS

Principal factor analysis was used to explore the dimensionality of instrument scores. Cronbach’s alpha and intraclass correlations were calculated to determine internal consistency and interrater reliability, respectively.

RESULTS

QIPAT-7 items comprised a single factor (eigenvalue = 3.4) suggesting a single assessment dimension. Interrater reliability for each item (range 0.79 to 0.93) and internal consistency reliability among the items (Cronbach’s alpha = 0.87) were high.

CONCLUSIONS

This method for assessing resident physician QI proposals is supported by content and internal structure validity evidence. QIPAT-7 is a useful tool for assessing resident QI proposals. Future research should determine the reliability of QIPAT-7 scores in other residency and fellowship training programs. Correlations should also be made between assessment scores and criteria for QI proposal success such as implementation of QI proposals, resident scholarly productivity, and improved patient outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1

Similar content being viewed by others

References

  1. ACGME Outcome Project. Available at: http://www.acgme.org/outcome/comp/compFull.asp. Accessed July 29, 2006.

  2. Ashton CM. “Invisible” doctors: making a case for involving medical residents in hospital quality improvement programs. Acad Med. 1993;68(11):823–4.

    Article  PubMed  CAS  Google Scholar 

  3. Headrick LA, Richardson A, Priebe GP. Continuous improvement learning for residents. Pediatrics. 1998;101(4 Pt 2):768–73; discussion 773–4.

    PubMed  CAS  Google Scholar 

  4. Parenti CM, Lederle FA, Impola CL, Peterson LR. Reduction of unnecessary intravenous catheter use. Internal medicine house staff participate in a successful quality improvement project. Arch Intern Med. 1994;154(16):1829–32.

    Article  PubMed  CAS  Google Scholar 

  5. Frey K, Edwards F, Altman K, Spahr N, Gorman RS. The ‘Collaborative Care’ curriculum: an educational model addressing key ACGME core competencies in primary care residency training [see comment]. Med Educ. 2003;37(9):786–9.

    Article  PubMed  Google Scholar 

  6. Amin AN, Rucker L. A systems-based practice curriculum. Med Educ. 2004;38(5):568–9.

    Article  PubMed  Google Scholar 

  7. Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents: addressing core competency, improving systems. Acad Med. 2004;79(10 Suppl):S65–7.

    Article  PubMed  Google Scholar 

  8. Ogrinc G, Headrick LA, Morrison LJ, Foster T. Teaching and assessing resident competence in practice-based learning and improvement. J Gen Intern Med. 2004;19(5 Pt 2):496–500.

    Article  PubMed  Google Scholar 

  9. Ziegelstein RC, Fiebach NH. “The mirror” and “the village”: a new method for teaching practice-based learning and improvement and systems-based practice. Acad Med. 2004;79(1):83–8.

    Article  PubMed  Google Scholar 

  10. Allen E, Zerzan J, Choo C, Shenson D, Saha S. Teaching systems-based practice to residents by using independent study projects. Acad Med. 2005;80(2):125–8.

    Article  PubMed  Google Scholar 

  11. ACT (Achieving Competence Today) Online. Available at: http://www.actcurriculum.org/. Accessed June 27, 2006.

  12. Langley GJ, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance. San Francisco, CA: Jossey-Bass; 1996.

    Google Scholar 

  13. Cleghorn GD, Headrick LA. The PDSA cycle at the core of learning in health professions education. Joint Comm J Qual Improv. 1996;22(3):206–12.

    CAS  Google Scholar 

  14. Berwick DM, Nolan TW. Physicians as leaders in improving health care: a new series in Annals of Internal Medicine [see comment]. Ann Intern Med. 1998;128(4):289–92.

    PubMed  CAS  Google Scholar 

  15. DeVellis RF. Scale Development: Theory and Applications. London: Sage Publications; 1991.

    Google Scholar 

  16. Gorsuch RL. Factor Analysis, Second Edition. Hillsdale: Lawrence Erlbaum Associates, Inc.; 1983.

    Google Scholar 

  17. Landis JR, Koch GG. The measure of observer agreement for categorical data. Biometrics. 1977;33:159–74.

    Article  PubMed  CAS  Google Scholar 

  18. Swing SR. Assessing the ACGME general competencies: general considerations and assessment methods. Acad Emerg Med. 2002;9(11):1278–88.

    Article  PubMed  Google Scholar 

  19. Lynch DC, Swing SR, Horowitz SD, Holt K, Messer JV. Assessing practice-based learning and improvement. Teach Learn Med. 2004;16(1):85–92.

    Article  PubMed  Google Scholar 

  20. Messick S. Validity. In: Linn RL, ed. Educational Measurement. 3rd ed. Phoenix, AZ: Oryx Press; 1993.

    Google Scholar 

  21. Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003;37:830–7.

    Article  PubMed  Google Scholar 

  22. Standards for Educational and Psychological Testing. Washington, DC: American Education Research Association; 1999.

  23. Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166e7–16.

    Article  Google Scholar 

  24. Beckman TJ, Ghosh AK, Cook DA, Erwin PJ, Mandrekar JN. How reliable are assessments of clinical teaching? A review of the published instruments. J Gen Intern Med. 2004;19:971–7.

    Article  PubMed  Google Scholar 

  25. Beckman TJ, Cook DA, Mandrekar JN. What is the validity evidence for assessments of clinical teaching? J Gen Intern Med. 2005;20:1159–64.

    Article  PubMed  Google Scholar 

  26. Moss F, Thompson R. A new structure for quality improvement reports [comment]. Qual Health Care. 1999;8(2):76.

    Article  PubMed  CAS  Google Scholar 

  27. Davidoff F, Batalden P. Toward stronger evidence on quality improvement. Draft publication guidelines: the beginning of a consensus project [see comment]. Qual Saf Health Care. 2005;14(5):319–25.

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

We are grateful to Gordon T. Moore, M.D., M.P.H, Maryjoan Ladden, Ph.D., R.N., C.S., Antoinette Peters, Ph.D., and the faculty of the other academic medical centers involved in the Achieving Competency Today initiative for their input at the original forum discussing content that formed the basis for the first iteration of QIPAT-7. We also extend our thanks to Mr. Gregory J. Engstler and Ms. Pamela J. Nelson for their technical assistance with the computerized version of the assessment tool.

There were no external or internal funding sources for this work.

Conflict of Interest

None of the authors has any conflict of interest related to this work.

Author information

Authors and Affiliations

  1. Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, 55905, USA

    James L. Leenstra MD & Bryan J. Krajicek MD

  2. Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, 55905, USA

    Thomas J. Beckman MD, William C. Mundell MD & Furman S. McDonald MD MPH

  3. Division of Primary Care Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, 55905, USA

    Darcy A. Reed MD, MPH & Kris G. Thomas MD

  4. Division of Biostatistics, Mayo Clinic College of Medicine, Rochester, MN, 55905, USA

    Stephen S. Cha MS

  5. Division of Gastroenterology and Hepatology, Mayo Clinic College of Medicine, Rochester, MN, 55905, USA

    Joseph C. Kolars MD

Authors
  1. James L. Leenstra MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thomas J. Beckman MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Darcy A. Reed MD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. William C. Mundell MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kris G. Thomas MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bryan J. Krajicek MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Stephen S. Cha MS
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Joseph C. Kolars MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Furman S. McDonald MD MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Furman S. McDonald MD MPH.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Leenstra, J.L., Beckman, T.J., Reed, D.A. et al. Validation of a Method for Assessing Resident Physicians’ Quality Improvement Proposals. J GEN INTERN MED 22, 1330–1334 (2007). https://doi.org/10.1007/s11606-007-0260-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11606-007-0260-y

KEY WORDS

Search

Navigation