Skip to main content
Log in

Assessing the Written Communication Skills of Medical School Graduates

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

The ECFMG® Clinical Skills Assessment (CSA®)was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's ability to summarize and synthesize the data collected. On a yearly basis, approximately 75,000 patient notes are reviewed and scored by physician raters. Recent changes to the PN scoring rubric, combined with enhancements to quality assurance procedures, mandate that additional evidence be provided to support the intended use of PN scores. The purpose of this study was to further investigate the psychometric adequacy of PN scores. Generalizability analyses suggest that while variability in PN rating scan be attributed to the choice of rater, candidate scores are reproducible over the 10-encounter CSA. The relationship of PN scores with other related ability measures and select candidate characteristics provides additional evidence to support the validity of the written exercise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Educational Research Association, American Psychological Association, National Council on Measurement in Education (1999). Standards for Educational and Psychological Testing. Washington, DC: Author.

    Google Scholar 

  • Australian Council for Educational Research (1998). ACER ConQuest: Generalized Item Response Modelling Software. Melbourne, Australia: Author.

    Google Scholar 

  • Ayers, W.R. & Boulet, J.R. (2001). Establishing the validity of test score inferences: Performance of 4th-year U.S. medical students on the ECFMG Clinical Skills Assessment. Teaching and Learning in Medicine 13(4): 214-220.

    Article  Google Scholar 

  • Boulet, J.R., Ben David, M.F., Ziv, A., Burdick,W.P., Curtis,M. & Peitzman, S. et al. (1998a). Using standardized patients to assess the interpersonal skills of physicians. Academic Medicine 73(10 Suppl.): 94-96.

    Article  Google Scholar 

  • Boulet, J.R., Friedman Ben-David, M., Hambleton, R.K., Burdick, W.P., Ziv, A. & Gary, N.E. (1998b). An investigation of the sources of measurement error in the post-encounter written scores from standardized patient examinations. Advances in Health Sciences Education: Theory and Practice 3: 89-100.

    Article  Google Scholar 

  • Boulet, J.R., Friedman Ben-David, M., Ziv, A., Burdick, W.P. & Gary, N.E. (2000). The use of holistic scoring for post-encounter written exercises. In D. Melnick (ed.), Proceedings of the Eighth Ottawa Conference on Medical Education and Assessment (pp. 254-260). Philadelphia, PA: National Board of Medical Examiners.

    Google Scholar 

  • Boulet, J.R., McKinley, D.W., Norcini, J.J. & Whelan, G.P. (2002). Assessing the comparability of standardized patient and physician evaluations of clinical skills. Advances in Health Sciences Education: Theory and Practice 7: 85-97.

    Article  Google Scholar 

  • Boulet, J.R., van Zanten, M., McKinley, D.W. & Gary, N.E. (2001). Evaluating the spoken English proficiency of graduates of foreign medical schools. Medical Education 35(8): 767-773.

    Article  Google Scholar 

  • Brennan, R.L. (2001). Generalizability Theory. New York: Springer-Verlag.

    Google Scholar 

  • Cradock, J., Young, A.S. & Sullivan, G. (2001). The accuracy of medical record documentation in schizophrenia. Journal of Behavioral Health Services & Research 28(4): 456-465.

    Article  Google Scholar 

  • Crossley, G.M., Howe, A., Newble, D., Jolly, B. & Davies, H.A. (2001). Sheffield Assessment Instrument for Letters (SAIL): Performance assessment using outpatient letters. Medical Education 35(12): 1115-1124.

    Article  Google Scholar 

  • Friedman Ben-David, M.F., Boulet, J.R., Burdick, W.P., Ziv, A., Hambleton, R.K. & Gary, N.E. (1997). Issues of validity and reliability concerning who scores the post-encounter patientprogress note. Academic Medicine 72(10 Suppl. 1): 79-81.

    Article  Google Scholar 

  • Friedman Ben-David, M.F., Klass, D.J., Boulet, J., De Champlain, A., King, A.M. & Pohl, H.S. et al. (1999). The performance of foreign medical graduates on the National Board of Medical Examiners (NBME) standardized patient examination prototype: A collaborative study of the NBME and the Educational Commission for Foreign Medical Graduates (ECFMG). Medical Education 33(6): 439-446.

    Article  Google Scholar 

  • Grace-Farfaglia, P. & Rosow, P. (1995). Automating clinical dietetics documentation Journal of the American Dietetic Association 95(6): 687-690.

    Article  Google Scholar 

  • Hallock, J.A. ECFMG and the Challenges Facing International Medical Graduates (2002). Reporter 11(8): 2-3. Washington, DC: Association of American Medical Colleges.

    Google Scholar 

  • Harden, R.M. & Gleeson, F.A. (1979). Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education 13(1): 41-54.

    Google Scholar 

  • Howell, J., Chisholm, C., Clark, A. & Spillane, L. (2000). Emergency medicine resident documentation: Results of the 1999 American Board of Emergency Medicine in-training examination survey. Academic Emergency Medicine 7(10): 1135-1138.

    Article  Google Scholar 

  • Larimore, W.L. & Jordan, E.V. (1995). SOAP to SNOCAMP: improving the medical record format. Journal of Family Practice 41(4): 393-398.

    Google Scholar 

  • Medical Council of Canada (2002). Qualifying Examination Part II, Information Pamphlet. Ottawa, Canada: Author.

    Google Scholar 

  • Prentice, D.A. & Miller, D.T. (1998). When small effects are impressive. In A.E. Kazdin (ed.), Methodological Issues and Strategies in Clinical Research (pp. 163-173). Washington, DC: American Psychological Association.

    Google Scholar 

  • Regehr, G., MacRae, H., Reznick, R.K. & Szalay, D. (1998). Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine 73(9): 993-997.

    Article  Google Scholar 

  • Rollnick, S., Kinnersley, P. & Butler, C. (2002). Context-bound communication skills training: Development of a new method. Medical Education 36(4): 377-383.

    Article  Google Scholar 

  • SAS Institute, Inc. (1989). SAS/STAT User's Guide: Version 6 (4th ed.), Cary, NC: Author.

    Google Scholar 

  • Shapiro, J. (1999). Correlates of family-oriented physician communications. Family Practice 16(3): 294-300.

    Article  Google Scholar 

  • Slater, S.C. & Boulet, J.R. (2001). Predicting holistic ratings of written performance assessments from analytic scoring. Advances in Health Sciences Education: Theory and Practice 6(2): 103-119.

    Article  Google Scholar 

  • Sleszynski, S.L., Glonek, T. & Kuchera, W.A. (1999). Standardized medical record: A new outpatient osteopathic SOAP note form: Validation of a standardized office form against physician's progress notes. Journal of the American Osteopathic Association 99(10): 516-529.

    Google Scholar 

  • Swanson, D.B. & Norcini, J.J. (1989). Factors influencing reproducibility of tests using standardized patients. Teaching and Learning in Medicine 1(3): 158-166.

    Article  Google Scholar 

  • van Dalen, J., Kerkhofs, E., Verwijnen, G.M., Knippenberg-Van Den Berg, B.W., van Den Hout, H.A., Scherpbier, A.J. et al. (2002). Predicting communication skills with a paper-and-pencil test. Medical Education 36(2): 148-153.

    Article  Google Scholar 

  • van der Vleuten, C.P., Norman, G.R. & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity: Issues of reliability. Medical Education 25(2): 110-118.

    Article  Google Scholar 

  • Weber, D.O. (2002). Charting a course toward legible medical records: Perfect paperwork can mean financial savings, better patient care. Physician Executive 28(1): 8-13.

    Google Scholar 

  • Williams, S., Weinman, J. & Dale, J. (1998). Doctor-patient communication and patient satisfaction: A review. Family Practice 15(5): 480-492.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to John R. Boulet.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Boulet, J.R., Rebbecchi, T.A., Denton, E.C. et al. Assessing the Written Communication Skills of Medical School Graduates. Adv Health Sci Educ Theory Pract 9, 47–60 (2004). https://doi.org/10.1023/B:AHSE.0000012216.39378.15

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/B:AHSE.0000012216.39378.15

Navigation