Article Text

Download PDFPDF
Development and validation of an A3 problem-solving assessment tool and self-instructional package for teachers of quality improvement in healthcare
  1. Jennifer S Myers1,
  2. Jeanne M Kin2,
  3. John E Billi3,4,5,
  4. Kathleen G Burke6,7,
  5. Richard Van Harrison8
  1. 1Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, USA
  2. 2Quality, Michigan Medicine, University of Michigan, Ann Arbor, Michigan, USA
  3. 3Medicine and Learning Health Sciences, Michigan School of Medicine, University of Michigan, Ann Arbor, Michigan, USA
  4. 4Health Management and Policy, School of Public Health, University of Michigan, Ann Arbor, Michigan, USA
  5. 5Integrative Systems and Design, College of Engineering, University of Michigan, Ann Arbor, Michigan, USA
  6. 6Biobehavioral Health, University of Pennsylvania School of Nursing, Philadelphia, Pennsylvania, USA
  7. 7Leonard Davis Institute of Health Economics, University of Pennsylvania, Philadelphia, Pennsylvania, USA
  8. 8Learning Health Sciences, University of Michigan Health System, Ann Arbor, Michigan, USA
  1. Correspondence to Dr. Jennifer S Myers, University of Pennsylvania Perelman School of Medicine, Philadelphia, PA, USA; jennifer.myers2{at}


Purpose A3 problem solving is part of the Lean management approach to quality improvement (QI). However, few tools are available to assess A3 problem-solving skills. The authors sought to develop an assessment tool for problem-solving A3s with an accompanying self-instruction package and to test agreement in assessments made by individuals who teach A3 problem solving.

Methods After reviewing relevant literature, the authors developed an A3 assessment tool and self-instruction package over five improvement cycles. Lean experts and individuals from two institutions with QI proficiency and experience teaching QI provided iterative feedback on the materials. Tests of inter-rater agreement were conducted in cycles 3, 4 and 5. The final assessment tool was tested in a study involving 12 raters assessing 23 items on six A3s that were modified to enable testing a range of scores.

Results The intraclass correlation coefficient (ICC) for overall assessment of an A3 (rater’s mean on 23 items per A3 compared across 12 raters and 6 A3s) was 0.89 (95% CI 0.75 to 0.98), indicating excellent reliability. For the 20 items with appreciable variation in scores across A3s, ICCs ranged from 0.41 to 0.97, indicating fair to excellent reliability. Raters from two institutions scored items similarly (mean ratings of 2.10 and 2.13, p=0.57). Physicians provided marginally higher ratings than QI professionals (mean ratings of 2.17 and 2.00, p=0.003). Raters averaged completing the self-instruction package in 1.5 hours, then rated six A3s in 2.0 hours.

Conclusion This study provides evidence of the reliability of a tool to assess healthcare QI project proposals that use the A3 problem-solving approach. The tool also demonstrated evidence of measurement, content and construct validity. QI educators and practitioners can use the free online materials to assess learners’ A3s, provide formative and summative feedback on QI project proposals and enhance their teaching.

  • continuing education
  • continuing professional development
  • graduate medical education
  • healthcare quality improvement
  • health professions education
  • lean management

Statistics from


  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Patient consent for publication Not required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data availability statement All data relevant to the study are included in the article or uploaded as supplementary information.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.