Table 2

Development of an A3 assessment tool and self-instruction package for QI project proposals: (a) overview of five cycles and (b) examples of adjustments between cycles

(a) Overview of five cycles
 ActivityCycle #1
Summer 2017–Spring 2018
Cycle #2
Spring 2018–Summer 2018
Cycle #3
Summer 2018–Fall 2018
Cycle #4
Fall 2018–Spring 2019
Cycle #5
Spring 2019–Fall 2019
Development and revisionsLiterature review
Created initial A3 materials
  • Template

  • Content guide

  • Assessment tool

    shared with A3 teachers for comments

Revised materials
Added instructions for use of the self-instruction package and assessment tool
Revised materialsRevised materials
  • Description of rating options

  • Exemplary and deficient A3 examples with rating explanations

Revised materials
Added another deficient A3 example with rating explanations
Added automated functions to assessment tool
ChecksFeedback from two raters who assessed one A3Feedback from two experts who reviewed materialsTest of agreement for 4 raters×4 A3s and rater feedbackTest of agreement for 12 raters×6 A3s and rater feedbackFinal test of agreement for 12 raters×6 A3s and rater feedback
(b) Examples of adjustments between cycles
Document Cycle #1 to cycle #2 Cycle #2 to cycle #3 Cycle #3 to cycle #4 Cycle #4 to cycle #5
A3 template Within section. Removed question: ‘What residual issues can be anticipated?’ Across sections. Moved analysis section to after goal section to match original order used by Toyota. Within section. Added prompt: ‘What is contributing to the problem?’ Within section. Added question: ‘What will be monitored, by whom, when?’
A3 content guide(No adjustments) Within section. Added illustration of criteria matrix to countermeasures. Within section. Elaborated: ‘process map use’ and ‘strength of countermeasures’. Across sections. Graphics changed to similar set of colours.
A3 assessment tool Across sections. Better visual distinction between items ratable from A3 only or require context knowledge Within section. Eliminated vague question (‘How often is information clearly conveyed in each section of the A3?’). Within section. Wording improvement: from ‘Are timeframes identified …’ to ‘Are completing dates identified …’ Within section. Two items re-categorised from ‘ratable from A3 only’ to ‘requires contextual knowledge’.
  • Part (a) of this table provides an overview of each development cycle, including when initial versions of documents were developed and the checks performed at the end of each cycle. We created the A3 template, A3 content guide and A3 assessment tool during the first cycle. Part (b) of this table provides examples of adjustments to these documents that were based on comments and testing at the end of one cycle and incorporated in the next cycle. Documents are listed in hierarchal order, with an adjustment to a document often resulting in parallel adjustments (not shown) to subsequently listed documents. In each cycle, minor wording changes (not shown) were made to the documents to improve clarity of language.