Table 2

Development of an A3 assessment tool and self-instruction package for QI project proposals: (a) overview of five cycles and (b) examples of adjustments between cycles

(a) Overview of five cycles
Activity	Cycle #1 Summer 2017–Spring 2018	Cycle #2 Spring 2018–Summer 2018	Cycle #3 Summer 2018–Fall 2018	Cycle #4 Fall 2018–Spring 2019	Cycle #5 Spring 2019–Fall 2019
Development and revisions	Literature review Created initial A3 materials Template Content guide Assessment tool shared with A3 teachers for comments	Revised materials Added instructions for use of the self-instruction package and assessment tool	Revised materials	Revised materials Added: Description of rating options Exemplary and deficient A3 examples with rating explanations	Revised materials Added another deficient A3 example with rating explanations Added automated functions to assessment tool
Checks	Feedback from two raters who assessed one A3	Feedback from two experts who reviewed materials	Test of agreement for 4 raters×4 A3s and rater feedback	Test of agreement for 12 raters×6 A3s and rater feedback	Final test of agreement for 12 raters×6 A3s and rater feedback
(b) Examples of adjustments between cycles
Document	Cycle #1 to cycle #2	Cycle #2 to cycle #3	Cycle #3 to cycle #4	Cycle #4 to cycle #5
A3 template	Within section. Removed question: ‘What residual issues can be anticipated?’	Across sections. Moved analysis section to after goal section to match original order used by Toyota.	Within section. Added prompt: ‘What is contributing to the problem?’	Within section. Added question: ‘What will be monitored, by whom, when?’
A3 content guide	(No adjustments)	Within section. Added illustration of criteria matrix to countermeasures.	Within section. Elaborated: ‘process map use’ and ‘strength of countermeasures’.	Across sections. Graphics changed to similar set of colours.
A3 assessment tool	Across sections. Better visual distinction between items ratable from A3 only or require context knowledge	Within section. Eliminated vague question (‘How often is information clearly conveyed in each section of the A3?’).	Within section. Wording improvement: from ‘Are timeframes identified …’ to ‘Are completing dates identified …’	Within section. Two items re-categorised from ‘ratable from A3 only’ to ‘requires contextual knowledge’.

Part (a) of this table provides an overview of each development cycle, including when initial versions of documents were developed and the checks performed at the end of each cycle. We created the A3 template, A3 content guide and A3 assessment tool during the first cycle. Part (b) of this table provides examples of adjustments to these documents that were based on comments and testing at the end of one cycle and incorporated in the next cycle. Documents are listed in hierarchal order, with an adjustment to a document often resulting in parallel adjustments (not shown) to subsequently listed documents. In each cycle, minor wording changes (not shown) were made to the documents to improve clarity of language.