Tool* | Miller's level† | Safety competencies‡ | Validity evidence§ | |||||||||
1 | 2 | 3 | 4 | 5 | 6 | Content validity | Response process | Internal structure | Relationships to other variables | Outcomes¶ | ||
Medical | ||||||||||||
A knowledge, skills, and attitudes questionnaire23 24 | 1 | ✓ | ✓ | No | No | No | Pre/post intervention | No | ||||
Unnamed online test25 | 1 | ✓ | ✓ | Yes | No | IC, TRR | Learner level | 1 | ||||
Unnamed questionnaire26 ** | 1 | ✓ | ✓ | Yes | No | IC, TRR | Learner level | No | ||||
Unnamed questionnaire27 | 1 | ✓ | No | No | No | No | No | |||||
Computer-based case management of USMLE Step 328 29 | 2 | ✓ | No | No | Reliability coefficient, inter-case reliability | Concurrent28 | No | |||||
Unnamed checklist30 | 3 | ✓ | No | No | No | No | No | |||||
8-station OSCE31 32 | 5 | ✓ | ✓ | ✓ | ✓ | ✓ | Yes | Yes | IC, IRR, inter-station correlation coefficients, G | Concurrent validity | 132 | |
6-station OSCE33 | 5 | ✓ | ✓ | ✓ | ✓ | ✓ | No | No | No | Learner level | No | |
10-station OSCE34 | 5 | ✓ | ✓ | ✓ | ✓ | No | No | No | No | 1 | ||
Anaesthesia | ||||||||||||
Anesthetists' Non-technical Skills System35–41 | 3/4 | ✓ | ✓ | ✓ | Yes | Yes | IC, IRR, IRA, rater reference rater difference | Concurrent validity,35 learner level,36 41 pre/post intervention37–39 | 171 | |||
Rating scale42 | 3 | ✓ | No | Yes | G | No | No | |||||
4-station43 | 5 | ✓ | ✓ | ✓ | ✓ | Yes | Yes | IC, IRA | Predictive validity | No | ||
Surgery | ||||||||||||
Non-technical Skills for Surgeons44 | 3 | ✓ | ✓ | ✓ | Yes | No | IRR | No | No | |||
Non-technical Skills System45–49 §§ | 3 | ✓ | ✓ | ✓ | Yes | Yes | IC, IRR, IRA | Concurrent validity,45 47 learner level45 48 49 | 145 46 48 49 | |||
PAR matrix50 | 3 | ✓ | ✓ | ✓ | No | No | No | No | 1 | |||
Observational teamwork assessment for surgery51 | 3 | ✓ | Yes | Yes | No | No | No | |||||
Emergency/resuscitation | ||||||||||||
Stanford University for Crisis Management Behaviours52 | 3 | ✓ | ✓ | ✓ | ✓ | No | No | IRR | Pre/post intervention | No | ||
A behaviour grading sheet53 | 3 | ✓ | ✓ | No | No | IRR | Leaner level | No | ||||
Non-technical scorecard54 | 3 | ✓ | ✓ | ✓ | No | No | IC, IRR | No | No | |||
A behaviourally anchored team skill rating scale55 | 3 | ✓ | ✓ | No | No | IRA | Concurrent validity | No | ||||
Modified ANTS and anti-air teamwork observation measure56 †† §§ | 3 | ✓ | ✓ | Yes | Yes | IRR | Concurrent validity | No | ||||
Ottawa Crisis Resource Management Global Rating Scale57 58 | 3 | ✓ | ✓ | ✓ | Yes | Yes | IC, IRR | Leaner level | No | |||
Ottawa Crew Resource Management Checklist58 | ||||||||||||
Teamwork behaviour frequencies form59 | 3 | ✓ | ✓ | ✓ | ✓ | Yes | Yes | IRR | Learner level | No | ||
Observational Skill-based Clinical Assessment Tool for Resuscitation60 †† §§ | 3 | ✓ | Yes | No | IC, IRR, inter-observer reliability | No | No | |||||
Unnamed scale61 †† §§ | 1 | ✓ | No | No | No | Pre/post intervention | No | |||||
Nursing | ||||||||||||
Quality Improvement Knowledge, Skills, and Attitudes62 | 1 | ✓ | ✓ | ✓ | ✓ | Yes | No | IC | Learner level | No | ||
Patient Safety Attitudes, Skills, and Knowledge Scale63 ‡‡ | 1 | ✓ | ✓ | Yes | No | IC | No | No | ||||
Essay64 | 2 | ✓ | ✓ | ✓ | No | No | No | No | No | |||
Situation Awareness Check list65 | 2 | ✓ | No | No | No difference in scores by two raters tested | Concurrent validity | No | |||||
Clinical Simulation Evaluation Tool66 †† | 3 | ✓ | ✓ | No | No | No | No | No | ||||
Knowledge, skills, attitudes criterion67 | 3 | ✓ | ✓ | ✓ | ✓ | No | No | IC | Pre/post intervention | No | ||
Unnamed checklist68 | 3 | ✓ | ✓ | ✓ | ✓ | ✓ | No | No | No | No | No | |
Clinical Performance Evaluation Tool69 | 3 | ✓ | ✓ | No | No | No | No | No | ||||
Unnamed checklist70 | 4 | ✓ | No | No | IC, IRR | No | 1 |
↵* Tools were labelled as unnamed if the tool was not named in the study.
↵† Miller's levels: 1=Knows; 2=Knows how; 3=Shows how; 4=Does; 5=Competencies assessed at multiple levels.
↵‡ Canadian Safety Competency domains: 1=Contribute to a culture of patient safety; 2=Work in teams for patient safety; 3=Communicate effectively for patient safety; 4=Manage safety risks; 5=Optimise human and environmental factors; 6=Recognise, respond to, and disclose adverse events.
↵§ Refers to whether each validity component was evaluated. Boldface denotes an acceptable level of validity.
↵¶ Outcomes were evaluated using the lowest level (1=participation: learners' or observers' views on the tool or its implementation) of a modified Kirkpatrick hierarchy.
↵** Article authors reported that this is not appropriate for determining individual competency standards because the tool was designed for program improvement.
↵†† Conference report or pilot study.
↵‡‡ Seventeen items come from ‘the Knowledge, Skills, and Attitudes Questionnaire’.25
↵§§ These tools assessed the competencies of both medical and nursing professionals.
ANTS, Anaesthetists' Non-Technical Skills System; G, generalisability; IC, internal consistency; IRA, inter-rater agreement; IRR, inter-rater reliability; OSCE, Objective Structured Clinical Examination; TRR, test retest reliability.