Source | Evaluation* | Time of evaluation | Outcome | Comments | Risk of bias† |
Awad et al18 |
|
|
|
| High |
Fisher et al19 |
|
|
|
| High |
France et al20 |
|
|
|
| High |
Grogan et al21 |
|
|
|
| High |
Haller et al24 25 |
|
|
|
| High |
Halverson et al26 |
|
|
|
| High |
Leonard et al27 |
|
|
|
| High |
McCulloch et al28 |
|
|
|
| High |
Morey et al29 |
|
|
| Six out of 16 contacted sites immediately agreed to participate (cases). Three agreed later and were assigned as controls. Counting of patient safety incidents and team dimension rating conducted by unblinded unit staff but re-evaluated by blinded raters (kappa: 0.69). No indication of response rates to any of the questionnaires. | Moderate |
Nielsen et al30 | 4a. Cluster analysis of 11 clinical process measures at case and control units before and after implementation. 4b. Before–after cluster analysis of adverse outcomes of 28.536 deliveries at case and control units. | Up to 5 months after | 4a. One out of 11 additional process measures had significant better score among cases. 4b. No significant before–after difference in Adverse Outcome Index at case or control units. | Low | |
Pettker et al32 | 2c. SAQ 4b. AOI | 2c. Two years before and 1 year after. 4b. Three years before, during and after training. | 2c. The percentage of respondents reporting a ‘good teamwork climate’ and a ‘good safety climate’ improved from 38.5% before to 55.4% after and 33.3% before and 55.4% after respectively. 4b. From initiation of intervention the AOI showed a significant decrease. | 2c. No indication of response rate. 4b. No information about trend in AOI before intervention. | High |
Pratt et al33 | 2c. Hospital level SAQ 4b. AOI, WAOS and SI 4b. Malpractice claims and cases | 2b. Four years after training. 4b. Three years before and 4 years after training. 4b. Before–after. No indication of period. | 2b. A higher percentage of staff from the intervention department strongly agreed to five items from the SAQ compared with the rest of the hospital. 4b. Before intervention: AOI: 5.9%, WAOS: 1.15, SI 19.59. After intervention: AOI: 4.6%, WAOS: decreased 33.2%, SI: decreased 13.2% (no absolute numbers reported). 4b. Before: 21 cases, 13 of high severity. After: 16 cases, 5 of high severity. | 2c. No indication of response rate | High |
Rivers et al34 | 1. Five point Likert scale end-of-course critique. 2c. Before–after TAQ. 4a. Before–after analyses of surgical count-errors. | 1. Immediately after. 2c. Immediately after. 4a. 6 months after. | 1. 75% perceived knowledge obtained in course as useful or very useful. 81% perceived that the course strongly or very strongly would increase their effectiveness. 2c. ‘The surveys revealed that the training had a significant effect on desired behaviours’34 (no further information included in article). 4a. 50% reduction in surgical count errors. | Limited description of methods and results including response rates | High |
Sax et al35 | 2c. 10-item SAQ (Rhode Island site). 3a. Reporting of incidents by staff (New York site). 3b. Use of checklist (New York site) Training took place from 2003 to 2006 (New York) and 2005 to 2006 (Rhode Island). | 2c. Immediately before, immediately after and 2 months after course. 3a. Difference from 2002 to 2008. 3b. Difference from 2002 to 2007. | 2c. Significant increase from before to immediately after. Remained stable at 2 months except for one item which further improved significantly. 3a. Upward trend on 28-point run chart from 709 incidents per quarter in 2002 to 1481 per quarter in 2008. Reporting of near misses (as indication of stronger safety culture) increased from 15.9% to 20.3%. 3b. From 75% in 2003, 86% in 2004, 94% in 2005 to 100%. | The reported evaluations stem from two different interventions. SAQ 80% immediately after course and 40% 2 months after. | High |
Sehgal et al36 | 1. 21-item five point Likert scale and open-ended questions end-of-course critique | Immediately after | 1. Overall training rating: 4.49 (±0.79) ((nurses 4.71 (±0.52), pharmacists 4.64 (±0.49), physicians 4.31 (±0.61)). 99% would recommend course to peers. Course likely to change the way the participant communicate: 4.37 (±0.71) and participate in teamwork 4.31 (±0.56). | No other results from the end-of-course critique reported | High |
Stead et al37 |
|
|
|
| High |
Watts et al38 | 2c. 53-item TAQ. | Before, 4 months after and 8 months after | 2c. TAQ-score increased significantly after 4 months. Rating by 42 participants after 8 months showed sustained results. | 90% response rate before, 81% after 4 months and 53 after 8 months | High |
Weaver et al39 |
|
|
|
| Moderate to High |
↵* Evaluation level: (1) reactions to course, (2) learning (a) knowledge, (b) skills and (c) attitudes, (3) observed change in individuals' behaviour, (4) results: (a) Changes in organisational processes and (b) results for patients.
↵† Risk of bias: ‘high’ for high risk of bias, ‘moderate’ for moderate risk of bias and ‘low’ for low high risk of bias based on (1) study design (controlled/uncontrolled; randomized/not randomised; prospective/retrospective), (2) loss of participants to follow-up and (3) blinding of observers.44
AOI, Adverse Outcome Index; SAQ, Safety Attitude Questionnaire; SBAR, the Situation-Background-Assessment-Recommendation communicative framework; SI, Severety Index; TAQ, Teamwork attitude questionnaire; WAOS, Weighted Adverse Outcome Score.