Table 2

Evaluation methods and outcomes of studies evaluating class-room based multiprofessional team training in hospitals

SourceEvaluation*Time of evaluationOutcomeCommentsRisk of bias
Awad et al18
  • 2c. SAQ

  • 3. Counting of operations preceded by briefings. Method not indicated.

  • 4a. Before–after preoperative antibiotic administration.

  • 2c. 4 months after.

  • 3. 1 and 4 months after.

  • 4a. 4 months after

  • 2c. Significantly better for anaesthetists and surgeons but not for nurses.

  • 3. 64% after 1 month, 100% after 4 months. 4a. Significant increase in prophylaxis administration.

  • No of participants who underwent training or responded to SAQ is not indicated.

  • 3. Method for assessment of briefings not indicated.

High
Fisher et al19
  • 2c. SAQ to all crew members. 144 crew members who had or had not participated in CRM training returned surveys.

  • Not stated

  • 2c. Participants of intervention had significantly better scores

  • Limited information on training and survey. Low response rate (21%).

High
France et al20
  • 3. Observation after training of participants' skills during 30 surgical cases

  • One year after training (weeks after brief retraining)

  • 3. Teams were compliant with 60% of practices enforced in the training programme

  • 3. No information of blinding of observers. No before-measures.

High
Grogan et al21
  • 1. 11-item Likert scale end-of-course critique

  • 2c. 23-item before–after Likert scale TAQ.

  • 1. Immediately after.

  • 2. Before and immediately after.

  • 1. Positive evaluation of course content: Scores on 3.91 to 4.58 (max. 5).

  • 2c. Training had significant positive impact on 20 items.

  • 1. 95% response rate

  • 2c. 69% response rate.

High
Haller et al24 25
  • 1. 10-item Likert scale end-of-course critique.

  • 2a. Before–after 36-item Likert scale TAQ.

  • 2c. 57-item Likert scale before–after SAQ.

  • 1. Immediately after.

  • 2c1. Immediately after.

  • 2c2. One year after.

  • 1. 90% positive of course organisation, 63.5–71% positive of content, 79–81% positive of teaching methods and 69–79% positive of group dynamics.

  • 2c1. 35 of 36 items with improved scores (27 significantly).

  • 2c2. Positive changes in safety climate: 2.9 [CI 1.3–6.3] to 4.7 [1.2–17.2].

  • 2a. 74% response rate.

  • 2c. 95% response rate.

High
Halverson et al26
  • 2c1. TAQ.

  • 2c2. Questionnaire on utility of briefings among 156 ‘selected individuals.’

  • 3. Before–after observation of briefings.

  • 4a. Wrong site surgery events, timely antibiotic administration, cases starting on time, turnover time between cases.

  • 2c1. Immediately after.

  • 2c2. Not stated.

  • 3. Up to 6 months after.

  • 4a. Not indicated.

  • 2c. Improved perception of teamwork on 19 out of 19 items (14 significantly).

  • 2c. 37% reported useful information shared at briefing, 75% reported greater sense of teamwork (higher among nurses than among anaesthetists and surgeons).

  • 3. Compliance with formal briefings before intervention: 0%. 2 weeks after: 86%. 6 months after: 66%. Compliance with all required elements of briefing increased from 47% before to 86% after 6 months.

  • 4a. No significant change in prophylaxis administration or operation time (no further information about this evaluation included in the article).

  • 2c1. No response rate indicated.

  • 2c2. No response rate indicated.

  • 3. No information of blinding of observers.

High
Leonard et al27
  • 3. Numerical results from one site on use of checklists.

  • 3 months after.

  • 3. Two checklists made information from hospital to skilled nursing facility (on important data like anticoagulants, code status and preferred intensity of care) available 44–100% of the time.

  • The article mainly describes qualitative effects and critical aspects of implementation at three sites

High
McCulloch et al28
  • 2c. SAQ

  • 3. Observation of teamwork skills during 48 surgical procedures before and 55 procedures after training

  • 4a. Before–after technical errors, procedural errors, complications, operating time and length of stay.

  • 2c. Before training and 3 months after.

  • 3. Not stated.

  • 4a. Not stated.

  • 2c. Significantly improved teamwork climate score for B but not A. No effect on other SAQ-components.

  • 3. Significant increase in team non-technical skills: for A but not B; For nurses but not for anaesthetists and surgeons; on teamwork/cooperation and problem-solving/decision-making but not on leadership and management or situation awareness.

  • 4a. Mean technical error rate reduced significantly for A but not for B. Procedural errors reduced significantly for A and B. No significant affection of operating time, length of stay or number of complications.

  • 2c. Response rate not indicated.

  • 3. Two observers were not blinded but a third observer was independent/uninvolved.

High
Morey et al29
  • 1. End-of-course critique

  • 2b1. Before–after 3 item staff perception of support

  • 2b2. Before–after evaluation of handover to unit by unit nurse

  • 2c1. Before–after 15 item TAQ

  • 2c2. Before–after 6-item individual subjective workload experience

  • 3. Before–after 5-item team dimension-rating 4a. Before–after clinical error rate. 4b. Before–after 12 item patient satisfaction survey.

  • Up to 8 months after

  • 1. Results not included in article.

  • 2b1. Significantly higher perception of support at case units. No change at control units.

  • 2b2. No significant before–after difference in quality of handover at case or control units.

  • 2c1. Significant higher teamwork perception score among cases. No change among controls. 2c2. No significant before–after difference at case or control units.

  • 3. Significant increase in score among cases. No change among controls. 4a. Significant reduction in clinical error rate for cases: 30.9% before intervention to 4.4% after intervention. No significant change for controls (16.8% to 12.1%).

  • 4b. No significant before–after difference in patient satisfaction at case or control units.

Six out of 16 contacted sites immediately agreed to participate (cases). Three agreed later and were assigned as controls. Counting of patient safety incidents and team dimension rating conducted by unblinded unit staff but re-evaluated by blinded raters (kappa: 0.69). No indication of response rates to any of the questionnaires.Moderate
Nielsen et al304a. Cluster analysis of 11 clinical process measures at case and control units before and after implementation. 4b. Before–after cluster analysis of adverse outcomes of 28.536 deliveries at case and control units.Up to 5 months after4a. One out of 11 additional process measures had significant better score among cases.
4b. No significant before–after difference in Adverse Outcome Index at case or control units.
Low
Pettker et al322c. SAQ
4b. AOI
2c. Two years before and 1 year after.
4b. Three years before, during and after training.
2c. The percentage of respondents reporting a ‘good teamwork climate’ and a ‘good safety climate’ improved from 38.5% before to 55.4% after and 33.3% before and 55.4% after respectively.
4b. From initiation of intervention the AOI showed a significant decrease.
2c. No indication of response rate.
4b. No information about trend in AOI before intervention.
High
Pratt et al332c. Hospital level SAQ
4b. AOI, WAOS and SI
4b. Malpractice claims and cases
2b. Four years after training.
4b. Three years before and 4 years after training.
4b. Before–after. No indication of period.
2b. A higher percentage of staff from the intervention department strongly agreed to five items from the SAQ compared with the rest of the hospital.
4b. Before intervention: AOI: 5.9%, WAOS: 1.15, SI 19.59. After intervention: AOI: 4.6%, WAOS: decreased 33.2%, SI: decreased 13.2% (no absolute numbers reported).
4b. Before: 21 cases, 13 of high severity. After: 16 cases, 5 of high severity.
2c. No indication of response rateHigh
Rivers et al341. Five point Likert scale end-of-course critique.
2c. Before–after TAQ.
4a. Before–after analyses of surgical count-errors.
1. Immediately after.
2c. Immediately after.
4a. 6 months after.
1. 75% perceived knowledge obtained in course as useful or very useful. 81% perceived that the course strongly or very strongly would increase their effectiveness.
2c. ‘The surveys revealed that the training had a significant effect on desired behaviours’34 (no further information included in article).
4a. 50% reduction in surgical count errors.
Limited description of methods and results including response ratesHigh
Sax et al352c. 10-item SAQ (Rhode Island site).
3a. Reporting of incidents by staff (New York site).
3b. Use of checklist (New York site) Training took place from 2003 to 2006 (New York) and 2005 to 2006 (Rhode Island).
2c. Immediately before, immediately after and 2 months after course.
3a. Difference from 2002 to 2008.
3b. Difference from 2002 to 2007.
2c. Significant increase from before to immediately after. Remained stable at 2 months except for one item which further improved significantly.
3a. Upward trend on 28-point run chart from 709 incidents per quarter in 2002 to 1481 per quarter in 2008. Reporting of near misses (as indication of stronger safety culture) increased from 15.9% to 20.3%.
3b. From 75% in 2003, 86% in 2004, 94% in 2005 to 100%.
The reported evaluations stem from two different interventions. SAQ 80% immediately after course and 40% 2 months after.High
Sehgal et al361. 21-item five point Likert scale and open-ended questions end-of-course critiqueImmediately after1. Overall training rating: 4.49 (±0.79) ((nurses 4.71 (±0.52), pharmacists 4.64 (±0.49), physicians 4.31 (±0.61)). 99% would recommend course to peers. Course likely to change the way the participant communicate: 4.37 (±0.71) and participate in teamwork 4.31 (±0.56).No other results from the end-of-course critique reportedHigh
Stead et al37
  • 1. End-of-course critique

  • 2abc. 23-item knowledge, skills and attitudes questionnaire.

  • 2c. 42-item SAQ. 3. Use of SBAR and observation of team behaviours and performance.

  • 4a. Seclusion (∼isolation of patients) rates.

  • 1. Immediately after.

  • 2abc. Not indicated.

  • 2c. Not indicated.

  • 3. SBAR: 1 month after. Team: Before–after.

  • 4a. 9 months after.

  • 1. ‘Virtually all’ participants found that training was appropriate, would improve patient safety, facilitate leadership and improve communication.

  • 2abc. Significant increase.

  • 2c. Significant improvement in 2 of 12 domains.

  • 3. Multidisciplinary use of SBAR in ‘practically all’ patient presentations and writing. Improved team structure and process of meetings, improved role clarity and reduced unnecessary team membership.

  • 4a. Significantly reduced seclusion rates after implementation.

  • 1. No quantitative measures indicated.

  • 2c and 2abc. No information about response rate.

  • 3. External observers. No quantitative data on observations.

High
Watts et al382c. 53-item TAQ.Before, 4 months after and 8 months after2c. TAQ-score increased significantly after 4 months. Rating by 42 participants after 8 months showed sustained results.90% response rate before, 81% after 4 months and 53 after 8 monthsHigh
Weaver et al39
  • 1. 11-item Likert scale end-of course critique.

  • 2a. 23-item knowledge questionnaire.

  • 2c1. TAQ (controlled)

  • 2c2. Operating room management questionnaire (ORMAQ) (controlled).

  • 3. Surgical team observation of 10 procedures per team before and after (total 60) (controlled).

  • 1. Immediately after training.

  • 2a. Immediately after training. 2c1. One month before and 1 month after training.

  • 2c2. 1 month before and 1 month after training.

  • 3. 1 month before and 1 month after training.

  • 1. 52–94% of respondents reacted positively to items.

  • 2a. Respondents had an average of 92% correct answers.

  • 2c1. Both cases and controls improved significantly from before to after.

  • 2c2. Attitude to teamwork improved significantly among cases after training.

  • 3. Trained teams engaged in significantly more precase briefings than controls. Trained team members were significantly more willing to speak up and engage in contingency plan discussions. Trained staff improved significantly on communication and mutual support but not on leadership and situation monitoring. The proportion of team members who received training was significantly correlated to debriefing participation ratio. Two out of three trained teams demonstrated a significant increase in perception of teamwork after training.

  • 2c1. and 2c2. A very low number of controls (N=7) answered the questionnaire after training of cases.

  • 3. No information about blinding or neutrality of observers.

Moderate to High
  • * Evaluation level: (1) reactions to course, (2) learning (a) knowledge, (b) skills and (c) attitudes, (3) observed change in individuals' behaviour, (4) results: (a) Changes in organisational processes and (b) results for patients.

  • Risk of bias: ‘high’ for high risk of bias, ‘moderate’ for moderate risk of bias and ‘low’ for low high risk of bias based on (1) study design (controlled/uncontrolled; randomized/not randomised; prospective/retrospective), (2) loss of participants to follow-up and (3) blinding of observers.44

  • AOI, Adverse Outcome Index; SAQ, Safety Attitude Questionnaire; SBAR, the Situation-Background-Assessment-Recommendation communicative framework; SI, Severety Index; TAQ, Teamwork attitude questionnaire; WAOS, Weighted Adverse Outcome Score.