Article Text

PDF
The Surgical Safety Checklist and Teamwork Coaching Tools: a study of inter-rater reliability
  1. Lyen C Huang1,2,
  2. Dante Conley3,4,
  3. Stu Lipsitz5,
  4. Christopher C Wright6,
  5. Thomas W Diller6,
  6. Lizabeth Edmondson1,
  7. William R Berry1,
  8. Sara J Singer7,8,9
  1. 1Ariadne Labs: a joint center for health system innovation at the Brigham and Women's Hospital and Harvard School of Public Health, Boston, Massachusetts, USA
  2. 2Department of Surgery, Stanford University School of Medicine, Stanford, California, USA
  3. 3Department of Health Policy and Management, Harvard School of Public Health, Boston, Massachusetts, USA
  4. 4Tanana Valley Clinic, Fairbanks, Alaska, USA
  5. 5Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA
  6. 6Greenville Health System, Greenville, South Carolina, USA
  7. 7Department of Health Policy and Management, Harvard School of Public Health, Boston, Massachusetts, USA
  8. 8Department of Medicine, Harvard Medical School, Boston, Massachusetts, USA
  9. 9Mongan Institute for Health Policy, Massachusetts General Hospital, Boston, Massachusetts, USA
  1. Correspondence to Dr Sara J Singer, Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Kresge Building 3, Room 317, Boston, MA 02115, USA; ssinger{at}hsph.harvard.edu

Abstract

Objective To assess the inter-rater reliability (IRR) of two novel observation tools for measuring surgical safety checklist performance and teamwork.

Summary background Data surgical safety checklists can promote adherence to standards of care and improve teamwork in the operating room. Their use has been associated with reductions in mortality and other postoperative complications. However, checklist effectiveness depends on how well they are performed.

Methods Authors from the Safe Surgery 2015 initiative developed a pair of novel observation tools through literature review, expert consultation and end-user testing. In one South Carolina hospital participating in the initiative, two observers jointly attended 50 surgical cases and independently rated surgical teams using both tools. We used descriptive statistics to measure checklist performance and teamwork at the hospital. We assessed IRR by measuring percent agreement, Cohen's κ, and weighted κ scores.

Results The overall percent agreement and κ between the two observers was 93% and 0.74 (95% CI 0.66 to 0.79), respectively, for the Checklist Coaching Tool and 86% and 0.84 (95% CI 0.77 to 0.90) for the Surgical Teamwork Tool. Percent agreement for individual sections of both tools was 79% or higher. Additionally, κ scores for six of eight sections on the Checklist Coaching Tool and for two of five domains on the Surgical Teamwork Tool achieved the desired 0.7 threshold. However, teamwork scores were high and variation was limited. There were no significant changes in the percent agreement or κ scores between the first 10 and last 10 cases observed.

Conclusions Both tools demonstrated substantial IRR and required limited training to use. These instruments may be used to observe checklist performance and teamwork in the operating room. However, further refinement and calibration of observer expectations, particularly in rating teamwork, could improve the utility of the tools.

  • Checklists
  • Patient safety
  • Surgery
  • Teamwork

Statistics from Altmetric.com

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.