Article Text

PDF

Comparing NICU teamwork and safety climate across two commonly used survey instruments
  1. Jochen Profit1,2,
  2. Henry C Lee1,2,
  3. Paul J Sharek2,3,4,
  4. Peggy Kan1,2,
  5. Courtney C Nisbet2,
  6. Eric J Thomas5,
  7. Jason M Etchegaray6,
  8. Bryan Sexton7,8
  1. 1Perinatal Epidemiology and Health Outcomes Research Unit, Division of Neonatal and Developmental Medicine, Department of Pediatrics, Stanford University School of Medicine and Lucile Packard Children's Hospital; Palo Alto, CA, USA
  2. 2California Perinatal Quality Care Collaborative; Palo Alto, CA, USA
  3. 3Center for Quality and Clinical Effectiveness, Lucile Packard Children's Hospital, Palo Alto, CA, USA
  4. 4Division of General Pediatrics, Department of Pediatrics, Stanford University, Palo Alto, CA, USA
  5. 5University of Texas at Houston – Memorial Hermann Center for Healthcare Quality and Safety, University of Texas Medical School, Houston, TX, USA
  6. 6The Rand Corporation, Santa Monica, CA, USA
  7. 7Department of Psychiatry, Duke University School of Medicine; Duke University Health System, Durham, NC, USA
  8. 8Duke Patient Safety Center, Duke University Health System, Durham, NC, USA
  1. Correspondence to Dr Jochen Profit, Department of Pediatrics, Stanford University, 1265 Welch Rd, x115, Stanford, CA 94305, USA; profit{at}stanford.edu

Abstract

Background and objectives Measurement and our understanding of safety culture are still evolving. The objectives of this study were to assess variation in safety and teamwork climate and in the neonatal intensive care unit (NICU) setting, and compare measurement of safety culture scales using two different instruments (Safety Attitudes Questionnaire (SAQ) and Hospital Survey on Patient Safety Culture (HSOPSC)).

Methods Cross-sectional survey study of a voluntary sample of 2073 (response rate 62.9%) health professionals in 44 NICUs. To compare survey instruments, we used Spearman's rank correlation coefficients. We also compared similar scales and items across the instruments using t tests and changes in quartile-level performance.

Results We found significant variation across NICUs in safety and teamwork climate scales of SAQ and HSOPSC (p<0.001). Safety scales (safety climate and overall perception of safety) and teamwork scales (teamwork climate and teamwork within units) of the two instruments correlated strongly (safety r=0.72, p<0.001; teamwork r=0.67, p<0.001). However, the means and per cent agreements for all scale scores and even seemingly similar item scores were significantly different. In addition, comparisons of scale score quartiles between the two instruments revealed that half of the NICUs fell into different quartiles when translating between the instruments.

Conclusions Large variation and opportunities for improvement in patient safety culture exist across NICUs. Important systematic differences exist between SAQ and HSOPSC such that these instruments should not be used interchangeably.

  • Safety culture
  • Critical care
  • Surveys

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/

Statistics from Altmetric.com

Introduction

Despite a renewed focus on patient safety over the last decade there has been limited progress in ensuring that patients receive safer care in hospitals.1 ,2 Development of a culture of safety is considered to be foundational to achieving safer care.3 The Joint Commission requires that hospitals assess their safety culture on an ongoing basis.4 A culture of safety is the shared values, attitudes, perceptions and patterns of behaviour that determine the observable degree of effort with which organisational members direct their attention and actions towards minimising patient harm.5

Several survey instruments have been developed to measure safety culture. The Safety Attitudes Questionnaire (SAQ), is one of the two most widely used.6 In prior work we used SAQ in a small sample of 12 neonatal intensive care units (NICUs) and demonstrated good psychometric properties and significant variation in safety culture.7 ,8 The other commonly used survey is the Hospital Survey on Patient Safety Culture (HSOPSC).9 This instrument has shown good psychometric properties in adult healthcare settings10 but has not yet been studied in the NICU setting. Head to head comparisons between these instruments are uncommon. Etchegaray and Thomas11 found similar reliability and predictive validity between SAQ and HSOPSC. However, this study was conducted within a single health system, limiting generalisability. Here we expand this literature by comparing these instruments across a large sample of NICUs.

The NICU setting is an excellent domain in which to explore these relations due to the compromised physiological state of patients and the complexity of interventions that are required. Preterm infants are fragile and exposed to complex and prolonged intensive healthcare interventions. Furthermore, care of preterm infants requires the coordination of and transition between prenatal, perinatal and postnatal caregivers from highly subspecialised medical, nursing and ancillary care disciplines. This study evaluates how well the two different survey instruments capture safety culture in this complex environment. The two objectives of this study were,

  1. to assess variation and characteristics of safety and teamwork climate in the NICU setting, and

  2. to compare NICU performance on safety and teamwork climate between SAQ and HSOPSC.

Methods

Overview

This survey study was performed among a voluntary sample of NICUs participating in a Delivery Room Management Quality Improvement Collaborative organised by the California Perinatal Quality Care Collaborative (CPQCC).12 For the current study of NICUs, we assembled a survey to investigate safety culture and workforce engagement using existing validated metrics from multiple instruments (detailed below) and combined them with routinely collected hospital and NICU characteristics.

Sample

Of 61 NICUs who participated in a quality improvement initiative organised by CPQCC, 44 accepted an invitation to complete the survey at its onset (between June and September 2011). Of these 44 NICUs, 10 (22.7%) were designated as regional NICUs, 28 (63.6%) as community NICUs and 6 (13.6%) as intermediate NICUs as defined by the California Department of Healthcare Services. These designations are roughly equivalent to designations by the American Academy of Pediatrics as levels IV, III and II, respectively.13

Staff with 0.5 full-time equivalent or more in the NICU for at least the four consecutive weeks prior to survey administration was invited to participate. Paper-based surveys were administered during regular staff meetings, together with a pencil and sealable return envelope to maintain confidentiality. Individuals not captured in pre-existing meetings were hand-delivered a survey, pencil and return envelope. The CPQCC organised administration of the survey and linked the data with routinely collected organisational data.

Measures

Patient safety culture

The SAQ is associated with clinical outcomes,14–17 and contains 30 items that load on six domains: teamwork climate, safety climate, job satisfaction, perceptions of management, stress recognition and working conditions with response scales ranging from 1 (disagree strongly) to 5 (agree strongly). Here we report on the safety and teamwork climate scales. SAQ also captures respondent characteristics including job position, years in specialty, gender and predominant work shift. Job position included attending physicians (Medical Doctors; MDs), fellow MDs, neonatal nurse practitioners, registered nurses, respiratory care practitioners and others.

HSOPSC18 is composed of 42 items that load on 12 domains. In order to limit respondent burden or the current study, we included four domains: Overall Perceptions of Safety, Teamwork within Units, Communication Openness, and Error Feedback and Communication. For consistency, and to save room on the paper-based survey administration, the HSOPSC items were scaled using the SAQ Likert options. This differed slightly from the HSOPSC standard, which uses several Likert options (eg, Never/Rarely/Sometimes/Most of the times/Always), but did not change the phrasing of the items. The change to a standard response scale improved the internal consistency reliability of the scales when compared with the literature.19

Organisational characteristics

We had access to the following organisational characteristics. NICU level was defined according to the 2012 American Academy of Pediatrics (AAP) Policy Statement13 and obtained in 2012 as part of a survey of NICU directors. Hospital ownership was also obtained from this survey. Annual number of NICU admissions, number of NICU beds and hospital teaching status were derived from the California Department of Public Health's Regional Perinatal Program of California data set.

Statistical analysis

Objective 1—Variation and characteristics: We used descriptive analyses such as frequencies, percentages, means (±SD) and graphs to describe respondent characteristics and organisational characteristics. Safety and teamwork climate scale and item scores were calculated for individual NICUs by taking the average of the scaled items, and then calculating the percentage of respondents within a NICU who reported positively (ie, proportion of those who agreed slightly or strongly).5 Scale reliability was measured via Cronbach's α. One sample two-tailed t tests were performed on the NICU-level scale scores.

Objective 2—Comparison between SAQ and HSOPSC: We assessed similarity in the rankings between SAQ and HSOPSC using Spearman's rank correlation coefficient. In addition, paired sample t tests were used to compare teamwork scales between instruments and safety scales, as well as conceptually similar items. We then ranked NICUs by their scale-level performance (per cent positive, ie, the per cent of respondents in a NICU reporting good climate) and classified them into quartiles. We then determined how often NICUs changed performance quartiles using the two instruments. Finally, we evaluated the effect of respondent and hospital characteristics on teamwork and safety climate scale scores using hierarchical multivariate regression models.

Results

Sample

Of 3294 administered surveys in the 44 NICUs, 2073 were returned for an overall response rate of 62.9%. NICU response rates ranged from 21.7% to 100% with an average of 69.7% (SD=19.8%), and the mean number of respondents per NICU was 47.1 (SD=24.7), ranging from 10 to 144. Table 1 lists respondent and hospital and unit characteristics of our sample. The majority of respondents were nurses (72%) and female (85%). A third of respondents had more than 20 years work experience. NICUs represented were of high acuity (88% level III or IV) and of medium to small size (96% <100 beds).

  • Objective 1—Variation and characteristics of safety and teamwork climate

Table 1

Respondent and neonatal intensive care unit (NICU) characteristics

Figure 1 shows a box and whisker plot of NICU performance on the safety and teamwork climate scales of SAQ and the four safety culture scales of HSOPSC. Respondent and NICU-level variability by scale and item are exhibited in table 2. Internal reliability was good for all scales, with Cronbach's α ranging from 0.71 to 0.86. With SAQ, on average, about two-thirds of the respondents in a given NICU reported good teamwork climate (66%) and good safety climate (65%). Yet, there is opportunity for improvement with regards to appropriately resolving disagreements, creating a culture that makes it easy to discuss and learn from errors.

Table 2

Item-level descriptive results

Figure 1

Safety and teamwork climate scales (Safety Attitudes Questionnaire, SAQ and Hospital Survey on Patient Safety Culture, HSOPSC) across 44 California neonatal intensive care units (NICUs). Box and whisker plots representing the mean, SD, and range per cent positive response for each scale.

With HSOPSC, on average, slightly more than half reported good overall perceptions of safety (56%). Teamwork within units was the scale with the highest scores with three quarters of respondents reporting good teamwork (74%). About half of respondents reported good communication openness (49%) and error feedback communication (49%), mirroring item-level concerns regarding these areas in SAQ. T tests demonstrated significant variation across NICUs (p<0.001) in each one of the teamwork and safety scales on SAQ and HSOPSC.

  • Objective 2—Comparison of safety and teamwork climate scales between SAQ and HSOPSC

We examined the degree to which SAQ and HSOPSC instruments provide similar information on safety and teamwork climate. NICU-level correlation coefficients for the SAQ and HSOPSC scale score per cent positives suggest that they are related but distinct metrics. Safety scales (safety climate and overall perception of safety) and teamwork scales (teamwork climate and teamwork within units) of the two instruments correlated strongly (safety r=0.72, p<0.001; teamwork r=0.67, p<0.001).

Figure 2 plots NICU-level performance for the safety climate and teamwork scales for both instruments. For each scale, using the two instruments resulted in similar NICU-level performance. However, for several NICUs performance differed strongly. For example, in comparing safety climate scales, one NICU, which ranked 23rd on HSOPSC, would have ranked 3rd on SAQ. Another, which ranked 25th on HSOPSC, would have ranked 40th on SAQ. Similar findings are true for the teamwork scale.

Figure 2

Per cent positive responses for safety (A) and teamwork (B) climate by NICU for the SAQ and HSOPSC instruments. Both figures are sorted by increasing performance on HSOPSC. While there is concordance for most NICUs on the two instruments, some (examples noted with an asterisk) have widely differing performance. SAQ, Safety Attitudes Questionnaire; HSOPSC, Hospital Survey on Patient Safety Culture; Safe Clim, Safety Climate; TW Clim, Teamwork Climate; OPS, Overall Perceptions of Safety; TW, Teamwork Within Units; NICU, neonatal intensive care unit.

To further examine the relation between NICU performance on the two instruments, we ranked NICUs by quartiles of teamwork climate and teamwork within units. We found that 23 (52%) fell into different quartiles of teamwork between the instruments, 5 of which were off by two or more quartiles. Similarly, for NICUs ranked by quartiles of safety climate and overall perceptions of safety, we found that 21 (48%) fell into different quartiles of safety between the instruments, 5 of which were off by two or more quartiles.

So about half the time NICUs were classified to a different quartile, based on the use of the most similar scale from a different instrument—the quartile disconnects were larger still between teamwork climate and communication openness and between safety climate and ‘non-punitive response to error’.

Table 3 shows comparisons between SAQ and HSOPSC safety and teamwork climate scales and select conceptually similar items from the two instruments; all comparisons showed significant differences (p<0.002). We investigated potential reasons for these findings based on respondent or hospital characteristics. Table A in the eAppendix shows results of hierarchical models created for each of the SAQ and HSOPSC safety and teamwork scales. This table reveals significant associations of respondent and hospital characteristics with these scales. Women, non-physician providers and those with work experience exceeding more than 1 year exhibited lower safety culture ratings. With regards to hospital characteristics, larger NICUs exhibited lower safety and teamwork climate ratings. Overall though, both safety culture instruments exhibited similar patterns, providing little explanation for the significant differences in scale scores as a result of hospital and respondent demographics.

Table 3

Comparisons of SAQ and HSOPSC teamwork and safety climate scales and select items

Discussion

This study of NICU safety and teamwork climate included more NICUs and providers than previous studies combined8 ,11 ,20 ,21 and contributes to the literature by directly comparing hospital performance across the two most commonly used safety culture surveys, SAQ and HSOPSC. While we found areas of congruence between the instruments, we highlight important distinctions that warrant avoiding direct comparisons for internal or external performance assessments.

For internal comparisons, this study provides a rough but practical translation from one safety culture instrument to the other. Although switching instruments is not encouraged, it is relatively common and leaves organisations with a lack of continuity and a current year of data that can be misleading when compared with prior years using a different instrument. Progress in quality and safety is often assessed through safety culture instruments and instrument transitions leave quality and safety professionals blind to the relative changes that are taking place, while providing an unfortunate reason to minimise the usefulness of current results.

We found that switching instruments translates to a change in quartiles for about half of the NICUs, and that about half of the variance in one scale is not accounted for in the conceptually similar scale on the other instrument. Moreover, every cross-instrument comparison whether scale level or item level, yielded t tests that were significant and often quite large. Simply stated, switching from SAQ to HSOPSC would appear to cause your per cent positive teamwork to improve over 8 points, while your per cent positive safety norms would appear to decrease by 9 points.

Conceptually similar item comparisons across instruments mirrored the scale score results. Inspection of item content did not reveal obvious sources in the variation of these associations. For example, it is unclear why the correlation coefficient of r=0.49 for the two seemingly similar ‘speak-up’ items fails to account for three-fourths of the variance in each other (r2 provides the per cent of the variance in one variable accounted for by the other). Perhaps it is due to the changing valence of the items, subtle phrasing differences, or the respondent interpretation, that cause these seemingly similar items to elicit such different results from the same respondents. In addition, commonly measured respondent and hospital characteristics did not explain these findings.

Etchegaray and Thomas11 had previously conducted a head to head analysis of SAQ and HSOPSC in a single healthcare system. Correlations between safety and teamwork scales between the two instruments were higher in this study compared with the study by Etchegaray et al (Safety Climate 0.78 vs 0.63; Teamwork 0.70 vs 0.52). This difference may be in part explained by the more homogenous sample consisting only of NICUs in our study, or by our transformation of HSOPSC response scales to fit SAQ's. We have previously reported that this transformation improved HSOPSC's psychometric properties for these scales.19

In addition to the direct comparisons between instruments, our findings also affirm and expand the current literature on safety culture assessments in the NICU setting. We found significant variation in safety culture between NICUs and opportunity for improvement across all measured dimensions of safety culture.

An understanding of safety culture is important because it has been linked to clinical and operational outcomes,22–27 as well as burn-out among NICU staff.28 In addition, effective interventions to improve safety culture exist19 ,29 and have proven sustainable.30 In previous work, we demonstrated significant variation and opportunity for improvement in safety culture among a small (n=12) group of NICUs8 and highlighted lower perceptions of safety culture among nurses. Here, we replicated these findings but found two notable differences to our prior work.

First, safety climate ratings in this sample of NICUs were about 6–10% higher than in the previous sample across all domains of SAQ. Since the two samples don't overlap, this finding may be due to differences in samples (the sample in this study self-selected to participate in a quality improvement initiative), time trends or repeat administration of the survey. Mean SAQ scores among participants who noted prior completion of the survey did not differ from those who completed it for the first time (71.6 vs 72.4, p=0.34), making this explanation less likely.

Second, safety culture ratings tended to decrease as the number of admissions and beds goes up, during evening shifts and after the 1st year on the job. These findings mirror those of Sexton et al26 who describe a similar relation with regards to hospital size among 127 adult ICUs who participated in the keystone project. This association is concerning, considering the recent trend of consolidation of hospital systems in the wake of the Affordable Care Act.31

Our findings must be viewed in light of the study design. Survey studies may be subject to bias. We used the well established administration procedures by Sexton et al6 to assure adequate response rates. In addition, our transformation of HSOPSC response scales to fit SAQ may change the behaviour and meaning of these scales. However, we have found improved psychometric properties without rewording the actual questions. Finally, our voluntary sample may have been biased in that these NICUs had signed up to participate in a quality improvement collaborative. Alas, whether this suggests better or worse than average safety culture ratings is not certain given that our collaboratives attract some NICUs that are particularly challenged in the clinical improvement topic for which they sign up. In addition, any bias could be expected to affect ratings on both survey instruments equally.

Conclusion

Safety culture assessments among NICUs vary widely. SAQ and HSOPSC yield strongly correlated results and the scales elicit conceptually similar content. Nevertheless, in this one-to-one comparison of the instruments, we failed to find a single set of scales or items that yielded interchangeable results. We recommend caution in translating and or transitioning between instruments.

Acknowledgments

The authors thank the local leaders and staff of the CPQCC NICUs who contributed their efforts to this study.

References

View Abstract
  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Contributors JP had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. JP acquired funding for this study, conceptualised and designed the study, selected data for inclusion in analyses, analysed the data, assisted with interpretation of the results, drafted the initial manuscript, and approved the final manuscript as submitted. PK assisted with designing the analysis and interpretation of the results, revised the manuscript, and approved the final manuscript as submitted. HL helped with conceptualising the paper, designing the analysis and interpreting the results. He revised the manuscript, and approved the final manuscript as submitted. PS helped conceptualise and design the study, was the local lead for CPQCC NICUs, helped select data for inclusion in the survey and analyses, assisted with interpretation of the results, revised the manuscript, and approved the final manuscript as submitted. CN coordinated data collection among CPQCC member NICUs, de-identified data, assisted with interpretation of the results, revised the manuscript, and approved the final manuscript as submitted. EJT helped acquire funding, conceptualised and designed the study, selected data for inclusion in the survey and analyses, assisted with interpretation of the results, revised the manuscript, and approved the final manuscript as submitted. JME helped with conceptualising the paper, designing the analysis and interpreting of the results. He revised the manuscript, and approved the final manuscript as submitted. BS helped acquire funding for this study, conceptualised and designed the study, selected data for inclusion in the survey and analyses, assisted with interpretation of the results. He revised the initial manuscript and approved the final manuscript as submitted.

  • Funding Texas Children's Hospital (33–126); Agency for Healthcare Research and Quality (1UC1HS014246); National Institute of Child Health and Human Development (K23 HD056298-01, K24 HD053771-01).

  • Author note At the time of the research, Dr Profit was on faculty at Baylor College of Medicine, Texas Children's Hospital, Department of Pediatrics, Section of Neonatology. He held a secondary appointment in the Department of Medicine, Section of Health Services Research and conducted his research at the VA Health Services Research and Development Center of Excellence. Dr. Etchegaray held an appointment at the University of Texas at Houston – Memorial Hermann Center for Healthcare Quality and Safety, University of Texas Medical School.

  • Competing interests None declared.

  • Ethics approval Institutional Review Boards at Stanford University and Baylor College of Medicine.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.