You are here

Article Text

Article menu
Original research
A theory-driven, longitudinal evaluation of the impact of team training on safety culture in 24 hospitals
Free
  1. Katherine J Jones1,
  2. Anne M Skinner1,
  3. Robin High2,
  4. Roni Reiter-Palmon3
  1. 1Division of Physical Therapy Education, School of Allied Health Professions, University of Nebraska Medical Center, Omaha, Nebraska, USA
  2. 2Department of Biostatistics, College of Public Health, University of Nebraska Medical Center, Omaha, Nebraska, USA
  3. 3Industrial/Organizational Psychology Program, Center for Collaboration Science, University of Nebraska at Omaha, Omaha, Nebraska, USA
  1. Correspondence to Dr Katherine J Jones, Division of Physical Therapy Education, School of Allied Health Professions, 984420 Nebraska Medical Center, Omaha, NE 68198-4420, USA; kjonesj{at}unmc.edu

Abstract

Background Effective teamwork facilitates collective learning, which is integral to safety culture. There are no rigorous evaluations of the impact of team training on the four components of safety culture—reporting, just, flexible and learning cultures. We evaluated the impact of a year-long team training programme on safety culture in 24 hospitals using two theoretical frameworks.

Methods We used two quasi-experimental designs: a cross-sectional comparison of hospital survey on patient safety culture (HSOPS) results from an intervention group of 24 hospitals to a static group of 13 hospitals and a pre-post comparison of HSOPS results within intervention hospitals. Dependent variables were HSOPS items representing the four components of safety culture; independent variables were derived from items added to the HSOPS that measured the extent of team training, learning and transfer. We used a generalised linear mixed model approach to account for the correlated nature of the data.

Results 59% of 2137 respondents from the intervention group reported receiving team training. Intervention group HSOPS scores were significantly higher than static group scores in three dimensions assessing the flexible and learning components of safety culture. The distribution of the adoption of team behaviours (transfer) varied in the intervention group from 2.8% to 31.0%. Adoption of team behaviours was significantly associated with odds of an individual reacting more positively at reassessment than baseline to nine items reflecting all four components of safety culture.

Conclusions Team training can result in transformational change in safety culture when the work environment supports the transfer of learning to new behaviour.

  • Team training
  • Safety culture
  • Evaluation methodology
  • Surveys

https://doi.org/10.1136/bmjqs-2012-000939

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    Safety culture can be defined as the learned, shared, enduring values and behaviours of organisation members regarding the organisation's willingness to detect and learn from errors.1–3 According to Reason,4 four components of organisational safety culture interact to achieve learning and high reliability: reporting culture, just culture, flexible culture and learning culture. The knowledge, skills, attitudes, language and coordinating mechanisms inherent in teamwork5 create the flexibility team members need to manage complexity6 and learn from experience.7–9 Effective team functioning is comparable to Reason's flexible component of safety culture. However, observed behaviours are often not consistent with stated organisational values3 because individuals lack the knowledge, skills and management support10–12 to sustain desired behaviours and achieve high reliability.13

    Team strategies and tools to enhance performance and patient safety (TeamSTEPPS) is a team training programme developed by the US Department of Defense and the Agency for Healthcare Research and Quality (AHRQ) to teach the knowledge and skills that comprise effective teamwork. These skills are: leadership, situation monitoring, mutual support and communication.14 There is evidence that successful team training and effective teamwork improve patient outcomes15–22 and team-related dimensions of safety culture.23 However, team training alone may not produce the desired results.24 ,25 A meta-analysis found that team training accounted for less than 20% of the variance in team performance.26 The primary determinant of team performance is what an organisation does after training to sustain or routinise team behaviours.11 ,27 To our knowledge, there are no rigorous evaluations of the impact of team training on all four components of safety culture. Given the ability of effective teamwork to facilitate collective learning,8 we hypothesise that the adoption of team behaviours within a hospital will positively impact all components of safety culture and contribute to transformational culture change.28

    The use of theoretical frameworks advances our knowledge of organisational factors that are fundamental to successful implementation and sustainment of innovations such as team training.29 The ‘Diffusion of innovations’ framework of Rogers30 identifies characteristics of innovations, individuals and organisations that are associated with the adoption of new ideas and stages of adoption. The cross-sectional distribution of adoption within individuals and organisations typically follows a normal distribution. Based on this distribution, Rogers30 described five categories that help to explain variation in adoption: innovators, early adopters, early majority, late majority and laggards. When used for evaluation purposes, these categories are standardised by deciding the number of categories to use, the proportion of organisations to include in each category and the method of defining categories. In general, innovator and early adopter organisations successfully implement and sustain innovations because they tend to have slack resources, remain connected to experts, and have champions who overcome barriers.30 These characteristics facilitate movement through organisational stages of innovation: agenda-setting to solve a problem, matching the problem to an innovation, redefining/restructuring the innovation to fit the organisation, clarifying the relationship between the organisation and the innovation, and making the innovation routine.30

    The four-level taxonomy of training criteria by Kirkpatrick and Kirkpatrick31 is widely used to evaluate the effectiveness of training programmes. The original taxonomy has been augmented as follows:32

    • Level 1: Reaction to training, including satisfaction and assessment of the utility of the training.

    • Level 2: Learning, including immediate post-training changes in knowledge, knowledge retention and demonstration of skill.

    • Level 3: Transfer of learning to behaviour that is applied in the work environment (transfer is synonymous with the adoption of new behaviours).

    • Level 4: Results, the degree to which desired outcomes are achieved as a result of transfer.

    Success at levels 1 to 3 increases the likelihood of achieving level 4—desired results such as transformational change in safety culture.32

    The purpose of this study was to evaluate the impact of a team training intervention on hospital safety culture. We used Kirkpatrick's taxonomy as the overall framework for our evaluation. As Rogers’ ‘Diffusion of innovations’ framework explains variation in the adoption of new behaviours (Kirkpatrick's level 3—transfer), we report intervention hospital characteristics, respondent characteristics and hospital survey on patient safety culture (HSOPS) results (Kirkpatrick's level 4) by Rogers’ adopter categories.30 The linkage between these two frameworks and a HSOPS outcome are illustrated in figure 1.

    Figure 1
    Figure 1

    Evaluation of the impact of team training on safety culture using Rogers’ adopter categories.

    Methods

    Setting and participants

    The 37 hospitals in this study were critical access hospitals located in three central US states. These hospitals served 37 distinct counties that had a median 2009 population of 7766.33 The category, ‘critical access hospital,’ was created in 1997 by the US government to maintain access to healthcare in rural areas by providing cost-based reimbursement for services provided to Medicare beneficiaries. These hospitals are limited to 25 beds and must be located more than 35 miles from another hospital;34 they comprise one-fourth of the general community hospitals in the USA.35

    From 2005 to 2007, 19 of the 37 hospitals in this study participated in our AHRQ-funded project that sought to implement the patient safety practice of organisational learning from voluntary medication error reporting.36 The remaining 18 hospitals became engaged in the improvement collaborative through the Nebraska Office of Rural Health and hospital networks. Of the 37 hospitals, 24 had an existing baseline assessment of safety culture conducted within 18 months before the beginning of the intervention in April 2008. These 24 agreed to participate in the intervention. The remaining 13 did not participate in the intervention and thus created a static group for comparison.37 This study was approved by the University of Nebraska Medical Center Institutional Review Board.

    Intervention description

    The intervention was conducted from February 2008 to March 2009 and was intended to improve safety culture by implementing and sustaining team behaviours in the 24 intervention hospitals. It included the following activities:

    • agenda-setting for improvement using the baseline assessment of safety culture to identify weaknesses in teamwork and communication;

    • matching safety culture weaknesses to TeamSTEPPS tools;

    • conducting the TeamSTEPPS train-the-trainer course14 in April 2008 to train master trainers for each intervention hospital;

    • conducting a workshop on 6 October 2008 to address disruptive behaviour;38

    • conducting a TeamSTEPPS fundamentals14 course on 7 October 2008 to train coaches to assist with implementation; and

    • conducting 17 one-half hour conference calls to share strategies and tools for redefining/restructuring, clarifying and routinising behaviours to sustain the innovation. Summaries of the calls were distributed via email to intervention hospitals.

    Strategies to redefine and clarify the innovation included focusing initial training efforts on supervisors who then role-modelled desired behaviours by conducting briefs, huddles and debriefs. The Situation Background Assessment and Recommendation (SBAR) format was used to structure email communication and fax forms to physician offices. Bulletin boards, newsletter articles and contests were used to clarify and provide additional learning opportunities after classroom training. Strategies to routinise behaviours included auditing the frequency with which tools were used, conducting hospital-wide briefs, integrating the use of TeamSTEPPS tools and strategies into new employee orientation, and including the use of TeamSTEPPS tools in job descriptions and performance appraisals.

    Instrument and data collection

    We conducted the AHRQ HSOPS in March 2009 as a reassessment for the intervention group of 24 hospitals and a baseline assessment for the static group of 13 hospitals. The HSOPS is a psychometrically sound39 ,40 instrument developed by AHRQ to provide healthcare organisations with a valid tool to assess safety culture.41 It is widely used in the USA with results from 1128 hospitals in the 2012 comparative database.42 It consists of 42 items categorised in 12 dimensions and two items that are outcome measures. These 44 items reflect Reason's four components of safety culture.36

    We customised the HSOPS to evaluate the impact of patient safety interventions on safety culture in critical access hospitals. In 2005, we revised the work areas and job titles in the demographics section to categorise staff accurately in these small hospitals.36 For this study, we added one item to measure respondents’ extent of TeamSTEPPS training, four items to measure learning (level 2) as knowledge of TeamSTEPPS tools, and five items to measure transfer (level 3) as a respondent's perception of the frequency with which TeamSTEPPS tools and strategies were used. The learning and transfer items were internally consistent and are available on request.

    To conduct the survey, key contacts at participating hospitals provided a list of eligible employees according to the criteria recommended in the survey user's guide.41 We assigned a unique identifier to each employee that was linked to their name and hospital. We used this linkage to: personalise the invitation to participate in the survey (personalisation increases likelihood of response);43 calculate accurate response rates by hospital; track change in individual responses at the item level from baseline to reassessment; and adjust for the correlated nature of the data. At baseline and reassessment, we used the Dillman tailored design methodology43 to conduct the survey as a self-administered paper instrument, as described elsewhere.36

    Study design and statistical analyses

    Returned surveys were scanned into a database and imported into SAS V.9.2 for analysis. We conducted a cross-sectional comparison of the HSOPS results from the intervention group of 24 hospitals to the static group of 13 hospitals and a pre-post comparison of HSOPS results within the intervention hospitals according to Rogers’ adopter categories.30 Due to our small sample of 24 intervention hospitals, we collapsed Rogers’ five adopter categories into three, and we used the quartile distribution of the perception of transfer to achieve three balanced groups: six hospitals were categorised as laggards, 12 as early/late majority and six as early adopters (figure 1).

    Statistical analyses to test for differences in hospital and respondent characteristics by groups were chosen based on the type and distribution of data. Tests used in tables 1 and 2 are described in the footnotes. We calculated the mean percent positive scores for HSOPS items and dimensions for each group and category of hospitals according to the method in the survey user's guide.41 We determined the internal consistency of the survey items by calculating Cronbach's α coefficient for each dimension at the hospital level, and then we averaged the hospital level coefficients for each dimension by group and adopter category.

    View this table:
    Table 1

    Characteristics of 37 hospitals participating in 2009 hospital survey on patient safety culture by group

    View this table:
    Table 2

    Characteristics of respondents participating in 2009 hospital survey on patient safety culture by group

    All data summarising HSOPS results presented in supplementary appendix A (available online only), tables 3 and 4 were considered representative values of a population of hospitals to support a broad inference. Consequently, all data were analysed using PROC GLIMMIX (generalised linear mixed model),44 which provides a choice of distributions for the dependent variable, accounts for random effects (clustering of respondents within hospitals) or correlated data (repeated measures), and provides either subject-specific or population-averaged inferences.44 The dependent variable was an individual positive response to a survey item, which was a binary outcome measured pre and post-intervention. Within PROC GLIMMIX, analysis of a binary distribution utilises a logit link that provides OR. We used this technique to model the OR of a respondent in an intervention hospital reacting positively to an item at reassessment compared to baseline (table 3), and for each 5% increase in the proportion of respondents reporting TeamSTEPPS training, learning and transfer (table 4). To calculate these OR in table 3, we used data from those respondents in the intervention group who had both baseline and reassessment results. Those respondents were significantly more likely to have been at the hospital for 6 years or longer compared to those respondents without baseline data (75% vs 37%, p<0.0001).

    View this table:
    Table 3

    OR of responding positively in 2009 compared to baseline by adoption category for HSOPS items reflecting four components of safety culture

    View this table:
    Table 4

    2009 OR of responding positively to a HSOPS item for each 5% increase in the proportion of respondents reporting TeamSTEPPS training, learning and transfer

    We chose 10 survey items to compare the odds of a respondent reacting positively at reassessment and baseline. These items were chosen because they assess behaviours that reflect the four components of safety culture such as near-miss reporting and teamwork, and they were the least positively perceived items within a dimension. All statistical tests were two-sided, and probability values less than 0.05 were considered statistically significant.

    Results

    Hospital and respondent characteristics

    Of the 4601 personnel in the 37 hospitals who were eligible to participate in the survey, 3465 (75.3%) responded. Hospital characteristics are reported in table 1. Laggard hospitals participated in a significantly fewer number of support calls than did the other two adopter categories. Although not statistically significant, laggard hospitals had smaller numbers of employees, smaller median bed sizes, and trained fewer coaches than did the other two adopter categories.

    Respondent characteristics are reported in table 2. There were statistically significant differences in the distribution of positions and work areas and the proportion of employees with direct patient contact between the intervention and static groups. Only the distribution of respondents by work area, tenure at the hospital and tenure in the profession differed significantly between the adopter categories. Of note, 19% of respondents in early adopter hospitals compared to approximately 27% in the other categories had worked at the hospital for 16 years or more.

    Cross-sectional comparison of HSOPS results by hospital group

    Supplementary appendix A (available online only) reports results for the survey dimensions, items, internal consistency of the dimensions, and the added training, learning and transfer items by hospital group for the 37 hospitals. The intervention group had significantly greater positive scores than did the static group in the survey dimensions organisational learning—continuous improvement (76% vs 71%), teamwork within departments (82% vs 80%) and teamwork across hospital departments (67% vs 62%). The early adopter hospitals had significantly greater positive scores than did the early/late majority and laggard hospitals in the survey dimensions frequency of events reported (71% vs 65% and 56%), staffing (76% vs 70% and 64%) and hospital management support for patient safety (89% vs 83% and 75%). The early adopter hospitals had significantly greater positive scores than did the laggard hospitals in all survey dimensions. The Cronbach's α coefficients for the dimensions varied across the groups from 0.57 to 0.87. The dimensions staffing and organisational learning—continuous improvement had coefficients below the suggested value for acceptable reliability of 0.70.45

    TeamSTEPPS training and learning were significantly more prevalent in the intervention group than the static group (59% vs 5% and 25% vs 2%, respectively). There was no statistically significant difference between the proportions of respondents who reported the adoption of team behaviours (transfer) in the intervention compared to the static group (15% vs 12%). Figure 1 illustrates the distribution of transfer in the intervention hospitals by adopter categories. Within the early adopter hospitals, an average of 83% of respondents reported receiving training, 44% demonstrated learning and 26% reported transfer. These values for training, learning and transfer were 62%, 27% and 18% in early/late majority hospitals, and 27%, 11% and 7% in laggard hospitals.

    Longitudinal comparison of HSOPS results by adopter category

    Of the 2137 respondents in the intervention group, 1190 (56%) had both baseline and reassessment data. Table 3 compares these respondents’ average baseline and reassessment scores for 10 HSOPS items representing Reason's four components of safety culture, and it provides the odds of a respondent reacting positively to an item at reassessment compared to baseline. The average baseline scores for early adopter hospitals were 5.8% greater than those in early/late majority hospitals and 10.4% greater than those in laggard hospitals. The average changes in scores from baseline to reassessment were 3.5%, 3.0% and −2.1% for the adopter categories, respectively.

    There was variability by adopter category in the odds of a respondent reacting positively to an item at reassessment compared to baseline.

    For early adopter respondents, the odds of reacting positively at reassessment compared to baseline were significantly:

    • greater for two items that represent flexible (A14) and learning (A9) cultures, and were

    • less for one item representing flexible culture (F7).

    For early/late majority respondents, the odds of reacting positively at reassessment compared to baseline were significantly:

    • greater for four items representing flexible (A11, C4) and learning cultures (A9, A10).

    For laggard respondents, the odds of reacting positively at reassessment compared to baseline were significantly:

    • less for two items representing flexible culture (F5, F7).

    Cross-sectional comparison of HSOPS results using Kirkpatrick's taxonomy

    Table 4 provides OR of responding positively to the 10 HSOPS items for each 5% increase in the proportion of respondents reporting training, learning (level 2) and transfer (level 3). For every 5% increase in the proportion reporting some training in TeamSTEPPS, there was a significant:

    • increase in the odds that a respondent would react positively to one item representing learning culture (A10).

    For every 5% increase in the proportion reporting that they had received training in all of the modules or that they were a master trainer, there was a significant:

    • increase in the odds that a respondent would react positively to one item representing learning culture (A9).

    For every 5% increase in the proportion demonstrating that they had learned TeamSTEPPS tools, there was a significant:

    • increase in the odds that a respondent would react positively to five items reflecting reporting (D15), flexible (A14) and learning (A9, F9, A10) cultures.

    For every 5% increase in the proportion reporting transfer, there was a significant:

    • increase in the odds that a respondent would react positively to nine items reflecting all four components of safety culture.

    Discussion

    We sought to rigorously evaluate the impact of the TeamSTEPPS team training programme on hospital safety culture by using two designs recommended for quasi-experimental research: comparison of results to a static group and a pre-post comparison.46 Results from the static group comparison (see supplementary appendix A, available online only) indicate that the team training intervention was associated with significantly greater hospital-level positive HSOPS scores in three survey dimensions reflecting flexible and learning cultures. Results from the pre-post comparison (table 3) indicate that the intervention was associated with changes in individual respondents’ perceptions of flexible and learning cultures, and that the direction and magnitude of these changes varies by adopter category. In particular, individual respondents in the early adopter and laggard hospitals were significantly less likely to have a positive perception of hospital handoffs and transitions post-intervention compared to baseline. This decrease is consistent with the phenomenon of response shift bias, which occurs in pre-post comparisons when participation in an intervention changes a respondent's frame of reference.47 Therefore, participating in the HSOPS and learning and/or adopting team skills may have changed respondents’ reference point regarding what defines an effective handoff. Participating in the HSOPS increases awareness of patient safety concepts and behaviours,48 and is considered a patient safety initiative.49 Finally, the fact that 12% of respondents in the static group reported the adoption of team behaviours indicates the potential for response shift bias at reassessment if these hospitals conduct team training.

    Results from assessing the discrete impact of training, learning and transfer on safety culture (table 4) reveal a dose–response relationship between the levels of Kirkpatrick's taxonomy: simply participating in training had minimal impact on perceptions of safety culture, learning TeamSTEPPS tools had a moderate impact on perceptions of safety culture, and transferring learning to behaviour had the greatest impact because it increased the odds of reacting positively to nine of the 10 survey items reflecting all four components of safety culture. This dose–response relationship is consistent with what is referred to as the ‘transfer problem’50 ,51—the fact that much of the training conducted by organisations does not produce transfer of learning to behaviour in the work environment.

    Grossman and Salas51 have concisely reviewed the factors that address the transfer problem. These factors include employee characteristics (eg, motivation, perceived utility of training), the design of the training (eg, role modelling, opportunities to practise), and the work environment (transfer climate, support from supervisors and peers, follow-up learning opportunities). These factors are consistent with characteristics associated with Rogers’ categories of adopters and with movement through the organisational stages of adoption. The design of the TeamSTEPPS training curriculum, which was used by all intervention hospitals, is standardised and incorporates factors such as behavioural modelling and error management known to facilitate transfer.

    There is evidence that employee characteristics and work environments differed across the adopter categories. Employees in laggard hospitals had significantly less employee turnover than did the other adopter categories, possibly indicating that employees who experienced job dissatisfaction in these hospitals remained on the job but were poorly motivated.30 ,52 This response is consistent with limited employment options in isolated rural areas. The work environment differed among the adopter groups in several ways. First, early adopter and early/late majority hospitals tended to be larger and remained significantly more connected to the collaborative by participating in conference calls than did the laggard hospitals. Larger size (indicating potentially slack resources) and a higher degree of interconnectedness are characteristics that are positively associated with organisational innovativeness30 and thus, transfer. Second, early adopter and early/late majority hospitals focused on training supervisors/managers who could then support transfer by providing feedback, and they implemented multiple follow-up learning opportunities (bulletin boards, newsletter articles, new employee orientation). Finally, as indicated by baseline HSOPS scores, the laggard hospitals began the intervention with less mature safety cultures than did the early adopter and early/late majority hospitals. The laggards did not have as strong a foundation of management support for patient safety and reporting and learning from error as did the other two adopter categories. Therefore, the laggard hospitals had the greatest need for team training but had the least success in implementation, which reflects Rogers’ innovativeness/need paradox—those organisations in greatest need of an innovation are the last to adopt.30

    Strengths and limitations

    There are several strengths that support our ability to draw a causal inference between the intervention and changes in perceptions of safety culture. We established that the reliability of the HSOPS in our sample of critical access hospitals was comparable to that reported for other populations.40 ,49 ,53 We used two quasi-experimental designs thus limiting potential confounding due to maturation and history. We conducted the HSOPS within each hospital in a standardised manner enabling us to pool the results validly. Because we linked baseline and reassessment responses at the individual level, we were able to compare post-training to pre-training results and assess the impact of discrete changes in training, learning and transfer on safety culture. We allowed nearly a year for the intervention to produce the desired result of change in safety culture.31 We used appropriate statistical techniques to account for the correlated nature of the data at the hospital and respondent levels,46 and we prospectively used two theoretical frameworks to guide our analysis and interpretation.29 Finally, the median response rates for all hospital groups were 76% and greater thus minimising non-response bias.

    There are limitations that may affect the interpretation of these results. As with many quality improvement initiatives, these hospitals self-selected to participate, they were not randomly assigned to the intervention and static groups, and we did not independently audit the prevalence of team behaviours. There was variability in the extent to which each hospital followed the TeamSTEPPS curriculum within their hospital and implemented their action plan for implementation and sustainment. This study was conducted in critical access hospitals, which may limit its generalisability to larger hospitals. However, the consistency of our findings with theoretical frameworks tends to mitigate these limitations. In addition, the small size of these hospitals made it reasonable to assume that culture change could occur within a year. Finally, we had limited power to detect differences between adopter categories due to small sample size.

    Conclusion and practical implications

    Team training can result in transformational change in safety culture when the work environment supports transfer of learning to behaviour. First, this work environment must include a foundation of reporting and learning from error. Second, managers and senior leaders should understand that we ‘learn to team’ because teamwork is essential for learning, adapting, and providing safe care in the current complex healthcare environment.54 Without this knowledge, mid-level managers will not role model team behaviours and senior leaders will not allocate the financial and human resources needed to sustain ongoing training that includes learning opportunities such as newsletters, new employee orientation, annual refresher courses and opportunities to practise including simulation. Senior leaders must also ensure that team behaviours are routinised by revising job descriptions and performance appraisals to include team behaviours. In summary, the laggards in our study were in greatest need of teamwork, but were least likely to adopt because their work environment did not support training, learning, or transfer. Additional research is needed to understand the barriers to establishing a work environment that supports the transfer of team training in hospitals.

    Acknowledgments

    The authors would like to thank Andrea Bowen, DPT, for her assistance in data management and editing and Victoria Kennel, MA, for her assistance in clarifying concepts in the discussion.

    References

    1. ↵
      Health and Safety Commission. Organising for safety: third report of the human factors study group of ACSNI. Sudbury, UK: HSE Books, 1993.
    2. ↵
      1. Wiegmann DA,
      2. Zhang H,
      3. von Thaden T,
      4. et al
      . A synthesis of safety culture and safety climate research. http://www.aviation.illinois.edu/avimain/papers/research/pub_pdfs/techreports/02-03.pdf (accessed 3 Jan 2013).
    3. ↵
      1. Schein EH
      . Organizational culture and leadership, 4th edn. San Francisco, CA: Josey-Bass, 2010.
    4. ↵
      1. Reason J
      . Managing the risks of organizational accidents. Aldershot, UK: Ashgate, 1997.
    5. ↵
      1. Salas E,
      2. Sims DE,
      3. Burke CS
      . Is there a ‘Big Five’ in teamwork? Small Group Res 2005;36:555–99.
      OpenUrlAbstract
    6. ↵
      1. Cannon-Bowers JA,
      2. Salas E
      . Team performance and training in complex environments: recent findings from applied research. Curr Dir Psychol Sci 1998;7:83–7.
      OpenUrlFREE Full Text
    7. ↵
      1. Senge PM
      . The fifth discipline: the art & practice of the learning organization. New York, NY: Doubleday, 1990.
    8. ↵
      1. Edmonson AC
      . Learning from failure in Health Care: frequent opportunities, pervasive barriers. Qual Saf Health Care 2004;12(Suppl. II):ii3–9.
      OpenUrl
    9. ↵
      1. Salas E,
      2. Rosen MA,
      3. Burke CS,
      4. et al
      .  The wisdom of collectives in organizations: an update of the teamwork competencies. In: Salas E, Goodwin GF, Burke CS, eds. Team effectiveness in complex organizations: cross-disciplinary perspectives and approaches. New York, NY: Routledge/Taylor & Francis Group, 2009: 39–79.
    10. ↵
      1. Frankel AS,
      2. Leonard MW,
      3. Denham CR
      . Fair and just culture, team behavior, and leadership engagement: the tools to achieve high reliability. Health Serv Res 2006;41:1690–709.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Helfrich CD,
      2. Weiner BJ,
      3. McKinney MM,
      4. et al
      . Determinants of implementation effectiveness: adapting a framework for complex innovations. Med Care Res Rev 2007;64:279–303.
      OpenUrlAbstract/FREE Full Text
    12. ↵
      1. Singer SJ,
      2. Hayes J,
      3. Cooper JB,
      4. et al
      . A case for safety leadership team training of hospital managers. Health Care Manage Rev 2011;36:188–200.
      OpenUrlCrossRefPubMedWeb of Science
    13. ↵
      1. Baker DP,
      2. Day R,
      3. Salas E
      . Teamwork as an essential component of high-reliability organizations. Health Serv Res 2006;41:1576–98.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      Agency for Healthcare Research and Quality. TeamSTEPPS: strategies and tools to enhance performance and patient safety. http://www.teamstepps.ahrq.gov  (accessed 12 Jul 2010).
    15. ↵
      1. Deering S,
      2. Rosen MA,
      3. Ludi V,
      4. et al
      . On the front lines of patient safety: implementation and evaluation of team training in Iraq. Jt Comm J Qual Patient Saf 2011;37:350–6.
      OpenUrlPubMed
    16. ↵
      1. Friedman DM,
      2. Berger DL
      . Improving team structure and communication: a key to hospital efficiency. Arch Surg 2004;139:1194–8.
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      1. Mayer CM,
      2. Cluff L,
      3. Lin WT,
      4. et al
      . Evaluating efforts to optimize TeamSTEPPS implementation in surgical and pediatric intensive care units. Jt Comm J Qual Patient Saf 2011;37:365–74.
      OpenUrlPubMed
    18. ↵
      1. Mukamel DB,
      2. Temkin-Greener H,
      3. Delavan R,
      4. et al
      . Team performance and risk-adjusted health outcomes in the Program of All-Inclusive Care for the Elderly (PACE). Gerontologist 2006;46:227–37.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Neily J,
      2. Mills PD,
      3. Young-Xu Y,
      4. et al
      . Association between implementation of a medical team training program and surgical mortality. JAMA 2010;304:1693–700.
      OpenUrlCrossRefPubMedWeb of Science
    20. ↵
      1. Strasser DC,
      2. Falconer JA,
      3. Stevens AB,
      4. et al
      . Team training and stroke rehabilitation outcomes: a cluster randomized trial. Arch Phys Med Rehabil 2008;89:10–15.
      OpenUrlCrossRefPubMed
    21. ↵
      1. Young MP,
      2. Gooder VJ,
      3. Oltermann MH,
      4. et al
      . The impact of a multidisciplinary approach on caring for ventilator-dependent patients. Int J Qual Health Care 1998;10:15–26.
      OpenUrlAbstract/FREE Full Text
    22. ↵
      1. Percarpio KB,
      2. Harris FS,
      3. Hatfield BA,
      4. et al
      . Code debriefing from the Department of Veterans Affairs (VA) Medical Team Training program improves the cardiopulmonary resuscitation code process. Jt Comm J Qual Patient Saf 2010;36:424–9.
      OpenUrlPubMed
    23. ↵
      1. Weaver SJ,
      2. Rosen MA,
      3. DiazGranados D,
      4. et al
      . Does teamwork improve performance in the operating room? A multilevel evaluation. Jt Comm J Qual Patient Saf 2010;36:133–42.
      OpenUrlPubMed
    24. ↵
      1. Riley W,
      2. Davis S,
      3. Miller K,
      4. et al
      . Didactic and simulation nontechnical skills team training to improve perinatal patient outcomes in a community hospital. Jt Comm J Qual Patient Saf 2011;37:357–64.
      OpenUrlPubMed
    25. ↵
      1. Siassakos D,
      2. Draycott TJ,
      3. Crofts JF,
      4. et al
      . More to teamwork than knowledge, skill and attitude. Br J Obstet Gynaecol 2010;117:1262–9.
      OpenUrlCrossRef
    26. ↵
      1. Salas E,
      2. DiazGranados D,
      3. Klein C,
      4. et al
      . Does team training improve team performance? A meta-analysis. Hum Factors 2008;50:903–33.
      OpenUrlAbstract/FREE Full Text
    27. ↵
      1. Eccles MP,
      2. Mittman BS
      . Welcome to implementation science. Implement Sci 2006;1:1.
      OpenUrlCrossRef
    28. ↵
      1. Leape L,
      2. Berwick D,
      3. Clancy C,
      4. et al
      . Transforming healthcare: a safety imperative. Qual Saf Health Care 2009;18:424–8.
      OpenUrlAbstract/FREE Full Text
    29. ↵
      1. Eccles M,
      2. Grimshaw J,
      3. Walker A,
      4. et al
      . Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. J Clin Epidemiol 2005;58:107–12.
      OpenUrlCrossRefPubMedWeb of Science
    30. ↵
      1. Rogers EM
      . Diffusion of innovations, 5th edn. New York: Free Press, 2003.
    31. ↵
      1. Kirkpatrick DL,
      2. Kirkpatrick JD
      . Evaluating training programs: the four levels, 3rd edn. San Francisco: Berrett-Koehler Publishers, Inc., 2006.
    32. ↵
      1. Alliger GM,
      2. Tannenbaum SI,
      3. Bennett W,
      4. et al
      . A meta-analysis of the relations among training criteria. Pers Psychol 1997;50:34158.
      OpenUrlCrossRef
    33. ↵
      US Census Bureau. Population Estimates. County Totals: Vintage 2009. http://www.census.gov/popest/data/counties/totals/2009/index.html (accessed 15 Feb 2012).
    34. ↵
      Flex Monitoring Team. About the Flex Program and Critical Access Hospitals . http://www.flexmonitoring.org/aboutcahflex.shtml (accessed 2 Dec 2010).
    35. ↵
      Flex Monitoring Team. CAH Information. http://www.flexmonitoring.org/cahlistRA.cgi (accessed 21 Feb 2012).
    36. ↵
      1. Jones KJ,
      2. Skinner A,
      3. Xu L,
      4. et al
      . The AHRQ hospital survey on patient safety culture: a tool to plan and evaluate patient safety programs. In: Henriksen K, Battles JB, Keyes MA, et al. eds. Advances in patient safety: new directions and alternative approaches. (Vol. 2: Culture and Redesign). Rockville, MD: Agency for Healthcare Research and Quality, 2008.
    37. ↵
      1. Campbell D,
      2. Stanley J
      . Experimental and quasi-experimental designs for research. Boston, MA: Houghton Mifflin Company, 1963.
    38. ↵
      1. Rosenstein AH,
      2. O'Daniel M
      . Invited article: managing disruptive physician behavior: impact on staff relationships and patient care. Neurology 2008;70:1564–70.
      OpenUrlCrossRef
    39. ↵
      1. Colla JB,
      2. Bracken AC,
      3. Kinney LM,
      4. et al
      . Measuring patient safety climate: a review of surveys. Qual Saf Health Care 2005;14:364–6.
      OpenUrlAbstract/FREE Full Text
    40. ↵
      1. Sorra JS,
      2. Dyer N
      . Multilevel psychometric properties of the AHRQ hospital survey on patient safety culture. BMC Health Serv Res 2010;10:199.
      OpenUrlCrossRefPubMed
    41. ↵
      1. Sorra JS,
      2. Nieva VF
      . Hospital survey on patient safety culture. http://www.ahrq.gov/qual/hospculture/hospcult.pdf (accessed 23 Feb 2008).
    42. ↵
      1. Sorra J,
      2. Famolaro T,
      3. Dyer N,
      4. et al
      . Hospital Survey on Patient Safety Culture 2012 user comparative database report. (Prepared by Westat, Rockville, MD, under Contract No. HHSA 290200710024C). 2012; AHRQ Publication No. 12-0017.
    43. ↵
      1. Dillman DA
      . Mail and internet surveys: the tailored design method. 2nd edn. Hoboken, New Jersey: John Wiley & Sons, Inc., 2007.
    44. ↵
      SAS. The GLIMMIX Procedure, June 2006. SAS The Power to Know Web site. http://www.support.sas.com/rnd/app/papers/glimmix.pdf (accessed 21 Feb 2012).
    45. ↵
      1. Cronbach L
      . Coefficient alpha and the internal structure of tests. Psychometrika 1951;16:297–334.
      OpenUrlCrossRefWeb of Science
    46. ↵
      1. Dawson JD
      . Quantitative analytical methods in translation research. Worldviews Evid Based Nurs 2004;1(Suppl. 1):S60–4.
      OpenUrlCrossRefPubMed
    47. ↵
      1. Howard GS
      . Response-shift bias: a problem in evaluating interventions with pre/post self-reports. Eval Rev 1980;4:93–106.
      OpenUrlAbstract/FREE Full Text
    48. ↵
      1. Nieva VF,
      2. Sorra J
      . Safety culture assessment: a tool for improving patient safety in healthcare organizations. Qual Saf Health Care 2003;12(Suppl. 2):ii17–23.
      OpenUrlAbstract/FREE Full Text
    49. ↵
      1. Vlayen A,
      2. Hellings J,
      3. Claes N,
      4. et al
      . A nationwide Hospital Survey on Patient Safety Culture in Belgian hospitals: setting priorities at the launch of a 5-year patient safety plan. BMJ Qual Saf 2012;21:760–7.
      OpenUrlAbstract/FREE Full Text
    50. ↵
      1. Baldwin TT,
      2. Ford JK
      . Transfer of training: a review and directions for future research. Pers Psychol 1988;41:63–105.
      OpenUrlCrossRef
    51. ↵
      1. Grossman R,
      2. Salas E
      . The transfer of training: what really matters. Int J Train Dev 2011;15:103–20.
      OpenUrlCrossRef
    52. ↵
      1. Zhou J,
      2. George JM
      . When job dissatisfaction leads to creativity: encouraging the expression of voice. Acad Manage J 2001;44:682–96.
      OpenUrlAbstract/FREE Full Text
    53. ↵
      1. Sarac C,
      2. Flin R,
      3. Mearns K,
      4. et al
      . Hospital survey on patient safety culture: psychometric analysis on a Scottish sample. BMJ Qual Saf 2011;20:842–8.
      OpenUrlAbstract/FREE Full Text
    54. ↵
      1. Edmondson AC
      . Teaming: how organizations learn, innovate, and compete in the knowledge economy. San Francisco: John Wiley & Sons, 2012.

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • Contributors KJJ and AMS contributed to the conception, design, analysis and interpretation of the data as well as drafting and revising the article. RRP contributed to the interpretation of the data, and to the drafting and revising of the article. RH contributed to the analysis and interpretation of the data as well as drafting and revising the article. All authors approved of the final revisions.

    • Funding This research was funded by the Agency for Healthcare Research and Quality (1 U18 HS015822) and the Nebraska Department of Health and Human Services.

    • Competing interests None.

    • Ethics approval This study was approved by the University of Nebraska Medical Center Institutional Review Board.

    • Provenance and peer review Not commissioned; externally peer reviewed.