Article Text

Effects of a team-based assessment and intervention on patient safety culture in general practice: an open randomised controlled trial
  1. B Hoffmann1,
  2. V Müller1,
  3. J Rochon2,
  4. M Gondan2,
  5. B Müller1,
  6. Z Albay1,
  7. K Weppler1,
  8. M Leifermann1,
  9. C Mießner1,
  10. C Güthlin1,
  11. D Parker3,
  12. G Hofinger4,
  13. F M Gerlach1
  1. 1Institute of General Practice, Goethe University Frankfurt am Main, Frankfurt, Germany
  2. 2Institute of Medical Biometry and Informatics, University of Heidelberg, Heidelberg, Germany
  3. 3Safety Culture Associates Limited, Leeds, UK
  4. 4Team HF Hofinger Forschung Beratung Training, Remseck, Germany
  1. Correspondence to Dr Barbara Hoffmann, Institute of General Practice, Goethe University, Theodor-Stern-Kai 7, Frankfurt am Main D-60590, Germany; Hoffmann{at}allgemeinmedizin.uni-frankfurt.de

Abstract

Background The measurement of safety culture in healthcare is generally regarded as a first step towards improvement. Based on a self-assessment of safety culture, the Frankfurt Patient Safety Matrix (FraTrix) aims to enable healthcare teams to improve safety culture in their organisations. In this study we assessed the effects of FraTrix on safety culture in general practice.

Methods We conducted an open randomised controlled trial in 60 general practices. FraTrix was applied over a period of 9 months during three facilitated team sessions in intervention practices. At baseline and after 12 months, scores were allocated for safety culture as expressed in practice structure and processes (indicators), in safety climate and in patient safety incident reporting. The primary outcome was the indicator error management.

Results During the team sessions, practice teams reflected on their safety culture and decided on about 10 actions per practice to improve it. After 12 months, no significant differences were found between intervention and control groups in terms of error management (competing probability=0.48, 95% CI 0.34 to 0.63, p=0.823), 11 further patient safety culture indicators and safety climate scales. Intervention practices showed better reporting of patient safety incidents, reflected in a higher number of incident reports (mean (SD) 4.85 (4.94) vs 3.10 (5.42), p=0.045) and incident reports of higher quality (scoring 2.27 (1.93) vs 1.49 (1.67), p=0.038) than control practices.

Conclusions Applied as a team-based instrument to assess safety culture, FraTrix did not lead to measurable improvements in error management. Comparable studies with more positive results had less robust study designs. In future research, validated combined methods to measure safety culture will be required. In addition, more attention should be paid to evaluation of process parameters. Implemented actions and incident reporting may be more appropriate target endpoints.

Trial registration German Clinical Trials Register (Deutsches Register Klinischer Studien, DRKS) No. DRKS00000145

Statistics from Altmetric.com

Introduction

Safety culture has an impact on how organisations deal with medical errors and how patient safety measures are implemented in daily practice.1 Safety culture is commonly defined as ‘the product of individual and group values, attitudes, competencies, and patterns of behaviour that determine the commitment to and the style and proficiency of an organisation's health and safety programme’.2

There is an ongoing debate on how to improve an organisation's safety culture and safety performance.3 Assessments are carried out via self-administered questionnaires that measure healthcare workers’ perceptions.1 Self-assessment and interventions to improve safety culture are combined in an approach based on a typology of organisational culture.4 This method is embodied in the Manchester Patient Safety Framework (MaPSaF).5 MaPSaF is a matrix of nine dimensions which can be assigned to five different levels of culture. It focuses mainly on the identification and investigation of patient safety incidents and on learning from them, while other dimensions address patient safety as reflected in, for example, communication and teamwork. Primarily, MaPSaF encourages reflection on the safety culture of one's practice team6 and the development of interventions to change its culture.5 MaPSaF was originally developed to be used in primary care in the UK and was later adapted for use in different sectors7 and countries.8 ,9 In a recent survey, MaPSaF has been identified as the most frequently used safety culture instrument in the English National Health Service in acute care trusts as well as in primary care.10

It is argued that safety culture is affected by organisational changes such as leadership or newly introduced systems and processes.11 In order to improve safety culture, other groups have assessed safety culture in healthcare organisations and followed up the evaluation with an intervention comparable to MaPSaF. This has had promising effects on safety culture in secondary care.12 ,13 However, these studies applied pre-post designs without randomly selected control groups. No research yet exists on the effects of MaPSaF. In a pilot study we translated and adapted the primary care version of MaPSaF into German (the Frankfurt Patient Safety Matrix, FraTrix), and results from the pilot study showed high acceptability and applicability of the instrument. In addition, after applying the matrix in one team session, 15 of the 19 practice teams in the pilot study decided on actions to improve processes affecting patient safety,14 and hence encouraged us to conduct a proper intervention study that looks at outcomes relevant to safety processes and culture.

We hypothesised that, by encouraging teams to think about their safety culture, FraTrix would lead to the implementation of specific measures to improve patient safety and safety culture. We thus carried out a randomised controlled trial of FraTrix to investigate whether it can improve safety culture —that is, whether the teams of general practices improved (1) their error management and other structures and processes supporting patient safety; (2) their general safety climate; and (3) their patient safety incident reporting.

Methods

Design and participants

We conducted an open randomised controlled parallel-group rater-blinded trial in general practices in the Hesse region of Germany. As FraTrix targets team processes, a minimum team size of three healthcare professionals was required.

Recruitment took place between May and September 2009. All registered general practices (statutory health insurance physicians specialised either in family medicine or general internal medicine) in southern Hesse were invited to participate. Practices that demonstrated interest were contacted by telephone to check eligibility and were given further details on the trial. After obtaining consent from the entire practice team (physicians and healthcare assistants (HCA)), practices were randomly assigned to intervention and control groups. Randomisation was stratified by practice model (single-handed or group practice). The course of the recruitment procedure and the study is shown in figure 1.

Figure 1

CONSORT flow diagram.

Practice teams participated voluntarily. As patients did not take part in the trial and patient data (when used) were anonymised, the institutional review board of the Medical Faculty of the Goethe University Frankfurt waived further ethical review.

Intervention and control treatment

The intervention began with an introductory team session during which the course of the intervention was described and instructions on the use of FraTrix provided. This first session was followed by three facilitated team sessions that used FraTrix. In the control group, we offered every practice team member the opportunity to attend a short seminar on patient safety and error management at the Institute of General Practice in Frankfurt. Both types of intervention are described in more detail in box 1.

Box 1

Detailed description of the interventions

Intervention group The Frankfurt Patient Safety Matrix consists of two elements:

  1. Team sessions under the guidance of an external facilitator using

  2. A matrix of five safety culture grades (from ‘generative’ as the most mature, through ‘proactive’, ‘bureaucratic’ and ‘reactive’ to ‘dismissive’ as the least mature grade) and safety dimensions.

Team members learned about the matrix during the introductory team session and by reading the FraTrix brochure which includes instructions for the use of FraTrix and a detailed description of the matrix. The nine dimensions were

  • Overall commitment to quality

  • Priority given to patient safety

  • Perception of critical incidents and their causes

  • Analysis of critical incidents

  • Learning from critical incidents

  • Communication as it relates to patient safety

  • Personnel management as it relates to patient safety

  • Staff education and training as they relate to patient safety

  • Team work as it relates to patient safety.

No later than 4 weeks after the introductory meeting, the first FraTrix team session was scheduled and followed by second and third sessions after 3 and 6 months, respectively. Follow-up sessions were always attended by the same facilitator. Team sessions generally took place during lunch breaks or after work.

Each team session lasted between 60 and 90 min. Before each team session, every team member worked through the brochure independently of his or her colleagues and without discussing any of the issues. At the beginning of the team session, all individual assessments of each dimension were reported and the results were marked on a chart (see online supplementary appendix). Thus, every team member's assessment was seen by the whole team. The practice team members then decided on the three dimensions they wanted to deal with first. In the following discussion, the team members reflected on the strengths and weaknesses of their safety culture with regard to the chosen three dimensions. The facilitator supported the team to work with the brochure, guided the discussion along the matrix and helped to achieve an action plan by the end of the team session.

The facilitator subsequently recorded the results of the team assessment and the agreed action plan. Furthermore, the team received a form on which to keep records of the planned actions (ie, improvement protocol) for further use. At the beginning of the second and the third sessions, the facilitator asked the teams whether they had achieved the aims of the previous action plans. The remaining dimensions were topics for discussion during team sessions 2 and 3. After 12 months, improvement protocols were copied for data collection during a final practice visit.

To ensure the facilitation was standardised and of high quality, both facilitators agreed on a facilitation guide. Every third team session was audiotaped and the recordings were used by the other facilitator to ensure that the facilitation guide had been adhered to. Both facilitators were well trained and experienced in facilitating FraTrix sessions as a result of their participation in the pilot study14 (BH, VM). BH is both a physician and a nurse, as well as a health scientist with substantial expertise in patient safety. VM is a healthcare assistant, a study nurse and a family counsellor.

Control group

Since we considered the attention given to the FraTrix group as an intervention, we offered every practice team member in the control group the opportunity to attend a short seminar on patient safety and error management at the Institute in Frankfurt. During a lecture of 90 min, the participants were instructed in the basics of patient safety and the main elements of error management. In addition, control practices were contacted twice by phone about 5 and between 8 and 9 months after the baseline assessment. The calls were concerned with issues of error management and gave the practice teams the chance to ask questions about patient safety.

Measurements

Patient safety culture

In safety culture research, a combination of different measures is recommended.15 We therefore assessed safety culture on different levels and by different mean, using the three-layer model developed by Schein16 (artefacts, espoused values, core assumptions; see figure 2).

Figure 2

Model of safety culture measurement based on the three-layer model of organisational culture.

Patient safety culture indicators

On the one hand we adapted 12 established quality indicators for general practice17 ,18 that reflect structures and processes relating to patient safety (for more details see online supplementary appendix 2, part 3) in order to investigate ‘artefacts’ or behaviours of the practice team as a whole—what the team does and how with regard to patient safety.

Patient safety incident reporting

On the other hand, the measurement of ‘artefacts’ was complemented by the evaluation of patient safety incident reporting in the practice. For this purpose we assessed the number and quality of incident reports. We considered incident reports valid if they reported on patient safety incidents on the basis of a definition adapted from the National Patient Safety Agency.19 We assessed their quality according to a method adapted from an instrument developed for the assessment of Significant Event Analysis in general practice20 (for details see online supplementary appendix 2, part 4).

Patient safety climate

Safety climate reflects staff perceptions of the priority of safety.2 To evaluate these ‘espoused values’ and attitudes of the team, we measured safety climate with the Frankfurt Patient Safety Climate Questionnaire for General Practices (FraSiK), as developed and validated by our research group.21 The instrument consists of 44 items with moderate to good internal consistency that circumscribe nine scales. One scale is addressed only to doctors (‘quality and safety of medical processes’) and one only to employees (‘staff perception of management’). Responses to items are given on a 5-point Likert scale ranging from 1 (‘strongly disagree’ or ‘never’) to 5 (‘strongly agree’ or ‘always’).

Primary outcome

The primary outcome was assessed 12 months after randomisation. We chose the patient safety culture indicators (PSCI) error management as the primary outcome because the management of patient safety incidents is a core aspect in patient safety and of FraTrix: it is covered by three of the nine dimensions—namely, ‘perception of critical incidents and their causes’, ‘analysis of critical incidents’ and ‘learning from critical incidents’. The indicator is derived from the European Practice Assessment (EPA) indicator system, has been found to be valid and reliable18 and consists of six criteria that were assessed with a total of 13 self-reported items (table 2).

Secondary outcomes

Secondary outcomes were 11 additional indicators of patient safety culture, data on patient safety climate (FraSik scale scores) and the volume and quality of incident reporting (see online supplementary appendix).

Course of the intervention

We assessed whether the intervention took place as planned. Facilitators made records of the team sessions (duration and participants in the session, individual assessments displayed on the chart, dimensions of FraTrix that were discussed including important aspects of the discussion, and content of the action plan). Furthermore, we assessed the quality of the improvement protocols (see box 1) by evaluating whether the action and its goal were described, which factors contributing towards errors22 were addressed, other dimensions of patient safety practices23 and whether the action was implemented and the goal achieved.

Sample size

The sample size was calculated using the primary endpoint —that is, error management at 12 months. The calculation was based on data from 302 practices from the EPA.18 A total of 45 practices would be needed to detect the proposed effect size for competing probability (CP)22 ,23 of 0.73 with a statistical power of 80% and at a two-sided 5% significance level (for details see online supplementary appendix 2, part 2). We adjusted the total sample size to 60 practices to account for possible withdrawals.

Data management and analysis

Since the indicators have different numbers of criteria, we standardised the scores by dividing the sum of fulfilled criteria by the total number of criteria per indicator. For the FraSiK questionnaire, scale scores on an individual level were calculated as the mean of the scale's items. Safety climate scores on a practice level were calculated as the mean of the respective team members’ scores. The quality of incident reports was independently evaluated by two blinded raters and consensus sought in case of disagreement. The duration of the trial from baseline to final data collection was 12 months. No outcome measure was changed during the trial.

Primary analysis followed the intention-to-treat principle, with baseline observation carried forward for missing values. Continuous data were summarised using means and SD, and categorical data with frequency counts and percentages. The Wilcoxon–Mann–Whitney (WMW) test was used to test for differences in error management between groups at 12 months. The effect of the intervention on the primary outcome is expressed as CP from the WMW test statistic with a corresponding 95% CI.24 ,25 Sensitivity analyses, including a non-parametric repeated measures analysis of variance (ANOVA), were conducted (see online supplementary appendix 2, part 2). All tests were two-sided with a significance level of 0.05 and were not adjusted for multiple comparisons.

Results

One hundred and sixty-eight general practices were assessed for eligibility; 14 practices (8.3%) did not meet the inclusion criteria, 89 teams (53.0%) declined to participate and 65 practice teams (38.7%) agreed to take part in the trial. These practices were randomly allocated to either the intervention or control group. Since a sample size of 60 practices was required, the five practices that last consented to participation were transferred to a waiting list following randomisation. In the end, 60 practices took part in the trial, comprising 28 in the intervention group and 32 in the control group (figure 1).

Participating practices

Almost one-third of practices were group practices, the remaining 41 practices were single-handed (see table 1). Practices were located in small towns (41.7%), mid-sized towns (30.0%) and cities (26.7%). Only one practice was located in a rural community. Compared with non-participating practices, we found no differences in location, practice model, team size and implemented quality management system. In addition, practice characteristics (practice model, location) did not differ from the entire set of general practices in the area. However, participating physicians were, on average, younger (50 years) than general practitioners in the state of Hesse (53 years).26 The average team size was comparable to practices participating in a recent survey on safety culture in general practices in Germany.27

Table 1

Summary of characteristics of practices and healthcare professionals

Three practices were lost to follow-up after baseline data collection that took place between August and November 2009. One intervention practice team withdrew from the trial immediately after baseline data collection because the HCAs viewed the interview on PSCI as intrusive. One control group practice was closed and the final data collection (from August to December 2010) could not be conducted in another due to a lack of cooperation.

Course of the intervention

The initial meeting and three FraTrix team sessions took place in each of the final 27 intervention practices. On average, FraTrix team sessions lasted a mean (SD) of 86.2 (20.7) min and the entire team was present at most sessions (in only 7.2% of sessions was one or more team member absent).

With regard to the safety culture of their practices, healthcare professionals mostly assessed them as ‘proactive’ (the second most mature grade), followed by ‘bureaucratic’ and ‘generative’. The number of ‘dismissive’ or ‘reactive’ assessments was low (see online supplementary appendix). With regard to the aim of the team sessions, only 10 sessions ended without any action plan (12.3%). At the end of the other sessions, 265 actions had been planned (9.8 actions per practice), of which 124 actions (46.8%) were documented and followed up by the team. Of these improvement protocols, 97 actions could be rated with regard to their quality; 90 were rated as appropriate or fairly appropriate for improving patient safety. Most of the actions addressed organisational or environmental factors that may contribute to errors in patient care (93.8%) and/or individual characteristics of the healthcare professionals (48.5%) and only a minority addressed patient and medication or equipment factors. Nearly one-quarter of the actions consisted of one-time structural changes in the practice whereas three-quarters depended on repeated activities.

About 50% of these actions were fully implemented, and the practice teams judged the goal of the action to have been achieved in 32 cases —that is, about one action per intervention practice. Most actions targeted team communication, quality management in general, organisation of practice, education and training, error management and communication with patients.

In the control group, representatives from 16 practices (50%) participated in the seminar on patient safety and error management. The participants consisted of 17 physicians (72.2% of the physicians in the attending practices) and 38 HCAs (66.4%). As planned, every control practice was contacted twice by telephone during the intervention phase.

Primary outcome

At baseline, control group practices scored higher in terms of PSCI error management than intervention group practices (table 2). After 12 months the mean (SD) score had increased in both groups and was 0.68 (0.14) in the control group and 0.66 (0.18) in the intervention group. Thus, no significant difference between the intervention and control groups was observed in the primary outcome (CP=0.48, 95% CI 0.34 to 0.63, p=0.823). A non-parametric repeated measures ANOVA showed that the group-by-time interaction was not statistically significant (p=0.262).

Table 2

Primary outcome patient safety culture indicators (PSCI) error management

Secondary outcomes

Intervention and control practices did not show any statistically significant difference in scores of PSCI at 12 months (see online supplementary appendix 2, part 5). Sensitivity analyses did not reveal significantly different intervention effects over time.

Safety climate

Safety climate at baseline was perceived by all teams to be positive for eight of the nine scales. The majority of scales did not change from baseline to the end of the trial and, again, no significant differences were found between intervention and control groups at 12 months (table 3).

Table 3

Patient safety climate scale scores at baseline and 12 months after randomisation for intervention and control groups

Reporting of patient safety incidents

At baseline, 27 events were reported in 11 practices (from one to eight incidents in the preceding 12 months), and 24 of these were assessed as proper patient safety incidents. The mean number and quality of incident reports did not differ between the intervention and control groups. By the end of the trial, 362 events had been reported in 39 practices (1–26 incidents per practice team), and 225 of these were considered to be patient safety incidents. At the end of the trial, more patient safety incidents (mean (SD) 4.68 (4.94) vs 2.91 (5.29), CP=0.65, p=0.045) had been reported in the intervention group, and the reports made were of significantly better quality (2.19 (1.94) vs 1.40 (1.66), CP=0.65, p=0.038; table 4). Reported patient safety incidents mainly related to office administration, communication, investigations and medication (table 5).

Table 4

Quality scores for incident report data at baseline and 12 months after randomisation for both intervention and control groups

Table 5

Types of patient safety incidents reported by intervention and control practices at baseline (in total n=24) and 12 months after randomisation (in total n=225)

Discussion

This is the first study to investigate the effects of a MaPSaF-like instrument on safety culture in primary healthcare within the framework of a randomised controlled trial. No differences between the intervention and control groups could be shown after 12 months in patient safety culture indicator error management and 11 further indicators and safety climate scales. Nevertheless, better reporting of patient safety incidents was observed in intervention practices, as shown by more reports and reports of higher quality than in the control practices. In the interventions practices, about one action plan per practice was successfully implemented.

Comparison with other research investigating safety culture interventions

Few studies have used a MaPSaF-like approach to improve safety culture in healthcare. One trial used an adaptation of MaPSaF over the course of 5 years. Fifty departments at three hospitals in Sweden repeated three cycles of preparation, meeting, assessment, reporting and improvement. Contrary to our intervention, this trial demonstrated improvements in patient safety outcomes (eg, hospital-acquired infections) and in safety culture assessment measures.12 In another trial conducted in 23 intensive care units in the USA, the results of a survey on safety climate were discussed in the units to identify areas of improvement in central line-associated bloodstream infection rates and ventilator-associated pneumonias.13 Both safety climate scales and clinical outcomes improved. However, both studies had pre-post designs without any randomly selected control groups in place.

Generally, studies evaluating interventions that aim to improve safety culture have shown increases in safety climate scores,29–35 while one study also reported improved observed safety behaviour and performance.36 Unfortunately, these studies either had uncontrolled pre-post designs, small sample sizes of selected participants or were case reports from single healthcare organisations.37 In addition, all studies were conducted in an inpatient setting. In a recent systematic review on strategies for improving patient safety culture in hospitals, only leadership walk rounds and multifaceted unit-based interventions were found to show ‘some stronger evidence’ of a positive impact but, again, these studies generally had a number of methodological limitations.3

Strengths and weaknesses of the intervention and trial

A strength of the intervention is that it takes into account proposed barriers to culture change such as lack of ownership,38 as well as facilitating factors such as workforce empowerment, engagement and support by leadership.39 FraTrix is based on team processes and supports mutual understanding and agreement on perceptions of safety culture and actions for improvement. It enables the teams to develop their own action plans according to their needs, thus empowering all practice staff. As shown, intervention teams decided on plenty of sensible actions as a result of the team sessions. These actions can be perceived as appropriate since they are mostly directed towards systemic change and consist of structural changes to a considerable degree —that is, changes that are generally regarded as being more sustainable than changes directed at the individual healthcare professional or activities that require constant repetition. However, less than half of the measures were followed up by the team and an even smaller portion was fully implemented, resulting in around one successfully implemented action per practice. On the other hand, the implementation of all planned actions could not be expected in view of the teams’ heavy workload.

The intervention, however, may be rather weak as it encourages staff to reflect on safety culture and possible actions plans rather than the implementation of specific practical changes. In spite of their very positive self-assessment, the participating teams were probably in an early phase of the change process. According to the Transtheoretical Model of change (TTM), individuals40 or organisations41 go through stages of change when altering their behaviour (pre-contemplation, contemplation, preparation, action, maintenance). The teams in our study may have been in the phase of pre-contemplation that is characterised by the lack of any real intention of taking action in the near future. Therefore, too much (in terms of real actions) may have been expected of what was basically mere reflection. In order to achieve real progress, accompanying measures to support the teams41 may have been needed. Furthermore, the reflection process within the teams may not have been as open as would have been desirable, as HCAs may have found it difficult to conduct an open discussion. Consequently, the ‘real’ problem may not have been identified. However, as most of the participating practices were single-handed, we decided against a solution involving team sessions solely for HCAs for practical reasons.

Recently, facilitators of quality and safety improvements have demonstrated the importance of safety culture for successful change processes. Reductions in hospital-acquired infections in the Michigan Project were explained, among other factors, as being due to a ‘networked community’, ‘reframing the subject of the intervention as a social problem with a solution’ and the systematic feedback of data on clinical outcomes.42 Furthermore, knowledge about best practices43 and education and training in error reduction techniques were identified as other change-promoting factors.44 The impact of a FraTrix or MaPSaF-like intervention may be strengthened by such educational or organisational amendments.

We conducted a randomised controlled trial with a predefined sample size of 60 practices and a combined method to measure safety culture. To date, no widely accepted and validated method for the measurement of safety culture has existed. Even though there are a significant number of questionnaires assessing safety climate, the supplementary use of richer data is recommended.2 In other high-risk industries, safety culture is measured by means of staff questionnaires that are usually complemented by audits evaluating risk management documents45 and interviews with staff on how they actually conduct their work.46 We therefore followed recent recommendations2 ,15 and combined both methods (incident reporting documents and PSCI) with a recently validated safety climate questionnaire (FraSiK)21 that was developed for German general practice.

A further strength is the low dropout rate. In addition, by choosing the PSCI error management as a primary outcome, we selected a parameter that varied considerably between practices in terms of the data collected at the beginning and the end of the trial (table 2). We would therefore expect this indicator to be sensitive to change. We had expected the small team size to result in a shorter time frame being required for the implementation of changes than in hospital care.

Unfortunately, established instruments for studying interventions concerned with patient safety or safety culture in general practice were not available so we applied quality indicators as outcomes. These are regularly used for auditing but have not been validated for use in controlled trials. It was therefore somewhat unclear whether the indicators were sufficiently sensitive to changes or whether they would register changes induced by the FraTrix intervention. The suitability of error management as the primary outcome also requires some discussion. It is possible that this outcome is too demanding since it requires more patient safety proficiency on behalf of the team than simply planning and implementing actions such as changes in team communication or in the appointment system. In future evaluation studies of MaPSaF/FraTrix-like interventions, outcomes should be chosen that reflect behaviours that are more proximal to everyday care. In our study, the implementation of action plans would be an example of such an outcome.

The method we used to assess incident reports20 is not a recognised measurement tool for research. We could demonstrate an improvement in terms of reporting rate and quality of reports, but this improvement could only be shown on an exploratory level of analysis. However, incident reporting is one of the dimensions of the matrix and the starting point for error management, and can therefore be regarded as another potential outcome for interventions like FraTrix.

One drawback when studying safety culture interventions is that every method for measuring safety culture constitutes an intervention itself.3 ,47 At the end of the trial, participants reported that the interviews during data collection visits helped to identify problematic areas, and that changes had then been initiated without any further intervention. Moreover, discussions during the team sessions may also have had an unexpected impact on the data. The criteria for almost all indicators relied in part on self-reporting (see table 2 and online supplementary appendix 2, part 3), and participants in the intervention group reported that they were much more critical of their practices when again confronted with the topic of data collection 12 months later. Thus, it could be argued that we underestimated a possible positive effect of the intervention. Furthermore, we applied a minor intervention to the control group—although with a much smaller degree of involvement of the team members—that could have had a positive impact on the results, making it even more difficult to detect a difference in outcomes.

In addition, the time frame of the intervention may account for the lack of an intervention effect. As cultural changes require a considerable amount of time to emerge, 1 year may have been too short for a significant change to occur.48 ,49 A short time frame was also thought to partly explain the lack of any greater improvement in another recent controlled trial; measurable effects may have been delayed and are possibly only detectable at a later date than that of the final measurement.50

Although we randomly allocated practices to the intervention and control groups in our trial, participation alone can be interpreted as demonstrating a commitment to quality51 and patient safety. The participating teams in our trial perceived an overall positive safety climate at baseline and left little room for improvement. We cannot rule out that participating teams were already advanced in terms of safety culture. This selection bias, as well as the low participation rate, should be considered when generalising the results of the trial.

Implications for practice and future research

When it comes to cultural change, self-assessment with FraTrix appears to be a good starting point. Self-assessment and team-based reflection alone led to a considerable number of actions aimed at improving safety culture but did not result in a measurable change in structures, organisational processes and climate.

In conclusion, appropriate and realistic study outcomes and measures to support the potential effects of FraTrix may be needed. The latter could include team support by a supervising coach, and education and training in patient safety in order to strengthen the abilities of teams to successfully design and implement safe structures and procedures in primary care. To truly learn about strengths and weaknesses in safety culture, validated quantitative and qualitative methods45 and adequate time frames for interventions are further required for the evaluation of safety culture in primary care.

Acknowledgments

We would like to thank each practice team that participated in our trial and its pilot phase and are particularly grateful for their confidence in our project team. In addition, we thank the AQUA Institute, Goettingen, the National Association of Statutory Health Insurance Physicians (KBV), their AQUIK project, and Dr Susanne Kleudgen, Berlin, for their support in the design of the trial. Furthermore, we thank Phillip Elliott who helped enormously with the writing of the manuscript.

References

View Abstract

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Files in this Data Supplement:

Footnotes

  • Contributors BH led the study design with contributions from VM, GH, FMG, CG, JR and DP. ZA contributed to the development of data collection tools. ZA and BH led the data collection with contributions from BM, CM, ML and KW. BH led the analysis with contributions from JR, MG, BM, CG, KW, GH and DP. All authors contributed to the data interpretation and the revisions to the manuscript. The first draft was prepared by BH. All authors had full access to the data derived from the trial and take full responsibility for the integrity of the data and the accuracy of the data analyses. BH is guarantor for the trial.

  • Funding A grant provided by the Federal Ministry of Education and Research (grant no. 01GK0702) enabled this project. The funder had no role in the study design, collection, analysis and interpretation of data, writing of the report or the decision to submit the article for publication.

  • Competing interests http://www.icmje.org/coi_disclosure.pdf None.

  • Ethics approval Patients did not take part in the trial and, when used, patient data were anonymised. The institutional review board of the Medical Faculty of the Goethe University Frankfurt stated that no review of ethical issues for this quality improvement trial was required.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Previous presentations Preliminary results of this trial were presented at the 2011 annual conference of the German and Austrian Colleges of General Practitioners and Family Physicians in Salzburg, Austria, 22–24 September 2011, as well as at the 2011 International Congress of the Patient Safety Foundation of Switzerland, 29–30 November 2011.

  • Trial protocol The full trial protocol can be provided by the corresponding author.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.