Article Text
PDF
Abstract
Background Safety culture has been identified as having a major impact on how safety is managed in healthcare. However, it has not received much attention in general practices. Hence, no instrument yet exists to assess safety climate—the measurable artefact of safety culture—in this setting. This study aims to evaluate psychometric properties of a newly developed safety climate questionnaire for use in German general practices.
Methods The existing Safety Attitudes Questionnaire, Ambulatory Version, was considerably modified and enhanced in order to be applicable in general practice. After pilot tests and its application in a random sample of 400 German practices, a first psychometric analysis led to modifications in several items. A further psychometric analysis was conducted with an additional sample of 60 practices and a response rate of 97.08%. Exploratory factor analysis with orthogonal varimax rotation was carried out and the internal consistency of the identified factors was calculated.
Results Nine factors emerged, representing a wide range of dimensions associated with safety culture: teamwork climate, error management, safety of clinical processes, perception of causes of errors, job satisfaction, safety of office structure, receptiveness to healthcare assistants and patients, staff perception of management, and quality and safety of medical care. Internal consistency of factors is moderate to good.
Conclusions This study demonstrates the development of a patient safety climate instrument. The questionnaire displays established features of safety climate and additionally contains features that might be specific to small-scale general practices.
- Safety culture
- patient safety
- family medicine
- questionnaires
- psychometrics
- family medicine
- patient safety
- safety culture
Statistics from Altmetric.com
- Safety culture
- patient safety
- family medicine
- questionnaires
- psychometrics
- family medicine
- patient safety
- safety culture
Introduction
Throughout the world, most patients receive healthcare in primary care. However, patient safety in this healthcare setting has not yet received as much attention as in secondary care. The frequency of critical incidents in primary care is not well studied and is a subject of debate.1 Some aspects of patient safety in primary care and in general practice in particular appear to have been studied in some depth, for example, errors in medication2 and in the follow-up of test results.3 Compared with these subjects, however, patient safety culture has been of lesser concern in this setting.
Patient safety culture is ‘an integrated pattern of individual and organisational behaviour, based upon shared beliefs and values that continuously seeks to minimise patient harm, which may result from the process of care delivery’.4 Adapted from a model of organisational culture by Edgar Schein, safety culture consists of a three-layer model5:
Deep-rooted core assumptions that are not accessible for measurement
Values and beliefs regarding safety
Artefacts as the procedures and actions that are the most superficial expression of culture.
According to this model, core assumptions determine values and beliefs and these determine the artefacts.
Support for a culture of safety is one of the main recommendations in the Institute of Medicine report ‘To err is human’.6 Safety culture has been identified as having a significant impact on how safety is implemented in healthcare as well as in other industries.7 There is growing evidence of a link between safety culture and safety.8–10 The evaluation of safety culture in healthcare is therefore a necessary step in the process of improving patient safety, and safety culture instruments increasingly become important diagnostic tools when trying to assess patient safety. According to a recent survey, one-third of the acute and primary care trusts in the English National Health Service currently use a culture assessment instrument.11
Safety climate is often referred to as the superficial and temporal appearance of a culture that describes what is happening, whereas culture is rooted in history and explains why things happen.12 Zohar described safety climate as the ‘shared employee perceptions of the priority of safety at their unit and organisation at large’.13
A frequently used method to measure safety climate is the application of self-administered questionnaires to professionals14–16 in order to examine their perception of the safety culture of the respective healthcare organisations they work in. Only few instruments with moderate to sound psychometric properties have been developed or used in a non-secondary setting or general practice.17–22
The German system of general practice care is different from that in countries like the UK or USA since most practices have only one or two doctors. Doctors act as owners or partners and usually employ two to four professionals, in most cases healthcare assistants, resulting in a team size of three to eight individuals. Healthcare assistants are mainly responsible for administrative and receptionist's tasks, but are also responsible for medical procedures such as measuring blood pressure or phlebotomies.23 No safety culture instrument exists for use in this system.
We therefore developed an instrument to measure patient safety climate24 and pilot tested it on a large sample of healthcare professionals in general practice in Germany. The purpose of this study was to explore the factor structure of this questionnaire and to assess the internal consistency of the factors.
Methods
Selection of an instrument and adaptation process
In developing a questionnaire there are at least three different ways to proceed: adoption, adaptation or new development. If there is no appropriate instrument to adopt or adapt, then the development of a totally new instrument is recommended.25
In 2007, various safety climate instruments had already been developed for use in other healthcare settings. We checked their content and quality and decided to adapt the Safety Attitudes Questionnaire (SAQ) because of its psychometric properties, its validation in ambulatory care17 and because its content seemed to be adaptable to the context of small general practices. The SAQ-Ambulatory Version (SAQ-A) contains 62 items of which 30 collapse into six factors: teamwork climate, safety climate, perceptions of management, job satisfaction, working conditions and stress recognition.
Translation of the SAQ-A was carried out according to the recommendations for cross-cultural translation and the adaptation of socioscientific surveys.26 Thereafter, we carried out interviews with experts in quality improvement in general practice in order to evaluate whether the questionnaire covers all aspects of safety culture in German general practice. We interviewed eight experts for quality improvement in ambulatory care/general practice from Germany. They consisted of three healthcare assistants and five doctors. Interviews were carried out via telephone. Mainly, we requested feedback on the content of SAQ-A and what they regarded as relevant for patient safety culture in general practice. They held the view that the SAQ-A contained relevant aspects with respect to safety culture and practice organisation. However, they identified the absence of aspects like the involvement of patients. Based on these interviews, the questionnaire was revised by modifying or deleting items and adding new ones. By means of cognitive interviews, the questionnaire was then tested on 10 primary care physicians and healthcare assistants. The findings of these interviews led to further refinements of the 68-item instrument, which was then pilot tested in a random sample of general practices in the German federal state of Hesse in 2008. A sample size of at least three times the number of items was calculated.27 This sample constituted of 400 randomly selected practices. Four hundred and fifty-three individuals participated in the initial sample (38% response rate). The detailed process of development has been published elsewhere.24
After an initial psychometric analysis of the pilot test, two items were removed, 19 items reworded and the order of items modified. Filter items were added and the questionnaire was again tested on a different sample in autumn 2009 (see the section ‘Data collection’ for details). The questionnaire consisted of 72 items. This paper reports on the method and results of this second phase of testing of the final version of the questionnaire, named Frankfurt Patient Safety Climate Questionnaire for General Practices (Frankfurter Fragebogen zum Sicherheitsklima in Hausarztpraxen, FraSiK).
Data collection
The sample consisted of general practices in the Rhine-Main region of Hesse that had participated in a randomised controlled trial on the effect of team reflection on safety culture in general practice. Recruitment for this study was carried out by inviting all 1629 registered general practices in the region of interest (mandatory registration). Eligibility criteria were the use of electronic health records and a team size of at least three healthcare professionals. The necessary sample size of the trial was estimated to be 60 practices so that it would have the power to demonstrate an effect on measures of error management. Between August and December 2009, we employed FraSiK to measure safety climate at baseline by means of a self-administered anonymous questionnaire for all practice team members. When visiting the practice, questionnaires were handed to one member of the practice team who handed them to the rest of the team. Two to six weeks later, questionnaires were collected on site by members of the project team during another visit for baseline data collection. If questionnaires were missing, practice teams were reminded once. Practice teams were offered a financial incentive to compensate for the time spent on data collection and the study intervention. The psychometrics presented here are based on data from the baseline measurement.
Data analysis
Data entry occurred automatically by means of a document reader. One negatively worded item (no. 72) was reverse coded for analysis.
Responses at an item level were analysed by assessing distribution (skewness and excess with recommended values of lower than 2 and 7, respectively).28 Item difficulty was computed by relating the number of ‘correct’ answers (achieving five points on the scale) to an item to the number of all eligible answers to that item ×100 (with recommended values between 0.2 and 0.8).28 The appropriateness of the data for an explorative factor analysis was assessed by applying the Kaiser–Meyer–Olkin (KMO) criterion and Bartlett test. Items were disregarded if more than 10% of values were missing. As this was an exploratory analysis, we refrained from any imputation of missing values. In addition, items that were inappropriate to the whole sample (eg, those following a filter item or to be answered only by a subsample) were not included in the factor analysis but analysed separately. Principal component analysis was carried out as follows: first, the number of factors to be extracted according to the KMO criterion and scree test had to be determined. Then, factor extraction using an orthogonal varimax rotation—with the listwise deletion of items—was conducted. Our aim was to obtain a simple factor structure; thus, we conducted several factor analyses and deleted items with factor loadings of less than 0.40.29 After interpreting the factor loadings, the identified final components were determined and labelled. The internal consistency of these factors was calculated by means of Cronbach's α. Internal consistency of at least 0.7 was regarded as the minimal achievable, and above 0.8 the recommended level.30 Discriminant validity was evaluated by examining the correlation of factors using Pearson's correlation coefficient, with a value of above 0.85 seen as indicating overlapping concepts.31 The analyses were conducted using IBM SPSS Statistics 19. Data with 5-point Likert scales were treated as interval data.
Results
As the registration list presents only names and addresses, we checked the inclusion criteria only for those practices who contacted the project team by return fax. We stopped recruiting when 60 practices had enrolled, even though more than 100 practices were interested in participating. We received questionnaires from 332 (97%) of the 342 healthcare professionals working in the 60 practices. The respondents' demographic data are displayed in table 1.
Basic demographic characteristics of the study sample
The final version of the questionnaire consisted of a total of 72 items: 60 items were scaled using a 5-point Likert scale (from 1=strongly disagree to 5=strongly agree or from 1=never to 5=always); nine items employed a dichotomous scale, some of them were used as filter items, and three items had to be answered in free text. Since 14 of the Likert-scale items were directed to a subsample they were not included in the factor analysis (see figure 1).
Flowchart showing FraSiK items with respect to types of scale.
Results of analysis
Item analysis
Item means ranged from 1.68 to 4.85 and SD from 0.461 to 1.494. One item had missing values of more than 10% (mean 1.9%; for all items, see online appendix table AI). Interestingly, the number of missing values was considerably higher for items related to the staff perception of management. For most items we found a low to moderate skewness of less than −1.0 (data not shown), for nine items we calculated a difficulty index of less than 20% or more than 80%.
Exploratory factor analysis
Forty-six items were appropriate for factor analysis. The Bartlett test of sphericity proved to be highly significant and the KMO criterion (0.88) showed the data to be meritorious32 and thus suitable for factor analysis. Communalities ranged from 0.249 (item no. 51) to 0.715 (item no. 12) with the majority between 0.4 and 0.6. As a result of the scree test, seven factors had to be extracted (data not shown). Of the 46 items, seven further items had to be excluded because of low (<0.4) or of equivocal factor loadings.
After considering the scree plot and interpreting the factors with regard to item content, a preliminary seven-factor structure was found that explained 49.7% of total variance. Factors showed mostly moderate to good internal consistency (for details see table 2).
Result of factor analysis: distribution of factor loadings (restricted to loadings >0.3), interpretation of factors and result of internal consistency for each dimension (Cronbach's α)
The items covering staff perception of management were analysed separately (KMO test 0.755; Bartlett test highly significant). All five items comprised one factor explaining 51.4% of total variance (data not shown) with an internal consistency of α=0.753. The sample for this analysis was reduced to the 250 responses made by practice employees.
The three items directed at doctors only (subsample of 89 individuals) were analysed accordingly (KMO test 0.57). They comprised one factor with an internal consistency of α=0.563.
In the end, seven factors were found that reflected the patient safety climate as perceived by the whole team, as well as two separate factors that addressed the two relevant groups of healthcare professionals in this setting. Of the items eligible for factor analysis, seven items were not included in any of these nine factors. These were items on instructions about new procedures (no. 7), critical appraisal of quality of work (no. 16), work pressure (no. 17), adherence to hygiene guidelines (no. 32), the team's responsibility for safety (no. 44), disclosure of adverse events to patients (no. 51) and the relevance of equipment causing errors (no. 58) (see table AI, items that are not included in the factor structure are displayed in italics).
Mostly low to moderate correlation coefficients (between 0.24 and 0.52) demonstrated moderate to good discriminant validity with the exception of the factor perception of the causes of errors that was negatively correlated with all other factors (data not shown).
Discussion
This study reports on the structure and psychometric properties of the Frankfurt Patient Safety Climate Questionnaire for General Practices, FraSiK. For validation purposes it was used in the German healthcare system. Overall, the instrument has a seven-factor structure. However, two factors were analysed separately due to the inclusion of different perspectives in the measurement of safety climate: those of managing doctors and those of healthcare assistants. All nine factors represent a wide range of dimensions associated with patient safety in general practice: teamwork climate, error management, job satisfaction, perceptiveness to healthcare assistants and patients, perception of causes of errors, safety of office processes, safety of office structure, staff perception of management, and quality and safety of medical care. The internal consistency of the factors is predominantly moderate to good, and correlation between factors shows moderate to good discriminant validity.
This measurement instrument is specifically suited to healthcare settings that are similar to the German setting in the following way: very small teams of rarely more than 10 individuals (doctors and healthcare assistants) work together, doctors are simultaneously front line staff and employers. In Germany, healthcare assistants have less medical training and authority than nurses. Therefore, some of the items specifically address the perception of employed staff towards management, and some other items the perception of doctors towards quality and safety of medical care. In order to account for the different staff perspectives (employees, doctors) the respective items were analysed separately and indeed emerged as contingent factors in this sample.
Admittedly, the dimension of safety climate that has been identified in literature as being the most important—the perception of management attitudes in relation to safety33—was analysed separately. Future analyses should answer the question whether different survey instruments are needed for each type of healthcare professional. Evidence exists that the management perspective is different from that of front line staff.34 Differing perceptions of safety issues probably also led to the adaptation of the German version of the HSOPSC for use with hospital managers, in particular.35
Comparison with other safety climate instruments
Recent reviews have identified common features among safety culture instruments.14–16 36 37 In a systematic review the most frequently cited dimensions were leadership, commitment to safety, open communication founded on trust, organisational learning, a non-punitive approach to adverse event reporting and analysis, teamwork and the shared belief in the importance of safety.37 Apart from the latter, each of these is also represented in FraSiK.
In addition, we identified another dimension that might be specific to small practices—we named it receptiveness to healthcare assistants and patients.
Why is this specific? Traditionally, in German ambulatory practices the doctor is perceived as having a higher rank than healthcare assistants, and paternalistic attitudes towards patients are still sometimes present in the medical profession. Until recently, healthcare assistants and patients did not have a great impact on medical services at all. Now, the involvement of patients is becoming more and more important in order to improve patient safety in the ambulatory setting where patients are much more autonomous than patients in hospitals. Against this background, the factor reflects the fact that the openness of doctors to suggestions made by patients and healthcare assistants may be conceived to be part of an open climate of safety.
Workload in FraSiK was represented by only one item and did not lead to a separate factor in our analysis but is represented by other instruments developed for use in ambulatory care (SAQ-A17 and the Medical Office Survey on Patient Safety Culture18) and primary care (PC-SafeQuest).21 Furthermore, these instruments have been employed and analysed in relation to considerably bigger teams. The application of the Medical Office Survey on Patient Safety Culture is recommended for practices with at least three medical professionals, for the other instruments there are no known recommendations.17 18
Considering all the aspects of safety climate questionnaires described so far, FraSiK contains established features to which the dimension of perceptiveness to healthcare and patients was added.
Most safety culture surveys present a staff perspective.38 In our experience team members in small-sized practices often do not see themselves as staff since they are more actively involved in management and willing to identify with the practice. They focus more on ‘us’ than on ‘them’, which means items have to be worded differently. For example, nobody thinks of doctors as ‘senior management’, because no line management exists. The item ‘I know the proper channels to direct questions regarding patient safety in this office’. (item from SAQ-A) reflects task responsibilities in larger organisations but not the multi-tasking teams of three to six healthcare professionals who are personally responsible for a wide range of procedures. Therefore, this item had to be removed. In comparison with the SAQ-A, only 17 of 30 items and two of six dimensions (team climate and job satisfaction) remained. Although, we started out with the SAQ-A, the different interpretation and content of factors show FraSiK to be a newly developed questionnaire.
In addition to the wording of items, results may differ when using safety climate questionnaires in small-scale practices. In a study evaluating culture in group practices the size of the practice team impacted perceptions of the practice culture in that the bigger the team the less collegial and cohesive the culture was perceived to be.39
Patient safety outcome ratings
We did not include subjective patient safety outcome ratings in the questionnaire. On the basis of our experience during cognitive testing, we were worried about considerable ceiling effects and therefore decided not to include such items as general patient safety ratings. We preferred to include items that ask respondents to reflect on the safety of relevant procedures in patient care (eg, items 31 and 33) and the contribution of factors leading to adverse events (eg, items 56–59).
Perception of causes of adverse events and the role of stress
Interestingly, the scale perception of causes of errors is not correlated with other factors in the questionnaire. In contrast to the other items, this factor evaluates more individual aspects (‘Wenn ich müde bin, mache ich eher Fehler=When I am tired, I am more prone to make errors’.) than aspects relating to the practice or practice team. Different semantics might contribute to a different pattern of answers and thus account for the low correlation.40 In a hospital setting, the SAQ-A factor stress recognition (three of its items are part of the six-item perception of causes of errors in our instrument) showed a similar pattern of correlation.41 Whether the perception of causes of adverse events (with stress being one of various causes) is a component of safety climate needs to be demonstrated in future studies.
Limitations
Although it seems obvious that the dimensions of any safety climate concept are correlated, we decided to analyse the data using an orthogonal rotation technique. First, the different dimensions covering the concept of safety climate represent a wide range of subconcepts that may not be interrelated to any great degree, for example, teamwork climate and safety systems. Second, if the aim of factor analysis is to identify a simple factor structure then orthogonal rotation is the recommended method because it does not allow for correlation between factors.28 However, we undertook oblique rotations too and could not find relevant differences in factor structures (data not shown).
The initial version of FraSiK has been applied to a random sample of general practices. In order to evaluate the modified final version we applied it to a non-random sample of practices (as practices had actively decided to take part in a patient safety intervention study beforehand). Due to this approach we achieved an extremely high response rate. However, our sample possibly represents general practice teams interested in patient safety. Thus, it should be kept in mind that the overall results regarding safety climate may be higher than in a random sample.
Future application of the instrument
FraSiK is the first instrument for the measurement of patient safety climate in German general practice—the setting in which most patients in Germany are cared for. In a first step, it was developed and pilot tested in a random sample of general practices, and in a second step in a sample of selected practices interested in the improvement of safety culture. Both sample sizes (451 and 332 participants respectively) were sufficiently large for psychometric analysis though not for concurrent hypothesis generation and testing. Certainly, the overall validity of the instrument regarding construct validity and responsiveness requires further analysis in the future.
The question still remains whether safety climate questionnaires are suitable for benchmarking and for capturing deeper aspects of culture. Some argue that other qualitative methods are needed in order to add further information on the safety culture of the respective organisation.42 43 The value of FraSiK as an instrument to evaluate interventions aimed at improving patient safety in this setting will be determined in the above-mentioned ongoing study on improving safety culture (ID of trial registration DRKS 00000145).
Future prospects
When selecting an instrument the following factors should be considered: the healthcare setting, the dimensions of safety climate to be assessed, the healthcare professionals to be addressed and the quality of the tool. A broad range of culture areas should be included if a more diagnostic tool aimed at identifying areas for improvement is to be chosen.38 In FraSiK, an instrument with moderate to sound internal consistency is now available in Germany. Nevertheless, its overall validity and responsiveness still need to be determined.
Acknowledgments
We thank Natalja Menold and Jürgen Hoffmeyer-Zlotnik from the GESIS-Leibniz-Institute for Social Sciences, who assisted with the development of the questionnaire and data analysis, and our colleague Juliana Petersen who made valuable contributions in the preparation of the manuscript. We are grateful to Phillip Elliott who helped enormously with the writing of the manuscript.
References
Supplementary materials
Web Only Data
Files in this Data Supplement:
Footnotes
Funding A grant provided by the Federal Ministry of Education and Research (project no. 01GK0702) has enabled this project.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.