Article Text
Abstract
Aim: To review quantitative studies of safety climate in health care to examine the psychometric properties of the questionnaires designed to measure this construct.
Method: A systematic literature review was undertaken to study sample and questionnaire design characteristics (source, no of items, scale type), construct validity (content validity, factor structure and internal reliability, concurrent validity), within group agreement, and level of analysis.
Results: Twelve studies were examined. There was a lack of explicit theoretical underpinning for most questionnaires and some instruments did not report standard psychometric criteria. Where this information was available, several questionnaires appeared to have limitations.
Conclusions: More consideration should be given to psychometric factors in the design of healthcare safety climate instruments, especially as these are beginning to be used in large scale surveys across healthcare organisations.
- safety climate
- patient safety
- worker safety
- questionnaires
- psychometric factors
Statistics from Altmetric.com
In response to growing concern about patient safety, the Department of Health in the UK1 and the Institute of Medicine in the USA2 advised that healthcare organisations should consider adopting the safety management techniques used in other industries. The UK industrial safety regulator, the Health and Safety Executive, recommends that organisations operating in high risk industries should regularly assess their safety culture.3 Safety culture is “the product of individual and group values, attitudes, perceptions, competencies and patterns of behaviour that determine the commitment to, and the style and proficiency of, an organisation’s safety management” (page ).4 This is usually measured in industry by workforce questionnaire surveys to assess what is called the “safety climate”.5 Safety climate can be regarded as the surface features of the underlying safety culture.6 It assesses workforce perceptions of procedures and behaviours in their work environment that indicate the priority given to safety relative to other organisational goals.7–9 As organisations are inherently hierarchical in structure, there are multiple levels at which safety climate can be investigated—for example, individuals, work groups, departments, organizations.10 Safety climate data are generally collected at the individual level, then aggregated to a higher level. The degree of homogeneity of workers’ perceptions, as a measure of climate strength, can also be considered.11,12
A number of different instruments are used to measure safety climate in industry.6,13 The resulting data offer managers an additional perspective on the state of their safety management systems and can also be used for benchmarking purposes and trends analysis.14 It has been argued15,16 that the essential dimension is management commitment to safety: while this is probably fundamental, industrial researchers do measure other aspects. The most common6,13 are shown in box 1. In industry, workforce perceptions of safety climate have been linked to safety outcomes such as workforce injuries,17–21 and to safety processes such as workers’ behaviours.22
Box 1 Safety climate features in industry and health care
-
Management/supervisors
-
Safety systems
-
Risks
-
Work pressure
-
Competence
-
Procedures/rules
Health care
-
Management/supervisors
-
Safety systems
-
Risk perception
-
Job demands
-
Reporting/speaking up
-
Safety attitudes/behaviours
-
Communication/feedback
-
Teamwork
-
Personal resources (e.g. stress)
-
Organisational factors
Safety climate surveys are now being increasingly used in healthcare organisations23 and several instruments have been developed. This paper reviews quantitative studies designed to investigate safety climate in health care, with particular attention devoted to the questionnaires. It provides a complementary analysis to a recent review of survey instruments,24 with some overlap in the studies examined.
METHODS
Four databases were searched: MEDLINE, PsychINFO, EBSCO, and Web of Science using the search terms “health care workers”, “hospital safety”, “patient safety”, “safety climate”, and “safety culture”. Relevant papers were retrieved and papers were also retrieved from patient safety conferences. A total of 29 studies were initially identified. The criteria for inclusion for detailed scrutiny were: (1) use of a questionnaire for individual response designed to measure safety climate or safety culture in a healthcare setting; (2) details provided of the measuring instrument; (3) tested on a sample of over 50 respondents; and (4) report published in English. From the 29 papers retrieved, 12 studies were identified as suitable for review.25–36 Studies reporting different components of the same data set37–41 were only included once, and those that examined general organisational culture or climate variables (such as work pressure or role ambiguity) in relation to safety in health care42–44 were not included. The analysis extracted information on the survey location and sample, safety climate measure, safety outcome variables, and the main findings. The specific psychometric properties45,46 considered (box 2) were the content validity, criterion validity, as well as the internal factor structure of the instrument.
Box 2 Psychometric criteria
-
Content validity is the degree to which “elements of an assessment instrument are relevant to and representative of the targeted construct for a particular assessment purpose” (page ).47 Determination of whether the scales or item set of a safety climate questionnaire have good content validity can be made from a number of sources, e.g. relevant theory, empirical literature, expert judgement.
-
Criterion related validity should be established by correlating the climate scores with outcome data, preferably collected by some method other than the questionnaire instrument. In the case of safety climate, these can be safety outcomes such as individual or organisational accident rates, or safety processes such as rates of behaviours that are deemed to be precursors of accidents (e.g. risk taking, rule breaking). In practice, industrial safety climate researchers frequently use self-report measures of accident rates or unsafe behaviours collected on the questionnaire. This is not ideal because of common method bias.48 However, there can be difficulties in accessing confidential accident data and, because questionnaires are completed anonymously (making it impossible to identify individual safety records), self-report measures are sometimes the only means by which individual level criterion related validity can be established.
-
Factor analysis reveals the underlying structure of a scale and shows whether there are distinct factors or themes being measured. It requires reasonably large data sets (of about 100) or a sample where there is a 10:1 ratio of participants to items.46,49 This ratio becomes less relevant for sample sizes above 300.50 Factors with three items or fewer are too close to being variable specific and should be discarded.51,52 The internal reliability data for proposed/identified factors can also be assessed. A Cronbach’s alpha score of 0.7 or higher is usually regarded as indicative of acceptable internal reliability.53
The safety climate factors/dimensions given in each study (table 1) were independently categorised (with acceptable inter-rater agreement) by two industrial psychologists (CB, SY) into 10 themes corresponding to distinct aspects of safety management. This was carried out by examining the content of the items loading on each subscale/factor where these were available. As most of the studies had based their definitions of safety climate on the literature from industry or had adapted industrial instruments, the themes were labelled in a similar fashion to those most commonly measured in industry (box 1).
RESULTS
The 12 studies are described in table 1 in terms of the survey sample, instrument details (with any psychometric data), identified safety climate factors, outcome measures, and results. Most of the studies were from the USA and most sampled medical staff in different occupations (response rates 26–91%). The first nine studies used different safety climate measures while the other three used different versions of the Operating Room Management Attitudes Questionnaire (ORMAQ).54 The ORMAQ was not originally designated as a general healthcare safety climate measure but recently it has been used for this purpose, so these studies were included for review. (Studies with the ORMAQ that did not specifically claim to be measuring organisational safety climate55,56 were excluded.)
Drawing from the information presented in table 1, the questionnaires were examined with respect to content validity, factor structure and internal reliability, and criterion related validity. The level of analysis used in the study is briefly considered below.
Content validity
Nine of the studies set out to measure safety climate, the remainder used the term safety culture, and one30 used both terms. Four did not define either term. Definitions of safety climate were usually a version of shared perceptions of safety.10 A theoretical basis or model to outline proposed causal influences between safety climate and the safety outcome measures was rarely specified. One exception32 stated that their survey items were based on “elements of a culture of safety articulated by high reliability theory” (page ). They listed seven components derived from this theory58 but it is not clear how these relate to their questionnaire items and the five extracted factors did not provide confirmation. A circular model was given in another study26 but it did not articulate any explanatory mechanism between safety climate and safety behaviours.
The 73 safety climate dimensions (table 1) were categorised into 10 safety management themes (box 1). Not surprisingly, given the origins of several of the measures, there was considerable overlap with the features measured in industry. Management commitment to safety emerged as the most frequently measured safety climate dimension in health care with nine studies including this. Three included supervisor commitment to safety. Safety systems—for example, personal protective equipment and safety training—were included in seven studies. Unlike the industrial sector, general attitudes to risk were not specifically addressed.
Work pressure is an important safety climate feature in industry and three healthcare instruments included a job demands/workload dimension. Unlike the industrial sector, competence did not emerge as a separate dimension, although two studies included measures about training and one included knowledge about universal precautions. Lastly, and in contrast to industry, compliance with procedures/rules did not emerge as a separate dimension in the healthcare measures, although two studies measured whether unsafe work practices were corrected by supervisors and workmates.
From the comparison it seems that at least three “core” dimensions from industry are being measured as components of safety climate in health care—management/supervisory commitment to safety, safety system, and work pressure. Sorra and Nieva34 included two of these dimensions but did not assess perceptions of the safety system. Pronovost et al36 also included two of these dimensions but their measure of the safety system was quite limited in scope as their safety climate scale only contained 10 items. Most of the other studies included one or two of these “core” dimensions each. This lends some weight to the argument for a set of universal or core variables that underpin safety climate across work sectors, although these probably need to be complemented with sets of sector specific factors.8,11
Factor structure and internal reliability
For this review, studies with a sample size of less than 300 and factors consisting of three items or less were regarded with caution (box 2). Two studies conceptualised safety climate as a unidimensional construct but did not report a factor analysis to confirm this. Only six studies reported the results of a factor analysis. DeJoy et al25 conducted separate exploratory factor analyses on their 35-item measure of safety climate for each of the occupational groups studied (nurses, physicians, technicians). Each analysis yielded eight similar factors based on the same 23 items, as shown in table 1 (α coefficients ranged from 0.61 to 0.83 apart from general work organisation which was 0.39). However, more than half of these factors (feedback, knowledge and information, perceived risk, response efficacy, work organisation) were based on three items or less, which is usually regarded below minimal. Gershon et al26 conducted an exploratory factor analysis on their 46-item safety climate scale, yielding six factors (based on 20 items) that did not correspond to their nine hypothesised safety climate dimensions. An internal reliability analysis of the factor scales yielded acceptable α coefficients. Singer et al32 found a five-factor solution that did not match their original thematic groupings. In two studies the data sets were rather small for the factor analyses that were conducted.29,30
Only one study provided a comprehensive report of scale development. For their Hospital Survey on Patient Safety, Sorra and Nieva33 conducted an exploratory factor analysis to explore the dimensionality of their survey data. The results of the exploratory factor analysis revealed the existence of 14 distinct factors. A subsequent confirmatory factor analysis was conducted and the final confirmatory factor model contained 42 items in 12 factors (two factors which measured outcomes and 10 which measured safety climate). This model fitted the data well with good α coefficients.
Three studies used versions of the ORMAQ.34–36 Despite sample sizes of more than 300, none of them reported a factor structure or internal reliabilities for the hypothesised dimensions. Itoh (personal communication, 2003) indicated that the results of an attempted factor analysis were not interpretable; a similar result has been reported elsewhere.56
Criterion validity
Criterion or outcome measures of safety in health care could include worker behaviours, worker injuries, patient injuries, or other organisational outcomes (such as litigation costs). As shown in table 1, four studies had no outcome measures. Self-reported worker rule compliance or event reporting behaviours were recorded in five studies, with two including independent measures of occupational injuries.28,29 Only one study had an independent measure of injuries or annoyances to patients,34 although this was broadly conceptualised and included events such as losing artificial teeth. Other studies used the term “patient safety culture”32,35 but included no patient outcome measures.
The three studies using independent outcome measures reported significant associations between climate scores and outcomes. Two28,30 found evidence that positive perceptions of organisational safety in healthcare settings were related to fewer worker injuries. Itoh34 measured the relationship between rates of nurses’ reports of patient incidents from one of the participating hospitals from the preceding year and aggregated scores on the questionnaire. There was no correlation between questionnaire responses on incident reporting and rates of adverse events to patients. A significant negative correlation was reported between questionnaire scores on “recognition of inevitability of human error” and rates of incident reporting. This represents one of the few attempts to include an independent measure of worker safety behaviour but, because of the breadth of the measure and the small sample of units, these results would require to be interpreted with a degree of caution. Correlations of climate scores with self-reported safety behaviours generally showed positive relationships.25,27,33
Some care needs to be taken with the level of analysis for measuring and testing predictors against outcomes (that is, individual worker, team, department, hospital, NHS trust).59–61 Only one study34 attempted to examine the relationship between safety climate of work units and their safety outcomes. Most aggregated questionnaire responses from the entire sample which often included more than one hospital.25,28,32,33 While Gershon et al26 aggregated responses to the organisational level, their sample only included hospital employees who were at risk of exposure to blood and body fluids. Neal et al27 and Pronovost et al35 aggregated responses from representative samples of hospital workers to the organisational level. Although it can be argued that these studies produced more meaningful safety climate data, they did not examine the relationship with organisational safety outcomes such as worker or patient injury rates.
DISCUSSION
The UK National Audit Office65 has recently reported on the state of patient safety in NHS trusts. While offering an encouraging prognosis, this is far from a clean bill of health. The report states that “The safety culture within trusts is improving … However, trusts are still predominantly reactive in response to patient safety issues and parts of some organisations still operate a blame culture” (page ). Measuring safety climate in health care helps to diagnose the underlying safety culture of an organisation or work unit. The prevailing culture influences safety behaviours and outcomes for both healthcare workers and patients. Safety climate questionnaires need to achieve as high a standard of measurement as possible so that healthcare managers can use the resulting data to design effective safety management systems and interventions.
We have reviewed the psychometric properties of instruments used to measure safety climate in 12 studies based in healthcare settings. None of these had achieved full scale testing and it is recognised that some instruments were at an early stage of development. The Hospital Survey on Patient Safety33 met more of the specified psychometric criteria due to more systematic testing of internal structure than the other instruments reviewed. Some of the scales of this instrument—such as organisational learning/continuous improvement and teamwork (within, and across hospital units)—should not perhaps be considered part of the safety climate11 unless their relationship with safety outcomes can be confirmed. This study only had a 29% response rate which was rather low compared with the other studies and may signal issues of usability or weaknesses in their survey method. Medical staff have limited time to complete and return questionnaires, so instruments for health care may need to be parsimonious and made available electronically as well as on paper to maximise response rates.
Several of the instruments had been developed from measures used in other industries (aviation, oil, nuclear). Considerable care needs to be taken when adapting measures from these very proceduralised high risk industries. Not only is the nature of the work very different, but the organisations have well defined hierarchical management structures with clear reporting relationships. Leadership issues are much more problematic to measure in health care as the managerial reporting relationships are subject to different interpretation by each professional group, thus introducing a degree of ambiguity. This is particularly true for doctors.66 Moreover, the safety climate studies in industry all focus on worker injury rather than product (cf patient) damage. Determining reliable outcome measures for these healthcare studies appears to be challenging; sometimes the focus is on workers’ behaviours, which might be regarded as safety process measures, and in other cases some kind of adverse event is used. As more patient safety indicator and outcome measures are being introduced, these should permit stronger data based on work unit and organisational performance to give appropriate higher level criterion metrics.
The data sets were drawn from different levels of organisational analysis and, as mentioned earlier, nested data of this type need to be analysed with some care—for example, by using multilevel modelling statistical techniques. Aggregating safety climate data across hospitals and, indeed, across healthcare systems in different countries64 is not entirely advisable unless the questionnaire is measuring sector rather than organisational features. Zohar11 has argued that safety climate can be meaningfully construed only at the group and organisational levels so as to reflect a particular supervisor’s and senior management’s influence on safety, respectively.
This is a preliminary review and it should be acknowledged that many of these research teams are now engaged in larger scale questionnaire studies. In future, meta-analysis based on effect sizes will be needed to compare their results and to determine the validity and generalisability of the climate measures. In general, these studies have begun to confirm that safety climate scores can be associated with healthcare workers’ safety behaviours or workers’ injuries, replicating earlier findings from industry, although few independent measures were used. Very few of the reviewed studies considered the mechanisms that mediate the relationship between safety climate and safety outcomes (that is, worker injury or patient harm). In the wider literature on safety climate there are now models that attempt to explain the psychological mechanisms linking safety climate and worker behaviour.9,11 In these models the relationships between safety climate, safety behaviour, and safety outcomes are focused on individual worker injury. In the healthcare sector there is an additional need to establish whether a different set of antecedents influences processes (worker behaviours) that affect patient safety outcomes as opposed to worker injury. In other words, are there different motivating factors that determine the safety behaviour of healthcare workers in relation to preventing personal harm compared with harm to a patient? The Institute of Medicine report1 “To Err is Human” stated that “workers’ safety is often linked with patient safety. If workers are safer in their jobs, patients will be safer also” (page ). In fact there is little evidence to support this claim, although emerging evidence is encouraging.67 So future safety climate research in health care should elaborate and test models that attempt to explicate the mechanisms influencing not only patient safety but worker safety as well.
Finally, while questionnaires offer an efficient and anonymous method of collecting safety climate data, researchers need to consider alternative techniques for sensing organisational culture. Ethnographic approaches based on observation and interviewing68 can be expensive but they can provide valuable qualitative data to test the validity of the survey methods.
REFERENCES
Footnotes
-
This research was funded by Grampian University Hospitals NHS Trust Endowment Fund. The views presented are those of the authors and should not be taken to represent the position or policy of the funding body.
-
Calvin Burns is now at the Business School, University of Strathclyde.
Linked Articles
- Quality lines