METHODS

Survey instrument

The survey instrument, initially constructed by PSCI, was adapted with permission from five existing surveys (OR management attitudes questionnaire, anesthesia work environment survey, naval command assessment tool, risk management questionnaire, and safety orientation in medical facilities^16,18–22). A review of these instruments produced partially overlapping questions covering 16 topics plus demographic information (see box 2). Many of these topics have been identified as important to a culture of safety.^{10,18,23–25}

The preliminary survey instrument was tested extensively in pilot studies at VA facilities before implementation at the Consortium hospitals. After modification based on feedback about specific questions in the pilot studies, a revised instrument (82 questions plus demographic information) was distributed to Consortium hospitals in two mailshots. For the third and final mailshot an abridged version (30 questions plus demographic information) was created by eliminating overlapping questions while retaining at least one (and usually two) questions for each of the 16 topics. All questions were close ended. Three scales were used: a 5-point Likert scale; a yes/no/uncertain scale; and a 5-point frequency scale (always/frequently/sometimes/rarely/never). Unless otherwise specified, results reported here pertain only to the 30 questions included in all three mailshots.

Approval to conduct this survey was granted in advance by the relevant institutional review boards of all of the participating institutions.

Survey sample

At each participating hospital the target sample was 100% of the hospital’s attending physicians, 100% of senior executives (defined as department head or above), and a 10% random sample of all other employees. For logistical reasons two hospitals surveyed less than 100% of physicians (one large hospital sampled 250 of their top admitting physicians, 25% of all physicians, and the other hospital randomly sampled 20% of their physicians). In a third hospital too few responses were received from physicians to analyse them as a separate category. The initial mailing list contained 6906 names; 347 duplicates were removed and 227 were returned as undeliverable (employee retired, was a temporary worker, or no longer employed), leaving a total of 6312 eligible individuals in the 15 hospitals.

Administration of survey

Between April and June 2001 the hospitals distributed survey packs to recipients via inter-office or US mail in three mailshots. There was some variability in starting dates between institutions for logistical reasons, and successive mailshots were separated by approximately 4 weeks. The packs included a cover letter co-signed by an investigator and a senior executive from the applicable hospital, a paper survey instrument, a business reply return envelope, instructions for completing the survey on the Internet if desired, and a separate questionnaire completion notification (QCN) postcard (first two mailshots only). A secure web version of the survey was offered for the first two mailshots. The QCN postcard allowed us to track non-responders to each of the first two mailshots without compromising the anonymity of the survey responses.

Weighting of data

Two weights were applied multiplicatively to the raw data to allow accurate comparison between participating hospitals and job types and to correct in part for subject non-response.

Sample weighting

To correct for the different sampling strategy for executives and physicians (100% sampling) versus other employees (10% sampling), we weighted the responses of other employees by a factor of 10 relative to executives and physicians. For hospitals that surveyed fewer than 100% of physicians or senior managers the sampling weights were adjusted accordingly. This weighting was applied whenever responses from different job types were aggregated together.

Non-response weighting

To correct for the different rate of non-response to the survey from employees of different job types, the primary data set was created by weighting the respondents for each of three job types by the inverse of the non-response rate for that job type from all mailshots.^26,27 For example, if the overall response rate across all mailshots of management, physicians, and other employees was 66%, 33%, and 50%, the relevant respondents would be weighted by factors of approximately 1.5, 3, and 2 to account for non-responders. We also created alternative data sets that accounted for the non-responders by weighting the mailshots separately. Responses to the first mailshots were significantly more likely to suggest the absence of a culture of safety—that is, they were more “problematic” about safety culture than were those to the third mailshot. Weighting respondents only from the first and second mailshots to account for those who did not respond to any mailing yielded a data set that gives a relatively pessimistic estimate of safety culture by the non-responders. Weighting only the data from the third mailshot to account for non-responders created a more optimistic estimate of non-responder attitudes.

Analysis of data

To allow aggregation of the different survey questions and comparison between hospitals and job types, the “problematic responses” to each question were compared. The answer choice that constituted a “problematic” response varied by question, depending on how the question was phrased. For example, in response to “I will suffer negative consequences if I report a patient safety problem”, to agree or strongly agree was considered problematic. In contrast, in response to “I am rewarded for taking quick action to identify a serious mistake”, to disagree or strongly disagree was problematic. We also examined the frequency of neutral responses, as these might also imply a lack of safety culture (L Kearney, personal communication). “Neutral” responses were neutral on questions using a 5-point Likert scale, uncertain on questions offering yes, uncertain or no responses, and sometimes on questions using a 5-point frequency scale.

Principal factor analysis was conducted on the responses to the 82 question survey implemented in the first two mailshots (with a varimax rotation to maximize the loadings on each factor). This yielded five factors which accounted for 80% of the systematic variation across questions. Based on the questions that loaded most heavily to each factor, we labelled these factors as “organization”, “department”, “production”, “reporting/seeking help”, and shame/self-awareness”, respectively.

We also analysed responses to questions by respondent hospital, job class, and clinician status. Although this report contains primarily a descriptive analysis, we also conducted multivariate regression models with interaction terms and hospital fixed effects. We comment briefly on these results where they add to information provided by the descriptive analysis. In addition, by comparing responses in our primary data set with those in the pessimistic and optimistic data sets (imputing different patterns of response to the non-responders), we conducted a form of sensitivity analysis on the probable impact of non-response bias.

Because the tables for the multivariate regressions and alternative non-response weighted data sets are extensive, they are not included but are available electronically from the authors.

Meeting the objectives of the study

Measuring the problematic response to survey questions enabled us to meet our principal objectives—to measure attitudes towards safety culture and to determine how safety culture varies among hospitals and between types of personnel. Our findings establish a baseline for future benchmarking and identify opportunities for improvement in participating hospitals.

RESULTS

Survey response

A total of 2989 surveys were returned over a 6 month period giving an overall response rate of 47.4%. The response rate excluding physicians was 62%, and the response rate for physicians was 33%. The overall response rate varied by hospital, ranging from 38% to 71%. The response rate also varied by mailshot. Of the 6312 subjects, 25% responded to the first mailshot, 9% (12% of the remaining subjects) responded to the second mailshot, and 13% (20% of the remaining subjects) responded to the third mailshot. The increase in response to the third mailshot may be attributed to the shorter survey instrument used. Respondents returned nearly all surveys by US mail; only 1% of respondents used the Internet.

Demographic information

Overall, there were slightly more women among the respondents (54%, table 2). The sex of the respondents varied with job class, reflecting the different sex profiles of the medical and nursing professions; 29% of physicians, 51% of senior managers, and over 70% of other employees were women. 64% of the sample were aged between 40 and 60 years. 65% were physicians, nurses, pharmacists, or other core clinical staff; 31% were non-clinicians, and the remainder (4%) were other clinicians such as nursing assistants or therapists.

Attitudes toward patient safety and organizational culture

The average problematic response for the 30 questions common to all three mailshots was 18% (table 3). Problematic responses to individual questions ranged from 4% to 52%. For 21 of the 30 survey questions problematic responses were more than 10%; for eight questions problematic responses were more than 25%. Problematic responses were higher for questions that were phrased as hypothetical or impersonal, and lower to questions that were personal or time delimited.

When neutral responses were included, problematic + neutral responses averaged 36.5% across all questions (range 8–70% for individual questions). Not surprisingly, personal questions relating to quality of care yielded more neutral responses than did those concerning coworkers.

Many respondents indicated a lack of rewards and fear of punishment for identifying and making mistakes. Approximately 33% of respondents said they were not rewarded for taking quick action to identify a serious mistake, and 28% believed they would be disciplined if a mistake they made was discovered. Respondents also felt that pressure impacted on patient care, noting a small but appreciable incidence of unsafe acts. Almost 52% believed that loss of experienced personnel had negatively affected their ability to provide high quality patient care. More than 39% reported that they had witnessed a coworker do something that appeared (in their opinion) to be unsafe, and 8% admitted that in the last year they had done something that was not safe for the patient.

Variation between hospitals

There was wide variation between participating hospitals, despite the presumed attention to patient safety by all of them. The average overall problematic response across all questions varied in individual hospitals from 13% to 22%, a 9% spread. For individual questions the range in problematic response varied from 6% to 38% between institutions. Problematic responses between hospitals varied most in answer to “Individuals in my department are willing to report behavior which is unsafe for patient care” and “Asking for help is a sign of incompetence”. While the hospital with the best survey results had only 13 questions with more than 10% problematic responses, the hospital with the worst results had 25 questions above this threshold (data available from authors).

Variation between types of hospital personnel

When the problematic response rates were compared by job class (senior managers, non-senior managers) and clinician status (clinician, non-clinician), clinicians in general were found to be more likely to provide problematic responses than non-clinicians (table 4). In addition, senior managers in general were less likely to give problematic responses than non-senior managers. Senior managers reported up to 15% fewer problematic responses on individual questions. The only question to which senior managers responded with more problematic responses than front line workers was “Telling others about my mistakes is embarrassing”.

Multivariate regression analysis confirmed these conclusions (p<0.01) and further suggested that responses by clinician senior managers were more similar to responses by other clinicians than to those by non-clinician senior managers (p<0.01)—that is, clinicians were generally more negative than non-clinicians, regardless of management status. Among clinicians, nurses were the most negative (p<0.01) and almost always responded significantly more negatively than non-clinicians (25 out of 30 questions). Similarly, sensitivity analysis of the effects of non-response bias (using the pessimistic and optimistic estimates of non-responders) showed that these findings were unlikely to be due to selection bias.

DISCUSSION

Key findings

Because these data were acquired using systematic sampling of a full cross section of employees in a diverse set of institutions, the results provide the most complete available information on the attitudes and experiences of workers about safety culture in hospitals. While most responses were consistent with a culture of safety, a substantial minority of respondents gave a problematic response (box 3). Across all questions and institutions the aggregate problematic response was 18% (37% including neutral responses). For 70% of the questions in our survey the aggregate problematic response was above 10%, and for 27% it was above 25%. Respondents also reported a small but appreciable occurrence of unsafe acts.

One key finding of this study is that there was a definite discrepancy between the attitudes and experiences of senior managers (particularly non-clinicians) and those of non-managers. Non-clinician senior managers answered more often in ways consistent with a culture of safety than did personnel who actually take care of patients. Clinicians, particularly nurses, gave more problematic responses than non-clinicians, regardless of management status. These findings have implications for managerial strategies in hospitals (see below).

Relationship with previous research and patient safety initiatives

Compared with the findings of Sexton and colleagues,¹⁰ the levels of problematic response reported in this survey are no more problematic than responses provided by operating room and intensive care clinicians to questions dealing specifically with fatigue, stress, hierarchy, and teamwork. However, those studying HROs believe that a high uniformity of safety attitudes and experiences is necessary to realize a working safety culture. They raise concerns when problematic attitudes exceed about 10% of a cohort (A Ciavarelli and K Roberts, personal communications). By this standard the number of problematic responses in participating hospitals was still worrisome, even in the hospital with the best survey results (overall problematic response of 13%).

Limitations

The aggregate subject response rate in this survey (47%) was not as high as desired, largely because of a low response among physicians. However, we predicted a lower response rate for physicians which is why we chose to sample nearly 100% of them. The actual number of physician respondents was over 1000. In addition, the responses of physicians were similar to those of other clinicians who had a much higher response rate. We believe the overall response rate was satisfactory for a complex survey administered to extremely busy individuals. We were able to make some estimates of possible non-response bias by incorporating the response wave into the analysis. This suggested that the overall results would probably not be affected strongly by non-response bias.

Our results may not be representative of hospitals generally as the hospitals surveyed in this study were members of a consortium of institutions with a specific interest in advancing their performance in patient safety. We presume that these are probably leading institutions in safety rather than laggards, although it is possible that some participated because they were aware of serious deficiencies and wished to correct them. The data from the survey are therefore more likely to reflect a best case rather than a worst case scenario. The small number of hospitals surveyed to date does not yet permit us to consider statistically significant differences between types of hospitals.

Implications

The discordance found between non-clinician senior managers and front line workers may reflect a tendency for front line workers and middle managers to gloss over patient care problems in briefings to senior managers.²⁸ This could make it hard for non-clinician executives to understand the true state of their organization, to determine changes needed, and to assess their attempts to create and maintain a culture of safety. In addition, these results could imply inadequate communication to the front line workers of the commitment of the management to patient safety. If further studies confirm these findings, reasonable strategies for improving safety culture might involve creating better linkages between executives and those directly concerned with patient care. Our findings also suggest that, while short term efforts to work on specific interventions (such as computerized physician order entry implementation) are laudable, they may be insufficient if the managerial structures and cultures of institutions do not sufficiently favor patient safety. Long term progress may need to include interventions specifically aimed at improving safety culture and breaking down barriers between managers and front line workers.

Further research

Further studies are required to determine more precisely the magnitude of the problematic response that indicates an unsafe culture. Using our instrument in a much larger set of hospitals over a larger geographical area could confirm the results in a broader range of personnel, and would allow assessment of the relationship between safety culture and hospital characteristics. Longitudinal studies could allow measurement of the impact of interventions designed to address safety culture. Comparing organizations in a variety of industrial domains and correlating the results with the actual safety performance of these organizations would provide important validation of the results. Such studies would provide a solid basis upon which to recommend significant policy changes and long term reform strategies to address safety culture.

Finally, since we cannot fully exclude non-response biases in our study, future studies may need to use intensive surveying of a subsample of non-responders (especially physicians) to test whether those who respond are systematically different from those who do not.