Article Text
PDF
Abstract
Objective The study had two specific objectives: (1) To analyse change in a survey measure of organisational patient safety climate and capability (SCC) resulting from participation in the UK Safer Patients Initiative and (2) To investigate the role of a range of programme and contextual factors in predicting change in SCC scores.
Design Single group longitudinal design with repeated measurement at 12-month follow-up.
Setting Multiple service areas within NHS hospital sites across England, Wales, Scotland and Northern Ireland.
Participants Stratified sample of 284 respondents representing programme teams at 19 hospital sites.
Intervention A complex intervention comprising a multi-component quality improvement collaborative focused upon patient safety and designed to impact upon hospital leadership, communication, organisation and safety climate.
Measures A survey including a 31-item SCC scale was administered at two time-points.
Results Modest but significant positive movement in SCC score was observed between the study time-points. Individual programme responsibility, availability of early adopters, multi-professional collaboration and extent of process measurement were significant predictors of change in SCC. Hospital type and size, along with a range of programme preconditions, were not found to be significant.
Conclusion A range of social, cultural and organisational factors may be sensitive to this type of intervention but the measurable effect is small. Supporting critical local programme implementation factors may be an effective strategy in achieving development in organisational patient SCC, regardless of contextual factors and organisational preconditions.
- Quality improvement
- patient safety
- collaborative
- breakthrough groups
- implementation science
- safety culture
- qualitative research
- health services research
- statistical process control
- accreditation
- health policy
- healthcare quality improvement
- management
- social sciences
- patient satisfaction
- patient-centred care
- patient education
- root cause analysis
- risk management
Statistics from Altmetric.com
- Quality improvement
- patient safety
- collaborative
- breakthrough groups
- implementation science
- safety culture
- qualitative research
- health services research
- statistical process control
- accreditation
- health policy
- healthcare quality improvement
- management
- social sciences
- patient satisfaction
- patient-centred care
- patient education
- root cause analysis
- risk management
Introduction
The challenge of ensuring the reliability and safety of care delivered to patients remains one of the most pressing yet complex and intractable problems for modern health service organisations.1 ,2 At the service level, the last 15 years have shown increasing use of large-scale, national programmes, drawing upon the quality improvement collaborative model, as a means of driving improvement on a broad scale.3 ,4 In the UK in 2004, the Health Foundation launched the Safer Patients Initiative which aimed to improve the safety and reliability of care within 24 participating hospitals, across four nations.5–11
The Safer Patients Initiative was structured in two phases: an initial pilot phase involving four healthcare organisations (2004–2006) and a main phase (March 2007–September 2008) involving 20 organisations selected from applications made by trusts in England, Wales, Scotland and Northern Ireland. The programme design was based upon the US Institute for Healthcare Improvement's Breakthrough Series Collaborative model.12 ,13 The initiative aimed to improve a range of clinical and hospital-level outcomes through development of a comprehensive package of measures for clinical care processes, visible leadership support, improved hospital communication and development of patient safety culture (box 1). Collaboratives draw upon a range of organisational theory, principles of evidence-based medicine and industry quality control to pursue the goal of sustained, effective change across multiple sites.14–16 The methodology involves bringing together multi-disciplinary teams from each organisation to focus upon a specific care issue or aim through a structured programme of iterative process change and regular reporting and dissemination of results within the collaborative.13 ,17–19 By actively pursuing broad staff engagement and addressing supportive organisational structures and processes, such programmes attempt to achieve lasting culture change in addition to improvement in patient outcomes.
Description of the Safer Patients Initiative aims, structure and components
The UK Safer Patients Initiative
Collaborators:
The UK Health Foundation; US Institute for Healthcare Improvement; 24 UK National Health Service Trusts
Main aims:
Mortality: 15% reduction
Adverse events: 30% reduction
Ventilator associated pneumonia: 0 (or 300 days between)
Central line bloodstream infection: 0 (or 300 days between)
Blood Sugars (Intensive Trauma Unit/High Dependency Unit (ITU/HDU)): 80% within range
MRSA bloodstream infection: 50% reduction
Crash calls: 30% reduction
Harm from anticoagulation: 50% reduction in Adverse Drug Events (ADEs)
Surgical site infections: 50% reduction
Timescale and main phases:
Phase 1 pilot (four sites 2004–2006)
Phase 2 main (20 sites 2006–2008; subsequently 19 organisations following merger)
Phase 3 Network (2008, ongoing)
Change package elements by work area:
Perioperative care: deep vein thrombosis (DVT) prophylaxis, β blocker use, focus upon surgical site infections, communication (Situation Background Assessment Recommendation (SBAR; safety briefings))
Medicines management: Medicines reconciliation, focus upon high risk medications (anticoagulants)
General ward care: Early warning systems and outreach/rapid response team, communication (SBAR, safety huddles), infection prevention and control, hand hygiene
Critical care: Ventilator bundle, central line bundle, multi-disciplinary ward rounds, infection prevention and control, daily goal sheets
Leadership: Leadership walkrounds, strategic prioritisation of quality and safety issues
Programme tools and methodological components:
Continuous quality improvement approach and philosophy: semiautonomous local quality improvement teams
Plan Do Study Act (PDSA) cycles and small tests of change
Incremental spread to successively larger work systems
Process measurement and analysis of run charts to determine effects of process changes
Expert faculty support from experienced clinical improvement leaders (site visits, conference calls, presentation seminars and online email support)
Large-scale learning sessions for multi-disciplinary improvement teams from each site (with educational and support components)
Online extranet for uploading and comparing process data generated by each site, with monthly faculty feedback
Collaborative learning community for networking and sharing best practices
Despite the popularity of the collaborative method in the UK and elsewhere, the research evidence for the efficacy of this type of programme remains a complex issue. Several authors have drawn attention to limitations in the evidence base, including over-reliance upon case study and anecdotal reports.20–25 Other authors have highlighted the research challenges that exist in studying complex longitudinal interventions which are subject to a range of sociotechnical and contextual influences beyond the control of programme developers.7 ,22 ,25 In the context of the Safer Patients Initiative, a robust multiple outcome evaluation of the main phase of the programme found no evidence of positive change in either clinical outcomes or care process quality metrics when compared with control sites.26 Broader synthesis of findings from robust study designs suggests that quality collaboratives may exert only modest positive impact upon quality of care outcomes at best.22 Despite these limitations, case studies commonly report several activities that are held to be important for the success of such programmes, including: obtaining the commitment of senior management, fostering receptivity to change, engaging clinicians, implementing quality reporting processes, developing safety awareness and fostering staff-driven process improvement.27–29 The collaborative programme may therefore act to develop local cultural and organisational conditions necessary for safety, raising the question of whether the Safer Patients Initiative had an effect upon organisational patient safety climate and capability (SCC).
Several considerations inform the design of the present study. There is clearly a need for further research to understand the impact of this type of programme on healthcare organisations and how development in local patient SCC may best be measured for research and evaluative purposes. A key issue for development of programme methodology is identification of the critical determinants of success for this type of programme and understanding the mechanisms of cause and effect which may be reproduced in other contexts to improve an organisation's capacity for continuous safe and reliable care. The purpose of the current study is therefore twofold, relating to the question of what impact the initiative had upon safety climate in participating hospitals and why. First, to determine to what extent sites which participated in the Safer Patients Initiative experienced the expected positive change in SCC, as measured by a comprehensive survey designed for this purpose. Second, to understand the role of a range of programme and contextual factors that research suggests are important for collaborative and quality improvement projects in predicting change in SCC scores.
The concept of ‘safety climate’ has received considerable attention in the healthcare quality and safety literature as a determinant of hospital level variation in patient safety outcomes and several survey measures have been developed to measure the construct.30 ,31 Drawing upon industry experience and review of emerging measures in this area, safety climate may be defined as those aspects of organisational culture that impact upon safety, including managerial and organisational behaviour, local workplace norms, communication, safety management, reporting systems and shared employee attitudes and values.32 ,33 Improving safety culture in healthcare organisations has been identified as an important component in improving patient safety across health systems.2 Improving local safety climate in hospitals was a stated aim of the Safer Patients Initiative and several of the programme components were designed to influence culture, including leadership walkrounds, multi-disciplinary collaboration, frontline engagement in development cycles, data-driven improvement, protocols for improved hospital communication (SBAR) and safety-specific communication (eg, theatre safety briefings and safety huddles).
In addition to analysing the impact of the initiative upon SCC, the present study sought to assess the influence of a range of variables upon change in SCC scores over time. Prior research and theory suggest a number of potential predictors of success in quality improvement collaboratives.34–38 Work in the science of improvement, for example, suggests that the effects of programmes such as the Safer Patients Initiative are context-specific and that a range of organisational characteristics, programme precondition factors and local implementation factors influence the local response to a programme.39–42 Research suggests that larger hospitals will not implement a programme as effectively as smaller hospitals due to diffusion of programme governance and that hospital teaching status may affect engagement and success in quality improvement.43–45 Programme preconditions such as organisational recognition and knowledge of local safety issues, systems to support improvement and measurement, stability in meeting external targets and availability of resources to support the programme may all lead to enhanced programme outcomes.46 ,47 Further studies and theory suggest that local experience in programme implementation in terms of the degree of clinician engagement, measurement of process variation, senior management support, programme governance, early adoption of process changes and compatibility with existing targets may be critical factors.29 ,48–50 Despite a large volume of descriptive case studies, few studies have assessed the relative importance of these factors upon longitudinal change in a programme outcome measure, such as safety climate, using a multi-site design.
Methods
Study design and hypotheses
The study aimed to determine the effect of the Safer Patients Initiative upon local safety climate and investigate the role of various factors that research suggests are important in influencing the effects of large scale collaborative programmes upon outcomes in participating sites. The research design focused upon the main phase of the Safer Patients Initiative and comprised a single group longitudinal study (similar to a cohort study) with repeated measurement at two time-points. The first time-point (June–August 2008) corresponded with the end of the structured implementation phase of the main SPI programme and the second time-point measure was taken after a 12-month follow-up period (June–August 2009). The interval between time-points represented the period over which it was expected that participating sites would implement, spread and consolidate the programme within local organisational structures and processes.
In addition to investigation of change in safety climate scores between the study time-points, a series of predictive models were fitted to the data to examine the degree to which a range of individual characteristics, hospital characteristics, programme preconditions and implementation factors could account for development in local SCC. Variables were entered into the models according to causal and temporal priority, informed by prior research and theory as outlined above. Figure 1 depicts the full study research model, which gives rise to the following five specific hypotheses:
Study research model linked to programme timeline for participating sites. At the centre of the model is the programme timeline split into three main periods: preprogramme, programme implementation and postprogramme development phases. The influence of the intervention itself and that of broader contextual factors is depicted above the timeline. The research model is illustrated below the timeline with each block of research variables linked to specific measurement points on the timeline (note that all preprogramme and implementation phase data were actually collected retrospectively at time-point 1). The dashed arrows indicate the five hypothesised causal relationships investigated in the study, including the main research questions concerning change in safety climate and capability between the study time-points (hypothesis 1) and which factors predict this change (hypotheses 2–5). The sequence of the research model in the figure represents the temporal priority of causal variables and illustrates the sequence of variable entry in the predictive models (ie, the unique contribution of each current variable set above and beyond that of prior variables is examined in the analysis).
Hypothesis 1: Organisational SCC will significantly improve over 12 months corresponding to the period in which the programme was to be spread and consolidated at each participating site.
Hypothesis 2: Respondent characteristics associated with experience (tenure), programme role (proximity to frontline) and length of programme involvement will influence perception of development in local SCC.
Hypothesis 3: Site characteristics such as variation in hospital size (number of beds) and type (teaching or district general hospital) will influence development in SCC once the effects of individual respondent characteristics have been controlled.
Hypothesis 4: The presence and degree of a range of programme precondition factors will influence development in SCC once the effects of individual and organisational characteristics have been controlled.
Hypothesis 5: Variation in a range of programme implementation factors will influence development in SCC once variance attributable to individual, organisational and precondition factors has been controlled.
Participants and data collection
Multi-site ethical approval was obtained for the study from the National Health Service (NHS) National Research Ethics Service in the UK. All 20 NHS acute care organisations participating in the Safer Patients Initiative second phase consented to take part in the study. During the course of the programme, two organisations merged into one with a combined programme governance structure prior to the first study time-point, resulting in a final sample that included 19 separate organisations. Due to the organic nature of development of the programme at each participating site, prespecification of individuals for inclusion was not possible.7 Instead, sampling was stratified according to criteria defined by the researchers to include representation of the full multi-disciplinary programme team at each site, including the subprogramme leads and the broader network of healthcare professionals, clinicians, nurses and health service managers involved in local implementation. Local recruitment was performed by local site coordinators nominated by the research team. Feedback was given by the research team to the coordinators during the process to ensure that all key programme and senior organisational roles were included at all sites. A combination of paper-based and online survey administration was used for data collection. In total, 635 responses were returned at the first time-point (an estimated response rate of 52% of individuals that met the inclusion criteria) and 284 were returned at the second time-point, representing a drop-out rate of 55%, in part due to mobility of staff over the intervening 12-month period. For the purposes of statistical analysis, only data from the true longitudinal sample (284) who responded at both study time-points were used (see table 1).
Characteristics of participants that responded to both study time-points
Measures
A 106-item survey was administered at the first study time-point, including varied scales designed to measure a range of individual characteristics, programme preconditions, programme implementation factors and the SCC scale. The SCC scale was readministered at the second study time-point. The survey items employed Likert response scales and were developed based upon pilot research in the first phase of the Safer Patients Initiative (SPI) programme.7–9 In the first survey, eight programme precondition items were measured using a scale worded to refer specifically to the period prior to the onset of the programme (the period up to March 2007). A further scale quantified a series of 14 local programme implementation factors in terms of the degree of positive or negative impact in the initial structured phase of the SPI. The individual items are listed in table 2 and further details may be found in a number of related articles.7 ,9 ,51 ,52 The SCC scale comprised 31 items which encompassed a range of sociotechnical factors designed to quantify cultural and organisational dimensions relating to: safety awareness, commitment to safe practice, senior management support, leadership for safety, communication, teamwork, information processes, organisational learning and improvement processes, with the majority of items based upon established safety climate surveys.31 ,32 The items displayed high internal reliability and were aggregated into an overall composite measure (Cronbach's α=0.952). In addition to the survey and SCC scale, data relating to organisation size (number of hospital beds) and type (district general or teaching hospital) were accessed from the Health Foundation's application records for the programme.
Descriptive statistics for SCC scale, programme precondition and programme implementation factors
Analysis
Movement in SCC was tested using a repeated measures t test. The subsequent research hypotheses were tested using blocked regression, with prior entry of covariates as determined by theoretical and causal priority (see figure 1). α Was set to 0.01 in order to adjust for the potential for increased family-wise error rates resulting from multiple testing. Full details of the method of statistical analysis and the results from fitting statistical models to the data are included within a technical appendix to this article, which is available online.
Results
The main findings relating to the study hypotheses are summarised below (see the technical appendix for more detailed results). The main characteristics of the 284 respondents that comprised the longitudinal sample for this study are described within table 1 and a summary of descriptive statistics for the observed research measures appears in table 2.
Change in SCC
In order to investigate hypothesis 1, change in SCC was investigated using a repeated measures t test. Between the two study time-points, mean SCC scores (as measured on a 5-point scale ranging from 0 to 4) increased by 0.14. This improvement was statistically significant (p<0.001) with a small effect size (d=−0.32).
Effects of respondent and hospital characteristics
The remaining research hypotheses were tested by fitting a series of linear models in which groups of related predictors were entered in sets, while controlling for covariates. The main summary statistics relating to model fit for each stage of the analysis are reproduced in table 3.
Summary of statistics illustrating model fit for each of the five stages of the final model, relating to study hypotheses 2–4
As a first step, an initial model was fitted in which the effects of baseline variation in initial SCC scores was controlled. The second hypothesis was then examined: that is, individual respondent characteristics would account for a significant proportion of variance in SCC scores, once baseline variation had been controlled. Individual characteristics explained a small but significant additional proportion of the variance (2%) (p<0.01). Neither organisational tenure nor length of programme involvement were found to be a significant predictor. Programme role, however, was a significant predictor, with programme leads and coordinators tending to rate SCC 0.11 higher than frontline staff (p<0.05). In order to examine the effects of hospital characteristics upon change in SCC (hypothesis 3), parameters representing hospital type and size were added to the model. No significant relationship between hospital size or type and SCC score was detected at the p>0.01 level and the expanded model accounted for no additional variance in SCC score.
Effects of programme preconditions and implementation factors
In a subsequent step, the effects of a range of programme precondition factors relating to organisational readiness for the initiative upon movement in SCC scores were investigated (hypothesis 4). The addition of the programme precondition factors accounted for a small additional proportion of the variance in SCC scores, once prior factors had been controlled, but this effect did not reach statistical significance.
In the final step, a series of programme implementation variables relating to local staff and organisational support were entered to see if they influenced development in SCC (hypothesis 5). Once the effects of prior factors had been controlled, the programme implementation factors collectively accounted for an additional 6% of the variance in safety climate scores (p<0.001). Examination of the influence of specific implementation factors revealed that the availability of early adopters to support the programme (p<0.05), collaboration between different professional groups (p<0.05) and the extent of process monitoring through data collection (p<0.001) were found to be significant. The extent of process monitoring was found to have the strongest effect, with every one point increase in score associated with a 0.17 higher SCC score. The availability of early adopters had a lesser but positive association and collaboration between different professional groups was found to be negatively associated with development in safety climate.
Discussion
This study aimed to determine if the UK Safer Patients Initiative had a positive effect upon local hospital SCC at participating sites, compared with baseline. It additionally sought to investigate the role of a range of contextual and programme-related factors in the development of safety climate through this type of collaborative programme by testing a series of predictive hypotheses.
Respondents reported a modest but statistically significant improvement in local SCC scores over the 12-month follow-up period during which sites were expected to spread, embed and consolidate the structured phase of the programme. Fitting predictive models to the data revealed that both the individual characteristics of respondents and perceptions of a range of programme implementation factors predicted perceived development in local SCC.
The results hold several important implications for understanding the impact of the Safer Patients Initiative and broader theory surrounding the effects of collaborative quality improvement programmes and their application. In terms of individual characteristics, the seniority of respondents' role in the programme was significant with programme leads rating the impact of the initiative more positively, a finding convergent with that from a previous study in this area which found that senior managers overemphasised the cultural impact of the Safer Patients Initiative compared with their clinical colleagues.52 Other safety culture research in healthcare has similarly detected elevated perceptions of an organisation's safety capacity by managers compared with the clinical frontline.53
In the analysis, a range of programme implementation factors were found to be significant predictors, with higher ratings of the availability of early adopters and the extent of reported process monitoring being associated with positive development in safety climate between the study time-points. Counter to established theory, neither hospital characteristics nor programme precondition factors representing hypothesised organisational readiness to undertake the initiative were significant predictors of change in safety climate. Studies have suggested that smaller organisations may be better able to implement quality and safety improvement mechanisms due to increased proximity of formal leadership to frontline care provision.54 No corresponding advantage was found for either larger or smaller hospitals in improvement in SCC in the present study. Taken at face value, this study's findings would suggest that context is not as important as intuition might lead us to believe. A broad range of hospitals may therefore benefit from a single programme model, regardless of organisation type, structure or design, in contrast to the findings of other studies of the determinants of patient safety improvement.41 The findings do not additionally lend support to theories of organisational readiness for change, which suggest that certain structural, resource, cultural and managerial preconditions must be in place for successful engagement in quality improvement programmes.9 ,37 ,47 ,55
The small but significant improvement in local SCC experienced by participating sites detected by this study is suggestive of a modest positive impact of the programme upon some organisational aspects of acute care delivery, including local organisational culture dimensions. Other studies have found that various forms of patient safety improvement initiatives have a positive impact upon patient safety culture when analysed using preintervention and postintervention designs.56 ,57 The findings from the present study, however, contrast with those from a recent mixed-method evaluation of the safer patients initiative which found that the effects of the programme upon a range of clinical processes and patient outcomes were negligible with a small improvement in organisational climate actually favouring the control group.6 ,26
It is difficult for researchers and service developers to interpret the apparent small or non-existent effects of this initiative to improve patient safety in the UK. It may be the case that the effects of programmes such as the Safer Patients Initiative are only detectable over long time frames, with the programme itself acting simply as the initiator for a more long-term development of an organisational focus upon patient safety. This is plausible and compatible with emergent theory in improvement science which suggests that quality improvement in healthcare may rely as much upon long-term developments in social and organisational processes as clinical technologies.58 Organisational SCC dimensions may mediate the relationship between intervention and outcome in organisational-level initiatives and collaborative programmes, a feature of recently proposed evaluative models for patient safety research.59 The results from the present study, which focused upon the 12-month period of development immediately following the structured phase of the programme, lend further weight to these potential explanations.
The fact that programme implementation factors accounted for a relatively large proportion of the explained variance in safety climate scores is of further interest. Implementation processes during the course of a programme represent the so-called ‘black box’ of quality improvement programmes, which have, until recently, been neglected by research that has favoured outcome evaluation.22 ,55 ,60 The experience of improvement teams during the implementation phase of an initiative, in terms of organisational and staff support along with attention to measurement, monitoring and evaluation, may be critical to programme success, irrespective of prior contextual or organisational conditions. This resonates with research into quality improvement processes which has shown, for example, that providing credible and timely data feedback to multi-disciplinary teams on process variations can support continuous improvement and serve to effectively direct remedial efforts.61
There are several practical and design limitations which must be considered when interpreting the findings reported here. First, the lack of a matched control group reduces the ability to attribute causality for observed movement in safety climate measures. History or maturation effects represent the main threat to internal validity of this type of study design.62 Indeed, further studies have found a general trend towards improvement in organisational climate in non-Safer Patients Initiative sites.6 ,26 Safety climate measures rely upon individual perceptions and are likely to be socially determined to a large degree, giving rise to a high degree of noise in this type of measure.32 ,63 Some of the scale items used relied upon respondent recall of organisational conditions present 15 months prior to completion of the survey, which may have been imperfect. The statistical design used controls for potential confounding factors for each hypothesis through prior entry of covariates, but this type of analysis is sensitive to the order of entry of sets of variables, which in turn is influenced by the researcher's ability to construct an unambiguous underpinning model that specifies temporal or causal priority.64 As in all longitudinal survey studies there was a drop-out rate of some 55% between study time-points and the possibility of some response bias operating must be acknowledged, regardless of the attention given to sampling strategy. Alternatively, the study may have been underpowered to detect any small effect of hospital size or type upon outcome. Finally, while the measures used in this study were piloted, with the challenge of capturing all relevant variation in a large programme of this type, the possibility cannot be excluded that unmeasured factors may account for significant variation in the outcome measure.
Conclusion
The modest nature of improvements in the outcome measure reported in this study highlights the challenges that organisations face in improving their capacity to deliver safe and reliable care. The study suggests that while development of local culture and organisational behaviour capable of promoting patient safety is possible, multiple complex processes and factors are implicated. Further research to understand the determinants of successful programme implementation and the constituents of a conducive, local organisational context for improving patient safety would be beneficial. Finally, the findings suggest that large investment and effort may be required to achieve even small improvements in patient safety climate and capability at the organisational or health systems level and therefore lasting positive change may be difficult to achieve where resources to support such programmes are limited.
Acknowledgments
The authors are indebted to Sandra Iskander for her contributions to this study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Files in this Data Supplement:
- Download Supplementary Data (PDF) - Manuscript file of format pdf
Footnotes
Funding This work was supported by funding from the Health Foundation and the National Institute for Health Research (NIHR).
Competing interests None.
Ethics approval Approval provided by the Leicestershire, Northamptonshire & Rutland Research Ethics Committee 2.
Provenance and peer review Not commissioned; externally peer reviewed.