Article Text
PDF
Abstract
Background Differences in contexts (eg, policies, healthcare organisation characteristics) may explain variations in the effects of patient safety practice (PSP) implementations. However, knowledge of which contextual features are important determinants of PSP effectiveness is limited and consensus is lacking on a taxonomy of which contexts matter.
Methods Iterative, formal discussions were held with a 22-member technical expert panel composed of experts or leaders in patient safety, healthcare systems, and methods. First, potentially important contextual features were identified, focusing on five PSPs. Then, two surveys were conducted to determine the context likely to influence PSP implementations.
Results The panel reached a consensus on a taxonomy of four broad domains of contextual features important for PSP implementations: safety culture, teamwork and leadership involvement; structural organisational characteristics (eg, size, organisational complexity or financial status); external factors (eg, financial or performance incentives or PSP regulations); and availability of implementation and management tools (eg, training organisational incentives). Panelists also tended to rate specific patient safety culture, teamwork and leadership contexts as high priority for assessing their effects on PSP implementations, but tended to rate specific organisational characteristic contexts as high priority only for use in PSP evaluations. Panelists appeared split on whether specific external factors and implementation/management tools were important for assessment or only description.
Conclusion This work can guide research commissioners and evaluators on the contextual features of PSP implementations that are important to report or evaluate. It represents a first step towards developing guidelines on contexts in PSP implementation evaluations. However, the science of context measurement needs maturing.
- effectiveness
- patient safety
- assessment
Statistics from Altmetric.com
Introduction
Many practices have been implemented to improve the safety of patients, such as use of a checklist to prevent blood stream infections. The effectiveness of these patient safety practices (PSPs) can vary markedly between different settings. Such variations in effectiveness are likely to be attributable to variations in a range of contextual factors affecting the implementation of PSPs, such as policies, regulations, or organisational characteristics.1–10 That is, some contexts are likely to affect the effectiveness of PSP implementations. However, as Eccles and colleagues (2009) noted, despite the effect that context can have on PSP implementation, ‘the role of context in intervention development needs to be better understood’.1 The lack of understanding in the field of patient safety is especially evident when considering the role that particular contextual factors have on PSP interventions. Our lack of understanding could in part be due to the complex nature of contexts. That is, the importance of particular context features might depend on the infancy or maturity of the PSP being implemented, the stage of the PSP implementation, the specific component when considering multi-component PSPs, the level at which the PSP is targeted (eg, national-level, hospital-level or unit-level efforts), and the type of PSP (ie, some contexts may be somewhat PSP specific). Our poor understanding of the effects of contextual factors on PSP interventions also could be because of the paucity of empirical evaluations of the issue, although contextual factors have been widely examined in the organisational science and business management fields.11 12
For us to better understand the effects of context on implementations, some contexts should be measured and analytically assessed. However, as Øvretveit and colleagues report, few studies assess the effect of context on PSP implementation and many studies only describe a few contexts.13 Some exceptions exist. For example, as part of the evaluation. For example, as part of the evaluation of the Agency for Healthcare Research and Quality's (AHRQ) patient safety contract portfolio, one study found that AHRQ grantees of health information technology implementations tended to report several contextual factors as contributing to their implementations' success: commitment from top management, dedicated staff and financial resources, an open process to encourage buy-in and enthusiasm by end users, and sheer persistence.14 Another study conducted as part of that evaluation reported on several structural components that AHRQ patient safety grantees considered essential to being in place prior to implementation, such as an institutional commitment and leadership, trust among participating stakeholders, a supportive organisational culture, and prior experience with performance improvement.15 Although a few additional studies specifically focused on patient safety16 or other types of intervention implementations have examined the effect of context,17 18 most do not.
Another potential reason we understand less than we do about the role of contextual factors on PSP implementations is that many evaluations report only basic information on contextual features. For example, a review of health information technology studies found that the interventions' contextual features were poorly described.19 Additionally, a recent paper noted that many published reports of clinical interventions in general fail to adequately describe the context in which interventions are implemented such that there is inadequate information for replication and/or implementation.20 Similarly, an analysis of quality improvement strategies identified a need for more detailed descriptions of both interventions and contextual features to inform interpretation, replication and local adaptation when choosing potential intervention strategies for particular circumstances.21 This would allow readers to answer the question, ‘Is this intervention likely to work in our circumstances?’
However, it is difficult to prioritise which contextual features to assess or only describe if there is a poor understanding of which contextual features are likely to modify the effectiveness of a PSP intervention. Given that such a wide range of contextual features exist, it is important to avoid calls for the reporting of more and more contextual features to an excessive level of detail, which risks paralysing researchers and readers alike. Therefore, the challenge is to identify which contextual features—given our current level of understanding—are most likely to inform understanding and generalisability. Admittedly, some contextual features are generally thought to be important (eg, having a culture of safety and leadership support) but the patient safety field is far from in agreement on whether these are specific to safety issues or any organisational change effort in general and what other types of patient, hospital unit, hospital, or external contextual features are important enough to assess or at least describe in research papers.
In sum, the role of contextual features in PSP implementations appears important, although there is little evidence or consensus around which contexts are the most salient for PSP implementations, subsequently hindering the generalisability and reporting of evaluations. The authors addressed this gap by presenting the conceptual domains of contexts and the specific contextual features within those domains that experts considered high priority for assessment or description. Experts were asked to distinguish between assessing the effects of contextual features and only describing contextual features in research papers. If researchers more richly describe the contexts in which PSP interventions are implemented, it might enhance readers' ability to determine if the study is likely to work in their own set of circumstances. However, assessing some potentially important contexts might help move the field forward by building the evidence for the importance of some contextual features to PSP implementations.
Methods
Throughout the study, the authors used an iterative process of formal group discussions with a 22-member technical expert panel composed of patient safety experts, clinical and healthcare system leaders, and methodological experts.These discussions were based on the results of targeted literature reviews and two surveys, with the purpose of identifying which contextual features are likely to have most relevance to and impact upon a diverse range of five PSP's. We used an expert panel, given that the issue of the importance of context in patient safety is relatively nascent. It was formed as part of a larger study and was composed of international experts in specific patient safety practices; methodologists from fields including epidemiology and statistics, program evaluation, organisational behaviour and human factors engineering; senior health system executives responsible for implementing safety practices; and leaders of national and international patient safety organisations. Throughout this study, an iterative deliberative process with the panelists was used, involving structured discussions (first by telephone and then face-to-face) and two surveys.
Selection of PSPs
The authors grounded their efforts to select contextual features using five diverse and representative PSPs. To select the PSPs, a list of possible PSPs was generated from a variety of sources, including the AHRQ Evidence-based Practice Report Making Health Care Safer 22, the National Quality Forum's Safe Practices 2009 update,23 and input from relevant organisations including AHRQ and the Institute for Healthcare Improvement.24 Through an iterative process of developing a framework to classify PSPs based on their diversity and representativeness, and based on input from the expert panel, five PSPs were selected:25
the universal protocol to prevent wrong site, wrong procedure, wrong person surgery;
medication reconciliation;
computer physician order entry and computer decision support systems;
practices designed to reduce the rate and risk of falls; and
use of a checklist to prevent blood stream infections. These were representative of PSPs in different settings (eg, hospitals, clinics and nursing homes), and addressed both PSPs that were subject to regulatory authority and those that were not, and those that were aimed at preventing very rare and those that targeted more common safety events.
Selection of initial set of contexts and first survey
To guide the panelist discussion and surveys, first, a list was compiled of contextual features reported in seminal PSP papers and these were grouped into nine broad conceptual domains. The authors recognised that some contexts might affect the implementation of only some PSPs while other contexts might affect the interventions of most PSPs. Given the long list of potential contexts that could have been examined, the authors tried to examine those contexts that might be important to many PSPs. (However, the panelists were specifically asked in the second survey (table 1) which of the five PSPs each context might be important to.) Also, some of the context features could be considered interventions (eg, the presence of information systems and decision support). When these already exist across both intervention and control sites in an experimental design, and the intervention being evaluated is independent of or additive to them, such features represent part of the contextual background to the evaluation. Given the purpose of this study was to examine how context might affect PSP implementations, the panelists’ discussions and survey responses of contexts were based on the features as contexts and not part of the intervention.
Contexts judged as high priority for assessing (A) their effects on PSP implementations or only for describing (D) in papers
The authors then discussed the domains and the specific contextual features during the first meeting with the panelists to help determine on which features panelists should be surveyed. After the meeting, a web survey was used to ask the panelists to rate the relative importance of the contextual features to each of the five PSPs, from ‘not very important’, ‘important’, ‘extremely important’, to ‘important or extremely important only in some settings’, and to ask them to consider nursing home, inpatient, outpatient, and community settings. The purpose of the first survey was to guide the development of the second survey and the context taxonomy (as described below).
Development of the context taxonomy
A taxonomy of important contextual features was developed using the features that were rated in the first survey as ‘important’ across all five PSPs or ‘very important’ across four or five PSPs. Lower scoring contextual features, while acknowledged as potentially important, were not discussed further. This taxonomy was first proposed by the project team and then discussed and agreed upon by the panelists as an easy way to conceptualise the various types of contexts that might affect PSP implementations. The panelist exchanges also resulted in some contextual features being divided into more homogenous groups and other new features being added and included in the second survey.
Second survey
The authors held a second intensive discussion about contexts with panelists and based on this discussion and the results of the first survey, a second web survey was designed for panelists to prioritise the list of contextual features. For each feature, the following question was asked: ‘Is it a high priority to collect data on [context] for describing the context and/or for assessing the effect of context on this PSP implementation?’ Response options were: 1. yes, it is a high priority for assessing the influence of context on the success of that implementation, 2. yes, it is a high priority only for describing the contexts, and 3. no, it is not a high priority. Both this survey and the first survey had a 95% (n=22) response rate. The authors conducted a descriptive analysis (frequencies) on the results of the surveys.
Results
Box 1 shows the 42 contextual features mentioned in the seminal PSP papers and how the project team organised them into nine conceptual domains. One of these domains was labeled ‘cultural organisational factors’, and one of the contexts within that domain was ‘safety culture’. It is worth noting that there are numerous definitions of organisational culture in general,26 27 including the informal ‘the way we do things around here’. On safe culture specifically, Pronovost and Sexton28 suggest that, ‘In a safe culture employees are guided by an organisation-wide commitment to safety in which each member upholds their own safety norms and those of their co-workers.’ When the panelists reviewed this list, they had a lively discussion on what contexts might affect PSP implementations. Some argued for using theory on which to select contexts because so little work has been done to determine which contexts are important; others acknowledged that some of the contexts mentioned could be part of the intervention in some circumstances, but recognised that our discussions would focus on the situations when those context features were external to the intervention; others mentioned that work has been done in other fields such as human factors and business management to examine the effect of contexts; and another panelist felt that ‘context’ should include anything one needs to know to replicate the study. The panelists agreed that our survey should include most of the contexts on our proposed list and they suggested a few additional contexts.
Contexts mentioned in published seminal evaluations of patient safety practices (PSPs)
External environmental factors
External PSP-related regulations or policies
Rural versus urban environment
Geographic location other than rural/urban
Medical payment environment
External incentives/motivators
Structural organisational demographics
Unit or organisational characteristic
Academic affiliation
Organisational size
Organisational affiliations other than academics
Economic status
Structural complexity
Internal organisational incentives
Cultural organisational factors
Safety culture
Teamwork
Innovativeness
Readiness for change
Need for change
Collaboration, resources and leadership
Collaboration across healthcare professionals
Collaboration with health services researchers
Collaboration with quality improvement experts or facilitators
Quality improvement or innovations funding
Leadership involvement
The presence of information systems and decision support
Information systems
Electronic medical record
Computer reminders
Guidelines
Training resources
Care coordination, management and patient self-management
Care coordination
Care management
Delivery system design
Professional staffing
Provider knowledge, attitude and skills
Knowledge/skill
Patient safety or quality improvement knowledge/skill
Attitudes
Evidence base
Ease of adoption
Type of healthcare provider
Patient and encounter complexity elements
Patient clinical complexity
Patient attitudes
Patient activation/engagement
Urgency of the patient/clinician encounter
Continuity of patient/clinician encounter
Patient transitions
The results of the first survey showed the panel considered 19 of the 42 contextual features as important or extremely important across the PSPs. The results of this survey and discussion with the experts were then used to shape the second survey (as noted below). The results were also used to develop a taxonomy of four conceptual domains of contextual features that are important for PSP implementations, as shown in box 2. When the panelists were presented with the results of the first survey, they agreed with the results. However, based on the results, the panelists also thought that the second survey should ask them about a few additional contexts and that the group of contexts should continue to be diverse in terms of their degree of universality, how tactical, psychosocial or mutable the contexts are, how much empirical evidence exists for the contexts' importance, and the number of PSPs to which the contexts related.
Results of first survey: four domains of contexts judged as important for patient safety practices (PSPs)
Safety culture, teamwork and leadership involvement.
Structural organisational characteristics (eg, size, organisational complexity or financial status).
External factors (eg, financial or performance incentives or PSP regulations).
Availability of implementation and management tools (eg, training resources or internal organisational incentives).
The results of the second panelist survey on the priority of a modified list of contextual features are shown in table 1. Panelists tended to rate the six patient safety culture, teamwork and leadership contexts as high priority for assessing their effects on PSP implementations rather than for only describing these features in evaluations or not a priority at all. One exception to this pattern is noteworthy: panelists tended to report that patient safety culture at the level of the hospital or clinic organisation is important only for describing in studies (not for assessing their effect). Moreover, in post-survey discussions with the panelists, they recognised that the items in the patient safety culture/teamwork/leadership domain likely represent distinct attributes of context.
Panelists tended to rate the 12 structural organisational characteristic contexts (eg, size or location), as high priority only for describing in PSP evaluations. However, again, two organisational contextual features were noteworthy exceptions: at least a third reported it was a high priority to assess the effects of existing quality/safety infrastructure and organisational complexity on PSP implementations.
For the five external contexts (eg, regulatory requirement and payments or penalties) and the 11 implementation or management tools (eg, internal incentives and staff training), panelists appeared somewhat split overall on whether the contexts are a high priority for assessment or a priority only description in evaluations.
Finally, of all the contexts examined, panelists tended to report that only two are not a high priority at all (not for describing or assessing): teamwork at the organisational level (for bloodstream infection checklists and for the universal protocol) and marketplace competition (for bloodstream infection checklists).
Discussion
This paper reports on the use iterative, formal discussions with a 22-member technical expert panel comprised of experts or leaders in patient safety, healthcare systems, and methods to determine what contexts are likely to have major influences on PSP implementations and which should be described in PSP evaluations. Given a long list of potentially important contextual features, the experts were able to reach agreement that four conceptual domains of contexts, and some specific contextual features within those domains, are important.
This study has several strengths, including the use of a heterogeneous, interdisciplinary panel of experts and the opportunity to have both face-to-face and telephone meetings with them to iteratively and intensely deliberate the contexts that might be important for patient safety interventions. Perhaps the biggest strength of this paper is that it aims to help build the consensus that has been lacking, given the infancy of the field, and creates categories of context that can be the focus of future research.
The limitations of this study include our potential omission of some contextual features that others consider important for assessing PSP interventions or for describing in evaluations. Admittedly, the process by which the contexts were selected and the opinions of the panelists were subjective, such that another set of experts might have derived a different set of context domains than the ones derived here. It is likely that the panelists' academic background, degree of medical experience, and other factors affected their decisions. However, the authors intentionally selected a heterogeneous group of panelists so that their overall opinions would be balanced. In addition, the taxonomy was not validated. However, the efforts presented in this paper should be considered the first step in a longer-term, iterative process of subsequent research. Of course, there is as yet scant evidence that the contexts selected actually do influence the effectiveness of patient safety interventions. However, the purpose of this study was to try and get the field to focus their measurement and reporting on those contexts believed to be potentially the most important. It is only by getting patient safety evaluators to focus on a common set of contexts that an evidence base can be built about which contexts really are important—the authors expect a future version of this process, informed by better evidence, to refine this list of contexts, possibly dropping some and adding new ones.
Several contextual factors within four conceptual domains of contexts were found to be important. Readers familiar with the evidence-based medicine literature might wonder about the applicability of the ‘PICO’ framework29 for categorising contexts as an alternative. In that framework, study questions are categorised into population, intervention, comparison group, and outcome groups (PICO). That appears to be a useful framework for categorising some contexts. For example, the ‘population’ category could include contexts that are characteristics of the organisational target group (eg, culture, teamwork, leadership) and setting (eg, structural organisational characteristics and external factors). However, the ‘intervention’ category in the PICO framework could include many of the contexts examined here because many could be the focus of an intervention. Likewise, the ‘comparison’ category in the PICO framework could include almost any context in the group not receiving the PSP intervention. The ‘outcome’ category might be less applicable to the contexts examined here. As such, the PICO framework might not lend itself to the efficient categorisation of contexts.
The authors suggest future PSP implementation studies measure, describe, and perhaps assess the effects of the contextual features that experts reported in this paper as high priority. Assessing some potentially important contexts would help build the evidence for the importance of some contextual features. If future evaluations more richly described the contexts in which PSP interventions took place, using some of the contextual features mentioned here, readers might be better able determine if the study is likely to work in their own set of circumstances. Guidelines such as CONSORT30 and SQUIRE31 32 have already been developed for reporting interventions. Perhaps these or other existing guidelines can be modified to include the reporting of contextual features when PSP implementations are reported. The results presented here could be the first step towards that effort.
Readers of research papers on PSP implementations need to know about context factors but also about the rigour of study design to draw conclusions about effectiveness. The technical expert panel and this paper focused on the first of these considerations, however additional research is needed to address the second. Also, despite consensus on the context domains presented here, the existing literature suggests the ability to accurately measure the contextual features comprising these domains is immature. Although the authors encourage researchers to describe or assess several contextual features in evaluations, they recognise the need for ‘basic research’ in patient safety to advance the science of measuring the contextual features.
References
Linked Articles
- Original research
- Original viewpoint
- Original research