The challenges in responding to external demands for performance and improvement

The Kelly paper addresses a narrow but important area of process and quality improvement—responding to external standards and expectations of performance. One of the key questions implicit in the Kelly paper is how similar or different the facilitators and barriers to successful implementation of externally demanded actions are from internally driven improvement efforts. For example, given that much of the implementation literature emphasises the importance of local champions and internal identification of the need for improvement, responding to external demands for change might be expected to impose additional organisational stresses and challenges than internally promoted efforts. However, despite the focus on externally driven efforts, many of the themes identified by Kelly and colleagues are similar to those identified in other implementation efforts. Among the authors’ findings are the importance of managerial leadership and commitment to implementing the standards, and the recruitment and availability of personnel who will act as champions and role models. Where the work is not internally driven, then creating internal commitment and engagement becomes one of the additional challenges to successful implementation. Also found to be important to implementing external standards are educational and other materials to explain the content of the standards, and credibility within the organisation that the standards will be associated with safety and quality improvements. Again, these challenges are not unique to externally imposed standards. While the synthesis notes that much of the implementation literature focuses on internally driven, that is, ‘bottom-up’, efforts and that more attention needs to be paid to ‘top-down’ processes, the synthesis offers only limited discussion of how implementation might differ for externally imposed standards compared with internally developed and championed efforts and how these might be studied.

For example, by definition, externally imposed standards relate to the ‘outer setting’ of the CFIR and the ‘outer context’ of Greenhalgh and colleagues’ framework. What is perhaps most interesting is that little is found in the Kelly paper about the strength of incentives, expectations of regulators or accreditors, or the perceived penalties of non-compliance with the standards as factors in motivating change or encouraging failure in implementation. The most significant finding in this realm is that monitoring and external assessments may be inconsistent across agencies and thus undermine standards’ credibility or view that the standards are broadly supported by the relevant external parties. By contrast, benchmarking and reporting were found to be important facilitators. Financial incentives as motivators were found to have only moderate support in the literature reviewed.

The authors note that there has been a lack of recognition across implementation science for characteristics of the political and outer context environments. More can be done with these findings and more broadly to better differentiate the organisational challenges of externally versus internally driven improvement.

Other factors influencing implementation

Across the themes describing barriers and facilitators, only a limited range of issues were identified with respect to the content or nature of the standards themselves. Adaptability, perceived impact on quality, concerns that implementing the standards might negatively affect care, and relationships among providers or between providers and patients stand out in the synthesis. Simplicity, feasibility and tailoring to implementation were identified as facilitators. Interestingly, the extent to which the standard calls for change of practice was not identified as a barrier.

Many of the identified facilitators and barriers relate to culture and resources. Some easily map to the CFIR and Greenhalgh and colleagues’ frameworks, like the roles of champions, collaboration and staff engagement. The biggest addition this study makes to the literature on implementation is highlighting as facilitators the importance of educational materials, of explicit efforts to educate staff on the standards and provide them formal support (via materials and staff education), and of staff expertise. While these are present in the formal frameworks, they do not stand out as clearly as they emerge in the current systematic review.

The value of a structured approach to systematic reviews

A key contribution of the systematic review by Kelly and colleagues is its formal implementation of the Sandelowski and Barroso framework for synthesising qualitative research and constructing a meta-summary.5 6 Adoption of this approach provides additional structure to the review process and creates additional understanding of the evidence base being summarised. The paper provides a powerful example of the value and use of this framework.

Once the literature to be reviewed was identified, Kelly and colleagues structured their work into four components. The first was assessment of the conclusions regarding implementation, which was drawn from the discussion sections of the papers reviewed. Conducting systematic reviews can be complex, and this approach avoids the need to reanalyse or reassess the data in the analysis. Given the large number of studies included in the review that were based on surveys of stakeholders or participants in the implementation process, relying on the original authors’ assessments of their findings rather than reviewing and interpreting reported survey results substantially reduces the workload. This approach is, however, dependent on the initial authors correctly interpreting and presenting their findings.

The second component is the qualitative analysis of this data set and construction of themes and other analyses. This draws upon standard approaches to qualitative analysis and is well reported by Kelly and colleagues.

The final two components add structure and information to the synthesis. The third component involves the authors evaluating the strength of each study based on the methodology employed. This methodology-specific assessment offers more detail and relevance than the use of a generic framework for evaluating research quality. The fourth component of the synthesis goes beyond the assessment of the strength of individual studies to provide a formal analysis of the strength and coherence of their overall findings, taking into account the number of studies, consistency of findings across studies and strength of the individual studies used as evidence for their constructed themes and subthemes. This is an important addition to the methods for systematic reviews. As one examines tables 1 and 2 in Kelly and colleagues’ paper, and the colour coding of the assessment of the strength of analysis, one is struck by how many of the conclusions merit high confidence, and how few are characterised as low confidence. Among facilitators, the conclusions with the lowest confidence are those on the facilitating value of budget and facilities. With respect to barriers, the findings with the lowest confidence are those related to judgements that the standards are not in fact norms for high-quality care and that inconsistent external judgements and use across monitoring agencies create barriers to implementation.

Kelly and colleagues also provide an analysis of their database, using two measures: the frequency effect size, that is, the proportion of studies in which the facilitator or barrier is reported, and the intensity effect size, which is a study-level measure of the proportion of identified facilitators or barriers that are identified in an individual study. Both provide information about the database for the systematic review, although neither should be interpreted as a measure of importance for a given facilitator or barrier. One of the strengths of the approach taken by Kelly and colleagues is that it identifies facilitators and barriers that are salient based on the extent to which studies have identified them as such. But salience is only imperfectly correlated with importance, and the frequency with which a factor is mentioned across studies is likewise only imperfectly correlated with importance. While we might draw the conclusion from the study of actual implementation activities that items in the CFIR or other frameworks that are never mentioned are unlikely to be important facilitators or barriers, the opposite conclusion cannot be drawn. Facilitators or barriers that are frequently mentioned may vary substantially in the extent to which they aid or impede implementation. We need different methods, analyses and metrics to measure how large the effect of a given facilitator or barrier is. That is one of the critical challenges for future research on implementation.

Moving forward

Future research or systematic reviews of past research need to further analyse how challenging or easy it is to overcome a barrier and how much tailwind having a specific facilitating factor is in moving implementation forward. There is also a need for studies that help organisations understand not only how important barriers or facilitators are, but particularly how barriers can be reduced and facilitators can be cultivated. Are some barriers more easily overcome, or can success be achieved without fully addressing some barriers? These are critical questions that organisations need guidance on, given the limited time and resources often available for implementation, and the need for organisations to demonstrate success in these activities.

More research is also needed on the variance in performance in successful implementation, to understand who is more or less likely to be successful in making and sustaining change. A recent study examining the determinants of regulatory compliance in health and social care services, largely drawing on studies of nursing homes, found higher compliance in smaller facilities and those with higher nurse staffing levels and lower turnover.7 More studies examining variation in performance, the factors associated with variance and examination of the characteristics of positive deviants will strengthen our understanding of implementation and improve the likelihood that organisations can succeed in meeting both external demands and internal drivers to increase safety and quality.