Article Text
PDF
Abstract
While the body of evidence-based healthcare interventions grows, the ability of health systems to deliver these interventions effectively and efficiently lags behind. Quality improvement approaches, such as the model for improvement, have demonstrated some success in healthcare but their impact has been lessened by implementation challenges. To help address these challenges, we describe the empowerment evaluation approach that has been developed by programme evaluators and a method for its application (Getting To Outcomes (GTO)). We then describe how GTO can be used to implement healthcare interventions. An illustrative healthcare quality improvement example that compares the model for improvement and the GTO method for reducing hospital admissions through improved diabetes care is described. We conclude with suggestions for integrating GTO and the model for improvement.
- Quality improvement
- Evaluation methodology
- Evidence-based medicine
- Healthcare quality improvement
- Implementation science
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
- Quality improvement
- Evaluation methodology
- Evidence-based medicine
- Healthcare quality improvement
- Implementation science
Why should efforts to implement evidence-based interventions in healthcare consider including an empowerment evaluation?
Improving healthcare quality has become a national priority in recent years. Landmark reports, especially the Institute of Medicine's ‘Crossing the Quality Chasm: A New Health System for the 21st Century’,1 and efforts, such as the Institute for Healthcare Improvement's vision for achieving the triple aim (improved safety, population health and decreased cost),2 have laid a foundation for transforming the healthcare system. Quality improvement initiatives based on industrial process improvement methods to implement evidence-based solutions and improve outcomes in clinical practice3 have grown in popularity. While interest in such initiatives is growing, quality improvement interventions often fail to achieve consistent outcomes.4–6 An emerging literature suggests that these disappointing results may be due, in part, to challenges related to implementation.6–8 Since quality improvement interventions are implemented within the context of complex dynamic systems,9 ,10 issues such as fit of the selected intervention to the organisational context, adaptation, engagement of staff and other stakeholders, and buy-in from leadership are all factors that affect the successful implementation and adoption of these interventions.6–8
In this article, we offer empowerment evaluation11 as a potential solution to address the above factors. We describe the empowerment evaluation approach that has been developed by programme evaluators and a method for its application (Getting To Outcomes (GTO)). We discuss how GTO (see acknowledgements) can be integrated with a widely used healthcare quality improvement approach (specifically, the ‘model for improvement’) to facilitate improvements in health outcomes. We also explain why those who want to successfully implement healthcare improvement interventions should consider including empowerment evaluation. We will use an illustration of shared medical appointments for persons with diabetes designed to reduce hospital readmissions and improve diabetes care as an example of the concepts. We then propose next steps for integration of the model for improvement and empowerment evaluation that builds on strengths of each approach.
What is empowerment evaluation?
Empowerment evaluation is defined as ‘…an evaluation approach that aims to increase the probability of achieving program success by providing program stakeholders with tools for assessing the planning, implementation, and self-evaluation of their program, which mainstreams evaluation as part of the planning and management of the program/organization’ (p. 28).12 Table 1 provides an overview of the empowerment evaluation definition and principles, and of the two main ‘how-to’ approaches. Empowerment evaluation integrates evaluation concepts and techniques within an empowerment framework in order to foster improvement, buy-in and self-determination among programme stakeholders.13 Empowerment evaluation is guided by 10 principles: improvement, inclusion, democratic participation, social justice, capacity building, organisational learning, community knowledge, community ownership, evidence-based strategies and accountability.12 In empowerment evaluation, the term ‘community’ can be conceptualised broadly to include geographical, social and organisational communities, including healthcare organisations and population health communities. A central premise of empowerment evaluation is that programmes are more likely to achieve desired outcomes if key stakeholders (eg, healthcare providers, clinic administrative staff) have the capacity to conduct and use their own evaluations. In this way, empowerment evaluation is designed to improve programme implementation and strengthen systems by building local capacity for planning more systematically, implementing with quality, self-evaluating and using the information for continuous quality improvement.11
Overview of empowerment evaluation
Empowerment evaluation differs from traditional evaluation approaches in several ways. In contrast to traditional evaluation, which typically views the evaluator as neutral about whether a programme achieves results or not and reports the facts (usually in retrospective reports),14 empowerment evaluation has a strong value to help the programme succeed; it works with programmes that have positive goals (eg, less homelessness, less child obesity) and would like the programme to be successful. Therefore, it is proactive in putting the logic and tools of evaluation into the hands of practitioners so that they can carry out the programme more successfully. If traditional evaluation is more like a postgame analysis in sports (ie, providing after the game analysis by an outside expert), then empowerment evaluation is more like providing the coaching staff and players with tools for planning for the game, monitoring implementation, making mid-course corrections, evaluating the results of the game and planning for future games. Empowerment evaluation shifts the roles of the evaluator and programme stakeholders. In empowerment evaluation, programme staff and participants are responsible for the programme and the evaluation, and the role of evaluator is shifted to that of a teacher, facilitator and coach.13 In this way, empowerment evaluation can alleviate the tension that often exists between evaluators and implementers.15
Some evaluators have raised a concern that empowerment evaluation or empowerment evaluators may be biased in their reporting because they would like the programme to achieve outcomes. This and other critiques of empowerment evaluation are discussed in depth by Fetterman and Wandersman.16 Fetterman and Wandersman address the bias concern by stating that the bottom line in empowerment evaluation is the same as in traditional evaluation, for example, honest reporting of patient-centered outcomes as measured by standard metrics.5 At the same time, empowerment evaluation recognises that the pathway to achieve these outcomes may differ by individual or organisation, and that stakeholders need to have the capacity to define needs, develop and conduct evaluations and implement and improve interventions to achieve intended outcomes while remaining alert to the quality of the implementation. Additional concerns or challenges in conducting empowerment evaluation include that it takes time and money to educate staff about the knowledge and tools for conducting such evaluations. Wandersman et al17 developed a model of support to build capacity to evaluate and implement with quality; the system provides tools, training, technical assistance and quality assurance/quality improvement. Any intervention that builds capacity of systems to improve takes effort, and the benefits and costs of conducting empowerment evaluation should be carefully weighed in advance. Though implementing empowerment evaluation can take additional financial and human resources, research indicates that investing in a system of support will improve the probability of achieving outcomes and promote the development of capacities needed for long-term sustainability.6
How-to approaches for incorporating empowerment evaluations into healthcare quality improvement initiatives
There are two major ‘how-to’ methods of empowerment evaluation: the Fetterman three-step approach18 and GTO. In this article, we focus on GTO because of its more comprehensive approach to planning, improvement and accountability. GTO incorporates 10 separate literatures in its comprehensive approach to planning, implementation, evaluation, continuous quality improvement and sustainability; therefore, it is more comprehensive than (and takes longer) than the three-step approach. GTO is a results-based approach designed to facilitate capacity building and accountability.19 Stakeholders apply principles of empowerment evaluation by engaging in a process of asking and answering 10 accountability questions in order to obtain and sustain positive results (table 2). Tools, training and technical assistance support the GTO 10-step process and are tailored to specific content areas. To date, GTO has been customised for several health contexts, including clinical settings (eg, behavioural health services, training for clinical psychology doctoral students) and public health initiatives (eg, home visiting programmes, substance abuse prevention). A book edited by Fetterman, Kaftarian, and Wandersman20 provides examples of how empowerment evaluation has been successfully implemented in schools, community-based organisations and community coalitions. A growing literature demonstrates positive results associated with using GTO, including better programme performance and greater fidelity.21 Emerging evidence also suggests that GTO is associated with improved outcomes, including reductions in underage drinking.21 In this article, we propose extending GTO to healthcare settings where it can supplement and enhance existing quality improvement approaches.
Getting To Outcomes accountability questions
GTO is a multilevel system approach that encourages collaboration. While responsibility for the 10 GTO steps can be shared across multiple system levels (eg, community, hospital, physician, patient), its enhanced power in healthcare settings includes its ability to bring evaluation principles to the individual level. For example, within a hospital, providers seeking to improve care may work collaboratively with patients to have individualised treatment plans that: identify needs and resources, develop goals and desired patient-level outcomes and identify how to apply evidence-based clinical practices in a way that fits the patient's values and health system's context, assesses for sufficient capacity to implement the treatment plan, evaluates the implementation of the plan and the outcomes, and has a plan for improving and sustaining the intervention. At the hospital level, GTO can provide support for implementing an evidence-based plan that fits the needs and resources of the hospital for patients with a particular diagnosis. GTO thinking can extend beyond the hospital to population health. Later in this article, we provide a multilevel illustration of the GTO method in a programme of shared medical appointments in a clinic designed to reduce hospital readmissions for persons with diabetes. We also illustrate how GTO enhances implementation, self-evaluation, buy-in and ownership.
Integrating tools and methods of empowerment evaluation with quality improvement efforts: an example
Traditional methods to encourage the adoption and implementation of evidence-based interventions in healthcare settings have been based on quality improvement approaches such as Lean, Six Sigma, Plan-Do-Check-Act/Plan-Do-Study-Act (PDCA/PDSA) and the model for improvement, which were adopted from industrial engineers.6 While the scientific principles behind these approaches are conceptually sound, the evidence about the best method to apply them remains inconclusive. For example, researchers undertaking a recent systematic review of the use of PDSA22 as a quality improvement tool in healthcare point to inconsistent implementation as one of the factors that make it difficult to rigorously assess PDSA's effects on improving health outcomes. Shortcomings include the lack of systematic implementation tools and a common vocabulary.
Since the model for improvement has been used most often for implementation of clinical interventions, we will use it as our example of a quality improvement approach; the points made in this article apply to the other improvement approaches as well. The model for improvement, shown in figure 1, has a long history, beginning with Shewhart and Deming in the first half of the 20th century.23 The model emphasises system change and focuses on reducing variation in process performance. It includes an experimental approach to improvement involving sequential identification and selection of small changes that are then tested in multiple PDSA cycles. We propose that empowerment evaluation, in combination with the model for improvement, could enhance the capability of healthcare systems to improve quality of healthcare delivery and outcomes.
Model for improvement.
To illustrate this combination, we present a hypothetical example of a primary care clinic that seeks to reduce admissions of patients with diabetes to hospitals that result from poor management of the disease in outpatient settings. We illustrate how the model for improvement and empowerment evaluation supports the quality improvement initiatives and then describe how integration of these approaches has the potential to produce a whole that is greater than the sum of the individual parts.
Reducing hospital admissions due to poor disease management using the model for improvement
A quality improvement team will begin by examining historical patterns of frequent hospital admissions of patients with diabetes using run charts or statistical process control charts. If these charts indicate that there is a higher hospital readmission rate relative to other clinics within the system, or if there are special non-random patterns over time such as spikes, runs or cycles, the quality improvement team will take on a project to tackle this issue at the clinic by improving the process of diabetes care. Following the steps of the model for improvement, data will be used to explore where problems occur in the diabetes care process and the drivers of these problems. Based on answers to these questions, the team will identify a set of potential solutions using a combination of internal brainstorming, literature review and benchmarking with other clinics. Some of these solutions could include individual self-management, group self-management or shared medical appointments. Through testing using Model for Improvement (MFI), shared medical appointments, an approach in which groups of patients meet regularly for comprehensive care, may be selected as the most promising solution.24
The question now becomes how to implement shared medical appointments in the clinic in a way that ensures that the desired outcome of reduced hospital admissions is achieved. PDSA cycles can also be used to do this. As an extension to the improvement cycles used to test multiple solutions, implementation cycles can be used to test implementation approaches for a single solution, such as shared medical appointments, and iterative improvements to implementation can be made until a satisfactory implementation model suitable to the context of the clinic is achieved.
Implementing shared medical appointments to reduce hospital admissions for patients with diabetes using GTO: a further illustration
We now illustrate how GTO would approach the same problem, following the steps in table 2. The first three evaluation steps correspond roughly to the three fundamental questions of the model for improvement in figure 1, and are designed to find an appropriate solution to the problem. To begin the GTO process at the health clinic, the leaders of the health system work with personnel in the health clinic to develop a GTO team, comprising clinic administrators, healthcare providers and patient representatives. The GTO team begins with Step 1: Needs and Resources Assessment and addresses the needs and resources of the clinic's patients. The team answers these questions using methods that include data compiled from electronic health records and interviews with health clinic staff and patients. The data suggest that the majority of persons with diabetes are diagnosed with type II diabetes and that providers find it difficult to address behavioural and psychosocial determinants of diabetes management in 15 min appointments. Resources identified include: staffing person/hours, administrative and healthcare support (including the use of the electronic health record system).
In Step 2, the GTO team sets goals and objectives to determine progress. Findings from the needs and resources assessment are used to identify goals and desired outcomes for the clinic's patients. The GTO team sets goals at both the patient (eg, reduce individual blood glucose levels within 6 months) and clinic levels (eg, decrease hospital admissions due to disease management, decrease healthcare costs). In Step 3: Best Practices, the GTO team identifies evidence-based practices that may be used to reach goals set in Step 2. The team reviews the literature and discusses options with other staff knowledgeable about strategies for addressing diabetes. Several options for meeting the goals include: (1) individual self-management, (2) group self-management and (3) shared medical appointments. Introducing shared medical appointments is attractive, since research suggests shared medical appointments have demonstrated positive patient benefits (eg, improved blood sugar values) and clinic benefits (eg, reduced clinic visits for diabetic patients).24 At the end of Step 3, the GTO team arrives at the same conclusion as the model for improvement team—shared medical appointments are a good best practice to consider using in the clinic.
Steps 4–10 are roughly analogous to how the model for improvement recommends the use of multiple PDSA cycles to test changes until there is enough learning to implement. However, GTO approaches implementation in a more systematic way and adds several tools that are not typically available in the model for improvement approach (see tools in the GTO manuals http://www.rand.org/health/projects/getting-to-outcomes.html). For example, in Step 4: Fit, the team discusses the fit of shared medical appointments with frontline staff and patients to assess the fit of shared medical appointments with patient's values, preferences and life situation along with the fit within the system and broader community context. This step requires deep understanding of the intervention and prevents implementers from rushing to implement a solution without truly understanding how the intervention works in a specific setting. Implementing without understanding, called ‘cargo cult science’ by Richard Feynman, may affect the effectiveness of even well-tested interventions.25 In Step 5, Capacity, the team uses tools to examine patient and healthcare system capacities (eg, technical, human capital) needed to implement shared medical appointments. After a review of results from Steps 4 and 5, the team decides that shared medical appointments fit well within the context at the clinic and that both the clinic and the healthcare system have the capacity (or can develop needed capacities) to implement shared medical appointments with quality.
In Step 6, the team works to develop a plan for implementing the shared medical appointments. It uses the Quality Implementation Tool,26 which includes defined action steps to complete the following components: (1) develop an implementation team, (2) foster supportive organisational climate and conditions, (3) develop an implementation plan, (4) receive training and technical assistance, (5) practitioner–developer collaboration in implementation and (6) evaluate the effectiveness of the implementation. The Quality Implementation Tool is based on a synthesis of 25 implementation science frameworks27 and uses implementation science to enhance the likelihood that desired outcomes are achieved. Through the use of the tool, the team develops a structured plan for quality implementation that operationalises implementation science components, including planning the logistics of holding shared medical appointments and how and when appropriate training should occur. The team also uses the Quality Implementation Tool in Step 7: Implementation and Process Evaluation, to guide monitoring whether treatment is being delivered as planned, assess whether the treatment is causing any side effects, and determine whether progress is being made towards treatment goals. The team monitors implementation science process indicators such as dosage, patient responsiveness and quality of delivery. Results from the process evaluation inform mid-course corrections. For example, feedback from the administrative staff may indicate that some patients are not attending appointments regularly or that patients indicate that they forget their appointment times. As a result, the team implements regular appointment reminder calls.
In Step 8: Outcome Evaluation, the team assesses attainment of both the patient and clinic outcomes described in Step 2. Using data collected through patient and clinic records, they evaluate if patients’ blood sugar levels are improving, if hospital admissions of patients participating in shared medical appointments are lower than rates of similar persons with diabetes who are not attending shared medical appointments, and if the overall cost for diabetes treatment is lower than before the group treatment was introduced. Results suggest that persons participating in shared medical appointments are demonstrating significantly improved blood sugar levels and have fewer diabetes-related hospital visits. Results gathered in Step 8 are used to improve the quality of the shared medical appointments in Step 9: Continuous Quality Improvement. The team discusses options for adjusting the shared medical appointments in order to improve the quality of implementation and overall patient satisfaction. Additionally, patients are consulted in order to understand how the shared medical appointment approach is working for them and individual treatment plans are performed to improve the quality of care and further improve outcomes. For example, what should be done about patients who drop out of shared medical appointments?
In Step 10: Sustainability, the GTO team at the clinic discusses plans for sustaining the progress. They are interested in understanding how changes made in treatment can be sustained for patients and how the clinic can continue to sustain the shared medical appointment approach. Using the sustainability toolkit, which is part of the GTO package and provides strategic processes for planning, implementing and evaluating sustainability plans, the team at the clinic is able to ensure that the capacities necessary for sustainability are in place to promote the lasting impact of the shared medical appointment approach.
Summary of the examples
As is evident from the two examples, the model for improvement and GTO emphasise tools at different places in the process of achieving outcomes. The model for improvement tends to be more systematic at the front end, especially relating to analysing the system, evaluating patterns of performance over time, understanding causes and sources of variation in performance, identifying drivers of poor performance and selecting solutions to address these drivers. In comparison, GTO offers more tools later in the process, where it is important to understand if the organisation has the appropriate capacities, whether the solution fits with the organisation's culture and whether the implementation takes place with quality. In addition, with its emphasis on ownership, participation and individual and organisational evaluation capacity building, GTO assesses the organisation's needs to have the capacity to adapt the solution to fit an individual patient. We propose that an integration of quality improvement methods along with GTO will ultimately result in more sustainable solutions.
How can empowerment evaluation fill gaps in current improvement approaches?
How specifically could GTO and improvement approaches be integrated in healthcare quality improvement initiatives? Since the details of the integration are areas for future development, we will restrict our closing comments to proposing how GTO, as a ‘how-to’ for empowerment evaluation, can enhance the model for improvement. Implementation, adoption and adaptation are three examples of opportunities for empowerment evaluation to add value to established improvement methods.
There are several limitations to the application of the model for improvement to optimise implementation. First, since there are no tools to guide the actual process of implementation in the ‘DO’ step of PDSA, as reported in the literature, there is the possibility that the model is applied inconsistently and incompletely, resulting in an implementation model that is not sustainable. While the lack of an implementation strategy may not be a major issue during the problem solving phase where multiple solutions are tested, it is a major barrier in the implementation phase where a clear understanding of local capacity and readiness is necessary. GTO can support systematic implementation of selected improvement solutions. In the clinic example, the detailed analyses performed by the improvement team to select shared medical appointments as the solution of choice will be wasted if they are not implemented well enough. The fit and capacity assessments and the Quality Implementation Tool are examples of GTO tools that can ensure that implementation elements that are critical to success are addressed.
Second, the empowerment evaluation framework provides an approach to local accountability that facilitates adoption of improvement solutions. At the end of multiple PDSA cycles, if the ‘Study’ step determines that the desired improvement has been achieved, the decision may be made in the ‘Act’ step to implement. This necessitates the development of standard documentation, training and operating procedures to stabilise process operations. However, no tools are provided in the model for improvement to drive these processes and guidelines into the organisational culture.28 GTO explicitly assesses organisational readiness and fit, its approach to implementation provides tools to increase organisational buy-in for supporting change efforts, and incorporate change management activities as an integral part of the implementation. This assists with organisational adoption of solutions, which is a critical requirement for sustainability.
Third, GTO can help with adaptation of evidence-based processes to specific contexts. The model for improvement will help to identify the optimal standard processes through which re-hospitalisations may be reduced, but does not provide the evaluation capability at the individual provider level to assess where adaptations to this process are acceptable and necessary, and to determine when and how to do this for an individual patient. In healthcare improvement, both standardisation and individualisation are important. Standardisation is important for efficiency, uniform adoption of safe practices and consistent patient experience at an organisational level. However, with emerging emphasis on patient-centred care, there is also a need for adaptive and proactive decision making at the individual patient–provider interface. Empowerment evaluation methods such as GTO, with their emphasis on accountability, ownership and self-determination can provide systematic guidance for providers to evaluate implementation of standard care processes and to make selective adaptive modifications for individual patient needs. Individual providers can be explicitly empowered to evaluate processes in day-to-day operations and identify improvements that can be shared with peers for sustained ongoing improvement.
Conclusion
In this article, we described two approaches (GTO and the model for improvement) that have been applied in healthcare and other fields to design, implement, evaluate and improve effective and efficient processes. Both approaches are used for improvement and are also self-evaluating, that is, they are evaluated by the team responsible for implementing the project.8 However, the methods typically used to evaluate quality improvement projects measure the change in the output of the improved process, but not how well the improved process was implemented. This is because there is no explicit implementation method associated with traditional process improvement methods. As a result, there is also little guidance provided to individual practitioners on how to adapt the processes if needed. In this article, we suggest that using empowerment evaluation in healthcare settings enables the capacity for intelligent adoption and adaptation to suit the particular circumstances of a patient and a provider by promoting ownership and building capacity for implementing evidence-based practices. Making accountability proactive and transparent by asking and answering the 10 accountability questions facilitates empowerment at the system and individual provider and patient levels.
We propose that an integration of empowerment evaluation with quality improvement approaches is likely to enhance the quality of healthcare delivery by increasing the ability of practitioners to improve as well as implement well, though future research is needed to test such an integration. We suggest that it would be valuable to (1) do the development work to integrate the methodologies with supporting tools that brings in best practices from the fields of empowerment evaluation and quality improvement; (2) evaluate the value addition of such combinations. We hypothesise that such integration will help improve health and healthcare in everyday practice.
Acknowledgments
We thank Charles Bennett, Bruce Dobkin, Gareth Parry and Brian Robson for their comments on draft versions of this manuscript. Getting To Outcomes and GTO are trademarks registered by the University of South Carolina and RAND Corporation. It is noteworthy that, while GTO is trademarked, resources are available free of charge (visit http://www.rand.org/health/projects/getting-to-outcomes.html for access to manuals and other GTO tools and resources).
References
Footnotes
Contributors AW was the lead author on this manuscript and was primarily responsible for the conceptualisation and overseeing manuscript preparation. KAA, BC and RR were all involved in drafting sections of the manuscript, revising the manuscript and providing input towards the conceptualisation of the manuscript.
Competing interests None declared.
Provenance and peer review Commissioned; externally peer reviewed.