Making the case for focusing on QI project outcomes

For most medical education interventions, it is extremely challenging, if not impossible, to impact patient level outcomes (Kirkpatrick Level 4) since these outcomes reflect many factors other than what a new learner knows or does (table 1). Even demonstrating behaviour change for clinicians (Kirkpatrick Level 3) through unobtrusive direct observation presents inherent logistical challenges. On the other hand, an experiential QI curriculum which involves a group of healthcare professionals represents a learning activity designed to change some aspect of the healthcare system, usually with the explicit goal of improving patient care outcomes. When this occurs, suddenly, Level 4 outcomes become attainable to the educator!

In addition to the theoretical advantage of potentially achieving Level 4 educational outcomes, focusing on project outcomes as a marker of curricular success offers practical advantages. First and foremost, healthcare institutions and schools for health professions are both more likely to fund educational programming when the training is mutually beneficial and aligned. By highlighting changes in clinical outcomes as the educational outcome of interest, the educator speaks in a language that the healthcare administrator understands. Focusing on the QI project’s potential for positive change from the start can also mitigate learner frustration and/or disengagement with QI.10 Learners are more likely to engage when they are innately interested in or care about the specific quality problem being proposed. Finally, careful consideration of the factors that increase the likelihood of QI project success (eg, stakeholder support, availability of data) helps inform curricular content and teaching methods.

Making the case for focusing on learner outcomes

Despite the advantages stated above, focusing exclusively on project outcomes as the barometer for a successful educational QI intervention can create problems. Imagine for a moment that you are an educator leading a team of students tackling inappropriate medication use in the geriatric patient population as their QI learning experience. Fast-forward 9 months and imagine your reaction and the students’ reactions, if the QI intervention falls flat. The students may become frustrated and the teacher dejected.

The reality is that many QI projects fail. This is not dissimilar to research. Many clinical trials show no benefit for the study treatment. We do not regard these ‘negative results’ as failures on the part of the investigators. The chance that an improvement project will fail to deliver significant benefit is at least as high as that in clinical research—probably higher because we know less about how to do QI effectively than we do about clinical research. Moreover, with clinical trials, investigators have only to worry about the evaluation, not developing the study treatment itself. In QI, we have to do both—develop an effective intervention and evaluate it well. The first part in particular presents numerous challenges, especially the first time. Thus, the fact that a QI project outcome is negative should not reflect poorly on the teacher, the students or the curriculum. Whether due to lack of stakeholder consensus, long wait times for an information technology change request or simply change fatigue by the individuals who are the target of the improvement, the contextual unknowns in the messy world of QI can never guarantee success.

Another problem with focusing on project outcomes is that many experiential QI curricula for students are forced to occur in an artificially shortened time frame due to competing curricula and rigid clinical schedules structured around rotations. In QI work, where learning is iterative and based on small tests of change, a successful project might require 10 Plan-Do-Study-Act (PDSA) cycles over 6–12 months.11 For their first project, however, learners might complete only one or two PDSA cycles, increasing the likelihood that they will not see positive results before the end of their educational experience. In the relatively young field of QI education where many faculty are still struggling with how to teach the topic, or teach it better, we risk educator attrition and lack of adoption of promising curricular innovations if we focus too heavily on project outcomes.

But perhaps the greatest threat of a singular focus on project outcomes is that it reinforces and relegates QI to ‘project work’—that is, something that is started and finished; completed or not completed; successful or not successful. This narrow perspective diverts attention away from our higher goal as health professions educators, namely an appreciation of and commitment to continuous improvement activities in daily practice among our learners. A focus on the long game will necessitate a shift in our approach to the organisation of QI education and how we define success.

Linking QI education to long-term improvements in outcomes through behaviour change

It has been proposed that educators must establish logical links between educational interventions and patient outcomes.12 O’Leary and colleagues in their article nicely demonstrate the causal link between QI education and the acquisition of QI knowledge and skills,8 and this link has been demonstrated previously by others.4 7 The next logical link is to determine if these newly acquired knowledge, skills and attitudes result in behavioural change for the learner. If so, the behavioural changes could impact the desired downstream outcomes of improved care beyond the curriculum and beyond the QI project outcome. Thus, assessing behaviour change as an outcome of QI education, while difficult, represents a critical target for educators.

What are the implications of a shift towards behaviour change as the main focus for QI education? We believe that there are several important considerations for educators.

1. QI education must adopt strategies known to influence behaviour change. But how does one organise QI education to engender ongoing participation in QI efforts? Much of the literature on behaviour change for healthcare professionals focuses on a concrete behaviour change linked to a specific clinical process or problem. For example, physicians are more likely to adopt guideline-concordant prescribing behaviours with audit and feedback interventions or default options embedded within the electronic health record.13 14 Not surprisingly, many of these interventions bear little relevance to encouraging health professionals to change their behaviours as they relate to QI. However, other interventions, such as educational outreach, may have greater relevance and could inform future QI educational efforts.15

2. We need to develop strategies to capture downstream QI behaviour change. Some simple ones are easy to imagine. Do learners take on a new improvement project after completing the curriculum?16 In this instance, we commend O’Leary and colleagues, as their study is one of a minority of studies that actually reports on future involvement in QI work (48% of their graduates reported taking on a new QI project within 18 months of completing the programme).8 However, other less tangible behaviours are difficult to measure, yet critically important to capture. For example, do learners review reports of their clinical practice outcomes? Do they take the time to actively reflect on them and design a small PDSA cycle in their daily work? Do they recognise how an isolated problem could be an opportunity for broader QI and take steps towards leading a change? Tracking these types of behaviours might involve the use of assessment methods such as multisource feedback or portfolios17 18 or require novel approaches that borrow from other fields, such as the social and behavioural sciences.19

3. The clinical learning environment plays a critical role in shaping future behaviours of health professions learners.20 Recent studies demonstrate that future physician practices and even the clinical outcomes experienced by their patients are strongly influenced by the quality outcomes achieved in the training environments where they learn.21 22 Given this relationship, one could argue that the clinical learning environment can also be manipulated to encourage desired behaviours. Such an approach would require QI education to move beyond the ‘QI project’ and instead embed learners in clinical environments where QI is core business and where continuous improvement cannot be disentangled from daily work.23 Learning would still be ‘experiential’, but the real value would lie in experiencing an optimal practice environment, as opposed to conducting an improvement project in any practice environment.

4. QI education research must be oriented to establish those links that connect QI knowledge and skills acquisition with behaviour change and QI behaviour change with longer-term improvements in clinical outcomes.12 Another study from this same issue of BMJ Quality and Safety provides an instructive example of the type of research that can begin to address such linkages. McNicholas and colleagues report their 3-year experience refining their QI programme to optimise the authentic use of PDSA. The authors used an established framework to measure authentic use of PDSA (ie, the desired QI behaviour change).24 They made iterative changes to their curriculum, and through qualitative document analysis and key informant interviews, explored how these curricular changes improved PDSA knowledge and skills and led to improved use of PDSA over time. The evaluative approach taken, which sought to draw linkages between programme design, immediate learning outcomes and downstream behaviour change, represents the type of QI education research that is needed.

As health professions educators, we are all striving to embed the value of continuous improvement in personal performance and system performance in our learners. As stated over a decade ago by Batalden and Davidoff, the goal in QI education is for everyone who works in healthcare to recognise that they have two jobs when they come to work every day: doing the work and improving it.25 As QI educators aspire to shift the professional identify of our learners towards one that values continuous QI, we believe that a greater focus on behaviour change as an outcome of QI education is urgently needed.