Article Text

Download PDFPDF

Theory-based and evidence-based design of audit and feedback programmes: examples from two clinical intervention studies
  1. Sylvia J Hysong1,2,
  2. Harrison J Kell3,
  3. Laura A Petersen1,2,
  4. Bryan A Campbell4,
  5. Barbara W Trautner1,2
  1. 1Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston, Texas, USA
  2. 2Baylor College of Medicine, Houston, Texas, USA
  3. 3Educational Testing Service, Morrisville, Pennsylvania, USA
  4. 4Normal Modes, Durham, North Carolina, USA
  1. Correspondence to Dr Sylvia Hysong, Center for Innovations in Quality, Effectiveness and Safety, Michael E. DeBakey VA Medical Center, Houston TX 77030, USA; Hysong{at}bcm.edu

Abstract

Background Audit and feedback (A&F) is a common intervention used to change healthcare provider behaviour and, thus, improve healthcare quality. Although A&F can be effective its effectiveness varies, often due to the details of how A&F interventions are implemented. Some have suggested that a suitable conceptual framework is needed to organise the elements of A&F and also explain any observed differences in effectiveness. Through two examples from applied research studies, this article demonstrates how a suitable explanatory theory (in this case Kluger & DeNisi's Feedback Intervention Theory (FIT)) can be systematically applied to design better feedback interventions in healthcare settings.

Methods Case 1: this study's objective was to reduce inappropriate diagnosis of catheter-associated urinary tract infections (CAUTI) in inpatient wards. Learning to identify the correct clinical course of action from the case details was central to this study; consequently, the feedback intervention featured feedback elements that FIT predicts would best activate learning processes (framing feedback in terms of group performance and providing of correct solution information). We designed a highly personalised, interactive, one-on-one intervention with healthcare providers to improve their capacity to distinguish between CAUTI and asymptomatic bacteruria (ASB) and treat ASB appropriately. Case 2: Simplicity and scalability drove this study's intervention design, employing elements that FIT predicted positively impacted effectiveness yet still facilitated deployment and scalability (eg, delivered via computer, delivered in writing). We designed a web-based, report-style feedback intervention to help primary care physicians improve their care of patients with hypertension.

Results Both studies exhibited significant improvements in their desired outcome and in both cases interventions were received positively by feedback recipients.

Summary A&F has been a popular, yet inconsistently implemented and variably effective tool for changing healthcare provider behaviour and, improving healthcare quality. Through the systematic use of theory such as FIT, robust feedback interventions can be designed that yield greater effectiveness. Future work should look to comparative effectiveness of specific design elements and contextual factors that identify A&F as the optimal intervention to effectuate healthcare provider behaviour change.

Trial registration number NCT01052545, NCT00302718; post-results.

  • Audit and feedback
  • Health services research
  • Quality improvement methodologies
View Full Text

Statistics from Altmetric.com

Background

Audit and feedback (A&F) is a common intervention for changing healthcare provider behaviour and improving healthcare quality.1 ,2 A&F can be defined as ‘a summary of clinical performance (audit) over a specific period of time, and the provision of that summary (feedback) to individual practitioners, teams, or healthcare organizations.’3 A&F is an appealing intervention because of its seemingly straightforward approach to behaviour change: by receiving current information about their performance (‘knowledge of results’), providers are more motivated to improve.4 ,5 A&F has been used to improve a wide range of behaviours in clinical practice across many settings,2 making it highly flexible. Further, A&F is often the first step in awarding financial incentives to providers; consequently, the design and implementation of an A&F intervention can materially impact quality of care, as well as costs for patients, providers and healthcare organisations.6 Thus, the nature and characteristics of both the outcome and the feedback intervention must be considered to achieve the desired results.

Studies within healthcare including a Cochrane review suggest that A&F can improve patient care effectively;2 however, others have found that A&F varies in effectiveness and have attributed this inconsistency to the design of specific interventions.2 Many A&F studies inadequately describe their interventions and do not detail their underlying causal processes, making it difficult to compare feedback interventions across different healthcare settings.1

Theory and conceptual frameworks are important yet underused tools in healthcare for successfully designing behaviour-change interventions. Michie and Abraham7 posit that, to design interventions that truly change behaviour, one must rely on empirically supported theories of behaviour change that identify both psychological antecedents and operating mechanisms of behaviour change. This contention is echoed in the A&F literature. Foy and colleagues suggest a conceptual framework is needed to organise the elements of A&F and explain observed differences in effectiveness.1 Ivers et al8 contend that using theory in designing, implementing or interpreting A&F interventions can help clarify the behavioural causal pathways and their underlying mechanisms, resulting in more effective feedback interventions. Unfortunately, explicit use of theories to design A&F interventions is rare, with little consensus as to appropriate theories.3

Another potential problem is that not all theories are well suited to aid in intervention design; at minimum, the theory must clearly describe the mechanism behind the behaviour change and be supported by independent work.7 For example, Rogers' Diffusion of Innovation Theory9 and Bandura's Social Cognitive Theory10 have been reported as the two theories most often used in A&F intervention design. Yet, the suitability of these theories for this purpose has been questioned due to their lack of specificity.3 Similarly, implementation-oriented frameworks such as PARIHSi can guide contextual factors to be considered to ensure successful implementation; however, they may not be able to help identify the kinds of features an intervention requires to be optimally effective in a given context or across multiple contexts. Often, it is necessary to draw from numerous theories to accomplish that goal. In the case of A&F, Kluger and DeNisi11 have already conducted such a synthesis to produce Feedback Intervention Theory (FIT). Specifically designed for explaining feedback's operating mechanisms, FIT draws from cognitive and motivational theories to formulate its tenets, including control theory,12 ,13 goal-setting theory,14 multiple-cue probability learning paradigm,15 social cognition10 and learnt helplessness theory, to offer clear propositions for how feedback works and what cues are likely to improve feedback effectiveness. Through two examples from applied research studies using FIT-based feedback interventions, the present article demonstrates how FIT can be systematically applied in healthcare settings to design better feedback interventions.

Feedback intervention theory: a brief primer

According to FIT, people regulate their behaviour by comparing it to goals, standards or benchmarks to which they have committed.14 When they observe a discrepancy between the standard and their actual behaviour, they try to resolve it by adjusting their level of effort to increase the likelihood that the benchmark will be met; the degree of increased effort an individual will undertake can be altered by providing performance feedback. For example, physicians who prescribe a particular medication for a given condition are likely to continue prescribing it until some external influence, such as a patient request for a cheaper alternative, compels them to change. Since individuals' attention is limited, only gaps receiving attention have the potential for change; for example, an alert in the computerised prescription order-entry system could bring a cheaper medication to the physician's attention, thereby increasing the likelihood of switching to the new medication. Thus, feedback interventions work by providing new information that redirects recipients' locus of attention either away from the task (eg, towards ourselves or towards irrelevant tasks) or towards the task. Kluger and DeNisi propose that three factors determine how effectively this attentional shift occurs: characteristics of the feedback itself (or ‘feedback intervention cues’), the nature of the task (‘task characteristics), and personality and situational variables.11 Feedback cues determine the direction towards which attention will likely shift; task characteristics determine how susceptible the task is to attentional shifts, and personality and situational variables determine how the feedback recipient chooses to change once the attentional shift occurs.

According to FIT, feedback cues that redirect the locus of attention towards the task itself reinforce extant motivational processes, thereby strengthening feedback's effect on task performance; similarly, cues that redirect attention towards details of the task tend to promote learning and also strengthen feedback's effect on task performance; conversely, information that shifts attention away from the task diverts cognitive resources away from trying to improve, thereby weakening feedback's effect.11 Figure 1 graphically depicts the basic tenets of FIT.

Figure 1

Schematic of tenets of Feedback Intervention Theory.

Kluger and DeNisi,11 Hysong16 and Ivers et al2 independently found via meta-analysis that feedback has a generally moderate effect on performance. Additionally, the Kluger and DeNisi and the Hysong meta-analyses explicitly tested the tenets of FIT and identified several feedback intervention characteristics that enhance feedback effectiveness, including correct solution information (information about how to do the task correctly), velocity (change from last period of measurement), graphical or written presentation formats, inclusion of normative information, neutral tone and other goal-setting components. Characteristics found to decrease feedback effectiveness included discouraging tone, praiseful tone and verbal feedback.11 Table 1 summarises and defines characteristics predicted by FIT to impact feedback effectiveness, and the effect reported by each meta-analysis. As shown in this table, these three independent meta-analyses exhibit several parallels. Although theory and evidence seem to disagree about which elements of feedback are ineffective, there appears to be good congruence about the more effective elements of feedback. Due to its versatile framework and demonstrable improvements in A&F effectiveness both inside and outside healthcare settings,17–19 FIT is a suitable guide for the design of A&F interventions across a variety of contexts.

Table 1

Factors predicted to impact feedback effectiveness by Feedback Intervention Theory and by Cochrane systematic review

The following case examples demonstrate how the principles of FIT were applied to design A&F interventions for two different healthcare scenarios, both of which led to improvements in two different healthcare quality outcomes. As the theory provides the most guidance about the effectiveness of feedback intervention cues, and feedback design is the element over which organisations and researchers have the most control, the cases concentrate on the feedback cues selected in the design of the feedback intervention.

Case 1: Reducing inappropriate diagnosis of catheter-associated urinary tract infection via case-specific feedback

We employed FIT to design an A&F intervention in the United States Department of Veterans Affairs Veterans Health Administration (VHA) to improve internal medicine resident and long-term care personnel's capacity to distinguish between asymptomatic bacteriuria (ASB) and catheter-associated urinary tract infection (CAUTI). ASB is frequently misdiagnosed as CAUTI, often leading to unnecessary use of antibiotics, in turn leading to resistant organisms, unnecessary cost, prolonged hospital stays and infections such as Clostridium difficile-associated disease.20 Details of the CAUTI study's design, including the A&F intervention, are published elsewhere;21 highlights are summarised here.

Participants

The A&F intervention was applied to 169 healthcare providers who make the decision to order a urine culture and prescribe antibiotics for a positive urine culture in two types of hospital units: acute-care medicine wards and long-term care units. In the acute-care medicine wards, internal medicine residents (trainees), usually interns or second-year residents, were generally responsible for making decisions about prescribing antibiotics for positive urine cultures, under supervision from a rotating pool of attending physicians. In the extended-care ward, the A&F intervention targeted the nurse practitioners, physician assistants and staff physicians who comprised the providers responsible for prescribing antibiotics for positive urine cultures.

Intervention

The intervention was based on a treatment algorithm derived from the Infectious Diseases Society of America guidelines for the non-treatment of ASB. This treatment algorithm describes steps providers should take when confronted with a patient with an indwelling urinary catheter and a subsequent positive urine culture.22 All participants' cases involving positive urine cultures were identified via a comprehensive chart review and examined by trained research staff for errors in diagnosis and treatment of ASB cases (defined as deviations from the treatment algorithm). For each clinician, the team identified a subset of cases that would teach the guideline principles clearly, keeping the ratio of erroneously to correctly managed cases to approximately 1:2. Research staff contacted the clinical providers involved to deliver the feedback intervention face to face, which consisted of a one-on-one feedback session reviewing each ASB case. For each case, an interactive hyperlinked PowerPoint presentation was used as an explanatory device during the feedback meeting (see online supplementary appendix A) and given to participants to keep.

Intervention design considerations

In developing the algorithm to be used for classifying cases in the study, we learnt that clinicians routinely caring for patients with urinary catheters use mental models that often deviate from guidelines when classifying cases of catheter-associated bacteriuria as either ASB or CAUTI.22 Thus, designing an intervention that shifted attention to the details of the task and fostered better learning of the guideline so that physicians' mental models could be altered was paramount. This drove our design over other factors, such as labour intensiveness or wide-scale deployment capabilities (all participants were physicians at local hospitals within 1.5 miles of each other and the research team). To accomplish our goal, we opted for an interactive, computer-based presentation of feedback, which FIT predicts would be less likely to shift attention away from the task than simple verbal presentation. The presentation's interactive nature was also intended to shift attention to the details of the task: it showed the physician the specific decision path he or she took, highlighted correct and incorrect decisions (a step towards correct-solution information) and provided correct solution information (one of the most effective cues according to the meta-analytic findings sizes reported earlier) by revealing the correct course of action when a participant clicked on a series of yes-or-no links in the presentation. Finally, because Ivers2 showed that providing both verbal and written feedback yielded better results than written feedback alone, research staff reviewed the treatment algorithm verbally with participants as they followed along on the interactive presentation; however, the research staff used a written, standardised but branching script to accomplish this task. The standardised script ensured that every participant received the same relevant feedback for each decision point in the diagnosis/treatment of ASB/CAUTI. Table 2 summarises how each feedback design characteristic was operationalised for the study.

Table 2

Operationalisation of feedback design characteristics Case 1

Measures

Reactions to feedback

After delivering personalised feedback to individual participants, research assistants completed Smither and colleagues' 11-item scale to capture feedback recipients' reactions to their feedback, specifically targeting recipients' perceptions of the value of the feedback, the recipient's sense of obligation to use the feedback and extent to which the recipient was concerned with others' perceptions of his/her performance.24 Feedback recipients also completed questions about the clarity, usability, tolerability and degree of acceptance of feedback received.

Effectiveness of feedback intervention

Postintervention chart monitoring at the ward level captured the effectiveness of the A&F intervention. For 6 months after the feedback intervention was implemented, charts for each ward were monitored for adherence to the treatment algorithm, specifically, rates of urine-culture orders and inappropriate use of antibiotics.

Results

Reactions to feedback

Most feedback recipients reacted positively to feedback received, indicating that they were open and accepting of the feedback and motivated to improve; mean feedback acceptance scores were high (M=5.18/7, SD=0.26), as were tolerability (M=5.81, SD=0.07) and usability scores (M=5.27, SD=0.04).

Based on research assistants' reports of recipients' reactions, the PowerPoint presentation received positive reactions, as it organised the guidelines for ASB and CAUTI classification and treatment in a simple, understandable way without loss of information fidelity. Though feedback recipients occasionally tried to justify their original decisions, the feedback session quickly turned to the evidence and a learning opportunity for the recipient.

Effectiveness of feedback intervention

The overall rate of urine-culture ordering in wards where participants received A&F decreased significantly during the intervention period (from 41.2 to 23.3 per 1000 bed-days; (incident rate ratio (IRR), 0.57; 95% CI 0.53 to 0.61; p<0.0001) and further during the maintenance period (12 per 1000 bed-days; IRR, 0.29; 95% CI 0.26 to 0.32; p<0.0001). At the comparison site, urine cultures ordered did not change significantly across all three periods. Overtreatment of ASB at the intervention site reduced significantly (1.6 to 0.5 per 1000 bed-days; IRR, 0.35; 95% CI 0.22 to 0.55, p<0.0001) during the intervention period; these reductions persisted (0.4 per 1000 bed-days; IRR 0.24; 95% CI 0.13 to 0.42) during the maintenance period. Overtreatment of ASB at the comparison site was similar across all time periods.25

Case 2: Helping physicians provide guideline-recommended hypertension care via summary feedback

In this second case, we used FIT to design an A&F intervention to help physicians meet their goals for controlling patients' hypertension, as part of a randomised controlled trial of financial incentives to improve delivery of guideline-recommended hypertension care.26 ,27 Despite evidence that hypertension treatment is effective, the condition is undertreated by physicians and controlled in less than one-fourth of US citizens.28 Hypertension is a risk factor for stroke, coronary artery disease and congestive heart failure.

Although the intervention of interest in this hypertension study was financial incentives, A&F constituted an important component of the financial-incentive intervention, as it served as the principal tool for justifying the size of the incentive awarded to any recipient. Moreover, A&F served as the comparable activity for the control group. All participants (regardless of study arm) received A&F on their hypertension care; only the various financial-incentive configurations tested in the study differentiated the study arms. The details of the financial-incentive intervention have been published elsewhere;26 only the A&F design is discussed here.

Participants

Eighty-three primary care physicians from 12 geographically dispersed VHA medical centres participated in this study. Additionally, non-physician clinicians (n=42) in trial groups receiving practice-based incentives (see below) also participated.

Intervention

Over 20 months, participants received web-based A&F reports every 4 months. Each report presented participants with percentage scores for patients receiving hypertension therapy, patients meeting goal blood pressure (BP) levels during last visit and patients receiving hypertension therapy that also met goal BP levels during last visit. Percentage scores were given, both for patients the physician treated during each period and for patients treated by other physicians at the same clinic during that period (in aggregate). Additionally, suggested performance goals for the subsequent performance period were also included.

Intervention design considerations

Several factors drove the design of this intervention. The nature of the task was an important consideration. FIT predicts that less complex tasks such as ordering guideline-recommended medication in the presence of consistently elevated BP29 are more susceptible to being positively influenced by feedback (see table 1); consequently, maintaining attention on the task and activating motivational processes (rather than trying to foster learning, as in the previous case) was a sufficient approach for this intervention. Practical matters also heavily influenced the design: this study involved hospitals geographically dispersed across the country, where often no prior relationship existed between the site and the research team. Target feedback recipients were busy primary care clinicians with little time to participate in a research study; in addition, feedback in this case was the control intervention, rather than the intervention of interest. Consequently, ease of delivery and minimal participant burden were key in this case—we sought to design feedback that could be deployed easily from a remote location, reviewed easily by participants and quickly show where gaps in their performance lay.

To accomplish our goal, we designed a web-based written report providing summary-based feedback. Each report depicted scores about each hypertension management aspect of interest; this provides some degree of correct-solution information by shifting attention to the specific aspect of hypertension management requiring most attention (eg, ordering new medication vs adjusting existing treatment). We also provided achievable goals for participants to strive towards for the following period, which work to activate recipients' motivational processes. Reports also contained group feedback and velocity information in the form of line graphs depicting individual and clinic score histories for hypertension treatment and meeting goal BP levels. Both velocity and group feedback cues improve feedback effectiveness by activating recipients' motivation processes and keeping attention focused on the task. Table 3 summarises how each feedback design characteristic was operationalised for the study. Appendix B presents an example of the feedback report used.

Table 3

Operationalisation of feedback characteristics for Case 2

Results

Reactions to feedback

A random subset of 28 physicians were interviewed at the 16-month mark to gain more in-depth knowledge of their hypertension management experiences during the study. Among the topics discussed was the quality of feedback received and suggestions for improvement. Interview respondents reported finding the feedback reports useful, as they were more detailed and focused than the performance measure reports they normally receive as part of their regular work duties:31I think it [the RCT feedback report used in this study] is a more accurate reflection, I don't find the [VA-generated] facility reports very useful.—Physician, Site #9We get reports but it's buried amongst, like, ‘Do you look at the diabetic's foot’ or whatever. I can't say I've really scrutinized those either. Yours is more targeted.—Physician, Site #2

Effectiveness of feedback intervention

Mean (SD) total payments over the study were US$4270 (US$459), US$2672 (US$153) and US$1648 (US$248) for the combined, individual and practice-level interventions, respectively. Relative to controls who received A&F alone, only the individual-incentives group exhibited significant changes in either BP control or appropriate response to uncontrolled BP. Change in guideline-recommended medication use was not significantly different than in the control group. The effect of the incentive was not sustained after a washout. However, we observed an average 11% increase (SD=2.87) in use of guideline-recommended antihypertensive medications between periods 1 and 5; this included the control group, who only received A&F reports.27

Discussion

A&F has been a popular tool employed to change healthcare provider behaviour and improve healthcare quality; however, it has suffered from inconsistent, atheoretical implementation, leading to mixed levels of effectiveness.1 FIT provides a practical, evidence-based framework with which to design feedback interventions. We have presented two different real-world feedback interventions designed using the principles of FIT, both positively accepted by their intended recipients and demonstrating significant improvements in desired clinical behaviours in the presence of well-designed feedback.

In both cases, feedback was presented in writing, via computer, using a standardised structure whose content was individualised for each participant. In addition, Case 1 featured very specific correct solution information; whereas Case 2 featured individualised goal setting and norms. Correct solution information and goal setting are two of the most powerful feedback design features for maximising effectiveness,11 ,16 and in some cases their combined use is warranted. In these two cases, however, the designers' decision to use only one or the other feature, as well as the designers' respective selections, were driven by the context in which the A&F intervention was being used (consistent with the PARIHS framework) and by the features of the desired outcome. In Case 1, the desired outcome was correct diagnosis and subsequently appropriate treatment of ASB many possible decisions could have led to a wrong outcome; consequently, providing correct solution information about where the physician should have made a different choice was imperative. Conversely, in Case 2 the desired outcome was BP control and antihypertensive medication prescription; in this case the path to the correct outcome was far clearer and simpler; consequently, the more important design feature was to provide the motivational effect of goal setting to remind providers to carry out the desired outcome they already know they should. These findings demonstrate the need for considering both the desired outcome and feedback characteristics when designing a feedback intervention for maximum effect.

The two cases presented here covered two different healthcare scenarios; consequently, the A&F interventions featured rather different characteristics. So how do A&F designers decide which components to choose for their interventions? The commonalities between the two cases can prove instructive. Both cases selected components that were individually tailored to the feedback recipient (consistent with FIT and Hysong et al),32 and that directed attention to the details of the task, consistent with FIT. The nature of which components to use (eg, written/computerised feedback, correct solution information, goal setting), in both cases, was driven by the situational and organisational context. The role of context on feedback effectiveness is an area for which FIT does not provide much guidance and is a limitation of the theory. However, newer research and theories can be drawn upon to fill this gap and are excellent opportunities for future research.33 For example, depending on whether feedback recipients tend to be more performance versus mastery oriented, research suggests feedback should be customised to align with those recipient characteristics.34 Recent research has also recommended encouraging feedback-seeking behaviours from employees and focusing on creating a culture of feedback where employees feel safe at giving and receiving feedback and have opportunity for reflection; this can take the form of both formal and informal forms of feedback.34 ,35

Limitations

The studies presented in the two cases, although guided by FIT to design a robust intervention that would be successful, were not designed for purposes of comparative effectiveness or to tease apart the effects of specific feedback elements; rather they were guided by FIT to design the intervention most likely to result in successful behaviour change, given the contextual constraints of the respective studies. Thus, the presentation of the two cases, while useful for presenting annotated examples of the design considerations for building successful, theory-based interventions using FIT, cannot speak to the matter of comparing the relative effectiveness of specific design elements of feedback.

The main limitation in our case presentation concerns limitations of the theory itself; FIT adopts an inherently cognitive perspective towards feedback; consequently, recipient motivations and contextual (situational) variables are outside the scope of the theory; additionally, because it adopts a cognitive perspective it is inherently individually based and does not address feedback in team contexts,36 as is the case with newer frameworks.37 Despite these limitations, newer research and theories (eg, DeShon et al37) have built upon the work of FIT precisely due to its clear and specific propositions and rationales; also, where the scenario lends itself to it, FIT can provide specific direction for designing A&F interventions, as it did with the two cases presented here.

Conclusion

We conclude that FIT can be effectively used to design feedback interventions to more effectively improve provider performance and quality of healthcare. Future work should focus on expanding understanding of specific contextual cues that impact feedback effectiveness (whether for individuals or teams) to provide more specific guidance towards A&F intervention design.

References

View Abstract

Footnotes

  • Contributors SH conceived and wrote the first draft of the paper and designed the A&F interventions presented in each case. BWT was the principal investigator of the study presented in Case 1; LAP was the principal investigator of the study presented in Case 2. HJK aided in the design of Case 1's audit & feedback intervention and designed the ‘reactions to feedback’ component of Case 1; BAC edited subsequent drafts of the paper and provided material intellectual input. All authors provided critical intellectual input and material edits for the paper.

  • Funding Veterans Administration, Health Services Research and Development Program (CDA 07-0181), (IIR 04-349) and (IIR 09-104); and National Heart Lung and Blood Institute (1R01HL079173-01), (R01HL079173-S2).

  • Competing interests None declared.

  • Ethics approval Institutional review boards for Baylor College of Medicine and Michael E. DeBakey VA Medical Center.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • i PARIHS—Promoting Action on Research Implementation in Health Services.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.