Article Text

Download PDFPDF

A qualitative study of the variable effects of audit and feedback in the ICU
  1. Tasnim Sinuff1,2,
  2. John Muscedere3,
  3. Linda Rozmovits4,
  4. Craig M Dale5,
  5. Damon C Scales1,2
  1. 1Department of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada
  2. 2Interdepartmental Division of Critical Care, University of Toronto, Toronto, Ontario, Canada
  3. 3Department of Medicine and Critical Care Program, Queens University, Kingston General Hospital, Kingston, Ontario, Canada
  4. 4Independent Qualitative Researcher, Toronto, Ontario, Canada
  5. 5Lawrence S. Bloomberg Faculty of Nursing, University of Toronto, Toronto, Ontario, Canada
  1. Correspondence to Dr Tasnim Sinuff, Department of Critical Care Medicine, Sunnybrook Health Sciences Centre, Toronto, Ontario, Canada M4N 3M5; taz.sinuff{at}sunnybrook.ca

Abstract

Background Audit and feedback is integral to performance improvement and behaviour change in the intensive care unit (ICU). However, there remain large gaps in our understanding of the social experience of audit and feedback and the mechanisms whereby it can be optimised as a quality improvement strategy in the ICU setting.

Methods We conducted a modified grounded theory qualitative study. Seventy-two clinicians from five academic and five community ICUs in Ontario, Canada, were interviewed. Team members reviewed interview transcripts independently. Data analysis used constant comparative methods.

Results Clinicians interviewed experienced audit and feedback as fragmented and variable in its effectiveness. Moreover, clinicians felt disconnected from the process. The audit process was perceived as being insufficiently transparent. Feedback was often untimely, incomplete and not actionable. Specific groups such as respiratory therapists and night-shift clinicians felt marginalised. Suggestions for improvement included improving information sharing about the rationale for change and the audit process, tools and metrics; implementing peer-to-peer quality discussions to avoid a top-down approach (eg, incorporating feedback into discussions at daily rounds); providing effective feedback which contains specific, transparent and actionable information; delivering timely feedback (ie, balancing feedback proximate to events with trends over time) and increasing engagement by senior management.

Conclusions ICU clinicians experience audit and feedback as fragmented communication with feedback being especially problematic. Attention to improving communication, integration of the process into daily clinical activities and making feedback timely, specific and actionable may increase the effectiveness of audit and feedback to affect desired change.

  • Audit and feedback
  • Implementation science
  • Qualitative research
  • Quality improvement
View Full Text

Statistics from Altmetric.com

Introduction

Audit and feedback has been proposed as an important approach for improving the quality of intensive care unit (ICU) services worldwide.1–3 As a means of attenuating practice variation contributing to poor clinical outcomes,4 ,5 audit and feedback has become an integral component of continuous quality improvement and patient safety initiatives.6 ,7 Outside the ICU setting, when included in multifaceted improvement strategies, audit and feedback is theorised to motivate providers to improve care, particularly when practice is identified as inconsistent with peers, institutions or accepted guidelines.8–10 However, in the ICU, this strategy is often linked to research-informed initiatives known to reduce patient mortality, duration of mechanical ventilation and ICU length of stay.11 ,12 In response to these advantages, considerable managerial commitments have been made to audit and feedback in ICUs internationally.6 ,13–24

Notwithstanding its potential benefit, a large gap exists in our understanding of why audit and feedback may or may not work in the ICU context.3 ,12 ,25 This uncertainty emerges, in part, from the current body of evidence outside the ICU, which describes wide variation in the effectiveness of audit and feedback.8–10 Analysis of systematic review data on audit and feedback26 using theoretical constructs of Feedback Intervention Theory27 and Control Theory28 suggests that the existing literature lacks a theory-driven approach to the development of audit and feedback interventions. Using theory to develop feedback interventions, it is proposed, may improve its effectiveness and thereby improve quality of care.29

Variable effectiveness within the ICU11 may be traced to published ICU reports whereby audit and feedback is undertaken in different ways: at the point-of-patient care in two,23 ,24 weekly in two,15 ,18 monthly in four,14 ,16–17 ,22 at a single time in one19 and not specified in two20 ,21 studies. Across reports, feedback is delivered through various channels including electronic, print and verbal processes. Moreover, feedback alternates between individual and group performance. Because these audit and feedback methods are complex and diverse, they may be difficult to operationalise. Uncertainties regarding the effectiveness and optimal approaches to audit and feedback suggest that further study is warranted.

Quality improvement in the ICU has been described as arduous and often emotionally laden work for ICU stakeholders.5 ,6 To understand why audit and feedback may or may not work in complex environments such as the ICU, we need to understand the relevancies and problems stakeholders identify in the processes that comprise this improvement practice. Hence, our objective was to understand both the experiences of ICU clinicians and leaders with audit and feedback across a range of community and academic ICUs and the core mechanisms that lead to the success or failure of the implementation of audit and feedback. We therefore conducted a qualitative study30 ,31 to understand factors that influence the effectiveness of audit and feedback in the ICU.

Methods

We wanted to understand the individual and organisational barriers to implementing audit and feedback and preferences for the types of audit and feedback interventions to support behaviour change from the perspectives of ICU clinicians and leaders. Although there are theoretical frameworks to explore the issues of implementation of strategies such as audit and feedback at both the individual and organisational levels,32–34 we approached this area without a pre-existing theoretical construct as audit and feedback is at its relative infancy in the ICU setting. Hence, we used a qualitative descriptive approach35 ,36 incorporating techniques from grounded theory.31

Setting, participants and sampling

We conducted this study in a representative sample of five academic and five community adult ICUs in Ontario, Canada. Participating units had between 10 and 20 ICU beds and all ICUs were managed under an intensivist or closed model of care.37 We included ICUs, which used audit and feedback variably across sites and across different best practices. The frequency of audit and feedback practices ranged from infrequently (less than quarterly) to routinely integrated into their culture (more than quarterly for at least 12 months). We included both academic and community hospitals as our prior work in this area suggested that different types of institutions have variable opportunities and experiences with audit and feedback.5

We sought participants with a variety of professional roles and insights. Hence, participants from each site included attending physicians, nurses, respiratory therapists, pharmacists and ICU administrators (physician directors, nurse and respiratory therapist managers). One academic centre served as the pilot site for interviews.

Initially, typical case sampling was used as a purposive strategy to recruit participants who represented common members of multidisciplinary ICU team including clinicians and administrators involved in the day-to-day management of critically ill, mechanically ventilated patients.30 ,31 Participants were recruited until no new information was forthcoming (saturation).31

Data collection

We used semistructured, in-person in-depth interviews for the majority of participants. For logistical reasons, 12 interviews with staff at two of the facilities were conducted by telephone. We developed our interview guide based on the audit and feedback literature8 ,10 and our study objective. Preliminary interview guides were developed for each clinician group (physician, nurse, respiratory therapist) and were piloted by conducting two interviews per ICU clinician group (intensivists, nurses, respiratory therapists), and with the nurse manager and physician director from the pilot site. The interview guide was revised following a preliminary analysis and team meetings (see online supplementary appendix).

The final number of participants per ICU clinician group was unequal across clinician groups and hospitals as the targeted sampling method was sensitive to the relative number of clinicians and their viewpoints at each participating ICU. All interviews were digitally audio recorded for verbatim transcription and were an average of 35 min in length.

Analysis

Analyses were conducted using constant comparative methods.31 Two investigators (TS, LR) developed the initial coding framework and subsequent necessary revisions. Initial coding was conducted by one of the investigators (LR) and then in duplicate by another investigator (TS) for a selection of codes considered more interpretive to ensure dependability of coding.38 Constant comparative methods included searches for disconfirming evidence. The codes and emergent themes were discussed in detail by two of the investigators (TS, LR). Subsequently, all emergent categories and themes were analysed (TS) to understand relationships between the codified phenomenon and the circumstances contributing to the core problem.31 Data were managed using HyperResearch software (Randolph, Massachusetts, USA).

Ethics

Approval to conduct this study was obtained from all 10 participating institutional research ethics boards. Written informed consent was obtained from each participant.

Results

We conducted 72 interviews, including 17 intensivists, 34 nurses (bedside and managerial/education roles), 17 respiratory therapists (bedside and managerial/education roles), one dietician and three pharmacists (table 1). Participants’ critical care experience varied and included novice and veteran clinical workers (table 2).

Table 1

Number of participants by participating hospital

Table 2

Participants by number of years in critical care

Eight ICUs routinely used audit and feedback. In two ICUs audit and feedback was infrequently used; in one, this was due to resource limitations and in the other it was a matter of the prevailing culture, which favoured mandatory approaches for implementation of best practices (eg, preprinted order sets, restriction of clinicians’ options, etc). Audit and feedback was used primarily for guidelines or protocols used to reduce central line infections and ventilator-associated pneumonia.

Discontinuous communication

Audit and feedback might have inconsistent or suboptimal effects in ICU due to the impaired nature of communication of the entire process. As identified by participants from the ICUs that conducted audit and feedback routinely, the core problem was that clinicians experienced audit and feedback as fragmented or discontinuous communication, because of the manner in which both processes were communicated to bedside clinicians (tacit, explicit, verbally and non-verbally implemented, enacted and perceived). The communication was fragmented and variable in its effectiveness.

Clinicians felt that audit, as it was generally conducted within their unit, lacked transparency and credibility, and that feedback was not timely, complete or actionable. Participants conveyed that the results of this discontinuous and variable nature of communication of the entire process of audit and feedback had negative ramifications—including perceptions of blame and feelings of marginalisation. Many felt disconnected from the entire process.

Audit process: lack of transparency, credibility and disenfranchisement

Understanding of audit processes was often limited to those responsible for conducting the audits and to a few individuals with direct involvement in quality improvement. For most participants, auditing was an opaque process carried out by unknown individuals for unspecified reasons. The lack of transparency often resulted in alienation of staff from quality improvement efforts and lost opportunities for learning. There was a sense that bedside ICU staff felt disenfranchised: their opinions about the process were not sought nor were they informed of the process. I'm aware there's people in the unit collecting data but I don't know if it's for audit and feedback or a research project … unless they need your authorization to do something they usually don't communicate with us directly. [Physician] [Participant eight] The forms just started showing up … nobody's really explained, “We're doing it for this purpose,” … the management or the educators are saying, “Well, why didn't you fill it out? Why didn't you fill it out?”…That goes to the bottom of my list because I don't know what it's about and I don't know what it's for. [Bedside Nurse] [Participant 25] We're missing an opportunity to actually improve the process because the auditors come and run away … if all these initiatives are really about helping save lives … then do the audit, get the objective measurement … mark it down as you see it and then help us change. [Physician Director] [Participant 55]

Clinicians also conveyed perceptions of surveillance and blame. For many and especially bedside nurses, the audit process held negative associations because they lacked information sharing. Clinicians often felt that they were under surveillance.

Others felt marginalised. Many targeted care practices were perceived to be nursing focused and information sharing about the audit process often did not reach colleagues outside of that professional group. Night-shift clinicians also felt overlooked. They expressed dissatisfaction at having to rely on impersonal forms of communication such as e-mail or posted notices rather than in-person communication or direct feedback about important best practice initiatives. It doesn't make you feel very good when you think your Big Brother's watching … I like to know the reasons why we're doing something and like to think that it's improving things. [Bedside Nurse] [Participant 29] They would go through my stuff and not tell me what they were doing …It's kind of sneaky … it feels like I'm not part of the team … [Bedside Nurse] [Participant 53] We're not really included, we're the other healthcare professional, we're not physicians, we're not nursing, we're in the other category, so they just kind of whoosh…bypass us … [Respiratory Therapist] [Participant 50] You never see the manager, you never see the educator, you don't see anybody and if the information about these best practices aren't clearly passed along they get lost on nights … we're left to rely on e-mail or stuff posted on the bathroom wall and that's about it … [Bedside Nurse] [Participant 25]

Feedback: need for more timely and actionable feedback

Feedback was clearly perceived to be an important tool for change, but was perceived to be more difficult to implement effectively than audit due to limited resources, logistical challenges, poor communication and change fatigue. Feedback data were rarely provided to front-line care providers in a timely, complete or actionable manner that could support behaviour change. When it was provided, there was a reliance on paper and electronic texts as opposed to direct communication. Feedback was often circulated by e-mail, posted notices or through staff meetings, none of which were felt to be particularly effective. Participants expressed a strong preference for direct face-to-face communication that would allow for education, discussion and feedback connected to specific patients. Many participants pointed out that while audit data might be fedback, this did not necessarily make for effective feedback, which they conceived as being timely, transparent and actionable information.

The lack of timeliness and specificity was of primary concern to clinicians. Overall, there was a strong sense of a disjointed process and incomplete communication as it relates to an active work process. Feedback was often perceived to be irrelevant because of delayed reporting. However, some participants cautioned against interpreting short-term data in the absence of longer term trends since short timescales also meant small denominators. Timeliness, in other words, was not just about the speed with which feedback was delivered but, more importantly, about a meaningful sense of temporality. To achieve this, both the short-term picture and the longer term trends needed to be addressed. How useful is it for me to know that in the month of April we had two line infections. Nobody knows whose patient that was, we don't know how it was breached and we don't know what happened to that patient [Physician] [Participant 40] If you feed back data to people quickly it's always on a small number … it's really hard for us to know what “your patients” means anyway, you'd be talking about feeding back data on something like twenty patients, many of whom the exposure wasn't just to me. [Physician] [Participant ten] Just having one month in isolation is not necessarily the most useful sort of thing … things go up, things go down – it's more useful to see that you've got a trend going on over time … [Pharmacist] [Participant 35]

In order to be useful, participants felt that feedback needed to be concrete, patient oriented and actionable. Feedback was difficult for many staff members to engage with because it was too abstract, passive and disconnected from care at the bedside. Bedside clinicians who were not research oriented, in particular, could not relate to the volumes of data presented. Overall, bedside clinicians indicated that feedback lacked directives for improvement and specific, clinically focused actions, which can be acted upon more immediately. If you find out about a problem after months after the fact it's not as useful to you as opposed to somebody coming to me right now and saying we have a violation of the VAP bundle … and then we say, “Okay, let's correct it.” [Physician] [Participant 40] So the feedback, positive or negative is more what I focus on, rather than the actual numbers … The actual numbers I have no idea … it takes a back seat for me because I'm not focused on that, I'm focused on what am I doing for these people. [Bedside Nurse and Charge Nurse] [Participant 62]

Strategies to improve the effectiveness of audit and feedback

To improve the effectiveness of audit and feedback, clinicians suggested improving information sharing, encouraging peer-to-peer discussions about quality improvement and engaging leadership. ICU clinicians felt that improved information sharing about all aspects of audit and feedback would address staff concerns about focusing on seemingly arbitrary or unjust measures and encourage greater sense of ownership of the results. Just to understand what it is you're going to be measured by, it'll help you to…personalize it, “I really do have a responsibility to do X because it will make a difference” [Patient Care Manager] [Participant 39] If it was more known…what they were actually going to be doing with the information it might be easier…to comply. [Bedside Nurse] [Participant 23]

Participants conveyed that a peer-to-peer discussions regarding quality improvement in the ICU would generate greater clinician engagement about quality improvement initiatives in general. A more peer-driven collaborative process would be better accepted and would more effectively achieve sustainable behaviour change through improved communication, greater receptiveness to education and increased compliance. Several ICUs also found that incorporating feedback into the daily rounds conversation resulted in greater consistency. This was seen to foster greater ownership for best practice implementation since these could be directly linked to patients. If you hear information…from your peer you're more likely to accept it … it [becomes] a conversation. [Bedside Nurse] [Participant 17] It leverages against people's sense of duty and ownership of their patients … it's very powerful in implementing change. [Physician Director] [Participant 55]

Engagement of leadership on levels, from nursing and physician to senior administration, was perceived to be a very important aspect of successful audit and feedback. In particular, the leadership role of physicians was a prominent factor for many. Although the ICU team was multidisciplinary, without physicians’ active and sustained engagement, change was difficult if not impossible. Although the flattened hierarchy within healthcare is advocated, it is not yet established within the work environment. Participants wished for nurses to embrace the opportunity for leadership. The capacity of nurses to implement change depended on the consistency of physician response. Being well supported and having recognition of achievements by hospital senior management were seen as imperative to increase clinicians’ confidence in, and willingness to engage with the audit and feedback process. There's still a widely-held view that the physician is the leader of the Critical Care team … they really drive it. [Nurse Educator] [Participant 17] Most of the things that we're trying to implement are nursing best practices … what physicians need to do is to create an environment … that allows nurses to … take leadership on those. [Physician] [Participant ten] There has to be … the institutional will to make this happen and to be supported by very senior people… At the corporate level it needs to be supported, championed and rewarded. [Physician Director] [Participant 45]

Discussion

We conducted a qualitative study of multidisciplinary ICU team members’ perspectives of audit and feedback in the ICU and found that the overall perception was that audit and feedback was potentially useful but often failed to affect clinical care because of the way that it was implemented. Audit and feedback was frequently experienced at the individual and professional level as discontinuous communication. The audit process was felt to lack transparency and credibility, and feedback was felt to be untimely, incomplete and often not actionable. The impaired nature of communication and its negative ramifications may be a significant mechanism underlying the suboptimal effect of audit and feedback in the ICU.

Several suggestions emerged about how to improve the process of audit and feedback in the ICU. These included improved information sharing about the audit and feedback process and rationale for change; implementation of peer-to-peer discussions about quality improvement to avoid a top-down approach and more positive engagement in audit and feedback by senior management.

In our study, participants strongly expressed the desire for timely and contextualised feedback. This is consistent with both Feedback Intervention Theory39 and existing empirical research outside the ICU setting. Feedback Intervention Theory39 posits that in order to change behaviour, feedback should be timely, actionable and provide data required to change behaviour. Empirical studies corroborate these tenets. For example, audit and feedback appears to be most effective for increasing adherence to clinical practice guidelines when feedback is timely,40 individualised41 ,42 and non-punitive.43–45 Feedback cues should follow the following sequence (from greatest to least barriers of feedback effectiveness): timely—individualised—non-punitive, which makes feedback increasingly actionable.29 Hysong et al29 posit a new sequential model, which adds customisability to these aforementioned three feedback intervention cues, to enhance feedback.

Our study has important implications for conducting audit and feedback in the ICU setting. First, how the audit process is communicated and how feedback is delivered to ICU clinicians play a significant role in the acceptance of the process and the results. Our results are consistent with Goal Setting Theory46 ,47 (feedback moderates individual motivation; goals need to be specific and feedback enhances performance) and Control Theory48 (behaviour is goal driven and feedback directs divergent behaviour back towards the desired performance goal). Clinicians in our study consistently felt that feedback must be specific, with specific and attainable outputs to be achieved. Accordingly, feedback may be enhanced by setting performance targets and implementing specific action plans.28 Second, from a practice perspective, whether for administrative or developmental purposes, audit and feedback is more than a simple reporting of performance data. Feedback needs to be both meaningful and tailored to the needs of the team members in order for clinicians to improve their performance.42 ,43 ,49 Some of the mixed findings in the ICU audit and feedback literature could be partially explained by differences in the audit and feedback characteristics.5–15 Future research should consider the limitations of prior ICU studies when designing audit and feedback interventions. Moreover, the aforementioned theories may be used to design appropriate feedback interventions to improve adherence to best practices and improve quality of care.28 ,29

This study is limited in that we interviewed clinicians once, which may not reflect their perspectives longitudinally nor describe the material circumstances of their work. However, since it was not our intention to conduct an ethnography of this area, our approach was an appropriate qualitative methodology to understand the social effects of audit and feedback. This method is helpful in identifying core problems and processes as they are experienced in everyday circumstances. We used a semistructured interview guide, with specific close-ended prompts rather than an unstructured interview guide. This allowed us to use the same key questions for all interviews, but also to explore more deeply into this area, especially the barriers the current audit and feedback process presented to bedside clinicians in the ICU.

Strengths of this study include involvement of all members of the ICU multidisciplinary team. This allowed us to approach the complexity and variation of audit and feedback implementation in the ICU to achieve the fullest understanding of this strategy. We also included both academic and community hospitals. In addition, we used a multicenter design and inclusion of 72 participants, which increases the credibility of our results as we have provided a large, inclusive and fuller perspective on a common and important quality improvement process in the ICU,6 which has not been described to date. Our study offers new insight into common dynamics that are encountered by many clinicians across several centres. From the ICU clinician perspective, this focus on their experience and understanding of audit and feedback captures a broad spectrum of the meanings and consequences attached to the use of this quality improvement strategy.

Conclusions

Greater attention to improving communication, integration of the process into daily clinical activities and making feedback timely, specific and actionable may influence the effectiveness of audit and feedback at changing desired behaviours. Future research could more fully explore the utility of customising clinical audit data so that it is meaningful to ICU clinician groups. To address the existing gaps in clinical practice, those in leadership positions at the unit level need to take steps to integrate feedback into the daily routine of the ICU and improve the timeliness and specificity of feedback.

References

View Abstract

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Files in this Data Supplement:

Footnotes

  • Correction notice This article has been corrected since it was published online first. The third author's surname has been corrected.

  • Acknowledgements We would like to thank all of the participants for their invaluable time and contribution to the interviews comprising this study.

  • Contributors TS and LR had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: TS, JM and DCS. Acquisition of data: LR. Analysis and interpretation of data: LR, TS and CMD. Drafting of the manuscript: TS, JM, LR, DCS and CMD. Critical revision of the manuscript for important intellectual content: TS, JM, LR, DCS and CMD. Obtained funding: TS, JM and DCS.

  • Funding This study was funded by the Physicians' Services Incorporated Foundation (PSI), University of Toronto Dean's Fund, and the University of Toronto Connaught Fund. DCS is supported by a Graham Farquharson Knowledge Translation Fellowship from PSI.

  • Competing interests None declared.

  • Ethics approval Research Ethics Boards of each of the participating institutions.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Data sharing statement All data from the study are included in the submitted manuscript.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.