Article Text

Download PDFPDF

Channelling the force of audit and feedback: averting the dark side
Free
  1. Eilidh M Duncan1,
  2. Noah M Ivers2,3,
  3. Jeremy M Grimshaw4,5
  1. 1Health Services Research Unit, University of Aberdeen, Aberdeen, UK
  2. 2Family Practice Health Centre, Women's College Hospital, Toronto, Ontario, Canada
  3. 3Family and Community Medicine, University of Toronto, Toronto, Ontario, Canada
  4. 4Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada
  5. 5Medicine, University of Ottawa, Ottawa, Ontario, Canada
  1. Correspondence to Dr Jeremy M Grimshaw, Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, K1H 8L6, Ontario, Canada; jgrimshaw{at}ohri.ca

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Audit and feedback is a strategy used to improve healthcare delivery and promote the implementation of evidence-based practice.1 A recipient, at individual, team or unit level, is provided with summary data of their performance over a specified period of time to act as a stimulus for behaviour change and quality improvement. Audit and feedback is used across clinical conditions and settings around the world. When used correctly, it is effective, scalable and relatively inexpensive.1 However, every healthcare improvement strategy has the potential to have a ‘dark side’2; the potential for adverse effects to arise from providing audit and feedback warrants careful consideration. How can we ensure that audit and feedback initiatives will lead to more benefits than harms?

Catlow et al3 in this edition of the journal investigated the unintended consequences of audit and feedback using an innovative approach. They undertook a theory-informed qualitative study of 19 endoscopists from six English National Health Service (NHS) endoscopy centres. The interviewees routinely received audit and feedback at least every 6 months, which provided their own performance data alongside within-site and national comparators. Key performance indicators for colonoscopy were used such as withdrawal time, completion rates, and polyp detection and removal. Colonoscopy involves the endoscopist inserting a camera into the large bowel then withdrawing the camera while looking for pathology. It is a key tool for detecting colorectal cancer but can be challenging, and poor performance can have serious consequences for patients.4 Catlow et al’s paper focuses on the paradoxical and harmful effects of audit and feedback. Benefits of audit and feedback, although probable, are not presented. The striking candour of the comments made by the interviewees is of note and may be because the interviewer was regarded as a sympathetic non-judgemental peer.

Some interviewees noted that they were not motivated to change if their performance was in line with local peers even if it did not attain the national minimum standards. This is an example of where a design feature of the audit and feedback approach (inclusion of more than one comparator) likely reduced its effectiveness.5 6 Other paradoxical effects included increased performance anxiety which could undermine endoscopists’ confidence, resulting in cognitive interference during task performance and may increase the chances of incomplete colonoscopy.

Harms that potentially affected patients were categorised as direct and indirect. Direct harmful effects included performing clinical actions to achieve higher scores on audit and feedback even when these went against patients’ best interests, such as persevering with a colonoscopy despite patient discomfort and performing unnecessary polypectomy. Indirect harmful effects included ‘fudging’ documentation and ‘gaming’ behaviours involving inaccurate reporting of data to artificially meet targets. The potential for harm to patients because of unintended consequences of audit and feedback in this specific context is striking and has been identified previously.7 Catlow et al include suggestions for mitigation strategies to avoid the ‘dark side’ of audit and feedback. These echo existing practice recommendations for optimising the effectiveness of audit and feedback based on theory, expert recommendations and systematic reviews.6

For example, providing feedback on withdrawal time for colonoscopy has previously been highlighted as unlikely to help facilitate improving inspection of the colon and identified as likely to increase the risk of gaming behaviours.7 It is of crucial importance that performance measures selected for feedback reflect aspects of care that really matter, that is, that prioritise patients’ outcomes and experiences and are measures that recipients of audit and feedback take seriously. We echo previous calls for audit and feedback designers to select measures that are based on sound evidence and that will facilitate practice changes that are going to benefit patients, rather than selecting based on ease of measurement.6 7

Also crucial to consider are the target behaviours that are reflected within the feedback. The effects of audit and feedback may be attenuated for physical, complex tasks.7 When the evidence–practice gap under target is due to a skills deficiency in conducting such tasks, then audit and feedback alone is unlikely to help; investment in skills-based training is needed. A behavioural diagnosis approach can help to ensure that the most appropriate solutions are chosen and implemented alongside (or instead of) audit and feedback.

As well as choosing the most appropriate measures and behaviours to audit, it is also essential to carefully consider the way the performance gap between current and recommended practice is communicated. The provision of two comparators in a feedback report (such as in the colonoscopy example) creates the risk that recipients will employ a defensive tendency to compare themselves with the comparator that is most comfortable for them; the one that allows them to view their current performance most positively.5 Existing best practice recommendations6 highlight the importance of choosing an aspirational comparator that will be more likely to reinforce desired behaviour change. Furthermore, providing feedback in a non-punitive tone, alongside support mechanisms such as facilitated feedback and action planning, can help to minimise defensive reactions and ensure recipients’ attention is focused more productively towards specific actions for improvement.1 8 9

Catlow et al’s paper illustrates why it is important for national audit programmes to consider the potential for unintended consequences and explore methods for monitoring and mitigating against these. Built-in monitoring systems that can detect gaming and other harmful effects in quantitative terms (for example, through data validation) could be one method for doing this.

Understanding audit and feedback recipients’ qualitative perceptions of adverse consequences is also worthwhile. However, the authors themselves acknowledge that interviewees rarely described their own negative behaviours and instead attributed these to unnamed others. Humans are prone to bias and, as such, relying on interviewees’ perceptions of how other people may behave could lead to an overemphasis on harms that are possible but may not be actualised. Furthermore, a broader understanding of harms from audit and feedback may be missing if only the perceptions of the audit and feedback recipients are considered. A wider conceptualisation of potential harms could include not only direct harm to patients, but also harm to audit and feedback recipients and healthcare systems (eg, the impact on health professionals’ anxiety, morale, team dynamics, professional culture and staff retention along with resource utilisation and costs), and to the wider patient community (eg, the consequences of acting on inaccurate or unreliable data such as redirection or reprioritisation of resources). Conversely, there is also a need to consider and balance the broader harms from not providing audit and feedback.

Replication of the dark logic model approach across other audit and feedback contexts is required along with an attempt to validate whether the perceived harms are real risks. The findings could also be validated through retrospective and planned subgroup analyses exploring paradoxical effects and harms in future audit and feedback evaluations. For example, it is plausible that audit and feedback leads to a regression to the mean phenomenon in top performers. Establishing the consistently identified risks and consequences across contexts could stimulate research into mitigation strategies. For example, involving the recipients of audit and feedback in its design intuitively makes sense, as they are likely to have the contextual knowledge that will ensure it is meaningful, understandable and useful.10 Prior work has applied human-centred design and co-design methods to engage recipients in audit and feedback intervention development.11 12 Including the expertise and views of a wider range of stakeholders such as policymakers and decision-makers, audit and feedback commissioners, experts in behavioural and implementation science, frontline clinicians and patients could help to ensure broader consideration of potential adverse effects and help to design interventions that maximise the chance of benefit.

Additionally, audit and feedback researchers, commissioners and providers need to establish what the boundary conditions and stopping rules are for audit and feedback intervention delivery, so that efforts are not wasted when benefits are limited or outweighed by potential harms. Audit and feedback laboratories,13 involving collaborations between large-scale providers of audit and feedback and researchers, are uniquely placed to be able to tackle these questions for further research and improve the evidence base on how to ensure audit and feedback leads to maximal benefit and minimal harm.

A stark message from the paper is how failure to consider the dark side of quality improvement interventions can result in a loss of focus on patient benefit, a disregard for patient needs and deficiency of protection from harms. We need to routinely consider these risks when planning any quality improvement interventions in practice or research settings.

Ethics statements

Patient consent for publication

Ethics approval

Not applicable.

References

Footnotes

  • Twitter @EilidhMDuncan, @noahivers, @GrimshawJeremy

  • Contributors EMD and JMG jointly wrote the first draft, NMI reviewed and revised drafts and all authors approve the final version.

  • Funding EMD is funded through a personal fellowship from The Healthcare Improvement Studies (THIS) Institute, supported by the Health Foundation’s grant to the University of Cambridge. JMG holds a Canada Research Chair in Health Knowledge Transfer and Uptake. NMI holds a Canada Research Chair in Implementation of Evidence-based Practice.

  • Competing interests None declared.

  • Provenance and peer review Commissioned; internally peer reviewed.

Linked Articles