Article Text

Download PDFPDF

Next-generation audit and feedback for inpatient quality improvement using electronic health record data: a cluster randomised controlled trial
  1. Sajan Patel1,
  2. Alvin Rajkomar1,
  3. James D Harrison1,
  4. Priya A Prasad1,
  5. Victoria Valencia2,
  6. Sumant R Ranji3,
  7. Michelle Mourad1
  1. 1 Department of Medicine, Division of Hospital Medicine, University of California, San Francisco, San Francisco, California, USA
  2. 2 Department of Internal Medicine, Dell Medical School at the University of Texas at Austin, Austin, Texas, USA
  3. 3 Department of Medicine, Division of Hospital Medicine, Zuckerberg San Francisco General Hospital, San Francisco, California, USA
  1. Correspondence to Dr. Sajan Patel, Department of Medicine, Division of Hospital Medicine, University of California, San Francisco, San Francisco, CA 94143, USA; sajan.patel{at}


Background Audit and feedback improves clinical care by highlighting the gap between current and ideal practice. We combined best practices of audit and feedback with continuously generated electronic health record data to improve performance on quality metrics in an inpatient setting.

Methods We conducted a cluster randomised control trial comparing intensive audit and feedback with usual audit and feedback from February 2016 to June 2016. The study subjects were internal medicine teams on the teaching service at an urban tertiary care hospital. Teams in the intensive feedback arm received access to a daily-updated team-based data dashboard as well as weekly inperson review of performance data (‘STAT rounds’). The usual feedback arm received ongoing twice-monthly emails with graphical depictions of team performance on selected quality metrics. The primary outcome was performance on a composite discharge metric (Discharge Mix Index, ‘DMI’). A washout period occurred at the end of the trial (from May through June 2016) during which STAT rounds were removed from the intensive feedback arm.

Results A total of 40 medicine teams participated in the trial. During the intervention period, the primary outcome of completion of the DMI was achieved on 79.3% (426/537) of patients in the intervention group compared with 63.2% (326/516) in the control group (P<0.0001). During the washout period, there was no significant difference in performance between the intensive and usual feedback groups.

Conclusion Intensive audit and feedback using timely data and STAT rounds significantly increased performance on a composite discharge metric compared with usual feedback. With the cessation of STAT rounds, performance between the intensive and usual feedback groups did not differ significantly, highlighting the importance of feedback delivery on effecting change.

Clinical Trial The trial was registered with (NCT02593253).

  • audit and feedback
  • hospital medicine
  • performance measures
  • quality improvement
  • randomised controlled trial

Statistics from


Quality improvement strives to bridge the gap between current practice and ideal practice. Audit and feedback, in which professional performance is measured (‘audit’), compared against targets or peers, and then fed back to the individual, is known to highlight that gap and is associated with improvements in care quality. However, there is considerable variability in the magnitude of the effect observed. A Cochrane Review meta-analysis on audit and feedback demonstrated only modest impact in improving professional practice, with a median adjusted risk difference of 4.3%.1

There is a robust body of literature on the effectiveness of audit and feedback to clinicians, most of which involve data feedback to practising clinicians in the context of specific quality improvement initiatives.2–5 Several guiding principles for audit and feedback have been proposed, including identifying metrics that are under a practitioner’s control, providing multiple instances of feedback frequently and at an individual level, closely linking the visual display and summary message, and constructing feedback through social interaction.6 7 Outside of medicine, audit and feedback employs these strategies by using real-time dashboards like those used to profile a user’s activity level in mobile activity applications (eg, FitBit). This is achieved by continuous data generation and processing that summarises large amounts of data in easily presentable formats. The rapid adoption of electronic health records (EHRs)8 now makes such personalised, real-time feedback feasible for clinicians9; however, it is not clear how to integrate such data feedback into a busy clinician’s daily practice in a fashion that promotes behaviour change without impeding clinician workflow or resulting in unintended consequences.

Several unique considerations also arise when feeding back performance data in an academic clinical setting. Physician trainees are extremely busy—working 80 hours per week in the USA—and have competing educational obligations, limiting their time available to review performance data. Additionally, because attendings and trainees provide care in teams, individualised feedback reports may not reflect this reality. Furthermore, since academic physicians rotate services frequently, providing feedback data weeks or months after episodes of care may make it difficult to influence behaviour. Finally, the high levels of burnout among trainees10 necessitate careful planning of data feedback interventions to avoid adding to the already considerable resident workload. 

We sought to combine known best practices of audit and feedback with newly available continuously generated EHR data to create a modern form of team-based audit and feedback with the goal of improving quality of individual patient care in an academic medical centre.


Study setting, participants and oversight

This study was a cluster randomised control trial of intensive audit and feedback as compared with usual audit and feedback, and took place from February 2016 through June 2016. The intensive audit and feedback consisted of an intervention followed by a washout phase. This study design was chosen given that care is organised by teams of physicians. This study was conducted on the internal medicine teaching service of the University of California, San Francisco (UCSF) Medical Center. The UCSF Moffitt-Long Hospital is a 650-bed, urban tertiary care hospital. The medicine teaching service comprised eight teams, with an average daily census of 80 patients. Each medicine team comprised an attending physician (usually a hospitalist), resident physician (postgraduate year (PGY) 2 or 3) and two internal medicine interns (PGY 1). Attendings, residents and interns were randomly assigned to a team at the beginning of the academic year to medicine teams. Attending physicians changed every 2 weeks, while trainees changed every 4 weeks. The trial was registered with (NCT02593253).

Prestudy baseline quality metric feedback

Inpatient medicine service teams were receiving feedback on quality metrics prior to our intervention. For the 19 months prior to the study period, all teams received twice-monthly feedback on six performance measures: phlebotomy use; telemetry use; per cent of discharges occurring before noon; a completed medication reconciliation; a completed after visit summary (‘High Quality AVS’); and a timely discharge summary (Figure 1).

Figure 1

Twice-monthly email feedback (‘Usual Feedback’). The email included the following descriptors: Medication reconciliation: entering home medications into the EHR and addressing all home medications during the admission with a stop, start or continue action.  Phlebotomy: reducing the number of patient ‘sticks’ or unique blood draws per day. Discharge before noon:  the percentage of patients discharged before noon. High-quality AVS: the percentage of patients with an ‘After Visit Summary’ that includes a principal diagnosis, patient-centred discharge instructions and a follow-up appointment or referral in the next 30 days. Telemetry until discharge: percentage of patients with a LOS >48 hours who are still on telemetry monitoring 24 hours prior to discharge. Timely discharge summary: the percentage of discharge summaries completed within 24 hours of discharge. EHR, Electronic Health Record; LOS, Length of Stay; MD, Medical Doctor; D/C, Discharge; PACU, Post-Anesthesia Care Unit; ER, Emergency Room

A completed medication reconciliation (‘Med Rec’) was credited when a patient’s prior to admission (PTA) medications were entered into the medical record and a corresponding action (order, do not order, change) was selected for each. Additionally, on discharge, the discharging physician was required to select an action for all PTA medications (stop, resume, modify) and for each hospital medication (prescribe, do not prescribe, modify). The AVS is a patient-centred record of the hospitalisation given to each patient on discharge. Credit for ‘High Quality AVS’ was given if the discharging provider completed all three of the following items: (1) a discharge diagnosis, (2) manually entered discharge instructions and (3) a qualifying follow-up appointment (either an existing appointment with the patient’s UCSF primary care provider (PCP) within 14 days, a completed referral to a UCSF specialist or a structured text manual entry of follow-up instructions for the patient’s non-UCSF PCP). A timely discharge summary was credited if the templated discharge summary note (written by any physician member of the team) was signed within the EHR (and thus automatically faxed to the patient’s listed PCP) within 24 hours of discharge.

Compliance with each metric was evaluated using custom-built EPIC Clarity reports that assessed whether each action (eg, medication reconciliation, follow-up appointment scheduling or the discharge summary completion) had truly occurred within the EHR for each alive, non-AMA (against medical advice) discharged patient. Performance was displayed graphically by team and was sent by email to the resident and attending providers on service during that time period.

Intervention group description

We first developed a programme theory11 for our intervention, seeking to identify the factors that would promote the use of timely data audit and feedback to facilitate behaviour change (‘intensive feedback’), as well as any potential barriers to the success of such an intervention (box 1). Representatives from the Division of Hospital Medicine and Internal Medicine residency programme developed explicit criteria for selecting improvement priorities for the medicine teaching service. Metrics to be fed back to residents and attending physicians were selected for the potential to be impactful—applying broadly to the patient population cared for by the team; meaningful—translating to improvements in the quality of patient care; actionable—under the control of the ward team; and timely—able to be fed back soon after the episode of care. Given the busy inpatient environment, we also recognised that data feedback had to be delivered in a fashion that respected the resident’s workload, time constraints and competing obligations. The Division of Hospital Medicine and Internal Medicine residency leaders (site director and chief residents) met quarterly to review the selection of metrics and methods for feedback, however they were not involved in the study design or dashboard development.

Box 1

Programme theory


  • Medicine teaching service clinicians (faculty and residents) receive data on performance metrics that are meaningful, actionable, impactful and fed back in a timely fashion.

  • These data are fed back to them in a fashion that is integrated into their existing workflow.


  • Review of objective performance data will become a routine part of providing care on the inpatient medicine service.

So that

  • Residents will perceive quality data as being meaningful to their education and professional development.

  • Residents and attendings will be able to change their practice in real time.

  • The inpatient service’s performance on key quality metrics will improve.

Our intervention consisted of intensive feedback, which included two key components in addition to usual feedback: (1) access to a team-based, contemporaneous online data dashboard; and (2) weekly inperson review of team data, which we referred to as ‘STAT rounds’. Both the dashboard and STAT rounds focused on compliance with three metrics: completion of medication reconciliation, a high-quality AVS and a timely discharge summary. These metrics were chosen for the following reasons: they were already part of housestaff performance improvement projects in preceding years and had been championed by housestaff as important to the quality of their care. Additionally, they were easily and accurately reported in the EHR, felt to be actionable by the team and are believed to improve discharge safety and quality,12–15 a priority for the year.

Each attending, resident and intern on the four teams randomised to the intensive feedback intervention received a link to an electronic dashboard at the start of their rotation. The electronic dashboard presented contemporaneous data regarding team performance of the three metrics above for the four intervention teams. Data were automatically extracted from the EHR (Epic, Verona, Wisconsin) at 05:00 and reflected the performance of the preceding 14 days. The landing page of the dashboard displayed aggregate data from all eight medicine teams and allowed for detailed drill-down of performance of the intervention teams as well as patient-level detail (Figure 2). In addition to displaying performance on each metric, an intervention team’s composite performance, the ‘Discharge Mix Index (DMI)’, was displayed. The DMI was achieved when all of the three selected discharge metrics were met for a patient (medication reconciliation, high-quality AVS, timely discharge summary). This strategy was used to decrease the cognitive load of visualising the data and engage teams equally in all three metrics. Each team was ranked according to their performance on the primary outcome, the DMI, which ranged from 0% to 100%, and a ‘scoreboard’ demonstrated relative performance. Data from the control group (‘usual’) feedback teams were displayed in an anonymised format. The raw data were processed and cleaned automatically on a secure server16 and then visualised with a QlikView dashboard (Qlik, Radnor, Pennsylvania) hosted by the medical centre.17

Figure 2

Electronic dashboard displaying daily and cumulative performance of Discharge Mix Index (DMI), a composite score of medication reconciliation (Med Rec), timely discharge summary and high-quality after visit summary (HQ AVS). DHM, Division of Hospital Medicine; DC, Discharge.

In addition to making the dashboard available, we hosted weekly inperson ‘STAT rounds’ for intensive feedback team members only, immediately following the department’s weekly morbidity and mortality conference from 1 February 2016 to 30 April 2016. Team attendance was voluntary, but intensive feedback providers received pages and emails reminding them to attend. STAT rounds lasted for 15 min and were moderated by a study investigator who also served as an attending physician (AR, SP, MM, SR). STAT rounds had the following goals: (1) introduction to the metrics and why they were chosen, (2) education on how to achieve the metrics and (3) a review of each of the intervention teams’ performance. The moderator was instructed to facilitate the session using a didactic approach. At each session, participating providers were surveyed anonymously as to whether STAT rounds helped improve their approach to discharging patients and whether they felt that reviewing their team data was worthwhile to their clinical practice.

Control group description

Each month, four control group teams were assigned to ‘usual audit and feedback’, which was the same as feedback sent to teams prior to the study (see prestudy baseline quality metric feedback description).

Washout period

Given the time-intensive nature of STAT rounds and our suspicion that STAT rounds were responsible for the majority of the impact on change in performance, STAT rounds were discontinued on 1 May 2016. During a ‘washout period’, 1 May 2016 through 30 June 2016, we continued sending emails to the intervention teams about availability of the dashboard and provided continued access to the dashboard, but conducted no inperson review of the data with teams.


Medicine teams were selected as units for stratified randomisation during the intervention period. To ensure balanced randomisation, teams were clustered to ensure that one team from each arm would be admitting daily, and patients would be randomly distributed between clusters. Cluster 1 was defined to be medicine teams A, B, C and D, and cluster 2 was defined to be medicine teams E, F, G and H. A coin flip by study investigators (SP and JDH) was used to randomly assign each to the intervention. Teams were informed of their group allocation by the study investigators (SP and JDH). Allocation concealment was not possible given the nature of the intervention. Throughout the study period, patients were randomly allocated to the four usual feedback and the four intensive feedback teams. During the day, patients were admitted 1 to 1 to a single intervention and single control medicine team until each team admitted six new patients or reached a total of 14 patients. At night, patients were admitted to a night team and were then randomly assigned in the morning to one of the remaining six medicine teams (three usual feedback and three intensive feedback), who would assume care for the rest of the patient’s hospitalisation. Given the random nature of the design, one attending and three interns were part of both a usual and intensive feedback team at different times during the study period. Informed consent was not required given the classification of this work as quality improvement.

Measures and outcomes

The primary outcome was performance on individual patients’ DMI for all non-AMA, alive discharges. The secondary outcomes were each individual element of the DMI (medication reconciliation, high-quality AVS and timely discharge summary), uptake of the intervention, and resident and attending perceptions of data feedback. To assess impact on patient outcomes, we also recorded 30-day readmission of patients discharged by the usual and intensive feedback teams. We monitored number of logins to the electronic dashboard (excluding study investigators and network administrators) using monthly usage logs provided by the medical centre. We also recorded attendance at STAT rounds. We surveyed study participants before and after the intervention to assess their perception of the utility of the data feedback and its effect on their workflow.

Statistical analysis

Analyses were conducted following intention-to-treat principles and blinded to group allocation. χ2 statistics were used to compare hospitalisation and patient characteristics of patients discharged by the two groups. We used descriptive statistics to summarise use of the dashboard, participation in and perceptions of STAT rounds. χ2 statistics were also used to assess the relationship between the intervention with the DMI and its component pieces. We also used generalised linear models to assess the adjusted effect of intensive audit and feedback on the primary outcome and readmissions. We accounted for clustering of performance during the intervention period of the 24 teams (eight teams for each of 3 months) using generalised estimating equations with exchangeable working correlation and robust SEs. Covariates were included for reasons of face validity. We performed the same model in the washout period for the 16 teams (eight teams for each of 2 months). Statistical modelling and analysis were completed using Stata V.14.


Patient and team characteristics

During the intervention and washout period, a total of 40 teams (eight each month for 5 months) were randomised to the intensive feedback and usual feedback arms (Figure 3). The 20 intensive feedback teams collectively discharged 942 patients and DMI was able to be calculated for 919 of the patients (97.5%). The 20 usual feedback teams discharged 897 patients and DMI was able to be calculated for 893 of those patients (99.6%). DMI was unable to be calculated for 27 patients due to missing data.

Figure 3
Figure 3

Study flow diagram. Reasons for inability to calculate Discharge Mix Index include unavailable data on medication reconciliation (Med Rec), high-quality after visit summary (HQ AVS) and/or timely discharge summary. 

The clinical characteristics of patients discharged from the intervention and control groups did not differ significantly between groups (online supplementary appendix 1).

Supplementary Appendix 1

Usage of dashboard and STAT rounds

There were 12 STAT rounds in the intervention period with an average attendance of 9.8 of 14 possible participants per round and at least one physician representative from each team, 47 out of 48 times (98.0%). During the intervention period, the dashboard was accessed 104 times by 40 users in February, 77 times by 33 users in March, and 55 times by 30 users in April. During the washout period, the dashboard was accessed 48 times by 20 users in May and 48 times by 24 users in June.

Primary and secondary outcomes

During the intervention phase, the primary outcome, DMI, was achieved in 79.3% (426/537) of patients in the intensive feedback group compared with 63.2% (326/516) in the usual feedback group (P<0.0001). Adjusting for age, gender, patient language, patient race, length of stay and discharge location, there was a statistically significant association between receipt of intensive feedback and DMI performance (OR=2.40, 95% CI 1.66 to 3.45). Results for the primary and secondary outcomes are listed in table 1. Out of the components of the DMI, only achievement of high-quality AVS was significantly increased in the intensive feedback arm (63% vs 79%, P<0.0001).

Table 1

Primary and secondary outcomes

Both teams had significantly higher performance during the intervention period on the DMI and individual components of timely discharge summary and high-quality AVS compared with teams in the baseline period. In the washout period, both usual and intensive feedback groups experienced decreased performance compared with the intervention period (Figure 4), and there was no longer a statistically significant difference in achievement of the DMI or any of its components between the two groups.

Figure 4
Figure 4

Run chart showing performance of Discharge Mix Index (DMI) at baseline, followed by intensive versus usual feedback during intervention and washout periods.

Rates of 30-day readmission were similar in all groups throughout the study (table 1).

Resident and attending perceptions

A total of 93 of 118 (78%) providers responded to the anonymous surveys assessing the utility of STAT rounds. Sixty of 93 (64.5%) reported that STAT rounds improved their approach to discharge quality and 70 of 92 (76.1%) indicated that the review of data was useful in achieving the metrics.


In this cluster randomised controlled trial performed on an internal medicine teaching service, intensive audit and feedback with an electronic dashboard and weekly data review significantly increased the achievement of a composite measure of three quality improvement measures compared with usual feedback. The improvements were largely driven by improvements in the completion of one of the three metrics. With the cessation of STAT rounds in the washout period, both visits to the dashboard and performance on the composite measure fell and the usual and intensive feedback groups did not differ significantly.

Our approach may serve as a generalisable model for effective performance feedback in academic settings. We believe we were able to achieve a higher level of performance improvement than most prior studies of audit and feedback1 by following key principles.6 First, we selected desired actions that were consistent with quality improvement goals set out by the department of medicine and internal medicine residency that were directly actionable by the providers involved. Second, before creating a dashboard, we ensured accurate measurement and attribution to the right level of intervention, which in our institution was the medical team. And lastly, we combined multiple metrics into a single index similar to what has been called ‘perfect care’18 that could be tracked easily over time and used to compare performance with a gamification-like leaderboard.19 The DMI allows for the use of clear graphs on the electronic dashboard that minimised cognitive load and linked the visual display and summary message.20

The randomised controlled design of this study allowed us to isolate the impact of intensive audit and feedback on performance change. Other factors associated with performance improvement, such as education on discharge safety and increasing trainee experience with discharge workflow, occurred during this time, but both intervention and control groups were exposed equally. We can therefore be fairly confident that the differences between the two study groups were due to differences in feedback intensity alone. A critical lesson was the importance of STAT rounds—the inperson facilitation and discussion of the metrics, centred on the dashboard. Our study confirms the findings of the Cochrane Review by Ivers et al 1 demonstrating that intensive feedback using face-to-face performance review is more effective. Our data add to the literature, suggesting that when the face-to-face feedback is discontinued, one risks performance falling. Indeed, with the cessation of STAT rounds in the washout period, the effect of the dashboard alone was not sufficient to demonstrate improvement in our primary outcome. Our usage logs indicate that STAT rounds drove dashboard usage and independent encouragement to use the dashboard was less successful. Our results are consistent with previously published best practices of providing feedback in multiple modalities, addressing barriers to feedback use, preventing defensive reactions and using social interaction.1 6 Of note, STAT rounds took only 15 min each week with minimal preparation because it centred around data automated by the dashboard itself.

Despite robust attendance and improvement in the targeted metrics, only 64% of participants felt that the intensive data feedback improved the quality of their discharges. There may be several reasons for this perception. Despite our efforts to ensure that our measures would be meaningful to clinicians and impactful for patients, physicians may not have been convinced of this. We intentionally used process measures rather than clinical or patient-reported outcomes in order to feed back data on metrics that could be measured accurately and followed daily, meaning changes in performance could be demonstrated immediately as opposed to outcomes that might not be apparent for weeks or months after discharge. However, physicians may not have been convinced of the link between the measured processes and their patients’ outcomes, which may have limited their adoption of sustained practice-changing behaviours.21 Defining ‘quality’ may seem straightforward, but it is challenging to find metrics that measure care effectiveness, as recognised by Donabedian almost half a century ago.22 Indeed, we received feedback from several participants that our metrics did not represent a broad enough view of the quality of their care, and that by focusing solely on the DMI, we may be missing other measured or unmeasured aspects of care quality.

Ultimately, defining standard work and measuring the associated processes are essential to quality improvement. But to be effective in improving care quality, the use of process measures requires that (1) the evidence tying the measure to an outcome is continuously evaluated and (2) providers have the opportunity to provide feedback on how meaningful they find the measures. These tenets reduce the feeling of excessive measurement and provider burnout. Ideally our future work would expand measurement to include more ‘patient-centered’ discharge process domains such as postdischarge medication reconciliation in the home, patient self-assessment of health status after return home or reliable transportation to appointments.

Our study had limitations. It was performed on a teaching service with heavy attending and trainee turnover during the study period. This may have diminished the effect of the intervention because teams may have had only a short time to improve performance and new team members would not have exposure to STAT rounds for up to 6 days after starting on a team. In addition, due to the random nature of team assignments, one attending and three interns crossed over from intensive feedback to usual feedback during the trial, which may have reduced the size of the effect of the high-intensity feedback on DMI performance. Due to technical limitations of our dashboard hosted through a UCSF-IT platform, we were unable to track the usage patterns of individual users of the dashboard to determine the relationship between dashboard use and team performance. Lastly, improvement in DMI was largely driven by improvements in the AVS, given the already high performance of the other two components of timely discharge summary completion and medication reconciliation.

In conclusion, we demonstrate that harnessing EHR data in an electronic dashboard for timely and inperson data review significantly improved performance on a composite measure of inpatient discharge processes. This intensive feedback was successfully incorporated into the workflow of a busy inpatient teaching service, providing a mechanism for optimising audit and feedback for practising physicians and trainees.



  • Contributors As the principal and corresponding author, I assert that every person who contributed to the writing of this manuscript is listed as an author, and that each person listed as an author had an active and significant impact on the conceptualisation, performance and analysis of research in this article. This work has not been previously published.

  • Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests None declared.

  • Ethics approval The University of California, San Francisco Institutional Review Board (UCSF Committee on Human Research) reviewed this trial and given the quality improvement focus was classified as not human subjects research.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.