Article Text
PDF
Abstract
Background In multisite trials evaluating a complex quality improvement (QI) strategy the ‘same’ intervention may be implemented and adopted in different ways. Therefore, in this study we investigated the exposure to and experiences with a multifaceted intervention aimed at improving the quality of intensive care, and explore potential explanations for why the intervention was effective or not.
Methods We conducted a process evaluation investigating the effect of a multifaceted improvement intervention including establishment of a local multidisciplinary QI team, educational outreach visits and periodical indicator feedback on performance measures such as intensive care unit length of stay, mechanical ventilation duration and glucose regulation. Data were collected among participants receiving the intervention. We used standardised forms to record time investment and a questionnaire and focus group to collect data on perceived barriers and satisfaction.
Results The monthly time invested per QI team member ranged from 0.6 to 8.1 h. Persistent problems were: not sharing feedback with other staff; lack of normative standards and benchmarks; inadequate case-mix adjustment; lack of knowledge on how to apply the intervention for QI; and insufficient allocated time and staff. The intervention effectively targeted the lack of trust in data quality, and was reported to motivate participants to use indicators for QI activities.
Conclusions Time and resource constraints, difficulties to translate feedback into effective actions and insufficient involvement of other staff members hampered the impact of the intervention. However, our study suggests that a multifaceted feedback program stimulates clinicians to use indicators as input for QI, and is a promising first step to integrating systematic QI in daily care.
- Audit and feedback
- Continuous quality improvement
- Critical care
- Quality measurement
- Performance measures
Statistics from Altmetric.com
- Audit and feedback
- Continuous quality improvement
- Critical care
- Quality measurement
- Performance measures
Background
In healthcare, there is a large variety of initiatives developing and reporting performance indicators.1–3 By providing care providers insight into their performance, these initiatives ultimately aim to improve patient outcomes. However, it is still much debated how to use indicators in order to achieve this.4–7
In 2006, the Dutch National Society of Intensive Care Medicine (NVIC) developed a set of quality indicators in order to evaluate and improve the quality at Dutch intensive care units (ICUs).8 The Dutch National Intensive Care Evaluation (NICE) registry facilitates the indicator data collection and analyses. As a regular NICE service, participants of the registry receive standard quarterly feedback reports on these indicators. Although structured feedback on analysed healthcare data is potentially effective in improving the quality of care,9–11 studies showed that multifaceted interventions are usually more effective than those consisting of a single element, in this case sending feedback reports only.9 ,11 Furthermore, interventions tailored to prospectively identified barriers are more likely to improve professional practice as compared with non-tailored interventions.12–15 Therefore, a barrier analysis was performed in Dutch ICUs with regard to the standard NICE feedback reports. This analysis showed that Dutch ICU healthcare professionals perceived several barriers to using performance data for systematic quality improvement (QI).16 ,17 Based on these results, we developed the Information Feedback on Quality Indicators (InFoQI) program, which is described in the Methods section.16 The effectiveness of this program was evaluated in a cluster randomised trial.16 In this trial, we evaluated the impact of applying for 1 year the InFoQI program on the patient outcomes ICU length of stay, mechanical ventilation duration, glucose regulation and all-cause hospital mortality as well as the impact on the organisational process measures bed occupancy rate and nurse-to-patient ratio.18 ,19
In multisite trials evaluating a complex QI strategy the ‘same’ intervention may be implemented and adopted in different ways, and the exposure to this intervention may vary considerably. Process evaluations can then be used to provide insights into to what extent the trial intervention was actually implemented and how it was experienced by the study participants.20 ,21 Such insights facilitate the interpretation of quantitative results of trials, and may be crucial in explaining why a QI strategy was effective or not. In addition, a process evaluation assesses the feasibility of the intervention in daily practice, and increases its reproducibility.21 ,22 However, publications on the results of formal process evaluations have been scarce.21 ,23 ,24
Therefore, besides assessing the effectiveness of the InFoQI program, we conducted a process evaluation as a fundamental part of our trial. With the current study we aimed to contribute to a better understanding of the implementation process of a multifaceted indicator feedback intervention, and its impact on the quality of ICU care.
Methods
The InFoQI program
The InFoQI program is a multifaceted intervention for the ICU setting, tailored to barriers identified prospectively and developed using evidence from literature, input by future users and expert knowledge.16 InFoQI aimed to promote the use of performance indicator data for systematic QI at ICUs. The main components of the InFoQI program included A) provision of comprehensive monthly and quarterly feedback reports, B) establishment of a local multidisciplinary QI team and C) two educational outreach visits (table 1).
Main components of the InFoQI program according to protocol
During these visits, the QI team was supported with formulating a QI action plan based on the performance data presented in the feedback reports. The main tasks of the QI team were to discuss their performance in monthly meetings and to communicate the main finding to the rest of the ICU staff.
To evaluate the impact of the InFoQI program on the quality of ICU care, we conducted a cluster randomised trial in the Netherlands from January 2009 until January 2011.16 ,18 ,19 ICUs were eligible for InFoQI if they participated in the NICE registry (n=80), were preparing to submit performance indicator data to the registry (n=46) and were able to allocate at least two staff members to form the QI team. We estimated that the average time investment per QI team member would be 4 h per month for implementing the intervention only, that is, excluding the time needed for executing QI initiatives. Finally, 30 ICUs fulfilled the inclusion criteria and gave informed consent. Fifteen ICUs were randomly assigned to the control arm receiving basic quarterly feedback reports, and fifteen were allocated to participating in the InFoQI program. The trial did not show a significant effect of the InFoQI program on the endpoints, which were nurse-to-patient ratio, bed occupancy rate, ICU length of stay, duration of mechanical ventilation, glucose regulation and hospital mortality.
Measurement and variables
We used the process evaluation framework described by Hulscher et al 20 to evaluate ‘the actual exposure to’ and ‘the experience with’ the InFoQI program. The information presented in this paper was collected among those exposed, that is, members of the multidisciplinary QI teams in the intervention ICUs. We used questionnaires and time registration forms to obtain data on the implementation process and the perceived barriers. These quantitative results were discussed in a focus group in order to verify the results and to support our conclusions.
Actual exposure to the InFoQI program
During the study period, all QI team members were asked to record the estimated time they invested in the various study activities. Each member received a time registration form—at 6 and 12 months after inclusion—for recording the time spent per activity, for example, reviewing and discussing a specific quarterly report. We distinguished hours invested in implementing the InFoQI program (eg, reviewing reports, local QI meetings) from hours spent on executing QI initiatives from the local QI plan.
Experience with the InFoQI program
In this paper, the experience with the InFoQI program refers to the barriers perceived by those exposed, as well as their satisfaction with the program. Between April 2010 and June 2011, electronic questionnaires were sent to all QI team members at the end of their follow-up period. The first part of the questionnaire addressed perceived barriers to using the InFoQI program for systematic QI at their ICU. We formulated 47 statements based on barriers identified beforehand,16 and based on factors influencing QI success mentioned in other studies.17 ,25 ,26 The majority of the statements asked for the respondent's opinion (eg, the definitions of the indicators are clear); some statements enquired after more factual circumstances (eg, at our ICU, someone is responsible for collecting and submitting the indicator data).
Based on the framework of Fleuren et al,25 the barriers were grouped into four categories: characteristics of the indicators, characteristics of professionals, characteristics of the environment and characteristics of the intervention. The second part of the questionnaire consisted of nine statements concerning satisfaction with the different components of the InFoQI program. Answers to all statements could be given on a 5-point Likert scale ranging from ‘1= strongly disagree’ to ‘5= strongly agree’. The questionnaire was pilot-tested by one ICU nurse and two intensivists with special interest in implementation to check the questionnaire for relevance and completeness.
In addition, we invited delegates of the local QI teams of the 15 intervention ICUs to voluntarily attend a 2 h focus group session in December 2010, to elaborate on the preliminary results of the questionnaire. Five intensivists, two ICU nurses and two managers from eight different ICUs were able to participate; five worked in teaching or academic hospitals. The session was chaired and co-chaired by two health services researchers (MdV and SvdV). A topic guide with open-ended questions, based on items included in the questionnaire, was used to structure the discussion. The session was audio-taped and transcribed verbatim by an independent research assistant.
Data analysis
We used descriptive analysis to analyse the data regarding time investment, and the response to the questionnaire. Responses to negatively formulated statements in the questionnaire were recoded to match those positively formulated. A score of more than three on the five-point scale was indicated as positive, less than three as negative, and a score of three was indicated as neutral. We considered a statement to reflect a barrier if less than 50% of the respondents scored the statement as positive.
Regarding the analysis of the focus group session, two researchers (MdV and SvdV) independently studied the transcripts, identified all barriers, and classified them into one of the four abovementioned barrier categories. Results were compared, and discrepancies were discussed until consensus was reached.
Results
Actual exposure to the InFoQI program
Table 2 shows the frequencies and time investments regarding the main components of the InFoQI program: monthly and quarterly feedback reports, the local multidisciplinary QI team and the educational outreach visits.
Exposure to the InFoQI program; frequencies and time investment regarding the main components of the intervention
Feedback reports
As planned, all intervention ICUs received 4 quarterly and 12 monthly feedback reports. The average number of monthly reports monitored per team was 8.3 (±3.7; range, 3–12), and the average number of quarterly reports was 2.6 (±1.4; range, 0–4). For ICUs that spent at least 4 h per month per team member (n=8) this was 10.4 (±2.3; range, 6–12) monthly, and 3.5 (±0.8; range, 2–4) quarterly reports. The average time spent to review 1 monthly report per team member was 1.0 h (±0.6; range, 0.4–2.5), and for a quarterly report this was 1.1 (±0.7; range, 0–1.9).
Multidisciplinary QI team
As prescribed by the InFoQI program, all 15 ICUs established a QI team with a minimum of two and a maximum of seven members. Most of the teams (53%) consisted of four members, including at least one intensivist and one ICU nurse. Most of these teams (n=11) added a (quality) manager as an additional member. The total average monthly time investment per team member for the study was 4.1 h (±2.3; range 0.6–8.1). On average, QI teams spent 62% of their time on activities pertaining to the InFoQI intervention itself (eg, reviewing feedback reports, QI team meetings, etc) (2.5±1.3 h; range, 0.6–4.8), and 38% on the execution of the QI initiatives as formulated in the local QI plan (1.6±1.5 h; range, 0–5.6). The average number of monthly QI team meetings was 5.7 (± 1.4; range, 0–12). Only one ICU organised the maximum number of 12 monthly meetings; three ICUs did not organise any QI meeting.
Educational outreach visits
All ICUs received both educational outreach visits. Most of the visits (87%) were attended by all the members of the QI team. The average duration of the first outreach visit was 3 h, and for the second outreach visit this was 2 h.
Experience with the InFoQI program
Of the 56 questionnaires, 43 (77%) were completed. Minimal one QI team member of each ICU responded. Most of the respondents were intensivists (33%) and ICU nurses (33%), followed by other healthcare professionals, for example, quality managers (9%). The majority of the 43 respondents were between 46 and 55 years of age (56%), and affiliated to teaching hospitals (56%).
Below we describe for each of the four categories the most important barriers identified, the barriers that were solved by the InFoQI program and a summary of the responses regarding satisfaction with the different components of the intervention. All results from the questionnaire discussed below were confirmed in the focus group, unless indicated otherwise.
Barriers
Twenty-two of the 47 statements (47%) were considered to be barriers. Top 10 of most important barriers to using the InFoQI program for local, systematic QI is presented in table 3. Table 4 presents the Top 10 of barriers that were targeted by the InFoQI program. The lower the percentage, the more a statement reflected a perceived barrier.
Top 10 of most important barriers to using the InFoQI program for local, systematic QI
Top 10 of barriers that were targeted by the InFoQI program
Characteristics of the quality indicators
Of the four categories, most perceived barriers were related to characteristics of the quality indicators. The most reported barrier was ‘lack of normative standards’ to determine whether improvements were needed (21%). In the focus group, participants stated that indicators that incorporate a normative standard are more often subject to discussion and to changes over time. For example, for the indicator ‘proportion out-of-range glucose measurements’ the upper threshold of 8.0 mmol/l was much disputed during our study. At first, ICUs considered this too high based on the study of Van den Berghe et al,27 while after the publication of The Normoglycemia in Intensive Care Evaluation — Survival Using Glucose Algorithm Regulation (NICE-SUGAR) study in 2009 28 many Dutch ICUs increased their local glucose target to exceed our threshold of 8.0 mmol/l. Other important barriers in this category were: indicators not being up-to-date (26%); the lack of useful benchmark data (30%); and the positive impact of the indicators not outweighing the required effort (31%). Also, 32% of the respondents stated that there was insufficient case-mix adjustment to facilitate determining the need for QI actions. Focus group participants mentioned the example of the benchmark for the indicator ‘duration of mechanical ventilation’ being difficult to interpret without adequate adjustment for case-mix.
Regarding the indicator ‘nurse-to-patient-ratio’, 74% of the respondents (strongly) agreed with the national guideline that one qualified ICU nurse should care for a maximum of two patients simultaneously.29 Prior to the development of the InFoQI program, one perceived barrier was the lack of reliable indicator data.16 At the end of the study, almost all respondents (91%) agreed that their confidence in the submitted data had improved, and that using structure and process indicators has resulted in improved quality of care (63%) (table 4). It was also stated in the focus group that participating in the study had improved data quality and increased awareness of data management processes. The participants emphasized that professionals will only act on data if they feel the data are reliable.
Characteristics of the professionals
In general, only few barriers concerned the characteristics of professionals. Respondents reported to be familiar with the indicators and their definitions (75%), although some lacked knowledge and skills on how to use the InFoQI intervention for QI at their ICU (22%). Some commented in the focus group that a list of potentially successful actions to improve their practice would have been useful. Most healthcare professionals were motivated to apply the InFoQI intervention to improve the quality of care at their ICU (83%), which was prospectively identified as a barrier. Focus group participants confirmed that their QI team was very motivated to use quality indicators for performance improvement efforts. The respondents of the questionnaire further stated that they were well informed regarding the definitions of the indicators (70%), and that they considered the definitions of the indicators to be clear (79%).
Characteristics of the environment
The most prominent environmental barrier was related to insufficient allocated time to implement the InFoQI program in daily practice (25%). Other barriers were related to insufficient staff for implementing the InFoQI program in daily practice (30%), and probability of interference of third parties outside the ICU (28%). Focus group participants mentioned that it was difficult to allocate time to review the reports since delivery of routine patient care always had the highest priority. Implementation of the InFoQI intervention was further hampered by the presence of temporary healthcare workers not being familiar with ongoing QI activities.
Furthermore, 50% of the respondents reported that there was good communication and cooperation within the ICU, and 62% felt that other staff members were willing to participate in the program. Most of the respondents reported that the responsibility to collect and submit indicator data was appointed to a specific person at their ICU (79%), and that incorporating the implementation in daily routines was no longer perceived a barrier (66%).
Characteristics of the InFoQI intervention
Overall, the statement with the lowest percentage respondents agreeing was ‘access to the feedback reports by other staff members’ (7%), despite encouragement during the outreach visits to share findings with the rest of the staff. In the focus group, participants stated that the reports were available, for example, put online, but not actively discussed with other staff members.
Most of the potential barriers identified in the prospective barrier analysis concerned aspects of the standard NICE feedback reports, for example, insufficient timeliness, the high level of data aggregation and a lack of intensity (ie, how much and how often information is sent).16 At the end of the study, results from the questionnaire showed that lack of intensity was no longer a barrier (77%), and respondents reported that the InFoQI program led to improved quality of care (63%). Also, the insufficient level of aggregation (51%) and applicability (58%) were no longer prominent barriers. In the focus group, some participants even stated that the intensity was too high, as they did not have enough time to discuss each monthly feedback report. Also, the improved applicability of the indicators was mentioned in the focus group. For example, one ICU had adjusted their protocols at the operating room based on analysing unexpected ICU deaths. Some participants mentioned that they would like to receive feedback about the relationships between indicators instead of looking at indicators separately because one can draw better conclusions when taking into account the interaction between structure, process and outcome indicator data; for example, the association between a high nurse-to-patient ratio and a lower death rate. Another barrier reported in the focus group was that participating in the InFoQI program was very time-consuming, especially the collection of valid and reliable indicator data. The QI team meetings were difficult to organise, and often only part of the QI team attended.
Satisfaction
All statements regarding satisfaction with the different components of the InFoQI program were indicated as positive by more than 50% of the respondents. The monthly feedback reports were rated least positive (58%). This was consistent with the abovementioned finding regarding the time investment to discuss each monthly feedback report. Respondents were most satisfied with the educational outreach visits (78%), and the local QI team meetings (71%) because it supported and stimulated them to use indicators for QI.
During the focus group, participants stated that the educational outreach visits helped them to look at their data and processes differently, and encouraged them to critically assess their practice. In addition, some participants mentioned that the establishment of a QI team was a good organisational structure to discuss the feedback reports and data within the team.
Discussion
Main findings
Our study results showed considerable variation among ICUs regarding the exposure to the InFoQI program. Most perceived barriers to using performance indicators for QI activities regarded characteristics of the indicators themselves. Clinicians’ trust in indicator data quality and their motivation to use indicators for QI were no longer perceived as problems. Respondents reported to be satisfied with all components of the InFoQI program.
Barriers successfully targeted by the InFoQI program
We found that the InFoQI program effectively targeted part of the prospectively identified barriers.16 After implementation of InFoQI, ‘lack of trust in data’ and ‘lack of motivation’ were no longer perceived a problem. This implies that a tailored multifaceted feedback intervention potentially removes concerns regarding data accuracy, and positively affects clinicians’ attitudes towards using performance indicator data to improve their practice. We feel that overcoming these barriers formed an essential first step when aiming to change practice based on indicator feedback. This was further supported in the focus group discussion. In addition, participants reported to be very satisfied with the InFoQI program, especially with the outreach visits. This is consistent with findings in other studies showing that healthcare professionals prefer educational interventions that are practice-based.30 ,31 However, social desirability bias might have resulted in an overestimation of the satisfaction with the visits since the researchers that facilitated the visits were also involved in the process evaluation.
Persistent barriers
This study also reveals persisting barriers, which might explain why the InFoQI program was not effective. First, many unsolved barriers were related to the characteristics of the indicators, for example, ‘lack of normative standards for deciding if improvement is needed’, ‘lack of case-mix adjustment’ and ‘lack of useful benchmark data’. These barriers potentially decreased the interpretability and actionability of the performance feedback.32–34 Our results warrant ongoing focus by organisations involved in the development of quality indicators on adequate case-mix adjustment of outcome measures, and providing benchmarks against a meaningful standard or comparison group to better facilitate effective goal setting.35
Second, only very few participants reported that other staff members had access to the feedback reports. On one hand, sharing the feedback more actively may have increased the involvement of the rest of the staff,36 which is pivotal to expedite actual changes in daily care. On the other hand, participants reported other staff members to be cooperative and positive with regard to the InFoQI program. This suggests that access to the feedback reports was not crucial in convincing colleagues to contribute. Unfortunately, we did not have data available on the exposure and experiences of other staff, impeding investigating this in more detail. In order to ensure broad participation within an organisation, however, we suggest that those involved in the development of feedback programs include explicit strategies to support QI teams in involving their organisation in the QI endeavour.
Third, characteristics of the environment were the second most reported category of barriers; they mainly referred to time and resource constraints to apply the InFoQI program in practice. This provides a plausible explanation for the fact that none of the participating ICUs implemented the intervention entirely as planned, hampering the impact of the program.37 On one hand, our study confirms that targeting the lack of allocated time and resources for activities other than direct patient care is complicated. On the other hand, the InFoQI program was perceived to facilitate the use of performance indicators as part of daily routines. This suggests that a multifaceted feedback intervention may partly—but not entirely—solve this problem.
Lastly, although most participants felt that the InFoQI program resulted in improved patient care at their ICU, our trial showed no significant impact of the intervention on the quality of care. This discrepancy between the perceived and the actual impact reinforces the need to evaluate the effectiveness of QI interventions by using objective (preferably blinded) measurements. In our case, the discrepancy might partly be explained by the ‘lack of information to initiate QI actions’, which was reported as a barrier. In addition, Davies et al38 suggested that clinicians might have a limited understanding of the concepts and methods underlying QI. Based on our study, we suggest providing QI teams with additional tools to translate performance feedback into effective actions. For example, cause-and-effect diagrams for systematic problem analysis,39 action plans that incorporate measurable targets,10 or evidence-based strategies to change daily practice.40 The effectiveness of such tools should be investigated in future research.
Strength and weaknesses of the study
The main strength of our study is that we used a combination of qualitative and quantitative data collection methods, which is increasingly used in health services research and evaluation.41 This enabled the triangulation of results, and assisted the process of exploring apparent discrepancies between findings.24
A limitation of our study is that a quarter of the participants did not complete the questionnaire. Because non-responders might have perceived more or different barriers than responders, some factors that hampered the implementation of the InFoQI program might have been missed. However, since all participating ICUs were represented by at least one QI team member, we believe that we identified the most salient barriers. Furthermore, the ICUs in our study volunteered to participate in the InFoQI program. This implies that they were motivated to change their practice based on performance feedback, and felt their organisational structure would facilitate this. Therefore, extrapolation of our results to settings with less motivated participants should be done with caution. In addition, we used a theoretically informed approach to conduct barrier assessment. However, an approach based on normative process theory might have emphasized different parts of our findings. Lastly, our study results do not allow the drawing of conclusions with regard to the causal relationship between identified barriers and the lack of impact of our intervention; this is a limitation inherent to process evaluations.
Conclusions
In conclusion, our study suggests that a tailored multifaceted feedback intervention potentially stimulates healthcare providers to use quality indicators as input for QI activities, and is a potential first step to integrating systematic QI in daily care delivery. However, time and resource constraints hampered the implementation of the intervention. Additionally, to further facilitate the translation of the feedback into effective actions, adequate case-mix adjustment, meaningful benchmarks and additional QI tools are required to successfully change healthcare practice. Also, promoting the active involvement of all staff members merits attention. We believe that our results contribute to a better understanding of how the use of quality indicators result in QI in ICU care, as well as in other clinical domains.
Acknowledgments
We would like to thank all healthcare professionals that participated in our study.
References
Footnotes
-
Contributors SvdV, MdV, NdK, PvdV, KJ, GW and WG were involved in the conception and design of the study. MdV and SvdV developed the questionnaire and provided the 2 h focus group session. MdV performed the statistical analysis regarding the questionnaire and MdV and SvdV analysed the results of the focus group session. MdV drafted the manuscript. NdK, KJ, PvdV, GW and WG supervised the study. All authors were involved in the analysis and interpretation of the data, the critical revision of the paper for intellectual content, and in its final approval before submission. NdK is director of the NICE registry, and a member of the NICE registry board. PvdV is chairing the committee on quality indicators of the Dutch National Society of Intensive Care Medicine (NVIC).
-
Funding The InFoQI study was funded by the National Institute of Public Health and the Environment (RIVM) and the National Intensive Care Evaluation (NICE). The views expressed are those of the authors.
-
Competing interests None.
-
Ethics approval The study was approved by the Institutional Review Board (IRB) following Dutch and European legislation.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement The data of the surveys and focus group are available to the original research team members of the InFoQI study.