Article Text
PDF
Abstract
Background Although clinical peer review is a well-established instrument for improving quality of care, clinical effectiveness is unclear.
Methods In a pragmatic cluster randomised controlled trial, we randomly assigned 60 German Initiative Qualitätsmedizin member hospitals with the highest mortality rates in ventilated patients in 2016 to intervention and control groups. The primary outcome was hospital mortality rate in patients ventilated fore more than 24 hours. Clinical peer review was conducted in intervention group hospitals only. We assessed the impact of clinical peer review on mortality using a difference-in-difference approach by applying weighted least squares (WLS) regression to changes in age-adjusted and sex-adjusted standardised mortality ratios (SMRs) 1 year before and 1 year after treatment. Recommendations for improvement from clinical peer review and hospital survey data were used for impact and process analysis.
Results We analysed 12 058 and 13 016 patients ventilated fore more than 24 hours in the intervention and control hospitals within the 1-year observation period. In-hospital mortality rates and SMRs were 40.6% and 1.23 in intervention group and 41.9% and 1.28 in control group hospitals in the preintervention period, respectively. The groups showed similar hospital (bed size, ownership) and patient (age, sex, mortality, main indications) characteristics. WLS regression did not yield a significant difference between intervention and control groups regarding changes in SMRs (estimate=0.04, 95% CI= −0.05 to 0.13, p=0.38). Mortality remained high in both groups (intervention: 41.8%, control: 42.1%). Impact and process analysis indicated few perceived outcome improvements or implemented process improvements following the introduction of clinical peer review.
Conclusions This study did not provide evidence for reductions in mortality in patients ventilated for more than 24 hours due to clinical peer review. A stronger focus on identification of structures and care processes related to mortality is required to improve the effectiveness of clinical peer review.
- cluster trials
- quality improvement
- randomised controlled trial
- critical care
Data availability statement
No data are available. Data are not publicly available due to confidentiality restrictions.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Key messages
What is already known on this topic
Clinical peer review is a well-established method that aims to improve the quality of inpatient care and patient outcomes. While observational studies indicated decreasing mortality rates in multiple patient groups after the introduction of clinical peer review in German hospitals, there is a lack of high quality confirmatory evidence.
What this study adds
We assessed the effect of clinical peer review on mortality in patients ventilated for more than 24 hours in a cluster randomised controlled trial. We did not find evidence for reductions in mortality due to clinical peer review. Impact and process analysis indicated few perceived outcome improvements or implemented process improvements following the introduction of clinical peer review.
How this study might affect research, practice and/or policy
A stronger focus on structures and care processes related to mortality is required to improve the effectiveness of clinical peer review.
Introduction
The COVID-19 pandemic has highlighted the crucial role of long-term ventilation in the care of critically ill patients.1 Due to different life-threatening indications and the risk of severe complications,2 3 treatment of intensive care patients is a complex task. Therefore, ensuring a high quality of care with respect to treatment of those patients is a main objective of intensive care medicine.
In addition to new medical therapies, structured ventilation protocols and improved infrastructure, the organisational aspects of care, interprofessional collaboration and quality improvement methods have increasingly been gaining clinical and scientific interest.4 5 One well-established quality improvement method is clinical peer review (clinical PR).6–8 In Germany, Initiative Qualitätsmedizin (IQM) hospitals introduced clinical PR in 2009. Within 10 years (2009–2019), 1208 clinical PRs were performed. With more than 400 member hospitals, IQM covers approximately 40% of all German hospital cases per year.9 Clinical PRs may be triggered by quality indicator results (eg, above-average mortality rates in specific patient groups) and by the voluntary participation of member hospitals.
Observational studies indicated decreasing mortality rates after the introduction of clinical PR.10–12 However, these findings may be driven by phenomena like regression to the mean bias,13 which implies the need for high quality confirmatory evidence on the effectiveness of clinical PR.14
Against that background, we conducted a pragmatic cluster randomized controlled trial to assess the effectiveness of clinical PR in reducing hospital mortality in patients ventilated for more than 24 hours.
Methods
Trial design and oversight
This was a pragmatic multicentre cluster randomised controlled trial embedded in a prospective cohort study (figure 1). All 385 IQM member hospitals in Germany were invited to participate in the study. After consent, 60 participating hospitals with the highest mortality rates in patients ventilated for more than 24 hours in 2016 were included in the cluster trial as these hospitals were expected to show the highest potential for benefit from clinical PR. All remaining hospitals formed a parallel observation arm. The 60 hospitals included in the trial were randomised into intervention and control groups with an allocation ratio of 1:1. Clinical PR was conducted in intervention group hospitals, and control group hospitals did not receive this intervention. The effectiveness of clinical PR was assessed by comparing intervention and control group hospitals regarding changes in the primary outcome 1 year before and 1 year after clinical PR. The intervention was introduced in the second half of 2017.15
Design of the ‘Effectiveness of the IQM peer review procedure to improve in-patient care—a pragmatic cluster randomized controlled trial’ (IMPRESS) study. IQM, Initiative Qualitätsmedizin.
While the exact pretreatment and post-treatment periods were defined by the clinical PR date for intervention group hospitals, these periods were set from July 2016 to June 2017, and July 2017 to June 2018 for all control group hospitals.
Patient characteristics were gathered from routine care data collected by each hospital according to German law (§21 Krankenhausentgeltgesetz (KHEntgG)). Hospital characteristics were derived from the German Hospital Directory provided by the German Federal Office of Statistics. Data linkage and anonymisation was conducted by the Koordinierungszentrum für Klinische Studien, Technische Universität Dresden (TU Dresden). Data were analysed at the Center for Evidence-Based Healthcare (ZEGV), TU Dresden. The responsible statistician was blinded throughout the trial conduct and analysis.
For impact and process analysis, we extracted information from the lists of recommendations for improvement made by the clinical PR teams visiting the intervention group hospitals and conducted a hospital survey at the end of the study period (January 2019 to June 2019). The study design is described in detail elsewhere.16
Randomisation
We allocated the 60 intervention study hospitals to intervention and control groups using block randomisation conditional on hospital ownership (public, private, non-profit) and the number of hospital beds (1–149, 150–299, 300–599, 600+) with a block size of 10. Randomisation was conducted at ZEGV. IQM was responsible for the enrolment and assignment of hospitals to intervention and control groups.
Intervention
The IQM clinical PR intervention includes three stages: preparation, implementation and follow-up (a more detailed description of IQM clinical PR is provided in the online supplemental material). The preparation stage includes patient case selection and a self-assessment by the hospital selected for clinical PR. The key element of the implementation stage is the external, on-site assessment. This assessment includes structured dialogue between reviewers and hospital staff with the objective to agree on measures for improvement of care. The main element of the follow-up stage is the reviewer report. This report considers the following criteria:
Supplemental material
Adequate and prompt diagnostics and treatment.
Prompt and targeted examination of treatment process.
Adequate and prompt indication.
Guideline adherence.
Control of the course of treatment.
Conflict-free interdisciplinary and interprofessional cooperation.
Coherent and complete documentation.
By focusing on these criteria, the intervention aims to improve inpatient care through an improvement of key structures and processes. Measures for improvement are mutually defined by staff of the visited hospitals and clinical PR reviewers and transferred into a feasible action plan by the reviewed hospital. Clear and precise formulation of potentials for improvement to derive this action plan is the main focus of the clinical PR report. The visited hospital is responsible for both the action plan and its implementation.
Our study followed the IQM methodology and conducted clinical PRs in line with international standards. The clinical PR included the ex-post review of 12–16 charts of patients who were ventilated for more than 24 hours and deceased in the reviewed hospital. Preference was given to deceased patients with characteristics that are associated with a relatively low probability of death (eg, younger age). The reviews were conducted by physicians and nurses from other IQM member hospitals, who had been trained in clinical PR. The training placed special emphasis on avoiding sham peer reviews.8 Potentials for improvement identified by the clinical PR team and recommended measures were discussed with the clinical management of the reviewed hospital and documented in a list of recommendations. Based on these recommendations, reviewed hospitals decided on implementation of specific measures (see online supplemental figure S1 for visualisation of the process).
Outcome
The outcome of the intervention study was in-hospital mortality in patients ventilated for more than 24 hours, excluding newborns aged ≤27 days. This outcome was chosen because of its high prevalence and large reductions in mortality in ventilation patients after clinical PR found in previous observational studies.10–12 Subgroup analyses were conducted for patients ventilated for more than 24 hours with myocardial infarction, stroke, pneumonia, chronic obstructive pulmonary disease (COPD) and colorectal resection because of their relatively large share in total ventilation patients and their higher degree of homogeneity compared with the overall patient population.16 The definitions of patient population and subgroups followed the German Inpatient Quality Indicators version 5.017 (indicator 56.1) (see online supplemental table S1).
Impact and process analysis
To gain insights into measures, processes and perceived outcomes during and after clinical PR, we extracted information from the intervention group hospitals’ lists of recommendations for improvement. These lists of recommendations were written by the leading reviewer and sent to intervention hospitals for their information. Information on documented potentials for improvement and recommended measures was included in an individualised hospital survey. All intervention group hospitals were invited to participate in this survey. In this way, we assessed whether specific measures recommended during the clinical PR process had been implemented in the relevant hospital. We also assessed perceived changes in structure, process and outcome due to clinical PR. The survey was conducted online using REDCap (V.8.5.8, Vanderbilt University, Nashville). Descriptive quantitative analysis of survey data was performed using SPSS (V.25.0.0.2). Qualitative analysis was conducted by two independent raters using MAXQDA (V.11.1.2) coding qualitative contents into iteratively defined subthemes. Both raters independently rated a sample of 10 questionnaires to define a common coding framework. After discussion and consensus, we applied this coding framework to all questionnaires.
In addition to implementation of measures recommended during clinical PR, the potential of these measures to affect mortality is a requisite for outcome improvements. The mortality relevance of the measures extracted from the lists of recommendations for improvement therefore was independently rated by four clinical experts. Each expert received the full list of recommended measures documented in the intervention study hospitals’ lists of recommendations for improvement and was asked to rate the mortality relevance of these measures on a 4-point scale (1: almost surely relevant; 2: likely relevant; 3: likely irrelevant; 4: almost surely irrelevant). The median rating by the experts was used to assess the potential effectiveness of the intervention and to explain potentially heterogeneous or null results of the treatment effect analysis. Details on this expert rating and the hospital survey are provided in the online supplemental material.
Supplemental material
Statistical analysis
Power simulation was conducted before the start of the study. Given a significance level of 5% and assuming 30 hospitals per study group, an expected number of 300 ventilation cases per hospital, an average baseline mortality rate of 38% and an intraclass correlation of 0.00408,18 a relative reduction in mortality by 8% due to clinical PR could be detected with a power of 80% using a t-test for unpaired samples.
Hospital and patient characteristics in the preintervention period were analysed using descriptive statistical techniques. Confirmatory analysis was based on a difference-in-difference approach considering the difference between intervention and control group hospitals with respect to changes in the age-adjusted and sex-adjusted standardised mortality ratio (SMR) in the postintervention period compared with the preintervention period. We applied weighted least squares (WLS) regression with the change in the hospitals’ SMRs as a dependent variable and a dummy for intervention group hospitals as an independent variable. A t-test of the intervention group dummy was used to assess the statistical significance of the treatment effect. The number of cases ventilated for more than 24 hours during the observation period at each hospital was used as weight to account for different precisions of SMR estimates. All observation arm hospitals in 2016 were used as an external reference group to derive expected mortality rates required for the calculation of SMRs. In this regard, we verified that expected mortality rates could be derived for all patient strata treated in intervention and control group hospitals within the study period. Subgroup analyses were conducted for patients ventilated for more than 24 hours with myocardial infarction, stroke, pneumonia, COPD and colorectal resection, respectively. For these analyses, we considered the point estimate of the coefficient of the intervention group dummy and its 95% CI. The confirmatory analyses reported in this paper were complemented by comprehensive sensitivity analyses, including adjustment for covariates, individual-level regressions, time lags and time-varying effects of the intervention (see online supplemental material). Statistical analyses were conducted using Stata V.15.1.
Supplemental material
Results
Treatment allocation and hospitals included in final analysis
Two hundred and thirty-seven hospitals agreed to participate (figure 2). Fourteen did not treat patients ventilated for more than 24 hours in 2016 and therefore were not eligible. From the remaining 223 hospitals, the 60 hospitals with the highest mortality rates in 2016 were randomised into intervention and control groups with an allocation ratio of 1:1. While all intervention group hospitals were included in the analysis, one control group hospital was ‘lost to follow-up’ due to a merger within the study period. Accordingly, the analyses were based on data from 30 intervention and 29 control group hospitals with 12 085 and 13 016 cases of patients ventilated for more than 24 hours during the pretreatment and post-treatment periods, respectively.
Flow chart showing allocation of hospitals to study groups and inclusion in final analyses. IMPRESS, ‘Effectiveness of the IQM peer review procedure to improve in-patient care—a pragmatic cluster randomized controlled trial’.
Baseline characteristics of hospitals and patients
Block randomisation resulted in similar characteristics in intervention and control group hospitals with respect to number of beds and hospital ownership in the preintervention period (table 1). The intervention group hospitals were characterised by a higher median number of ventilation cases than the control group hospitals but showed a similar IQR. The baseline mortality rates in ventilation patients were 40.6% and 41.9% in the intervention and control groups, respectively. There were no relevant differences in terms of age, sex and the shares of the considered subgroups (myocardial infarction, stroke, pneumonia, COPD, colorectal resection) in ventilation patients.
Baseline (preintervention period) characteristics of hospitals and patients ventilated for more than 24 hours
Main results
SMRs of the control group hospitals (SMR=1.28) and the intervention group hospitals (SMR=1.23) were similar in the preintervention period (figure 3). The SMRs of both groups remained almost unchanged in the postintervention period (control: SMR=1.29; intervention: SMR=1.26).
Evolution of standardised mortality ratios (SMRs) in intervention and control group hospitals with 95% CIs.
These findings were also reflected in the results of the difference-in-difference analysis. The WLS regression yielded a statistically insignificant point estimate (estimate=0.04, 95% CI= −0.05 to 0.13, p=0.38) of the intervention effect. Hence, there was no evidence for a beneficial effect of clinical PR on mortality in patients ventilated for more than 24 hours. All sensitivity analyses were in line with this result as they did not provide evidence for a treatment effect (see online supplemental material).
Subgroup analyses
We also estimated the potential effects of the clinical PR on mortality in patients ventilated for more than 24 hours with myocardial infarction, stroke, pneumonia, COPD and colorectal resection, respectively. In line with the main results outlined above, the point estimates and 95% CIs did not indicate a treatment effect in these subgroups (figure 4). More detailed information is provided in online supplemental table S2.
Results of weighted least squares (WLS) regressions for subgroups of patients ventilated for more than 24 hours with 95% CIs. COPD=chronic obstructive pulmonary disease.
Survey findings
We conducted the impact and process improvement survey from 7 January 2019 to 30 April 2019 and sent a reminder on 18 March 2019. The final response rate was slightly higher in the intervention group (26/30; 80%) compared to the control group (20/29; 69%).
A minority of intervention hospitals (5/26) and control hospitals (7/20) indicated that they had received a clinical PR on ventilation prior to participating in the current cluster randomized controlled trial. We extracted 132 recommendations for quality improvement from the intervention group hospitals’ lists of recommendations for improvement. The majority (116/132, 88%) of these recommendations were transferred unchanged (n=82) or modified (n=34) into an action plan.
Overall, 81 recommendations for quality improvement measures were already implemented or about to be implemented in the intervention group hospitals at the time of the survey. Most of the implemented recommendations aimed at improving procedures and structures of care including documentation (26/81), care strategies (16/81) and updated/additional standard operating procedures (11/81). Two-thirds of these implemented recommendations were rated by experts as ‘likely irrelevant’ (median rating=3–3.5: 21/81; 26%) or ‘almost surely irrelevant’ (median=4: 32/81; 40%) for in-hospital mortality in patients ventilated for more than 24 hours. Recommendations targeting improvements in documentation were rated as ‘likely irrelevant’ or ‘almost surely irrelevant’. Implemented recommendations targeting treatment protocols of ventilation or reanimation were rated as ‘likely relevant’.
The quantitative evaluation of implemented measures by participating intervention hospitals indicated a fairly high level of perceived structural and procedural (19/26; 73%) improvements. The qualitative evaluation described improvements due to structural recommendations, particularly in the form of new standards and infrastructures (11 quotes). Procedural recommendations leading to improvement were noted in 31 quotes addressing care processes or documentation. In line with the results of the main outcome analysis, perceived reductions in mortality of ventilated patients were rarely reported (three quotes).
Discussion
This study improves upon previous observational research10–12 on the effects of clinical PR as a measure to improve the quality of care in intensive care medicine. Neither the main results nor the subgroup and sensitivity analyses provided evidence for a beneficial effect of clinical PR regarding ventilation on mortality in patients ventilated for more than 24 hours. In line with these results, the hospital survey indicated that many of the recommendations made to improve the quality of care had been implemented, but that these measures were not considered to have a direct impact on the mortality rates in patients ventilated for more than 24 hours.
The survey results showed that clinical PR was perceived positively by the participating hospitals and many measures recommended during clinical PR were implemented in intervention group hospitals. The expert rating indicated that most of these measures were not seen as directly related to mortality in ventilated patients. This reflects that many potentials for improvement were identified which did not directly affect mortality but may have had other beneficial effects. If the above-average baseline mortality in the participating hospitals was driven by deficits in quality of care, clinical PR may not, therefore, have adequately identified these deficits.
Strengths and limitations
This study used a randomised design to prevent regression to the mean, unmeasured confounding and selection biases, and to ensure a high degree of generalisability. A limitation of the study is that, due to the study design, the effectiveness of clinical PR as a complex intervention could only be assessed as a whole and its impacts on outcomes other than mortality were not captured. The high number of measures recommended during clinical PR may primarily affect process quality, which was not considered as an outcome. However, in line with the value-based healthcare framework,19 patients are mainly interested in health outcomes rather than the structural or procedural quality of care.
The reliability of judgements by reviewers and the adequacy of recommended measures for improvement of inpatient care are crucial if clinical PR is to be effective. The IQM clinical PR is strictly standardised regarding both training and implementation. The IQM clinical PR report addresses specific topics related to the quality of care, including structures and especially processes. The clinical PR team represents different professions (physicians, nurses), specialties (eg, anaesthesiology, pulmonology) and hierarchical positions (eg, senior physicians, resident physicians). We believe that the high degree of standardisation and the interdisciplinary and interprofessional nature of the IQM clinical PR are likely to facilitate adequate judgements of potentials and flaws and facilitate the derivation of suitable measures for improvement. However, all these advantages did not translate into mortality benefit for patients.
Due to its focus on clinical structures and processes, the IQM clinical PR does not consider external drivers of poor quality and patient outcomes, including characteristics of the healthcare system. Therefore, identifying those external potentials for improvement requires different complementary approaches.
Another limitation of our study is the possibility that intervention effects may have evolved over time and that longer follow-up may have been valuable. However, sensitivity analyses did not provide support for time-varying intervention effects (see online supplemental material).
Despite the robustness of our findings against adjustment for multiple patient characteristics in sensitivity analyses (see online supplemental material), the persistence of above-average mortality in both intervention and control hospitals may be due to patient-associated risk not captured by our data. While our data provide information on patient-specific risk factors for mortality, for example, in terms of Elixhauser comorbidities, some clinical indicators of disease severity (eg, laboratory and imaging findings) could not be operationalised. However, since our study relies on a randomised design, we do not expect systematic differences between intervention and control groups regarding those characteristics.
In line with IQM methodology, the selection of intervention and control hospitals was based on the assumption that hospitals with the highest mortality rates in specific patient groups have the largest potential for improvement in terms of quality of care. Although previous observational studies on IQM clinical PR seemed to support this assumption, we cannot exclude that modifiable mortality in the participating hospitals was actually too low to induce a significant intervention effect.
This study did not investigate differences between hospitals with low and high mortality rates in patients ventilated for more than 24 hours in terms of care characteristics (eg, use of lung-protective ventilation or measures aimed at preventing ventilator-associated pneumonia). Investigating such differences could shed more light on mortality-relevant potentials for improvement and could be used to derive suitable quality standards for future clinical PRs.
Finally, this study examined the effectiveness of IQM clinical PR. The extent to which our results are generalisable to other variants of clinical PR may depend on specific methodological aspects, including the degree to which the considered clinical PR procedure focuses on measures related to mortality.
Implications
The COVID-19 pandemic has highlighted the role of long-term ventilation as a key element in the care of critically ill patients.1 Ensuring and improving the quality of care of ventilation patients is the main objective of this study. The lack of outcome improvements indicated by our analysis has major implications for the conceptualisation of clinical PR. Potential modifications of the clinical PR procedure include a more specific focus on outcome quality and stricter monitoring of changes in structures and procedures after clinical PR. Those modifications would require enhanced identification of mortality-relevant care processes and targeted measurement of mortality related to these processes.
Alternatively, the focus of future clinical PR may be shifted from outcome quality towards process quality. The latter would require measurement of process-related outcomes to assess effectiveness and different triggers for clinical PR. Modification of clinical PR may also require improved training of clinical PR teams, better documentation in lists of recommendations for improvement and more emphasis on implementation of sustainable measures. These issues have been recognised by the IQM steering group, which defined an action plan regarding the IQM clinical PR method. IQM clinical PR reports and trainings have been restructured with stronger focus on potentials to reduce mortality.
A general implication of this study is that the effectiveness of the clinical PR is under scrutiny. As with all complex interventions, reliable evidence on the effective components of clinical PR is required to improve the method and justify its continued use.
Supplemental material
Data availability statement
No data are available. Data are not publicly available due to confidentiality restrictions.
Ethics statements
Patient consent for publication
Ethics approval
Institutional Review Board (IRB) TU Dresden: Office for Human Research Protections (OHRP); identification numbers: IRB00001473 and IORG0001076.
Acknowledgments
The authors thank Jochen Strauß, Claudia Winklmair, the IQM executive board and all participating hospitals.
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
JS and MR are joint first authors.
JS and MR contributed equally.
Contributors JS and OS designed the study. MR and XG prepared the RCT data. MR analysed the RCT data. JS, FW and OS validated the analyses. FW and OS designed the hospital survey. FW analysed the survey data. PS, ME-G and RK provided clinical and methodological expertise regarding IQM peer review. All authors discussed and interpreted the empirical findings. MR drafted the manuscript and all the remaining authors critically revised the manuscript. JS is responsible for the overall content as guarantor.
Funding This study was funded by the Joint Federal Committee (Gemeinsamer Bundesausschuss, G-BA), Germany (grant number: 01VSF16013).
Competing interests All authors report grant from Innovationsfonds des GBA for the conduct of the study. JS reports grants from ALK, Novartis, Pfizer, Sanofi; personal fees from ALK, Novartis, Lilly, Sanofi outside the submitted work. OS reports personal fees from Novartis outside the submitted work. RK reports that as an employee of Helios, a subsidiary of Fresenius, he is working for a healthcare company and is holding Fresenius stocks and stock options. XG reports grants from the German Federal Ministry of Health during the conduct of the study. XG further reports grants from Dräger AG & Co KGaA, Aesculap, Gesundheitsforen Leipzig, Novartis Pharma, University of Rostock and medichema outside the submitted work.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.