Article Text
Abstract
Background In 2015, an academic-led surgical quality improvement (QI) programme was initiated in Japan to use database information entered from 2013 to 2014 to identify institutions needing improvement, to which cardiovascular surgery experts were sent for site visits. Here, posthoc analyses were used to estimate the effectiveness of the QI programme in reducing surgical mortality (30-day and in-hospital mortality).
Methods Patients were selected from the Japan Cardiovascular Surgery Database, which includes almost all cardiovascular surgeries in Japan, if they underwent isolated coronary artery bypass graft (CABG), valve or thoracic aortic surgery from 2013 to 2016. Difference-in-difference methods based on a generalised estimating equation logistic regression model were used for pre-post comparison after adjustment for patient-level expected surgical mortality.
Results In total, 238 778 patients (10 172 deaths) from 590 hospitals, including 3556 patients seen at 10 hospitals with site visits, were included from January 2013 to December 2016. Preprogramme, the crude surgical mortality for site visit and non-site visit institutions was 9.0% and 2.7%, respectively, for CABG surgery, 10.7% and 4.0%, respectively, for valve surgery and 20.7% and 7.5%, respectively, for aortic surgery. Postprogramme, moderate improvement was observed at site visit hospitals (3.6%, 9.6% and 18.8%, respectively). A difference-in-difference estimator showed significant improvement in CABG (0.29 (95% CI 0.15 to 0.54), p<0.001) and valve surgery (0.74 (0.55 to 1.00); p=0.047). Improvement was observed within 1 year for CABG surgery but was delayed for valve and aortic surgery. During the programme, institutions did not refrain from surgery.
Conclusions Combining traditional site visits with modern database methodologies effectively improved surgical mortality in Japan. These universal methods could be applied via a similar approach to contribute to achieving QI in surgery for many other procedures worldwide.
- quality improvement
- surgery
- audit and feedback
Statistics from Altmetric.com
Introduction
In the past several decades, various quality improvement (QI) efforts have been made for the purpose of improving surgical outcomes. In the 1980s, in the northern New England region of the USA, a QI programme including facility visits was implemented, after which the surgical outcomes in the area were dramatically improved.1 2 In the 1990s, Veterans Affairs began collecting clinical baseline and outcome data, evaluating the quality of medical care and developing benchmarking methods; these measures succeeded in improving the quality of surgery.3–6 These efforts have continued to the present day and have shifted to a collaborative QI framework that uses large-scale databases, such as the Society of Thoracic Surgeons (STS) database7 and the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP).8
However, even after all of these efforts, variations in surgical outcomes still remain. The previous QI programmes suffered from the following problems: (1) in collaborative QI programmes, each facility decides to participate; therefore, the facilities that truly need improvement do not always participate and (2) these programmes require considerable resources; especially for site visit programmes, it is completely impractical for a small number of inspectors to visit many facilities.
In 2000, the Japan Cardiovascular Surgery Database (JCVSD) was established with the support of the STS database. Beginning in 2012, almost all cardiovascular surgery data were registered because certification by the Japan Surgical Boards system was based on the JCVSD. Utilisation of the database has enabled us to conduct several clinical epidemiological9–12 and health service studies.13 14 To overcome the aforementioned problems, a joint QI programme between the JCVSD and the Japanese Society for Cardiovascular Surgery began in 2015. This QI programme involved the following process: (1) the identification of target institutions that need improvement in surgical outcomes based on database information and (2) audits of the institutions, chart reviews of cardiovascular surgical death cases and the presentation of improvement strategies by cardiovascular surgery experts.
The present study aimed to estimate the effect of the QI programme on surgical mortality using the JCVSD.
Methods
Design
Posthoc retrospective analyses for the implemented programme.
Database
The JCVSD adult division is a nationwide surgical registry that began in 2001 and includes approximately 460 000 cases from 590 institutions that were registered in 2001–2016.9 15–17 The database includes more than 95% of the cardiovascular surgeries performed in Japan. The database records preoperative status, surgical procedure, intraoperative events and postoperative outcomes in a standardised electronic format and is highly comparable to the STS National Database.18 The validity of the data was confirmed by an audit. Additionally, using a web-based system, each hospital was able to benchmark its own activities based on the input data.
Targeted population
The targeted population comprised all patients who underwent isolated coronary artery bypass graft (CABG) surgery, valve operations or thoracic aortic operations from 2013 to 2016 as well as candidates for the expected surgical mortality (JapanSCORE) calculation using previously established risk models.9
Selection of candidate hospitals for site visits
In 2015, a joint QI programme was initiated between the JCVSD and the Japanese Society for Cardiovascular Surgery. This joint programme used data entered in the aforementioned database in 2013–2014 and calculated both the observed surgical mortality and the expected surgical mortality (JapanSCORE) using previously established risk models9 for the three procedures of interest. For each institution and procedure, the risk-adjusted O/E ratio for surgical mortality and its 99% Poisson CI were calculated using the observed and expected mortality rates for the institution.8 19 The programme identified institutions that were high-mortality outliers for at least one procedure and selected these institutions as candidates for site visits. Finally, after the hospital surgery volume, the number of procedures that were outliers and the O/E ratio were accounted for, 10 hospitals were selected for site visits. Each of these hospitals was informed that it was a high-mortality outlier and a candidate for a site visit and asked to cooperate with a site visit by the Japanese Society for Cardiovascular Surgery committee. This process was implemented in the first half of 2015 (2015H1); the postprogramme period started after this process was implemented.
Intervention; details of the site visits
The site visits were conducted early in the second half of 2015 (2015H2) (table 1). Three or four physicians who were directors of the Japanese Society for Cardiovascular Surgery confidentially visited the selected hospitals. The process of inspection consisted of the following: (1) a description of the ‘structure’ of the hospital (eg, characteristics of the hospital and the numbers of beds, physicians, surgeons and nurses); (2) the identification of the ‘process’ used for perioperative management (eg, the use of preoperative antibiotics, the method of myocardial protection and cardiopulmonary bypass and the conduction of preoperative conferences and mortality and morbidity conferences); (3) chart reviews of all patient deaths that occurred after cardiovascular surgery during the study period and a discussion of whether each death was preventable and, finally, (4) a discussion of the results of the inspection, especially with regard for areas requiring improvement, and the determination of a plan for future improvement. The results of the site visits were summarised as reports and confidentially shared with the Japanese Society for Cardiovascular Surgery and each institution. Additionally, each hospital made voluntary improvements based on the report. The surgical performance of each hospital was monitored and tracked by a web-based electronic system.
Outcome measures
The primary outcome measure was surgical mortality. Surgical mortality was defined as (1) death within 30 days (regardless of hospitalisation) or (2) death during the index hospitalisation.9 This outcome measure has been used in several surgical database studies.20–22
Statistical analysis
The χ² test or Fisher's exact test was used to compare categorical data and their distributions. The Mann–Whitney U test or Kruskal-Wallis test was used to compare non-normally distributed data. All p values are two-sided, and p<0.05 was considered statistically significant.
The research question was whether the QI programme improved surgical mortality. To assess the impact of the QI programme, we used difference-in-difference methods (DIDs) to compare the risk-adjusted surgical mortality before and after the programme. In this model, we used a generalised estimation equation fitted with logistic regression to account for the clustering of the patients at the hospital level. We selected an exchangeable correlation matrix structure for the analysis. This analysis included surgical mortality as a dependent variable and the following independent variable: expected surgical mortality calculated based on previously established risk models for each procedure (JapanSCORE).9 We initially used the first half of 2015 (2015H1) as the cut-off point for programme effectiveness. Additionally, as a sensitivity analysis, we then also used the second half of 2015 (2015H2) as another cut-off point for considering the transitional effect of the site visit.
Furthermore, we assessed the time-series trend in risk-adjusted mortality between site-visit and non-site-visit hospitals. In this analysis, surgical mortality was included as a dependent variable, and expected surgical mortality (JapanSCORE) and the interaction between time (in half-year increments) and site-visit status were included as independent variables.
All statistical analyses were conducted with STATA 15 software (Stata, College Station, Texas, USA).
Results
Patient characteristics in the preprogramme and postprogramme periods
In total, 238 778 patients from 590 hospitals were identified between 2013 and 2016; these included 56 923 patients who underwent CABG procedures, 84 576 who underwent valve procedures and 68 129 who underwent aortic procedures (table 2 and online supplementary figure 1). The site visits were implemented in 10 hospitals that included 3556 patients (922 patients with CABG procedures, 1279 with valve procedures and 1355 with aortic procedures). During the preprogramme period, the observed surgical mortality rates were significantly higher in the site-visit group than in the non-site-visit group for all procedures (p<0.001). It should be noted that the expected surgical mortality rates (JapanSCORE) were comparable between the site-visit and non-site-visit groups in the preprogramme period for CABG, valve and aortic procedures (p=0.429, 0.904 and 0.779, respectively).
Supplemental material
Regarding pre-post comparisons of crude data, among CABG recipients, the observed surgical mortality was lower for site-visit hospitals than for those without site visits. Some improvements in complication rates were observed in parameters such as the length of stay for CABG surgery, newly required dialysis for CABG recipients and reoperation for haemorrhage in recipients of CABG and aorta surgery.
Effect of the QI programme and its time trend
The DID estimator showed that there was significant improvement in isolated CABG (OR 0.30 (95% CI 0.19 to 0.46), p<0.001) after the QI programme. However, when the same cut-off point was used, the DID estimators for valve and aortic procedures were 0.93 (95% CI 0.63 to 1.38, p=0.72) and 0.85 (95% CI 0.54 to 1.34, p=0.49), respectively. In the sensitivity analysis, when we set the cut-off period as the second half of 2015 (2015H2), we observed significant improvement in CABG and valve procedures (OR 0.29 (95% CI 0.15 to 0.54), p<0.001 for CABG procedures and OR 0.74 (0.55 to 1.00); p=0.047 for valve procedures). However, for aorta procedures, the DID estimator was not significant when the cut-off point was set as the second half of 2015 (2015H2) (OR 0.85 (0.52 to 1.39); p=0.51).
Table 3 and figure 1 show the time trends in the adjusted odds ratio (AOR) for surgical mortality adjusted by patient-level characteristics in the site visit and non-site visit groups. In the non-site visit group, the AORs were nearly stable for isolated CABG and valve procedures; however, significant improvement was observed in the aortic procedure. In the site visit group, the AORs were consistently high for all procedures during the preprogramme period. After the implementation of the QI programme, for CABG, the AOR rapidly improved and became comparable with that of the non-site visit group. For valve procedures, slight improvement was observed after the second half of 2015 (2015H2); however, the AOR of the site visit group remained higher than that of the non-site visit group. For the aorta procedure, improvement was delayed, and the AOR was comparable with that of the non-site visit group in the second half of 2016 (2016H2).
Changes in the severity of patient condition over time and the effect of the QI programme
Figure 2 shows the time trend in the distribution of expected surgical mortality (JapanSCORE) in the site visit group across the study period. For the isolated CABG procedure, the patient distribution in the early period after the programme (in 2015) was comparable to that reported during the preprogramme period. However, in the late period, the number of patients with intermediate-risk to high-risk conditions increased (p=0.084). For the valve procedure, the severity of patient condition was nearly constant during the whole period (p=0.850), while for the aortic procedure, the number of patients with high-severity conditions increased in 2016 (p<0.001).
Discussion
In the present study, with the cooperation of the JCVSD and the Japanese Society for Cardiovascular Surgery, we used a nationwide clinical database to efficiently select target hospitals and performed site visits to the target hospitals, and we observed QI in cardiovascular surgery without hindrance in the activities of the facilities. This approach of combining traditional site visits and modern database methodology successfully and effectively enabled QI in cardiovascular surgery in Japan.
Previously, QI projects have been implemented in various ways to improve surgical outcomes. A site visit programme conducted by the Northern New England Cardiovascular Disease Study Group succeeded in improving surgical outcomes in the 1990s.2 23 A collaborative QI project, the STS, enabled the visualisation of ‘real’ surgical performance that considered patient-level risk factors.7 Additionally, ACS-NSQIP conducted a national multispecialty QI programme8 22 that was initiated by the Department of Veterans Affairs.3–6 These programmes made it possible to visualise and compare surgical outcomes across the participating facilities and greatly contributed to improvements in surgical outcomes. However, previous approaches have some limitations. First, visiting all candidate institutions requires a great deal of resources and is not realistic. Second, institutions that do not choose to participate in the programme were not evaluated. As a method of promoting QI in such uncovered facilities, incentives such as pay-for-performance have been tried in some countries. However, although pay-for-performance programmes are very rarely implemented for surgery, positive results were obtained in the timing of surgery for hip fractures.24 Given that the study design included limitations and that many negative results have been reported for the pay-for-performance strategy across the entirety of medical treatments, the positive effects of this approach are limited.25–28 A feature of Japanese medical care is that all citizens have universal healthcare insurance coverage,29 and patients are often discharged directly to their homes after improvement; hence, the length of hospital stay is longer in Japan than other countries. Additionally, the population is rapidly ageing, and the prevalence of high-risk cardiovascular surgery in elderly patients is increasing. In the present study, the crude surgical mortality of CABG was comparable or slightly higher than the results obtained using the STS database;30 however, the patients in the JCVSD dataset were older and had higher NYHA classes than were found in the STS data.31 Considering the medical environment in Japan (as described above), the results of the present study would be feasible and within the expected range.
Additionally, the present study shows that the trend in QI varied depending on the type of operation. For the CABG procedure, significant improvement was observed starting soon after programme implementation. However, the quality of the valve and aorta procedures did not improve immediately after the implementation of the programme. For the CABG procedure, the educational effect of site visits and presentations by experts may have been effective because these procedures are simpler than those required in valve or aorta procedures and because the management of cardiopulmonary bypass is also generic, while valve and aorta procedures are highly individualised. For valve procedures, the timing and indication for the procedure are important. For example, heart failure is often accompanied by preoperative conditions, cardiopulmonary bypass must be sustained longer than the time required for CABG, and its postoperative treatments are more complicated than those used following CABG. Furthermore, aorta procedures are often performed as emergency surgeries, and their indications are also essential; it is therefore often necessary to forgo surgical therapy. Since the strategies used for the operations vary, the management of cardiopulmonary bypass is also complicated by the need to protect multiple organs at different temperatures. Additionally, the postoperative management of valve and aorta surgery is complicated and requires a multidisciplinary approach to treating various complications. Therefore, these differences among the procedures may have affected the variability of timing and the effects achieved in the present programme.
Strengths and limitations
First, the present QI programme enabled us to overcome weaknesses of previous QI programmes; the present programme was based on a ‘high-risk approach’ to the efficient use of resources in the public health field,32 which enabled us to respond to challenging disparities in surgical outcomes. Because it was based on a large-scale database, it was possible to visualise the situation in all of Japan, which has a massive sample size and a high registration rate. Additionally, from outlier detection to postintervention follow-up, consistent follow-up can be achieved when the database is updated each year.
Another important strength of the programme was that it was implemented without disrupting the medical system around the hospitals: the number of surgeries remained stable, and there was no strong evidence indicating that hospitals refrained from conducting surgeries due to the programme. Regarding this point of view, few studies to date have evaluated the side effects of improvement programmes. If such a programme has a significant negative impact, and if the target hospital monopolises local medical care to the exclusion of any alternative institutions, the local healthcare system could be disrupted. This programme therefore presents a great advantage in medical economics and medical sociology because it is compatible with the continuation of usual medical care.
This study also has some limitations. First, it was not free from the phenomenon of regression to the mean. However, during the preprogramme period, surgical mortality rates were consistently high in the site-visit hospitals. Additionally, the parallel trend assumption was satisfied, as shown in figure 1. Therefore, we believe that the effect on the present analyses was minimal and the results are valid.33 Second, the programme was implemented at the institutional level, not at the level of the surgeon. We therefore could not identify whether the observed improvement was due to hospital-level factors or surgeon-level factors. Third, our follow-up period was relatively short, and long-term effects were not measured in the present study. Additionally, the present approach was based on the ‘high-risk approach’; however, to achieve universal QI, it will be necessary to use a ‘population approach’ based on the knowledge gained from this programme.
Conclusion
Combining traditional site visits and modern database methodologies effectively enabled the creation of academic-led surgical QI in Japan without impeding the functions of medical systems. Because the methods used in this programme are universal, a similar approach could also be implemented for many other procedures worldwide, thereby contributing to QI in surgery.
References
Footnotes
Contributors HM, KT, HY, NM, YU and ST conceived the project. HY, HM and EF determined the analytical methods, handled the data and performed the statistical analyses. All authors contributed to the data interpretation. HY and HM wrote the first draft of the manuscript, and all authors reviewed and edited the drafts. All authors approved the final version.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests HY, HM and EF are affiliated with the Department of Healthcare Quality Assessment at the University of Tokyo. The department is a social collaboration department supported by grants from the National Clinical Database, Johnson & Johnson K.K and Nipro Co.
Patient consent for publication Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
Data availability statement Data are available on reasonable request.