Article Text
PDF
Abstract
Background Quality improvement (QI) campaigns appear to increase use of evidence-based practices, but their effect on health outcomes is less well studied.
Objective To assess the effect of a multistate QI campaign (Project JOINTS, Joining Organizations IN Tackling SSIs) that used the Institute for Healthcare Improvement’s Rapid Spread Network to promote adoption of evidence-based surgical site infection (SSI) prevention practices.
Methods We analysed rates of SSI among Medicare beneficiaries undergoing hip and knee arthroplasty during preintervention (May 2010 to April 2011) and postintervention (November 2011 to September 2013) periods in five states included in a multistate trial of the Project JOINTS campaign and five matched comparison states. We used generalised linear mixed effects models and a difference-in-differences approach to estimate changes in SSI outcomes.
Results 125 070 patients underwent hip arthroplasty in 405 hospitals in intervention states, compared with 131 787 in 525 hospitals in comparison states. 170 663 patients underwent knee arthroplasty in 397 hospitals in intervention states, compared with 196 064 in 518 hospitals in comparison states. After the campaign, patients in intervention states had a 15% lower odds of developing hip arthroplasty SSIs (OR=0.85, 95% CI 0.75 to 0.96, p=0.01) and a 12% lower odds of knee arthroplasty SSIs than patients in comparison states (OR=0.88, 95% CI 0.78 to 0.99, p=0.04).
Conclusions A larger reduction of SSI rates following hip and knee arthroplasty was shown in intervention states than in matched control states.
- quality improvement
- surgery
- evidence-based medicine
- nosocomial infections
Statistics from Altmetric.com
Introduction
In 2011, the Institute for Healthcare Improvement (IHI) launched Project JOINTS (Joining Organizations IN Tackling SSIs),1 a quality improvement (QI) campaign to disseminate a prevention ‘bundle’ aimed at reducing surgical site infection (SSI) following hip and knee arthroplasty. Project JOINTS was based on the spread theory, model and methods used in IHI’s previous 100 000 Lives and 5 Million Lives Campaigns, which aimed to accelerate hospital adoption of evidence-based practices to improve patient safety at state and national levels.2 3 This spread theory and model were adapted from previously published political campaign strategies, and the methods were predicated on spread of evidence-based practices beyond hospitals directly receiving the intervention.4 5
In states that participated in Project JOINTS, IHI leveraged a network of state ‘nodes’ (mainly state hospital associations and Quality Improvement Organizations supported by the Centers for Medicare & Medicaid Services). This so-called ‘Rapid Spread Network’ (RSN) was deployed to disseminate basic QI tools and to catalyse adoption of hip and knee arthroplasty infection prevention practices.
Hospitals that agreed to participate were encouraged to join campaign activities remotely (primarily via a webinar call series and an email listserv) or in person (‘town halls’ and site visits) and to use educational materials posted on the campaign website. IHI developed a variety of resources for participating hospitals, including: a how-to guide, improvement tools, a patient/family one-page overview, measurement tools, phone-based office hours and a business case analysis. Participating hospitals were also provided with instruction on using a multifaceted QI approach developed by IHI to motivate and support adoption of the SSI prevention bundle.6
The SSI prevention bundle included five evidence-based practices for all adult patients undergoing hip or knee arthroplasty.7 8 These were:
Preoperative screening for Staphylococcus aureus carriage (both methicillin-sensitive S. aureus and methicillin-resistant S. aureus (MRSA)) and decolonisation of carriers with 5 days of twice-daily intranasal mupirocin.9 10
Bathing or showering with 4% chlorhexidine gluconate for 3 days prior to surgery.11 12
Use of an alcohol-containing antiseptic agent for perioperative skin preparation.13
Avoidance of razors for hair removal.14
Appropriate timing of prophylactic antibiotics (including the addition of vancomycin in patients with known MRSA colonisation).15
A prior study showed that Project JOINTS increased adoption of the prevention bundle by 23%.16 Based on the known effectiveness of the bundle elements in reducing the risk of SSI following hip and knee arthroplasty,7–15 we hypothesised that campaign states would also show reductions in SSI. This paper presents an analysis of the change in SSI outcomes in five states that participated in Project JOINTS interventions beginning in May 2011. We compared these states with five matched comparison states that did not participate in the QI campaign.
Methods
Study population
Using Medicare Provider and Analysis Review (MedPAR) Part A Inpatient Claims data, we studied two clinical cohorts (hip arthroplasty and knee arthroplasty). The hip arthroplasty cohort included fee-for-service Medicare beneficiaries who underwent primary hip arthroplasty (International Classification of Diseases, Ninth Revision (ICD-9) codes 81.51 and 81.52) between May 2010 and September 2013, and the knee arthroplasty cohort included fee-for-service Medicare beneficiaries who underwent primary knee arthroplasty (ICD-9 code 81.54) between May 2010 and September 2013.17
Intervention states included Arkansas, Colorado, Michigan, New York and Tennessee. Prior to intervention, a comparison state was selected for each intervention state from a list of 18 states that had previously participated in the IHI RSN and expressed interest in participating in Project JOINTS. Matching was based on Medicare data for each state summarising the number of hospitals performing orthopaedic surgery and their orthopaedic surgical volume. This was done to balance the populations in intervention and control states and to reduce bias that might arise because of volume–outcome relationships. The five comparison states were Connecticut, Illinois, Missouri, Texas and Washington. These states did not receive the campaign interventions and thus served as a control.
We included data from all hospitals in the intervention and comparison states and did not limit analysis to hospitals that formally committed to participating in Project JOINTS campaign activities via a letter of commitment from hospital leadership (intention-to-treat analysis). This was decided a priori because of the likelihood of the spread of evidence-based practices beyond participating hospitals in the same state (so-called ‘spillover effect’), as seen in prior IHI campaigns.2 3 Indeed, such spread was encouraged by the state nodes and is considered desirable when conducting a campaign. To support this spread, the Project JOINTS team designed interventions to take advantage of existing professional and informal social networks within and across organisations. It was predicted that these networks would promote some spread of best practices from hospitals that signed up for Project JOINTS to those that did not.
We did not believe that there would be spread to states other than the intervention states because: (1) the selected states were geographically separated from one another; (2) only the intervention states were invited to participate in IHI’s campaign-associated activities; (3) only providers in intervention states were granted access to resources and tools on a dedicated Project JOINTS website; and (4) the professional societies supporting Project JOINTS agreed to disseminate guidance on practices included in the campaign only within the intervention states. As confirmation, we note that the randomised trial found no evidence of spillover of the intervention beyond the intervention states, while noting some uptake in non-participating hospitals in intervention states.16
Identifying SSI events
Following a hip arthroplasty or knee arthroplasty procedure, we analysed postoperative Medicare claims data for ICD-9 codes suggestive of an SSI within 90 days of the surgical procedure. We selected this surveillance window to be consistent with the current Centers for Disease Control and Prevention guidelines for SSI surveillance.18 Prior work has demonstrated that ICD-9 codes 996.66, 996.67 and 998.59 outperform traditional hospital-based surveillance and can be used to compare SSI outcomes across hospitals.19–21 Using linkers found in the MedPAR claims data, we analysed data both from the surgical admission and any readmissions to an acute care hospital in the 90-day surveillance window.
Main analysis
We defined three study periods: (1) preintervention period (12 months) from May 2010 through April 2011; (2) intervention (implementation) period (6 months)—a prespecified period from May through October 2011, during which hospitals were exposed to a series of Project JOINTS campaign activities; and (3) postintervention period (23 months) from November 2011 through September 2013. We excluded procedures from the last 3 months of 2013, as we did not have 2014 data to assess the full 90-day postoperative claims data for these procedures.
For the preintervention and the postintervention study periods, we calculated state-level SSI rates in the intervention and the comparison states, excluding data from the 6-month intervention (implementation) period. This was done separately for hip and knee arthroplasty.
We then used a generalised linear mixed effects model to predict the outcome of SSI. In this model, we adjusted for fixed effects at the patient level to ensure comparison of similar populations. These fixed effects included patient age, gender and each of the comorbidities in the Elixhauser comorbidity score handled dichotomously as yes/no variables.22 23 We also adjusted for random intercepts (random effects) for hospital and state to account for clustering,20 21 study arm (intervention vs comparison state), study period (postintervention vs preintervention) and an interaction term between the study arm and the study period. This interaction term provided the odds of an SSI in the postintervention versus preintervention period comparing outcomes in intervention versus comparison states. This is a difference-in-differences analysis, in which an OR <1 reflects a larger decline in the odds of an SSI in the intervention states relative to the comparison states following the intervention.
Finally, we used a linear regression model to test the parallel trends assumption in the preintervention period (May 2010 through April 2011). This was done to assess whether the slope of the monthly SSI rates was different across the 12 months in the intervention versus control states, so as not to attribute a lower SSI rate in the postintervention period to a trend that was occurring prior to the Project JOINTS interventions.24 25 We modelled the SSI rate, including terms for study arm (intervention vs comparison state), time (month) and an interaction term between the study arm and time. This interaction term assessed whether the slope over time was significantly different when comparing the trend in intervention versus comparison states.
Sensitivity analysis
To explore the impact on hospitals in intervention states that signed up to participate, we performed a sensitivity analysis. We modified the difference-in-differences analysis, limiting the intervention arm to hospitals that signed up to participate in the Project JOINTS campaign activities via a letter of commitment from hospital leadership during the intervention period. This allowed us to calculate the odds of an SSI in the postintervention versus preintervention period comparing outcomes in directly participating hospitals in intervention states versus all hospitals in comparison states.
Results
Study population
From May 2010 through September 2013, there were 125 070 Medicare beneficiaries who underwent primary hip arthroplasty in 405 hospitals in the five intervention states, and 131 787 Medicare beneficiaries who underwent primary hip arthroplasty in 525 hospitals in the five comparison states. In addition, there were 170 663 Medicare beneficiaries who underwent primary knee arthroplasty in 397 hospitals in the five intervention states, and 196 064 Medicare beneficiaries who underwent primary knee arthroplasty in 518 hospitals in the five comparison states.
Table 1 shows similar patient characteristics between Medicare beneficiaries who underwent hip arthroplasty table 1and knee arthroplasty table 1in intervention versus comparison states. These tables also include patient characteristics from hospitals that signed up to participate in the Project JOINTS campaign activities during the intervention period.
Patient characteristics for (A) hip arthroplasty procedures (May 2010 to September 2013) and (B) knee arthroplasty procedures (May 2010 to September 2013).
Project JOINTS enrolled 218 hospitals in the five intervention states, 171 (78%) by the May 2011 launch and 193 (89%) by the end of the intervention period. There were 25 additional hospitals (11%) that enrolled from November 2011 through May 2012. For the purpose of the sensitivity analysis focused on directly participating hospitals, we included the 193 hospitals in the five intervention states that enrolled during the intervention period (May to October 2011).
Overall, directly participating hospitals performed 50% of the hip arthroplasty procedures in intervention states in both the preintervention and postintervention periods. For knee arthroplasty, directly participating hospitals performed 52% of the procedures in intervention states in both the preintervention and postintervention periods.
Claims-based SSI rates
Table 2 shows the number of procedures, the number of SSIs and the SSI rate for the intervention and comparison states, based on Medicare claims. These data are broken down by study period (preintervention, postintervention) and by procedure type (primary hip arthroplasty, primary knee arthroplasty).
Claims-based SSI rates in the preintervention and postintervention periods
Figure 1 plots the SSI rates by month for hip arthroplasty across the three study periods. Figure 2 plots the SSI rates by month for knee arthroplasty across the three study periods. In both figures, we have added lines indicating the mean SSI rate in the preintervention and postintervention periods for intervention and comparison states.
Claims-based surgical site infection (SSI) rates for hip arthroplasty across study periods in intervention and comparison states.
Claims-based surgical site infection (SSI) rates for knee arthroplasty across study periods in intervention and comparison states.
For hip arthroplasty in the intervention states, the mean SSI rates in the preintervention and postintervention periods were 1.98% and 1.64%, respectively. In the comparison states, the mean SSI rates in the preintervention and postintervention periods were 2.18% and 2.14%, respectively.
For knee arthroplasty in the intervention states, the mean SSI rates in the preintervention and postintervention periods were 1.65% and 1.30%, respectively. In the comparison states, the mean SSI rates in the preintervention and postintervention periods were 1.64% and 1.45%, respectively.
For both procedures, the outcomes in individual intervention and comparison states are presented in online supplementary appendixes 1 and 2, along with an indication of which states were selected as pairs when selecting intervention and comparison states.
Supplemental material
Difference-in-differences analysis
In our difference-in-differences analysis comparing the risk-adjusted odds of developing an SSI following hip arthroplasty in the postintervention period compared with the preintervention period, we found a 15% greater decline in the odds of an SSI in intervention states relative to comparison states (OR 0.85, 95% CI 0.75 to 0.96, p=0.01). For knee arthroplasty, the risk-adjusted model found a 12% greater decline in the odds of an SSI in intervention states relative to comparison states (OR 0.88, 95% CI 0.78 to 0.99, p=0.04).
In our parallel trends analysis, the interaction term showed no significant difference in slope across the 12 months prior to intervention for either hip arthroplasty (p=0.91) or knee arthroplasty (p=0.27). This verified that the SSI rate trends were similar in the intervention and comparison states for both procedures prior to the Project JOINTS interventions.
Sensitivity analyses
When limiting the difference-in-differences analysis to directly participating hospitals in intervention states versus all hospitals in comparison states, we found a 23% greater decline in the risk-adjusted SSI rate for hip arthroplasty in intervention states relative to comparison states (OR 0.77, 95% CI 0.66 to 0.90, p<0.01). We did not detect a difference for knee arthroplasty (OR 0.97, 95% CI 0.84 to 1.12, p=0.69).
In the directly participating hospitals in intervention states, the mean SSI rates for hip arthroplasty were 2.19% preintervention and 1.63% postintervention. For knee arthroplasty, the mean SSI rates were 1.67% preintervention and 1.45% postintervention.
Discussion
We found that states participating in a QI campaign, leveraging IHI’s RapidSpreadNetwork and campaign methodology, had a significant reduction in SSIs following hip and knee arthroplasty, compared with a set of matched comparison states that were not participating in the QI campaign. These findings, along with a previous study demonstrating improved adherence to a ‘bundle’ of evidence-based practices known to reduce SSI,16 provide evidence that a well-conducted QI campaign, including a strong spread network infrastructure and multifaceted dissemination vehicles, can significantly impact patient outcomes on a large scale.
Our results are important because the evidence for the effectiveness of similar campaigns has been inconclusive and controversial. For example, a comparison of expected versus observed hospital mortality before and after the 100 000 Lives Campaign suggested that a reduction of more than 100 000 deaths was achieved. However, while this Campaign generated considerable enthusiasm, catalysed major national improvement initiatives in the USA and globally, and ‘changed the conversation’ about patient safety, the evaluation was limited because it did not include measures of adoption of recommended practices and did not include a control group to assess impact on associated outcomes.
The strategy for Project JOINTS was to publicise evidence-based elements of the SSI prevention bundle that were not yet in common practice, and to provide orthopaedic practices and hospitals with easy, free access to a campaign that provided practical tools and methods to help with adoption and implementation. Our assessment of Project JOINTS suggests that the IHI campaign methodology, including concerted, multifaceted dissemination of materials, engagement of state QI organisations and state hospital associations, and support from relevant professional societies, can successfully accelerate adoption and implementation of evidence-based practices and improve outcomes. This is important given long-standing evidence of delays in translating proven strategies from clinical trials into practice.26
It should be noted that our study design did not allow us to identify which components of the prevention bundle were most highly associated with the reduction in SSI rates at the patient or hospital level. We were also unable to definitively attribute potential indirect effects that the campaign may have had on hospitals not directly enrolled in Project JOINTS. The theory of the IHI spread network is that direct participation is not a requirement. Instead, non-participating organisations and professionals change their practices as they learn about or observe changes in practice of participating organisations and professionals.
To analyse the impact of these expected state-wide practice changes on SSI events, we compared outcomes in Medicare beneficiaries, as these patients’ claims data are accessible via a large, centralised database with good state penetration. We do not have information, however, on the per cent of the population covered by Medicare in each of the intervention and comparison states.
It is also important to note that our analyses relied on coding indicative of SSI using Medicare claims as a proxy for SSI events. While not all cases identified by claims would be expected to have a chart-confirmed SSI, previous nationally validated studies have shown that claims-based surveillance identifies a sizeable proportion of cases missed by traditional surveillance and that the SSI confirmation rate for records flagged by claims is highly consistent across hospitals.19–21 This is therefore a validated metric for comparing SSI performance, understanding the limitation that the aggregate codes are unable to differentiate superficial SSIs from deep and organ/space SSIs, and concerns about the sensitivity and specificity of billing codes used to identify SSI events.27 28 It should be noted, though, that these same studies highlight hip and knee arthroplasty as the procedures with the most data to support the use of claims-based surveillance, and we have no reason to suspect variability in the performance of these codes in the intervention versus comparison states in our study.
As for the observation that the decline in the risk-adjusted SSI rate for knee arthroplasty was no longer significantly greater in the sensitivity analysis comparing directly participating hospitals in intervention states versus all hospitals in comparison states, we have a few hypotheses why this might be. First, it is possible that hospitals that enrolled to participate in Project JOINTS in the intervention states were already working to reduce SSI rates prior to enrolment. Second, some of the hospitals that participated in Project JOINTS activities enrolled after the intervention period, such that impacts on these later participating hospitals would only be seen when looking at the data in our main analysis. Finally, the SSI rates for knee arthroplasty were lower than for hip arthroplasty, so there was less opportunity for improvement.
As for the observation that the decline in risk-adjusted SSI rate for hip arthroplasty was greater in the sensitivity analysis comparing directly participating hospitals in intervention states versus all hospitals in comparison states, it is worth pointing out that directly participating hospitals in the intervention states had a higher SSI rate in the preintervention period than non-participating hospitals in the intervention states. This may have been why these hospitals enrolled to participate.
Finally, data availability did not allow us to look at sustainability of the intervention beyond 23 months following Project JOINTS. While data from 2014 and beyond could help to look at sustainability, the data presented in figures 1 and 2 do not show a loss of impact over time based on a rising slope, although not tested statistically. The sustainability of this QI work is an important area for future study.
In conclusion, this evaluation of the Project JOINTS campaign demonstrated a significant reduction in SSIs in intervention states following the campaign in comparison with non-participating comparison states. The IHI RSN campaign approach provides a promising platform for large-scale spread and adoption of evidence-based practices, with improved patient outcomes.
References
Footnotes
Funding This study was funded by the Agency for Healthcare Research and Quality (grant number: R18 HS021424), and the Department of Health and Human Services (grant number: R18 AE00005).
Competing interests None declared.
Patient consent Not required.
Ethics approval Harvard Pilgrim Health Care IRB.
Provenance and peer review Not commissioned; externally peer reviewed.