Background Community hospitals, which care for most hospitalised children in the USA, may be vulnerable to declines in paediatric care quality when quality improvement (QI) initiatives end. We aimed to evaluate changes in care quality in community hospitals after the end of the Pathways for Improving Paediatric Asthma Care (PIPA) national QI collaborative.
Methods We conducted a longitudinal cohort study during and after PIPA. PIPA included 45 community hospitals, of which 34 completed the 12-month collaborative and were invited for extended sustainability monitoring (total of 21–24 months from collaborative start). PIPA provided paediatric asthma pathways, educational materials/seminars, QI mentorship, monthly data reports, a mobile application and peer-to-peer learning opportunities. Access to pathways, educational materials and the mobile application remained during sustainability monitoring. Charts were reviewed for children aged 2–17 years old hospitalised with a primary diagnosis of asthma (maximum 20 monthly per hospital). Outcomes included measures of guideline adherence (early bronchodilator administration via metered-dose inhaler (MDI), secondhand smoke screening and referral to smoking cessation resources) and length of stay (LOS). We evaluated outcomes using multilevel regression models adjusted for patient mix, using an interrupted time-series approach.
Results We analysed 2159 hospitalisations from 23 hospitals (68% of eligible). Participating hospitals were structurally similar to those that dropped out but had more improvement in guideline adherence during the collaborative (29% vs 15%, p=0.02). The end of the collaborative was associated with a significant initial decrease in early MDI administration (81%–68%) (adjusted OR (aOR) 0.26 (95% CI 0.15 to 0.42)) and decreased rate of referral to smoking cessation resources (2.2% per month increase to 0.3% per month decrease) (aOR 0.86 (95% CI 0.75 to 0.98)) but no significant changes in LOS or secondhand smoke screening.
Conclusions The end of a paediatric asthma QI collaborative was associated with concerning declines in guideline adherence in community hospitals.
- implementation science
- quality improvement
Data availability statement
Data are available on reasonable request. Deidentified participant data are available from the corresponding author (ORCID 0000-0003-2243-8268) on reasonable request. Reuse is not permitted.
Statistics from Altmetric.com
Asthma is a common cause of childhood hospitalisation in the USA, leading to over 100 000 hospitalisations and $1.6 billion in direct costs annually.1 2 Although well-established national guidelines for paediatric inpatient asthma care exist,3 4 there is variable clinician guideline adherence.5 6 Poor guideline adherence by clinicians contributes to worse health outcomes for children hospitalised with asthma, including longer recovery time and inpatient length of stay (LOS), higher rates of transfer to intensive care units and increased risk of hospital readmission.2 6–8
Prior research has established effective strategies for promoting initial adoption of guidelines and evidence-based practices,9 10 but to date, little research has focused on how to effectively sustain guideline adherence.11 A systematic review of sustainability of clinicians’ guideline adherence reported that the majority of studies show declines in adherence over time, likely due to competing priorities and decreases in resources.12 However, the studies included in this systematic review were primarily focused on adult populations.12 Most studies of paediatric asthma quality improvement (QI) initiatives are limited to ≤1 year and do not continue to collect data after the QI initiative ends.13 14 Thus, there is currently a limited understanding of whether guideline adherence is sustained after paediatric asthma QI initiatives and how to best promote sustained high-quality care.
Community hospitals, which care for over 70% of hospitalised children in the USA,2 may be especially vulnerable to declines in paediatric guideline adherence and care quality when QI initiatives end. Dedicated children’s hospitals have resources explicitly devoted to measuring, improving and maintaining high-quality care for children (eg, QI staff and data collection systems).15 Community hospitals primarily provide care to adults, and as a consequence, paediatric QI efforts may have limited access to such resources.16 17
The objective of our study was to evaluate guideline adherence and care quality in community hospitals for 2 years after the initiation of a national QI collaborative, specifically examining the effects of the collaborative ending. We evaluated sustainability after the Pathways for Improving Paediatric Asthma Care (PIPA) national QI collaborative. PIPA included a diverse, national sample of hospitals and demonstrated short-term improvements in multiple quality measures.18 19
Study design, setting and population
This was a longitudinal cohort analysis of the PIPA national QI collaborative.18 19 PIPA was led by the Value in Inpatient Paediatrics Network, the inpatient QI network at the American Academy of Paediatrics. PIPA included 85 hospitals, of which 45 were community hospitals. Community hospitals were defined based on self-reported data by site leader and verified using data from the American Hospital Association Annual Survey Database.20 21 Participating hospitals started the QI intervention in two groups, with half of sites starting in January 2018 and half starting in April 2018. The QI collaborative lasted 12 months for both groups. Thirty-four community hospitals (out of 45) completed the QI collaborative. Community hospitals that completed the PIPA collaborative were approached to participate in extended sustainability monitoring in fall 2018, prior to the end of the collaborative. Extended sustainability monitoring lasted through the end of 2019 (total of 21–24 months from collaborative start date, depending on group). Participating sites varied in terms of size, geographic region, type (eg, non-profit and government) and location (eg, urban). Study outcomes were determined via chart review of children aged 2–17 years hospitalised with a primary diagnosis of asthma, identified using International Classification of Diseases, tenth revision (ICD-10) codes.22 Exclusion criteria included chronic medical conditions that precluded pathway use (eg, cystic fibrosis, restrictive lung disease, bronchopulmonary dysplasia, congenital or acquired heart disease, airway issues, immune disorders, sickle cell anaemia or neuromuscular disorders).
QI collaborative phase
The QI collaborative took place from January 2018 to March 2019. Details on the supports provided by the collaborative and fidelity to the intervention are detailed in other manuscripts.18 20 Briefly, participating sites were provided with paediatric asthma pathways, educational materials and seminars, QI mentorship, monthly data reports, a free mobile application and peer-to-peer learning opportunities. Key driver diagrams were developed that identified five core intervention components: (1) guidance on dosing of bronchodilators via metered-dose inhaler (MDI); (2) pathway and/or instructions for standardised bronchodilator titration by nurses or respiratory therapists; (3) hospital discharge criteria; (4) reminders to screen for secondhand tobacco exposure; and (5) reminders to refer caretakers who screen positive for tobacco use to smoking cessation resources. In fall 2018 (near the end of the collaborative), local site leaders from all PIPA sites participated in an educational seminar focused on sustainability, as well as meetings with external QI mentors to facilitate sustainability planning.
During the sustainability phase, hospitals continued to have access to previously provided pathways, educational materials and the mobile app, but no longer had access to the remaining QI supports. Because our study aimed to understand performance changes after the end of QI collaboratives, we tried to simulate real-world situations as closely as possible. Therefore, we discontinued data collection on intervention fidelity, meetings with QI mentors, peer-to-peer learning sessions, educational seminars and distribution of monthly performance reports (activities that normally are not supported externally after the end of QI collaboratives).
Data were collected via chart review, which was performed locally at each site. Reviewers were trained based on the Medical Record Abstraction Quality Assurance and Control Framework.23 Reviewers were provided chart review manuals, trained via videoconference by the central research team and encouraged to ask questions during chart review. Charts from January to December 2017 were reviewed retrospectively to establish baseline performance. Charts from January 2018 to December 2019 were reviewed prospectively to assess performance during and after the QI collaborative. Charts were selected in chronological order each month, up to a maximum of 20 per month, per hospital. Data quality was audited monthly by the central research team and by quality audit functions within REDCap 8.5 (Nashville, Tennessee). Hospitals were blinded to their performance during sustainability monitoring (after the QI collaborative ended). Hospitals continued to perform chart review during the sustainability period but were no longer provided summary statistics from the collaborative for audit and feedback.
Study outcomes (table 1) included measures of guideline adherence3 4 and healthcare utilisation (ie, LOS) that were tracked during the PIPA QI collaborative.18 These outcomes were selected prior to the QI collaborative through a consensus process by a national expert panel on the basis of: (1) recommendations of evidence-based guidelines, (2) potential to improve health outcomes, (3) variability and/or room for improvements in performance and (4) feasibility of measurement. Guideline adherence was measured in relation to specific recommendations within national guidelines for the inpatient management of asthma, including early administration of bronchodilator via MDI (recommended by guidelines because MDIs are more cost-effective and have fewer side effects than nebulisers3 24 25), and screening for secondhand tobacco exposure and caregiver referral to smoking cessation resources (recommended by guidelines because tobacco exposure worsens asthma outcomes in children4). Guideline adherence was coded dichotomously (yes/no) for each admission. LOS was selected as a patient-centred outcome, given that asthma is a leading cause of illness-related missed school days and missed work for parents/caregivers.26 Prior studies have found that pathway implementation and standardised discharge criteria are associated with shorter LOS.27–31
Hospital and patient characteristics were summarised using descriptive statistics. Hospitals that participated versus dropped out of sustainability monitoring were compared using Fisher’s exact tests for structural characteristics and electronic health record (EHR) integration, Mann-Whitney U test for core intervention components implemented (measure of implementation fidelity) and Student’s t-test for per cent improvement in guideline adherence during the initial QI collaborative. Patient characteristics for admissions during versus after the QI collaborative were compared using multivariable regression models. Study size was predetermined by sites’ participation in the PIPA QI collaborative and willingness to participate in this follow-up study of sustainability.
For our primary analysis, we used multilevel regression models with an interrupted time series (ITS) approach. ITS accounts for pre-existing trends in each outcome and evaluates: (1) changes in an outcome at the time of an event and (2) changes in the rate of change in an outcome after versus before an event.32 Our primary event of focus was prespecified as the end of the QI collaborative, which was chosen because this timepoint was when sites no longer had access to several QI supports that were being provided externally. Thus, this event represented a decline in QI resources that commonly occurs at the end of such quality collaboratives33 34 but also may occur in single-centre QI initiatives as resources and priorities shift focus.12 Such declines in resources have been associated with declines in guideline adherence in prior literature.12
We conducted a secondary analysis examining changes in outcomes associated with initiation of the QI collaborative in order to compare care quality after versus during the collaborative. Our event of focus for this analysis was prespecified as the beginning of the QI collaborative. We used multilevel regression models with an ITS approach.
The unit of analysis was a single hospital admission. All regression models included patient characteristics including age at admission, sex, insurance type (proxy for socioeconomic status) and prior prescription of inhaled corticosteroids (proxy for chronic asthma severity) as fixed effects. This adjustment was intended to account for potential confounding bias due to case-mix differences over time. Time was modelled by adding a variable for month of participation (relative to QI collaborative start and end for that site); a first-order autoregressive covariance component was included to account for serial correlation. Hospital was included as a random effect to take into account clustering of admissions within hospitals and thereby potential similarities in practice by hospital site, as well as differences in duration of follow-up by hospital site. LOS was modelled using gamma regression, and a quadratic term was included to correct for seasonal trends. All other outcomes were modelled using logistic regression. In accord with a framework by Ramsay et al 35 that outlines proper use and reporting of ITS designs we specify that: (1) the interventions occurred independently of other changes over time, (2) the sources and methods of data collection were the same across all time periods examined, (3) the outcomes were measured objectively using prespecified criteria, (4) the dataset included data from ≥80% of eligible admissions during all time periods examined, (5) the events examined were prespecified and (6) the number of monthly time points examined was determined by the timeline over which we expected to see anticipated effects. All analyses were performed with SAS V.9.4. P values less than 0.05 were considered statistically significant.
Hospitals and study population
Twenty-three (68%) of the 34 eligible community hospitals participated in this study. Community hospitals that participated in extended sustainability monitoring were similar to those that dropped out in terms of teaching status, hospital size, geographic region, urban–rural location and ownership model (table 2). Community hospitals that participated in sustainability monitoring achieved high-fidelity implementation during the QI collaborative, implementing a mean of 4.6 of the five core intervention components. The majority of these hospitals also reported EHR integration of pathway recommendations (57%). During the initial QI collaborative, implementation was associated with higher magnitude improvements in guideline adherence for hospitals that participated in sustainability monitoring than for those that dropped out (29% vs 15%, p=0.02).
This study included a total of 2159 admissions (n=1256 during PIPA, n=903 after PIPA, 42–142 admissions per month). Patients were demographically and clinically similar during the QI collaborative and after the collaborative ended (table 3), and all patient characteristics were adjusted for in primary and secondary analyses.
Primary analysis: aggregate changes in outcomes associated with the end of the QI collaborative
Aggregate changes in outcomes associated with the end of the QI collaborative are illustrated in figure 1. The end of the QI collaborative was associated with a significant initial decrease in early MDI administration (81%–68%) (adjusted OR (aOR) 0.26 (95% CI 0.15 to 0.42)), which rebounded to 81% by the end of sustainability monitoring. The end of the collaborative was also associated with a change in the trend of caregiver referral to smoking cessation resources (2.2% per month increase to 0.3% per month decrease) (aOR 0.86 (95% CI 0.75 to 0.98)), which led to a decrease from 75% adherence at the end of the collaborative to 56% at the end of sustainability monitoring. The end of the collaborative was not associated with significant changes in LOS or screening for secondhand tobacco smoke exposure.
Secondary analysis: aggregate changes in outcomes associated with initiation of the QI collaborative
Initiation of the QI collaborative was associated with significant improvements in guideline adherence for this subset of community hospitals, in line with prior studies.18 19 Initiation of the QI collaborative was associated with significant increases in early MDI administration (16%–38%) (aOR 3.78 (95% CI 2.2 to 6.47)), screening for secondhand tobacco smoke exposure (78%–90%) (aOR 2.81 (95% CI 1.44 to 5.51)) and referral of caretakers to smoking cessation resources (23%–50%) (aOR 2.92 (95% CI 1.22 to 7)). There was, however, an increase in LOS (34–37 hours over collaborative) (adjusted rate ratio 1.01 (95% CI 1.01 to 1.03)).
In this multisite, national study, we found significant declines in clinicians’ guideline adherence at the end of a paediatric asthma QI collaborative. Although initiation of the collaborative was associated with improvements in guideline adherence, community hospitals experienced declines in performance after the QI resources provided by the collaborative were withdrawn. Similar reductions in QI resources are likely common in multisite quality collaboratives and in single-centre QI initiatives.12 These findings reinforce a pressing need to better understand how to promote sustained guideline adherence and care quality after QI initiatives.11
Our findings align with prior adult studies that demonstrate long-term declines in guideline adherence after initial QI implementation. A systematic review by Ament et al 12 reported that approximately half of studies show decreases in guideline adherence over time. Other studies have confirmed that, although QI initiatives are widespread, there is limited evidence that improvements in care quality are sustained long term.36 These declines have significant consequences, including lower quality care and decreased return on investments in research and QI.37
Although most studies of paediatric asthma QI initiatives have ≤1 year of follow-up,13 14 a study by Nkoy et al 27 examined guideline adherence for 5 years in multiple hospitals within a single integrated network (Intermountain Health). Nkoy et al 27 described approaches to sustaining guideline adherence that included many resource-intensive strategies (eg, leadership buy-in and support, external QI mentorship and facilitation, local champions, electronic order sets and technical assistance, audit and feedback). Our findings may have differed because such resource-intensive approaches may not have been not feasible in the community hospitals participating in our study.38
We found significant declines in two of four quality measures at the end of the national paediatric asthma QI collaborative. These declines may have been driven by decreases in QI resources, shifts in QI priorities and/or decreases in local QI activities (eg, performance monitoring, audit and feedback) after the end of the collaborative.38 39 A recent systematic review by Cowie et al 39 identified factors that may promote sustainability, including delegating roles and responsibilities, having local champions and securing organisational support. A qualitative study by Ament et al 40 reinforced the findings that delegating responsibilities and having local champions might promote sustained guideline adherence. Ament et al 40 additionally suggested other strategies to promote sustainability, including conducting audit and feedback with clinicians, providing educational booster meetings, reminding clinicians of evidence-based practices and changing physical structures and care processes. As part of an effort to promote sustainability, the PIPA QI collaborative promoted many of these potential strategies. Local site leaders participated in an educational seminar focused on sustainability that highlighted many of these strategies, including delegating responsibility, securing organisational support, reminding clinicians of evidence-based practices and changing physical structures and care processes. This seminar was followed by meetings with external QI mentors to facilitate sustainability planning. However, despite these efforts, we found declines in guideline adherence after the QI collaborative ended.
Paediatric QI efforts in community hospitals may be especially susceptible to declines in care quality over time.16 17 Our group’s prior research suggests that many strategies to promote sustainability may not be feasible in the community setting. In our prior study of 104 hospitals that participated in a national quality collaborative to improve care of infants with fever, we found that most hospitals did not sustain audit and feedback, educational meetings or organisational support.38 However, the majority of hospitals sustained EHR changes, local champions and reminders of evidence-based practices.38 Given that these strategies may be more feasible to sustain, it is important that future research determines the effectiveness of such strategies in promoting sustained guideline adherence for paediatric care in community hospitals.
We found different patterns of decline by outcome, which suggests that different evidence-based practices may be differentially susceptible to declines in performance over time. A qualitative study by Brewster et al 41 found that critical factors to sustainability varied by the type of practice: shifts in attitude and norms (for rewarding practices), revised performance standards (for complex practices) and automation (for simple practices). We found that early administration of MDIs had an initial decline at the end of the QI collaborative but then slowly rebounded over the sustainability monitoring period (eventually returning to the levels achieved at the end of the QI collaborative), while caregiver referral to smoking cessation resources steadily declined to near-baseline levels. Initial declines in MDI administration may have been related to the complexity of this practice, as our prior studies have found significant barriers to developing consensus for this practice (eg, concerns of increased time burden or need to wake up children overnight).17 42 Such barriers may require revised performance standards as Brewster et al suggest,41 or ongoing consensus building by local leaders that was initially less feasible or consistent in the sustainability phase. The rebound in MDI administration that we found may suggest that these barriers were eventually overcome during the sustainability period. Referral of caregivers to smoking cessation resources may have been more dependent on automation to consistently remind clinicians of this simpler practice.41 Identifying such practice-specific sustainability strategies may be critical to ensuring long-term, high-quality care for children with asthma in community hospitals. To identify these strategies, our team is conducting follow-up work to determine specific strategies that were enacted in hospitals that sustained gains in care.
This study has several limitations. First, we restricted our study to community hospitals, so our findings may not be generalisable to other settings such as free-standing children’s hospitals. However, our findings are clinically meaningful because community hospitals provide care for the majority of hospitalised children in the USA.2 Second, our findings may be subject to selection bias, as not all eligible community hospitals participated in extended sustainability monitoring. Hospitals that participated in sustainability monitoring had higher magnitude improvements during the PIPA collaborative than sites that did not participate, so it is possible that sites with leaders willing to continue collecting monitoring data were more likely to sustain guideline adherence. It is therefore particularly concerning that these sites had declines in performance after the end of the collaborative. Third, because only 68% of eligible sites participated in extended sustainability monitoring, our results may not be generalisable to other sites. Hospitals who participated versus dropped out of sustainability monitoring were structurally similar (table 2), but we may have been underpowered to detect structural differences between hospitals. Fourth, only two small community hospitals (<100 beds) participated in sustainability monitoring. As a result, our findings may not apply to small community hospitals. Fifth, we were unable to control for race/ethnicity or other clinical confounders due to dataset limitations. The PIPA collaborative (like many large-scale QI efforts) did not track race/ethnicity due to difficulties collecting accurate data across various sites. However, due to the known disparities in asthma outcomes,43 this will be a critical factor to measure in future studies. Finally, hospitals continued to perform chart review during extended sustainability monitoring (although no longer had access to summary statistics from the collaborative). This may have led to a Hawthorne effect where hospitals performed better than they might have otherwise.44 As the Hawthorne effect would likely cause hospitals to outperform what would have occurred without continued chart review, the declines in quality that we found are particularly concerning.
In conclusion, we found significant declines in guideline adherence in community hospitals at the end of a paediatric asthma QI collaborative. Given that QI interventions require substantial resources, it is critical to measure long-term performance to determine whether these QI resources are providing sustained returns on investment. In addition, it is critical that future research identifies feasible implementation strategies to promote sustained guideline adherence and high-quality care for children with asthma in community hospitals. This may require identifying both setting-specific and practice-specific strategies.
Data availability statement
Data are available on reasonable request. Deidentified participant data are available from the corresponding author (ORCID 0000-0003-2243-8268) on reasonable request. Reuse is not permitted.
Patient consent for publication
This study was approved by the Institutional Review Board at the University of California, San Francisco.
Value in Inpatient Pediatrics Network (American Academy of Pediatrics), Community Hospital Site Leaders.
Contributors Conception and design: SVK, BJ and MC. Data acquisition: community hospital site leaders, Value in Inpatient Pediatrics, SVK. Analysis: JR. Interpretation: all authors. Writing: SS and SK. Revision: all authors.
Funding This study was funded by the Agency for Healthcare Research and Quality (R03 HS027041).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.