Objectives To evaluate whether the Acute Frailty Network (AFN) was more effective than usual practice in supporting older people living with frailty to return home from hospital sooner and healthier.
Design Staggered difference-in-difference panel event study allowing for differential effects across intervention cohorts.
Setting All English National Health Service (NHS) acute hospital sites.
Participants All 1 410 427 NHS patients aged 75+ with high frailty risk who had an emergency hospital admission to acute, general or geriatric medicine departments between 1 January 2012 and 31 March 2019.
Intervention Membership of the AFN, a quality improvement collaborative designed to support acute hospitals in England deliver evidence-based care for older people with frailty. 66 hospital sites joined the AFN in six sequential cohorts, the first starting in January 2015, the sixth in May 2018. Usual care was delivered in the remaining 248 control sites.
Main outcome measures Length of hospital stay, in-hospital mortality, institutionalisation, hospital readmission.
Results No significant effects of AFN membership were found for any of the four outcomes nor were there significant effects for any individual cohort.
Conclusions To realise its aims, the AFN might need to develop better resourced intervention and implementation strategies.
- healthcare quality improvement
- evaluation methodology
- health policy
Data availability statement
Data may be obtained from a third party and are not publicly available. This paper was produced using Hospital Episode Statistics provided by NHS Digital under Data Sharing Agreement NIC-354497-V2J9P. This paper has been screened to ensure no confidential information is revealed.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
WHAT IS ALREADY KNOWN ON THIS TOPIC
Comprehensive geriatric assessment (CGA) saves lives and reduces institutionalisation for older people admitted to acute hospitals but is inconsistently and inadequately offered.
There have been few large-scale efforts to optimise the delivery of CGA in acute hospitals using quality improvement collaboratives (QICs).
Quasiexperimental evaluative designs of QICs are common but subject to methodological flaws in establishing baselines, identifying controls and defining objective outcome measures.
WHAT THIS STUDY ADDS
This study assesses the effectiveness of the Acute Frailty Network (AFN) by comparing outcomes for patients cared for in AFN sites with patients cared for in sites that were not part of the AFN.
Membership of the AFN and adoption of the best practice principles was not found to have a significant effect on length of stay, in-hospital mortality, institutionalisation or hospital readmission.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The AFN has attempted to improve processes across the National Health Service in order to improve outcomes; the result of this study suggests a review of structures (eg, staffing, age-attuned environments) and of implementation might be worthy of future enquiry.
Future research should evaluate the fidelity of CGA interventions and focus on optimal implementation strategies.
The global increase in the ageing population is a testament to advances in public health.1 2 However, long-standing efforts to compress morbidity have not yet been shown to be successful.3 4 As a consequence, many older people develop increasing degrees of frailty in the final decade of their lives.5 6 Solutions to minimise crises in older people living with frailty, and the often-associated hospital admission, remain elusive.7 8 Consequently, many older people living with frailty are admitted to hospitals across the world with crises such as falls, fractures or delirium.9–11 Patient experience12 13 and outcomes following crisis admission to hospital are poor,10 13 with data from many countries showing high rates of inpatient harm, readmission (often relating to inadequate assessment and care during the index admission14–17) and institutionalisation.18–20
Holistic care, namely comprehensive geriatric assessment (CGA), is one of the few evidence-based approaches shown to attenuate some of these poor outcomes and improve patient experience.21 22 CGA is most commonly defined as a multidimensional, multidisciplinary process which identifies each patient’s medical, social and functional needs, and involves development of an integrated/coordinated care plan to meet those needs.23 Typically, CGA involves a team undertaking a multidimensional assessment which should address: diagnoses, as there will usually be multiple interacting comorbidities with associated polypharmacy; physical function and activities of daily living; psychological function, especially confusion and mood; environment in which the individual functions; and social support networks, either present or required to maintain ongoing function.
Systematic reviews have consistently shown that CGA delivered to older people in acute care reduces admission to long-term care and saves lives.21 24–26 Yet, despite being supported by international expert reports,27–31 CGA is not routinely available for older people admitted to hospital in many countries.32 In England, there have been considerable efforts to support and promote CGA in the acute care context using a combination of policy levers33–35 and, in 2015, by establishing the Acute Frailty Network (AFN). To our knowledge, the AFN is the largest international quality improvement collaborative (QIC) focused on acute care for older people. The AFN has done some work in Australia, Holland, Canada and Ireland but these countries are at a much earlier stage in their journey compared with the UK. The AFN meets the five essential features of a QIC model36 as it: (1) focuses on an area with large variation between current and best practice; (2) includes experts providing best practice and improvement ideas; (3) involves multidisciplinary teams of health professionals from different places working together; (4) uses an agreed improvement model with targets, measurement and learning by doing; and (5) promotes collaborative activities and knowledge exchange among members.
The AFN focused on identifying and responding to frailty in the first 72 hours of an acute hospital attendance by using the 10 best practice principles of CGA (see box 1), with an emphasis on early discharge.37 The intention was that membership of the AFN and adoption of the best practice principles would: reduce length of stay (LoS) in hospital; reduce in-hospital mortality; help people return to their own homes rather than a care home or hospice; and reduce the risk of hospital readmission. The objective of this paper was to determine the causal effects of AFN membership on these four outcomes. To this end, we analysed patient-level administrative data and adopted a quasiexperimental event study design, comparing outcomes for patients cared for in AFN sites with patients cared for in sites that were not part of the AFN or before sites became members.
Acute Frailty Network best practice principles
Early identification of patients with frailty.
Multidisciplinary team response with comprehensive geriatric assessment initiated within an hour.
Rapid response system for frail older people in urgent care settings.
Adopt clinical professional standards to reduce unnecessary variation.
Developing a measurement mindset.
Strengthen links with services both inside and outside of hospital.
Put in place appropriate education and training for staff.
Identify clinical change champions.
Patient and public involvement.
Identify an executive sponsor and underpin with robust project management.
Similar to many other countries, the UK is slowly learning how to adapt its healthcare systems to the needs of a growing and increasingly frail older population.3 In the acute care context, a number of policy initiatives have called for a greater focus on the delivery of evidence-based care for older people with urgent care needs, notably CGA.27–29 31 35 Despite these policy initiatives, the delivery of CGA for older people in acute care remains highly variable and suboptimal,38 and service and patient outcomes remain persistently poor.10 12 13 18 39 In response, the AFN was initiated to apply quality improvement at scale to improve outcomes for older people with acute care needs. Described in detail elsewhere,37 the AFN intervention had two overarching aims: (1) to deliver CGA for older people with acute care needs and (2) to support sites with improvement and implementation methods.
The AFN best practice principles outline the key elements (as judged by expert consensus and informed by the evidence base) necessary to embed frailty-attuned pathways in the acute setting. These include defining the population (frailty identification), configuring teams to deliver CGA as soon as possible on the patient’s arrival, minimising unwarranted clinical variation and ensuring a person-centred approach. The AFN principles reflect both the delivery of evidence-based care (frailty identification, CGA and person-centred care) and implementation strategies (quality improvement, stakeholder engagement, leadership).
The AFN used a specific quality improvement approach, primarily the model for improvement,40 focusing on Plan-Do-Study-Act cycles to create change. Four national events were held annually, attended by all participating hospital teams, as well as a set of masterclasses and webinars to support teams and enable sharing of experience. Each hospital team had an allocated ‘coach’ and access to measurement expertise to help plan, deliver and measure change locally, as well as support from national clinical experts to guide the redesign of services. The AFN gained increasing prominence over time, becoming well known across the National Health Service (NHS). The factors that precipitated a site joining were multifactorial, including perceived benefit, local pride and ownership of services and the ability to persuade the site leaders to pay the membership fee (£20 000 per annum).
Regarding the selected outcomes, the link between CGA and reduction in mortality and in institutionalisation is drawn from the systematic reviews underpinning the AFN’s approach.21 41 The links to LoS and to readmission are more related to the policy context of minimising unnecessary time spent in hospital for older people in order to prevent iatrogenic harms such as deconditioning. The hope of the AFN was that by delivering CGA to older people as early as possible in their acute care episode, these joint outcomes might be achieved.
Hospital sites joined the AFN in six sequential cohorts, the first starting in January 2015, the sixth in May 2018 (table 1). Two other cohorts joined later, after the period covered by our data, so were not included in the analysis. We accounted for this differential phasing of entry into the AFN by employing a staggered difference-in-difference (DiD) panel event study approach. Traditional DiD estimates the effects of an intervention by comparing an exposed (treated) group to unexposed (untreated) group. The staggered DiD design generalises this traditional approach to multiple exposed groups that differ according to the timing of their exposure to the event,42 marked here by when sites became members of the AFN collaborative. In our application, we also adopted a recent methodological innovation by Callaway and Sant’Anna that recognises that effects may have varied across cohorts, according to when sites became members.43 Full technical details are provided in the online supplemental appendix but summarised here.
All patients seen at sites after they joined the AFN were exposed to the intervention and, therefore, are considered cases. Control patients were defined as patients from sites that never became AFN members (termed ‘never-treated’ sites) and patients cared for in sites before they became AFN members (termed ‘not-yet-treated’ sites). The analytical objective was to calculate the ‘average treatment effect on the treated’ (ATT) of the intervention by comparing outcomes for cases and controls.
The standard event study methodology assumes that the impact of the intervention is common across all cohorts. But this assumption may not hold, particularly for interventions that evolve over time. This might be the case for the AFN membership model, for two reasons. First, there may have been selection effects, such as organisational characteristics influencing when sites decided to join the network. For instance, those that joined early might have had pre-existing features that were more closely aligned with those of the AFN and may have been more enthusiastic about the network’s aims than those that joined later. Second, there may have been evolutionary effects, such as changes over time in the way that the AFN operated and worked with its members. In recognition of these possibilities we apply the Callaway and Sant’Anna estimator which relaxes the assumption of common cohort effects.43 This means that the ATT estimate of the intervention is calculated both for the AFN overall and for each cohort. The estimates are reported as average marginal effects (AMEs). For LoS, the AMEs can be interpreted as the difference in the number of days in hospital due to being treated in an AFN rather than non-AFN site. For the other three outcomes, the AMEs indicate the difference in the probability of dying, being institutionalised or being readmitted for those treated in AFN sites compared with those in control sites.
In our main analysis, we made the following decisions. First, we applied the same preintervention period of 36 months to all AFN cohorts. Second, we applied the same postintervention period of 11 months to all cohorts. This ensured that the postintervention period was the same across cohorts, the last month of available data being 11 months from enrolment for those sites in cohort 6. Third, patients in the control group were drawn from both ‘never-treated’ sites that never joined the AFN and from ‘not-yet-treated’ sites that subsequently enrolled.
We conducted three types of robustness check to assess the sensitivity of our results to our analytical choices. The first concerned the length of intervention period, set for all cohorts as 11 months in the main analysis. Instead, we allowed the intervention period to vary by cohort, thereby analysing (1) long-term effects, by running analyses to the last month for which data were available (March 2019), and (2) effects up to 12 months, though data for cohort 6 were censored at 11 months.
Our second robustness check restricted controls to ‘never-treated’ patients, those cared for in hospital sites that never joined the AFN. In this robustness check, patients in the ‘not-yet-treated’ sites were dropped from the analysis.
Our third robustness check applied a tighter definition of the AFN intervention, focusing on the most engaged sites, defined as those with above average adoption and implementation of the AFN best practice principles listed in box 1. On average, joining the AFN was associated with the adoption of an additional four of the principles. Using this information we ran two checks. We first restricted the intervention group to those AFN sites that adopted four or more of the principles after joining the collaborative network. Second, we ran a set of 10 analyses that considered adoption of each best practice principle in turn, with only those sites that adopted the specific principle being defined as intervention sites. This meant dropping from these analyses those AFN sites that had already adopted the principle prior to joining the AFN and those sites that did not adopt the principle upon joining.
We analysed anonymised patient-level data from the Hospital Episode Statistics (HES) which contains details of all admissions to NHS hospitals.44 The AFN focused on the acute care needs of older people living with frailty so we identified all patients in HES aged 75+ with a high Hospital Frailty Risk Score (HFRS), this being the age group for which the HFRS was originally developed,45 46 who had an emergency hospital admission to acute, general or geriatric medicine departments between 1 January 2012 and 31 March 2019. We constructed four outcome variables for each patient, namely: their LoS; whether or not they died in hospital; whether or not they were admitted from their own home but discharged to a care home or a hospice; and whether or not they were readmitted as an emergency to hospital within 30 days of being discharged.
The analyses controlled for patient age, gender and, in preliminary analyses, the relative socioeconomic conditions of the neighbourhood in which they lived.47 While all patients were categorised as having high frailty risk, we accounted for their actual HFRS score.45 46 Clinical complexity was captured using the Charlson Comorbidity Index,48–50 and counts of the number of diagnosis codes and of the number of emergency admissions in the previous year. Details about the construction of the outcome and control variables are provided in the online supplemental appendix.
As table 1 and online supplemental figure A1 show, the total data set consisted of 1 410 427 patient observations, though there were some missing values, particularly for 30-day readmission because data were censored on 31 March 2019. A total of 968 900 (68.7%) patients were cared for at hospital sites that never became AFN members, hence being ‘never-treated’ controls. A total of 336 236 (23.8%) patients were treated at sites before they became AFN members, so were considered ‘not-yet-treated’ controls. A total of 105 292 (7.5%) patients were treated at sites after they became AFN members and, hence, were subject to the intervention. The lower panel of table 1 reports the number by cohort.
Descriptive statistics for patients from the intervention and (‘never-treated’ plus ‘not-yet-treated’) control sites are provided in table 2. T-tests show that patients cared for at sites that were AFN members had significantly shorter LoS, and lower proportions died in hospital or were institutionalised. There are also significant differences in patient characteristics, notably in that the intervention group comprised patients with higher frailty risk scores, more Charlson comorbidities and more diagnoses. This necessitates balancing these characteristics between intervention and control groups via propensity score weighting when applying the Callaway and Sant’Anna estimator.
Main regression analysis
The results of applying the event study model to the four outcomes (LoS, mortality, institutionalisation and readmission) are presented in figure 1 and table 3 for the patient characteristics. Column 1 shows that older people and those with more Charlson comorbidities had shorter LoS, perhaps because these patients might disproportionately represent care home residents, for whom early supported discharge might be easier, as they came from and could return to a place of safety. LoS was longer for those with a higher HFRS and more diagnoses.
As shown in column 2, older people, men, those with a lower HFRS and those with a higher Charlson comorbidity burden were more likely to die in hospital. Column 3 reports results for the analysis of those that changed their institutional status, having been admitted from their homes but discharged to care homes or hospices. This change was more likely for older patients, women and those with a higher HFRS, but less likely for those with more Charlson comorbidities and diagnoses. Finally, column 4 shows that older people, women and those with a higher HFRS, more Charlson comorbidities, fewer previous admissions and more diagnoses were less likely to be readmitted within 30 days.
Figure 2 shows the difference in outcomes for intervention and control sites in the preintervention and postintervention periods. For all four graphs, preintervention trends in outcomes were the same in the intervention and control sites as confirmed by all 95% CIs overlapping ATT=0 in this period. All CIs also overlap ATT=0 in the postintervention period, implying that AFN membership had no significant impact on any outcome. This is confirmed in the upper panel of table 4 where the ATTs are not significant for any of the four outcomes. The middle panel of table 4 reports the ATTs for each cohort, none of which is significant, suggesting that there were no effects of the AFN intervention on the four outcomes for any of the six cohorts.
We subjected the overall effect of AFN membership on the four outcomes to a series of robustness checks, producing 56 test statistics reported in the upper panels of online supplemental tables A3, A6, A9 and A13 and in online supplemental table A14. Only one of these indicated a significant effect. This was from the subgroup analysis of the 27 most engaged AFN sites that met the threshold of having adopted four or more of the AFN principles. In this analysis (reported in the ATT row of online supplemental table A13), a higher percentage of AFN patients were discharged to a care home or hospice (0.004, p<0.05).
The main regression analysis revealed no significant cohort effects for the four outcomes. Our robustness checks for cohort effects yielded 96 test statistics reported in the middle panels of online supplemental tables A3, A6, A9 and A13, of which eight were significant. In the analysis of long-term effects (reported in online supplemental table A3), patients in cohort 4 had a longer LoS (1.201 days more, p<0.05), and a lower percentage of those in cohort 5 died in hospital (−0.008, p<0.05). In the analysis of 12-month effects (see online supplemental table A6), a lower percentage of patients in cohort 2 were readmitted within 30 days (−0.013, p<0.05).
In the three most engaged sites from cohort 2 (online supplemental table A13), a significantly lower percentage of patients were readmitted within 30 days (−0.024, p<0.001). Patients in the four most engaged sites in cohort 3 had a significantly longer LoS (1.75 days, p<0.05), and a higher percentage were discharged to a care home or hospice (0.011, p<0.05). Patients in the three engaged sites from cohort 6 had a significantly shorter LoS (6.579 days, p<0.001), and a higher percentage were discharged to a care home or hospice (0.014, p<0.001).
Online supplemental table A14 reports the results of analysing each best practice principle in turn, which found no statistically significant effects on the four outcomes associated with adoption of any particular one of these 10 principles.
This paper reports the analysis of the effect of AFN membership on LoS, in-hospital mortality, institutionalisation and hospital readmission for older patients living with frailty. The policy intention was that AFN membership would lead to reductions in each of these outcomes, as a consequence of members receiving support from national clinical and quality improvement experts and of their adoption of 10 principles of best practice. We applied staggered DiD panel event study methodology to determine the overall causal effect of AFN membership on these four outcomes. Baseline pairwise comparisons suggested that patients at AFN sites had significantly shorter LoS and were less likely to die or be institutionalised than those cared for in non-AFN sites. However, after controlling for patient and site characteristics and temporal factors, no significant effects of AFN membership were found for any of the four outcomes and nor were there significant effects for any individual cohort. The lack of an effect was generally robust to a series of robustness checks, with only 9 of the 156 (6%) tests proving significant at p<0.05.
The study has notable strengths compared with other QIC evaluations. First, the study is large and comprehensive, as it exploits data from the English HES. From this we were able to identify every person aged 75+ with high frailty risk admitted to every NHS hospital site in England between 1 January 2012 and 31 March 2019. This yielded a data set of 1 410 427 patients, of which 105 292 were subject to the intervention in 66 hospital sites and 1 305 135 were cared for in 248 control sites.
Second, we constructed four measures of outcome from the data. This overcomes a criticism of QIC studies that they might be subject to reporting bias, as many rely on self-reported clinical data rather than objective measures.51
The third strength was our research study approach. As AFN membership was not randomly determined, we adopted a quasiexperimental approach for the evaluation, applying an event study methodology designed to identify causal effects of interventions that are subject to phased implementation in non-randomised settings. Further, we implemented the Callaway and Sant’Anna estimator in order to identify differential effects across AFN cohorts. This improves upon QIC evaluations that suffer methodological flaws, with differences in baseline measurement, limited data about site characteristics, possible contaminating spillover effects to control sites52 and insufficient detail of randomisation and concealment procedures for randomised controlled trials.51
There are limitations with our study. First, we used a retrospective observational study design but the gold standard method would have been a stepped wedge cluster randomised trial.53 In the absence of randomisation the Callaway and Sant’Anna estimator attempts to approach a stepped wedged design by correcting for the confounding inherent in observational data. Our application of the estimator makes these corrections by controlling for patient characteristics via propensity score matching to ensure balance between intervention and control groups across cohorts, for site characteristics using fixed effects and for temporal effects by month of admission.
A second limitation is that sites joined the AFN voluntarily, raising the prospect of selection bias. However, intervention and control sites were subject to parallel trends during the preintervention period, implying selection was not a concern. Third, the hope was that the positive effects of joining the AFN would be revealed within 12 months of membership. However, only 11 months of data were available for cohort 6 so, for consistency, in the main analysis we adopted an 11-month postintervention period across all cohorts. Nevertheless, results were generally robust to applying longer follow-up periods. Fourth, two further cohorts of hospital sites subsequently enrolled in the AFN, but after the period covered by our data, so could not be included in the analysis. Fifth, the patient-level outcomes were limited to routinely available service metrics. These may not capture all potential benefits of the AFN, particularly those reported by patients themselves.
Finally, and perhaps most critically, we could not establish definitively whether the lack of significant effects was due to the intervention itself or a failure of implementation exactly as intended. We attempted to address this by examining the relationship between outcomes and adoption of the AFN’s best practice principles. But we found no evidence of a relationship, whether we focused on the most engaged sites (with above average adoption) or adoption of each principle individually. This, however, was an imperfect way to assess implementation of the best practices because the data were self-reported rather than objectively observed or measured.
Implications for practice and policy
There have been several systematic reviews of studies evaluating the effectiveness of QICs,36 51 52 54–56 which suggest that their effectiveness is ‘positive but limited’.36 But authors of these reviews note the possibility of publication bias, studies with non-significant findings (such as this one) being less likely to be published.55 56 There are two possible reasons for limited effectiveness. First, QIC processes alone might be insufficient to bring about improvements. Nadeem et al 54 identified 14 elements used in QICs, including learning sessions, data reporting, leadership, training in quality improvement methods and multidisciplinary quality improvement teams, all of which were included in the AFN. But even if QICs that adopt these elements improve care processes, this may not impact outcomes if the necessary structures are lacking. Adequate staffing, age-attuned environments and access to early supported discharge services are all necessary to improve outcomes for older people with frailty who have crisis admissions to hospital.57 Yet, these are not consistently available, even in relatively well-resourced healthcare systems such as the NHS.38 Expecting acute geriatric services to improve outcomes for older people without these structural elements is akin to expecting a surgeon to improve hip fracture outcomes without an operating theatre or associated team.
Second, even though something might appear effective, it may not always work in practice if the implementation strategy is inadequate.58 Although membership of the AFN did not lead to improved outcomes for older people in acute hospital settings, robust systematic review evidence demonstrates that the underlying intervention, namely CGA, is effective.21 This suggests that future research should consider how best to support those involved in implementation of CGA, perhaps by applying planning and evaluation frameworks designed by implementation scientists to help translate scientific advances into practice.59
In summary, this study found no clear evidence that the AFN succeeded in meeting its aims to support older people living with frailty in the UK to return home from hospital sooner and healthier, over and above the experience of patients cared for in hospital sites that were not part of the network. This implies a more concerted effort is required to configure acute care services to meet the needs of this vulnerable population. To realise its aims, the AFN might need to develop more tailored and better resourced implementation strategies. Given the importance of the issue of acute care for older people, further methodologically robust studies of mechanisms to implement evidence-based models of care to improve outcomes for older people living with frailty are warranted.
Data availability statement
Data may be obtained from a third party and are not publicly available. This paper was produced using Hospital Episode Statistics provided by NHS Digital under Data Sharing Agreement NIC-354497-V2J9P. This paper has been screened to ensure no confidential information is revealed.
Patient consent for publication
We thank Finbarr Martin, Neil Procter, and Emma Backhouse and Matt Tite for information about the AFN sites.
Contributors AS, LM and SC conceived and conceptualised the study. AS and LM developed the methodological approach and accessed and verified the data. LM performed the analyses and AS wrote the first draft of the manuscript. All authors contributed to interpretation of data and to critical revision of the manuscript and approved the final version. AS is the guarantor of the study.
Funding Funding was provided by NHS Elect.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.