Background The NHS England evidence-based interventions programme (EBI), launched in April 2019, is a novel nationally led initiative to encourage disinvestment in low value care.
Method We sought to evaluate the effectiveness of this policy by using a difference-in-difference approach to compare changes in volume between January 2016 and February 2020 in a treatment group of low value procedures against a control group unaffected by the EBI programme during our period of analysis but subsequently identified as candidates for disinvestment.
Results We found only small differences between the treatment and control group after implementation, with reductions in volumes in the treatment group 0.10% (95% CI 0.09% to 0.11%) smaller than in the control group (equivalent to 16 low value procedures per month). During the month of implementation, reductions in volumes in the treatment group were 0.05% (95% CI 0.03% to 0.06%) smaller than in the control group (equivalent to 7 low value procedures). Using triple difference estimators, we found that reductions in volumes were 0.35% (95% CI 0.26% to 0.44%) larger in NHS hospitals than independent sector providers (equivalent to 47 low value procedures per month). We found no significant differences between clinical commissioning groups that did or did not volunteer to be part of a demonstrator community to trial EBI guidance, but found reductions in volume were 0.06% (95% CI 0.04% to 0.08%) larger in clinical commissioning groups that posted a deficit in the financial year 2018/19 before implementation (equivalent to 4 low value procedures per month).
Conclusions Our analysis shows that the EBI programme did not accelerate disinvestment for procedures under its remit during our period of analysis. However, we find that financial and organisational factors may have had some influence on the degree of responsiveness to the EBI programme.
- quality improvement
- financial incentives
- health policy
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
What is already known?
In April 2019, NHS England launched the evidence-based interventions programme, a novel approach to disinvest in low value care which aims to reduce avoidable harm to patients, maximise value and avoid waste by reducing unnecessary interventions.
What does this study add?
By comparing trends against a control group, we provide early evidence that the evidence-based interventions programme has not accelerated disinvestment for low value procedures in the first 11 months after implementation.
How might this study affect research, practice or policy?
Despite the lack of effectiveness to date, the evidence-based interventions programme has developed a structured and transparent approach to identify candidates for disinvestment.
Moving forward, the evidence-based interventions programme needs to consider how to balance both bottom-up and top-down implementation to achieve sustainable and consistent reductions in low value care.
Low value care can be defined as the ‘use of an intervention where evidence suggests it confers no or very little benefit on patients, or risk of harm exceeds likely benefit, or, more broadly, the added costs of the intervention do not provide proportional added benefits’”1 Minimising low value care, and tackling unwarranted clinical variation is a major issue for all healthcare systems. It has been estimated that 25% of healthcare expenditure in the USA is spent on low value procedures.2 Equivalent figures for the UK do not exist, but disinvestment in low value care has been high on the policy agenda over the last two decades.3–8 Despite several initiatives to disinvest in low value care in the UK and internationally, analyses to date have indicated that many initiatives fail to achieve their objectives.9 This is typically attributed to factors such as poor engagement from the public and clinicians, and lack of data collection and monitoring arrangements.10 11 Over the last few years, NHS England has sought to develop a more coordinated and structured approach to disinvest in low value care through the evidence-based interventions (EBI) programme. The stated aims of the EBI programme is to reduce avoidable harm to patients, maximise value and avoid waste by reducing unnecessary interventions.12 After undertaking a review of evidence and consulting with the public, commissioners and healthcare professionals, the first phase of the programme identified 17 low value procedures within two categories: category 1 interventions which have been shown to be ineffective and should no longer be offered to patients, and category 2 interventions which are only appropriate in certain circumstances.
The EBI programme is distinctly different to previous initiatives to disinvest in low value care in several respects. First, targets have been set for each local commissioning organisation, known as clinical commissioning groups (CCGs), to reduce the number of category 1 interventions to ‘near zero’ and category 2 interventions to 25% of baseline levels nationally. Second, the EBI programme introduced a zero tariff for category 1 interventions, and recommended all CCGs implement a prior approval process for category 2 interventions. Finally, progress in meeting agreed targets is monitored and fed back to hospitals and CCGs using a publicly available dashboard to allow benchmarking.13 Statutory guidance for the first phase of the programme was published in November 2018,12 and the programme was officially launched in April 2019. A second phase of the EBI programme has since been developed involving 31 low value procedures, with statutory guidance published in November 2020.14 However, it has not been feasible to evaluate the effectiveness of the second phase of the EBI programme in this paper due to the significant impact of the COVID-19 pandemic on elective care provision in England.
To our knowledge, this paper is the first quantitative analysis of the EBI programme. The primary aim of this paper is to evaluate the impact of the first phase of the EBI programme in reducing the provision of low value procedures compared with a control group of other low value procedures unaffected by the EBI programme during our period of analysis. The secondary aims of this paper are to establish if the effect of the EBI programme differs according to different organisational or financial factors.
To test our primary aim, we used a difference-in-difference (DiD) approach, which is a quasi-experimental method commonly used to estimate the causal effect of a policy intervention against a comparator group that can adjust for both observable and non-observable confounding factors.15 While we cannot evaluate the effectiveness of the second phase of the EBI programme in this paper (the period of analysis in our paper ends in February 2020 which is several months before the second phase of EBI begins), the second phase does create an opportunity to construct a control group of low value procedures which were unaffected by the EBI programme during our period of analysis. To ascertain if there is evidence of adaptation after implementation of the EBI programme, we adapted a methodology used by Cooper et al to apply spline regression to analyse differences in trends between the treatment and control group.16
The two key assumptions which underlie the use of DiD analysis are that the treatment and control group have parallel trends before implementation of the policy intervention, and that the control group remains unaffected by the treatment after implementation.15 We relied on visual inspection of trends to test the parallel trend assumption. While data-driven approaches to test for parallel trends do exist, there is consensus among the literature that there is no perfect approach to test the parallel trend assumption and pre-trend testing is not a substitute for logical reasoning why parallel trends should hold for treatment and control group.17–19 In our case, we argue that the parallel trend assumption should hold as both the treatment and control groups are considered low value procedures. Moreover, one review of DiD studies found that pre-trend testing is rarely used in studies with large numbers of observations as even small differences in pre-implementation trends are likely to be significant.18 The authors of this review highlight how pre-trend testing is more often used in studies with small numbers of observations when the test could fail to reject parallel trends because it is underpowered rather than due to parallel trends.18 For the second assumption, we constructed a control group of low value procedures that should not have been impacted by disinvestment in the low value procedures targeted by the EBI programme during our period of analysis.
To test our secondary aims, we used a method developed to ascertain the difference between two DiD estimators known as triple difference estimation.20 This was used to expand on our DiD model to ascertain whether the implementation of the EBI programme was associated with a larger decrease in the provision in low value procedures in certain groups including between CCGs with different levels of financial performance, CCGs that were or were not part of a demonstrator community that volunteered to trial EBI guidance before implementation and NHS hospitals compared with independent sector hospitals.
Study cohort and data sources
We analysed pseudoanonymised individual patient level data between 1 January 2016 and 28 February 2020 retrieved from the NHS Digital Secondary Uses Services (SUS) database. We did not analyse data beyond this point as elective care activity was significantly impacted by the COVID-19 pandemic. Data access was provided by the NHS England EBI programme team to researchers from the London School of Economics and Political Science as part of a service development evaluation exercise. The identification of low value procedures was undertaken using combinations of primary and secondary procedure and diagnosis codes for each low value procedure, developed by the EBI programme, based on feedback from stakeholders including CCGs, hospitals and specialty organisations. These codes are publicly available and mentioned in EBI guidance.12 14A summary of these codes is also mentioned in online supplemental tables 1 and 2. The NHS SUS database is classified according to finished consultant episodes, which relates to the clinician responsible for the respective aspect of care, and hospital spells, which encompass all activity from hospital admission to discharge, including ward transfers of patients. To avoid multiple counting of low value procedures, procedures were identified by applying these criteria to each hospital spell rather than finished consultant episode. For each low value procedure, information on individual patient characteristics was extracted including age, gender, deprivation and comorbidities. Comorbidities were classified according to the Charlson Comorbidity Index (CCI), based on code written by Quan et al that uses International Classification of Diseases codes.21 Deprivation was classified according to the English Index of Multiple Deprivation (IMD) 2019.22
The primary treatment group for our analysis was activity data for category 2 procedures under the remit of phase I of the EBI programme (online supplemental table 3), with the exception of removal of benign skin lesions. Removal of benign skin lesions was removed from the primary analysis as it was seen as not comparable to other procedures under the remit of the EBI programme. It is a relatively minor procedure that often takes place in outpatient clinics rather than in surgical theatres, and as a relatively high-volume procedure its inclusion could bias results. The decision was made to not include category 1 procedures in the primary analysis as category 1 procedures are not recommended to be conducted in any circumstances, and therefore are likely to have experienced a significantly larger decline than category 2 procedures before implementation of the EBI programme. To ascertain if these omissions significantly alter our results, we include both removal of benign skin lesions and category 1 procedures in the treatment group in separate robustness analyses.
We also constructed separate treatment groups for procedures grouped according to whether they were classified as high-cost or low-cost procedures. This was determined by whether estimated potential annual savings for each procedure exceed £10 000 000 per annum. High-cost procedures in the primary treatment group included hysterectomy for heavy menstrual bleeding, shoulder decompression, carpal tunnel syndrome release and Dupuytren’s contracture release (online supplemental table 4). Estimated savings were calculated by the EBI team using baseline expenditure for each procedure (total expenditure for each procedure in 2017/18 divided by the number of associated hospital spells), and the assumption that all CCGs meet their targets to reduce provision. These estimated savings are mentioned in online supplemental table 4.
The control group for our analysis consists of four category A procedures: repair of minimally symptomatic inguinal hernia, surgical intervention for chronic rhinosinusitis, kidney stone surgery and surgical intervention for benign prostatic hypertrophy. These procedures were selected from category A low value interventions that are subject to phase II of the EBI programme (online supplemental table 3). Other category A procedures were not included in the control group as they are either diagnostics rather than surgical procedures, or considered to be vulnerable to potential spill-over effect of the first phase of EBI programme. Category B or C interventions were excluded as the EBI programme team have not yet developed reliable definitions to publish activity levels.14
Data on CCG financial performance was retrieved from NHS England.23 CCGs were coded as being in deficit if their expenditure exceed their allocation in the financial year before the implementation of the EBI programme. During our period of analysis, many CCGs underwent mergers and the number of CCGs reduced from 191 in 2018/19 to 135 in 2019/20.24 To overcome this, we consistently coded CCGs according to their CCG status in 2019/20 and the financial performance of CCGs that underwent a merger in 2019/20 was estimated using the total surplus or deficit for merged CCGs. The EBI team provided information on whether a CCG was coded as part of the demonstrator community that volunteered to trial EBI guidance before implementation. Hospital providers were coded as either NHS or independent sector hospitals according to their organisation code classified by the NHS Digital Organisation Data Service.25
Equation 1 shows the ordinary least squares (OLS) regression model for our DiD analysis that tests our primary aim:
is the dependent variable, the log of the number of total low value procedures undertaken at ccg, during month t. is the number of low value procedures in the treatment group; t indicates a running counter of months from January 2016 to February 2020. is the break point in the spline corresponding to the policy start point, which is 1 from April 2019, and 0 before (the month which phase I of the EBI programme was launched). indicates a running counter of months from April 2019 to February 2020. is a vector of CCG controls including aggregate patient characteristics (age, gender, CCI and IMD). The model was run using fixed effects which differenced out all time-invariant CCG characteristics out of the equation. Year and month dummies were also added to account for year and seasonal variation.
Setting gives rises to our preferred standard DiD specification when the coefficient captures the treatment effect of the EBI programme, specifically the difference in the average change in volumes of low value procedures in the treatment group before and after the implementation of the EBI programme minus the difference in the average change in volumes of low value procedures in the control group. We ran this model with all phase I category 2 low value procedures with the exception of removal of benign skin lesions, and then separately according to whether procedures were classified as high or low cost. Setting gives rises to our spline-based DiD regression specification when the coefficient captures the difference in the average monthly rate of change in volumes of low value procedures in the treatment group before and after the implementation of the EBI programme minus the difference in the average monthly rate of change in volumes of low value procedures for the control group. Relaxing these assumptions allows a combination of these estimators. This specification allows a step change in policy, and a change in trends. As these additional specifications did not show significantly different results, we did not repeat these specifications with procedures classified according to whether they are high or low cost.
A number of robustness checks were performed to test the reliability of our results. First, with the treatment group including volumes for both category 1 and 2 procedures. Second, with the treatment group including volumes for removal of benign skin lesions. Third, to account for potential anticipatory behaviour change before implementation of the EBI programme, the implementation period was changed to November 2018 (the month which the statutory guidance for the EBI programme was published).
Equation 2 shows the OLS equation which uses a triple difference estimator to test our secondary aims:
is the dependent variable, the log of the number of total low value procedures undertaken at CCG or hospital, i, during month t. X is a binary variable which reflects different CCG or hospital characteristics. The equation was run three separate times. First, with X being 1 for CCGs which posted a deficit in financial year 2018/19 and 0 for those which did not. Second, with X being 1 for CCGs which were part of the demonstrator community that volunteered to trial EBI guidance before implementation and 0 for those which were not. Third, with X being 1 for NHS hospitals and 0 for independent sector providers. The coefficient captures the difference in the average change in low value procedures in the treatment group between these CCG and hospital characteristics. The other components of equation 2 are the same as equation 1. We also performed the same robustness checks for equation 1 and equation 2. As classifying procedures in the treatment group according to their cost did not produce significantly different results in our standard DiD specification, we did not repeat this with the triple difference estimator.
Patient and public involvement
No patients were involved in the development of the research question, design or implementation of this study. The authors plan to disseminate results to patients and policymakers through outreach activities and platforms provided by NHS England, and the Royal College of Surgeons in England.
When visually inspecting pre-implementation trends between the treatment and control group (figure 1), the assumption of parallel trends appears to hold in the pre-treatment period. Similarly, the pre-implementation trends between treatment and control group do not appear to significantly differ for alternative compositions of the treatment group discussed in the ‘Methods’ section (online supplemental figures 1–4).
When focusing on the 11 months before and after the implementation of the EBI programme (table 1), the proportion of procedures undertaken after implementation was similar in the treatment group (48.4%), and control group (49.2%). There was more variation in the proportion of individual procedures undertaken after implementation in the treatment group (41.3%–53.4%), compared with the control group (48.5%–50.9%).
There were no significant changes in the proportion of patients, which were male or female before and after the implementation of the EBI programme for any of groups of procedures (online supplemental table 5). Most patients in the control group were male, which likely reflects how prostate surgery is performed exclusively for male patients and hernia repair is performed more frequently for male patients. There were significant increases in the average age of all groups of procedures, with largest increases in category 1 procedures at 1.42 years (95% CI 1.07 to 1.77), and the smallest increase in the control group at 0.79 years (95% CI 0.64 to 0.94). The average IMD score increased for all groups of procedures, although this increase was not significant in any groups of procedures. There were significant increases in average CCI for all groups of procedures. However, these increases were very small and the largest increase was only 0.09 points (95% CI 0.07 to 0.10) in category 1 procedures.
While 38 out of 135 CCGs (28.1%) posted a deficit in the financial year 2018/19 before implementation, a higher proportion of low value procedures were undertaken in these CCGs for both the treatment (37.0%) and control groups (39.0%) (online supplemental table 6). Forty-eight out of 135 CCGs (35.6%) were coded as having volunteered to be part of the demonstrator community to trial EBI guidance before implementation, and a similar proportion of low value procedures were undertaken in these CCGs for both the treatment group (36.2%) and control group (35.6%). While 226 out of 388 hospitals (58.2%) were independent sector hospitals, only 18.2% of procedures in the treatment group, and 20.5% of procedures in the control group were conducted in these hospitals. Although it should be noted that independent sector hospitals are typically smaller than NHS hospitals and have much less capacity.
In our primary DiD model, the coefficient that reflects the treatment effect of the EBI programme was 0.10 (95% CI 0.09 to 0.11), and significant at the p<0.001 level (table 2). This indicates that reductions in the provision of low value procedures in the treatment group were 0.10% smaller than reductions in the control group, which is equivalent to 16 low value procedures per month. This coefficient remained positive for the high-cost and low-cost treatment groups, indicating that the effectiveness of the EBI programme was not influenced by estimated potential annual savings for individual procedures. In our spline-based and combination-based DiD, the coefficients that reflect differences in monthly changes in volume were not significant. The co-efficient that reflects differences in the step-change in policy was 0.05 (95% CI 0.03 to 0.06) and significant at the p<0.001 level. This indicates the step-change reduction in the provision of low value procedures in the treatment group was 0.05% smaller than the control group, which is equivalent to 7 low value procedures. These findings are consistent across all robustness analyses outlined in our ‘Methods’ section (online supplemental tables 7–9).
Triple difference estimation
When focusing on the results of the triple difference estimation, we found that the coefficient which reflected differences in reductions in the treatment group after implementation of the EBI programme for CCGs that posted a deficit in the financial year 2018/19, was −0.06 (95% CI −0.08 to –0.04), and significant at the p<0.001 level (table 3). This indicates that reductions in low value procedures after the implementation of the EBI programme were 0.06% larger in CCGs posting a deficit in the baseline year (2018/19) than those which did not, which is equivalent to 4 low value procedures per month. We found no significant differences after implementation of the EBI programme for CCGs that were part of the demonstrator community compared with those that were not. The co-efficient which reflected differences in reductions in the treatment group for NHS hospitals after implementation of the EBI programme was −0.35 (95% CI −0.45 to –0.26), and significant at p<0.001 level. This indicates that reductions in low value procedures after the implementation of the EBI programme were 0.35% larger in NHS hospitals compared with independent sector hospitals, which is equivalent to 47 low value procedures per month. These findings are consistent according to all robustness analyses outlined in our ‘Methods’ section (online supplemental tables 10–12).
Summary of findings
This analysis demonstrates that for the first 11 months after implementation, the EBI programme did not successfully achieve its aim of accelerating disinvestment for the low value procedures under its remit. Conversely, on the understanding that the control group provides a counterfactual scenario whereby the EBI programme did not exist, we found statistically significant evidence that the implementation EBI programme was associated with a small increase in the volumes of low value procedures under its consideration. This finding is consistent irrespective of whether we change the composition of the treatment group according to procedures with estimated potential annual savings of above or below £10 000 000. When analysing organisational and financial factors which may have influenced implementation of the EBI programme, we found that CCGs which posted a deficit in the financial year before implementation had larger reductions in low value procedures than CCGs which did not. This may be because CCGs which posted a deficit in the year before implementation felt the need to more pro-actively engage with the EBI programme as one mechanism to save costs and reduce their deficit in the subsequent year. Despite approximately a third of CCGs volunteering to be part of a demonstrator community which trialled EBI recommendations before implementation, there were no significant differences between changes in volumes of low value procedures between demonstrator and non-demonstrator CCGs. Finally, we found that NHS hospitals had significantly larger reductions in low value procedures than independent sector hospitals. This may be because NHS hospitals have an institutional culture which is more amenable to NHS England-led national quality improvement initiatives, whereas independent sector hospital hospitals may be more motivated by their respective corporate-level objectives and strategies.
Strengths and limitations
There are several strengths to this analysis. First, to our knowledge this is the only quantitative evaluation of the impact of EBI programme. This provides important information on the early impact of the EBI programme so far which can be used to inform its planned expansion. Second, this evaluation uses a DiD analysis which is a robust method for casual inference. This is a valuable addition to the literature on empirical evaluations of national disinvestment initiatives which typically use weaker methods such as interrupted-time series, or before and after analyses.9 Third, we extend our analysis to focus on a range of organisational and financial factors which may have influenced implementation and use a series of robustness tests to assess the reliability of our findings.
Despite these strengths, the robustness of this analysis is heavily dependent on the ability of our selected control group to meet the two key assumptions to undertake a DiD.15 First, that there are parallel trends pre-implementation. Second, that the control group remains unaffected by the treatment after intervention. For the first assumption, the degree to which parallel trends exist is frequently debated in DiD analyses.18 However, we can be reassured that on visual inspection of pre-implementation trends the assumption of parallel trends appears to hold. Moreover, the coefficient which reflects differences in trends between the treatment and control groups is very small (table 3). For the second assumption, we constructed the control group to minimise the potential for any spill-over effect of the EBI programme. However, we cannot completely exclude any possibility of a spill-over effect as the EBI programme may have generally encouraged a culture of disinvestment in hospitals and CCGs.
There are other minor limitations of our analysis to consider when interpreting the findings of this study. First, we used a coding algorithm for the CCI developed by Quan et al.21 There is a modified version developed for use with hospital administrative data collected in England which is arguably more applicable to our dataset but this was not available on STATA.26 Second, we classified spells according to hospital spells which do not take account of transfers between hospitals. Therefore, it is possible we have counted some procedures twice if an interhospital transfer has occurred. Although, this is unlikely to have substantially impacted our results as we expect very few interhospital transfers took place for procedures in our analysis as they are generally of low complexity.
Policy implications and conclusions
The findings of this analysis are consistent with many other empirical analyses of national disinvestment initiatives that often conclude they have not successfully achieved their aims.9 Despite broad consensus from the healthcare community in the UK on the importance of disinvestment in low value care,8 other evaluations on the effectiveness of interventions such as the National Institute for Health and Care Excellence guidance and ‘do not do’ recommendations have also found limited impact.4 27 28 While it is frustrating that early evidence from the EBI programme has indicated limited effectiveness, the EBI programme still represents a major step forward as it has developed a structured and transparent approach to identify candidates for disinvestment that involved broad consultation with specialty organisations, patient groups and CCGs. It is possible that the EBI programme has taken an approach to implementation that is too top-down. The barriers to disinvestment in low value care are complex and involve a range of patient, clinician, organisational and health system factors.29 Moreover, the success of the EBI programme is heavily reliant on successful collaboration between CCGs, hospitals and primary care in developing and effectively implementing prior approval processes. Moving forward, the EBI programme will need to focus on developing strategies to facilitate and monitor these collaborations at the local level to balance both bottom-up and top-down implementation in a manner that could foster more sustainable and consistent reductions in low value care.
Patient consent for publication
We would like to express our gratitude to members of the NHS England evidence-based interventions programme team who provided access to NHS England Secondary Uses Service data for the purposes of this analysis and provided insights regarding their experiences interacting with hospitals and commissioning bodies during implementation of the EBI programme.
Correction notice This article has been corrected since it was first published online. In the abstract, 0.35% (95% CI 0.26% to 0.445%) was amended to 0.35% (95% CI 0.26% to 0.44%) and 0.06% (95% CI 0.045% to 0.08%) was amended to 0.06% (95% CI 0.04% to 0.08%).
Contributors MA undertook the analysis and drafted the manuscript. All other authors were collaboratively involved in the design of the analysis and provided comments and edits to iterative versions of the manuscript. MA is the guarantor for this manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests AM, was until August 2021, Clinical Lead of the EBI Programme. MA was also granted the honorary position of clinical fellow at NHS England between January 2021 and December 2021. The other authors have no relevant conflicts of interest to declare related to this manuscript.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.