Article Text
PDF
Abstract
Postoperative adverse events occur all too commonly and contribute greatly to our large and increasing healthcare costs. Surgeons, as well as hospitals, need to know their own outcomes in order to recognise areas that need improvement before they can work towards reducing complications. In the USA, the American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP) collects clinical data that provide benchmarks for providers and hospitals. This review summarises the history of ACS NSQIP and its components, and describes the evidence that feeding outcomes back to providers, along with real-time comparisons with other hospital rates, leads to quality improvement, better patient outcomes, cost savings and overall improved patient safety. The potential harms and limitations of the program are discussed.
- Health Services Research
- Surgery
- Report Cards
- Quality Improvement
- Quality Measurement
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/
Statistics from Altmetric.com
The problem
Over 30 million operations are performed in the USA each year.1 Postoperative adverse events occur frequently. Even in common, non-complex cases such as colectomy (250 000 cases annually), surgical site infections occur in 10% of patients.2 These adverse events increase hospitalisation length and cost. For example, one study estimates that the cost of a surgical site infection is over US$27 000,3 while a urinary tract infection can cost from US$675 to US$2800.4 A single case of ventilator-associated pneumonia can add US$50 000 to the cost of an admission.5 ,6 Length of stay increases when complications occur, with an extra 3–11 days in hospital required for respiratory events.7 Payments are being reduced for some adverse events, such as central line infections and surgical site infections. The Affordable Care Act of 2012 established the Hospital Readmission Reduction Program and in turn the Centers for Medicare and Medicaid Services began implementing a process to reduce payments for certain 30-day readmissions.
The largest and best known intervention for measuring and reporting surgical outcomes in the USA is the American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP). Risk-adjusted postoperative outcomes reported by NSQIP provide benchmarks intended to spur local quality improvement efforts to produce better patient outcomes. This review assesses the evidence to determine whether ACS NSQIP has improved surgical care and patient safety.
Patient safety strategies
The multi-component ACS NSQIP in part grew out of efforts initiated by Veterans Affairs (VA) health system researchers and clinicians in the late 1980s. In response to concerns about high complication rates in VA hospitals, VA NSQIP was officially launched in 1994 to collect and report clinical variables and outcomes across all VA hospitals.8 ,9 Another factor contributing to ACS NSQIP was the success of programs like the New York State Cardiac Surgery Reporting System (CSRS) in the 1990s.10 Basic concepts of the New York State CSRS have been adopted by other states (California, Pennsylvania) and have spread across the USA through the efforts of the Society of Thoracic Surgery (STS) Registry, which incorporated public reporting. The measurement and public reporting of cardiac surgery outcomes has also spread to England.11
With growing attention being paid to systematic reporting of surgical outcomes, non-VA hospitals became interested in applying the VA experience to their data reporting and quality improvement programs in the late 1990s. A pilot study in three civilian hospitals (University of Michigan, Emory University and University of Kentucky) demonstrated the feasibility of this initiative in the private sector.12 Following this pilot, the ACS took the lead to expand efforts to a broader group of hospitals (14 sites) in 2001, and the formal ACS NSQIP began in 2004.13 An overview of the key aspects of the program are provided in figure 1.
Key aspects of the American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP). *Surgeons selected the content, make decisions about changes in the way data are collected and analysed, and they provide oversight.
The intervention consists of several basic components. First, a surgeon champion (an individual staff member at the participating site) assumes a significant role in establishing and overseeing the participation of each site. Second, a surgical clinical reviewer (SCR) is trained to collect data on preoperative clinical variables and on 30-day outcomes. The third component is the development of risk-adjusted models of expected mortality and morbidity by procedure type. Fourth, data are analysed and presented back to the individual sites alongside masked data for the other sites. Data are provided for each of 21 morbidities (such as unplanned readmissions, urinary tract infections and pulmonary embolism) in addition to mortality. Sites are displayed as being high (worse than expected) or low outliers (better than expected) for each category of morbidity and for mortality. Lastly, the participating sites are encouraged to address and correct problem areas of clinical care.
Originally, ACS NSQIP reported observed-to-expected (O/E) 30-day mortality and morbidity ratios. Over the past 8 years, risk adjustment and modelling has evolved to improve the accuracy of the reported outcomes.14 Recently, the system has been adapted to better account for surgical case complexity by introducing a procedure mix adjustment. Additionally, ACS NSQIP no longer reports O/E ratios but instead provides logistical ORs using hierarchical modelling with shrinkage adjustment (figure 2). This allows for more stable estimates for small sample sizes, such as lower volume procedures or smaller hospitals. Additionally, this allows for more valid and accurate reporting of individual provider rates. Currently, both facility and surgeon data are collected. Individual facility results are provided routinely, but those for individual surgeons are provided upon request.
Example reporting events by odds ratio using hierarchal modelling with shrinkage adjustment. Hospital specific bar plots. Each box shows the distribution of odds ratios for hospitals in the model; the top and bottom of each box give the smallest and largest values, and horizontal lines give the deciles. The point and vertical line within each box give the individual hospital's odds ratio and 99% CI. General and vascular surgery (GV) outcomes models are reported. DVT, deep venous thrombosis; PE, pulmonary embolism; ROR, return to operating room; SSI, surgical site infection; UTI, urinary tract infection. Adapted from Cohen, JACS, 2013.14
From reports generated by ACS NSQIP, institutions can see where they are a high outlier for certain procedures and identify where improvement is needed. An example of a de-identified report is provided in figure 2. Auditing by ACS NSQIP staff occurs randomly and for cause, that is if a site reported results too good to be true, and an audit for cause revealed upcoding of comorbidities and downcoding of complications or dramatic changes in results. The audits are also measures of data collection quality. Program audits reported low disagreement rates (between the auditor and site data collector) relatively early in program development (3.15% in 2005) and this rate has continued to drop (1.56% in 2008).15
While the responsibility for making changes remains with the individual sites, the administrative ACS NSQIP body provides support in terms of case report of successful interventions, provision of best practices, national meetings, and monthly supportive conferences calls with the surgeon champions and clinical reviewers. For example, ACS NSQIP staff will reach out to individual sites to assist them by connecting them with others who have made improvements in the area of concern or interest. Additionally, the annual ACS NSQIP meeting serves as a critical forum for sites to present and discuss a range of quality improvement efforts and successes.
Review processes
A Medline search up to November 2012 carried out using the search terms NSQIP and National Surgical Quality Improvement Project identified 169 studies. The review also included the Cochrane Registry and gray literature available on the ACS NSQIP website (http://www.acsnsqip.org/). Following title and abstract screening, the author selected any article that provided evidence for the benefits or harms of VA NSQIP or ACS NSQIP, or quantitative information describing how ACS NSQIP was implemented. There was no restriction on study design.
The author supplemented this literature review with interviews with the leadership and administrators in ACS NSQIP, as well as four surgeon champions. These surgeons were selected to represent different participants in the program: community, academic, leadership and a state-wide collaborative. The interviews explored weaknesses as well as strengths of the program.
Benefits and harms
Benefits
A VA NSQIP review of over 400 000 cases performed between 1991 and 1997 showed that 30-day mortality and morbidity rates for major surgery fell by 9% and 30%, respectively.16 Reductions in one postoperative complication alone, surgical pneumonia, are estimated to have saved the VA US$9.3 million annually, and the overall reduction in morbidity may have saved billions since the program was started.8 ,17 ,18
Two published longitudinal studies reached divergent conclusions on the effects of reporting in ACS NSQIP. They first looked at changes from 2005 to 2007 in ACS NSQIP-participating sites (N=183) for all outcomes measured and surgical specialties using risk adjustment and accounting for hospital procedure volume.19 For the most recent period of 2006–2007, 118 hospitals participated long enough to produce clinically useful data. The authors found that 82% of hospitals had improved morbidity and 66% had improved mortality. The adjusted absolute difference in O/E ratio was −0.114 for morbidity and −0.174 for mortality (negative numbers indicate less morbidity and mortality). Similar results were seen when the researchers accounted for institutional volume. They also found that the number of high outliers (those with worse outcomes) decreased over time and the number of low outliers (those with better outcomes) increased. Institutions with high outliers were more likely to improve and had larger mean changes in outcomes. It was estimated that an average of 200–500 complications and 12–36 deaths may have been avoided.19
The other study compared ACS NSQIP to a private sector collaborative based at the University of Michigan Medical Center.20 The Michigan Surgical Quality Collaborative (MSQC) includes 34, mainly community (68%) hospitals, unlike the ACS NSQIP-participating hospitals, which are primarily academic/teaching institutions. Sixteen MSQC hospitals were assessed over two time periods and compared to the 126 non-Michigan NSQIP hospitals over the same time periods. MSQC hospitals had a decrease in morbidity from 10.7% to 9.7% (9.0% reduction, p=0.002; OR 0.898) over 3 years, whereas morbidity did not change for the ACS NSQIP hospitals in either time period or between the periods (12.4%; OR 1.0).20
The potential impact of participating in ACS NSQIP on complication rates and mortality has been reported by individual hospitals and collaboratives. Although improvements in morbidities have been large, mortality changes have ranged from none to modest. Mortality for most general and vascular surgical procedures is generally low, leaving little room for improvement. In contrast, improvements in morbidities are commonly reported. One local initiative showed that targeted efforts reduced respiratory complications and that after 7 months the rate of postoperative pneumonia had been reduced to zero from a peak monthly rate of 2%.21 Another state collaborative showed improvements in a range of postoperative events.22 However, most of the reports of improvement in single institutions or later collaboratives have been presented in forums other than peer-review publications.
At the July 2011 ACS NSQIP national meeting, 20 presentations reported reductions in morbidity following an intervention. In all these cases, ACS NSQIP data enabled the hospitals to target an area with worse-than-expected outcomes and to intervene, with resulting improvement (table 1). These presentations were selected as representative of successful quality improvement efforts that developed as direct responses to outcomes shown to be in need of improvement. These collective results demonstrate how varying sites and collaboratives worked to make specific changes to improve outcomes.
Example of interventions and changes in outcomes in ACS NSQIP hospitals/collaboratives
Almost all these studies have a pre–post design and therefore have all the accompanying limitations, including regression to the mean. However, the results are unlikely to be explained solely by regression to the mean, primarily because of the size of the effect and the post-intervention result. Additionally, these changes occur in response to directed efforts for particular areas, such as decreasing deep venous thrombosis (DVT) rates or reintubation rates. Furthermore, Hall et al23 demonstrated that both bad (high outlier) and good (low outlier) hospitals improve. Taking into consideration the large body of pre-post work, which shows repeatedly each year of the program, support that directed efforts were largely responsible for the reported improvements.
The way NSQIP improves care is multifaceted. To improve care and reduce complications, surgeons must know the outcomes of their own procedures. The data must be of high quality and reliable, and risk adjustment must be adequate to allay concerns about comparing ‘apples to oranges’. This comparison allows surgeons and hospitals to see how they compare in terms of outcomes, which promotes accountability and stimulates work to correct the problems. Most sites (59% of those surveyed) were unaware of their hospital's adverse event rates, let alone how they compared to other hospitals, until after they enrolled in ACS NSQIP.3
A particular feature of the program is its use of detailed clinical data collected from the medical records. A study comparing administrative and claims data collected by the University Health System Consortium (UHC) program showed that ACS NSQIP identified a greater number of complications (61%) than UHC, including 97% more surgical site infections and 100% more urinary tract infections.24
Furthermore, ACS NSQIP identifies adverse events following discharge. Studies using ACS NSQIP show that more than 50% of complications occur after discharge. For colectomies, 45% of deep surgical site infections, 39% of organ space infections and 28% of DVT occur after patients have left hospital.25 Identifying complications that occur outside the hospital is the prerequisite first step to developing changes in care to help prevent them.19
Harms
Few published studies have assessed the potential and actual harms of this program, and most are speculative. A primary concern has been that surgeons will avoid high-risk cases for fear of adversely affecting their O/E outcomes assessments. This issue was raised early in the process of implementing report cards when anecdotal evidence appeared to suggest that as the result of implementing the New York CSRS, high-risk coronary artery bypass graft (CABG) patients were being diverted instead to the Cleveland Clinic.26 However, subsequent and more comprehensive analyses could not document any systematic exclusion of high-risk patients from CABG operations, and showed that, on the contrary, the severity of illness and comorbidities of operated patients has increased over the years.27 ,28 The longitudinal ACS NSQIP study also supported this finding, showing that the risk profile and illness severity for surgical patients has increased over time.19 Another concern is that the outcomes for outpatient cases or for a hospital or surgeon who performs a small volume of procedures might need longer follow-up, possibly for more than a year, to accurately assess quality.29 The question has been raised that surgeons could alter treatment plans for patients based on individual operative risk rather than giving the patient the option of a procedure with a potentially better long-term functional outcome. A theoretical example would be in vascular surgery, where a high-risk patient eligible for a distal bypass would be recommended an amputation instead.
Implementation considerations and costs
Implementation context
The program requirements include site administrative support, a surgeon champion, and participation in a series of conference calls and the national ACS NSQIP meeting. Data reporting is mandated to follow particular rules, such as accrual of particular data and 30-day follow-up information. ACS NSQIP personnel perform audits to help maintain data quality. For small hospitals, the effort and cost may be less than for larger facilities, depending on the volume of cases.
ACS NSQIP has been implemented in a variety of settings including large academic hospitals, smaller community hospitals and large and small state-wide consortia. It soon became apparent that a variety of program models were needed to accommodate differing clinical volumes. Program options vary in terms of number of variables collected, surgical specialty, if procedures are specifically targeted, and case sampling required. Currently, more than 500 sites are enrolled in ACS NSQIP, which represents roughly 10% of the almost 4500 hospitals in the USA. The distribution of the more than 525 sites that reported clinically useful data is shown in the map in figure 3.
Geographical distribution of American College of Surgeons National Surgical Quality Improvement Project (ACS NSQIP) participating sites. Number of sites by state, region and country included in the January 2013 NSQIP semi-annual report. Reprinted by permission of ACS NSQIP.
Overall, 49% of participating sites are teaching or academic centres. The majority of these hospitals are high volume, as only 3% perform less than 100 procedures per year, 7% perform 100–299 procedures per year, 43% perform 300–499 procedures per year, and 47% perform more than 500 procedures per year. This skewed distribution of hospital size means that the 10% of hospitals participating in ACS NSQIP represent 32% of the procedures performed.30 Certain complex procedures are captured at an even higher rate, for example, 57% of oesophagectomies and 53% of pancreatectomies billed to Medicare are performed at ACS NSQIP sites (table 2).
Percentage of Medicare surgical cases covered by ACS NSQIP
Collaboratives are a main feature of ACS NSQIP and have taken many different forms (some are a handful of geographically close hospitals while others are all hospitals in a state) that work as a team to implement the program and quality improvement. They also can represent a disease or patient population and so members of a collaborative need not be geographically close together. In addition to providing intellectual and practical support to each other for quality improvement initiatives, they provide a collective voice for bargaining with potential sources of funding. One reported approach is for the main insurer for the hospitals in the collaborative to pay 50% of the cost of the program over a set number of years. Sometimes an option to renew the financial support is given if certain milestones are met. Some payors have judged there is a business case for helping support ACS NSQIP participation because of perceived cost savings. Table 3 shows the current list of active collaboratives.
List of ACS NSQIP collaboratives including type, number of sites and payor
A pilot paediatric ACS NSQIP collaborative collects data for patients under 18 years of age.31 ,32 Variables have been modified for paediatric surgery practices and needs.
There are many examples of different collaboratives, along with unique challenges facing different hospital types. The hospitals proposing to develop a collaborative negotiate as a group for financial support from a variety of sources, shape the program for their own needs, and work together to make quality improvement changes. Two examples, one of a community of hospitals starting a small state-wide collaborative (Tennessee Surgical Quality Collaborative, TSQC) and another where state-wide participation in a collaborative was developed (Florida Surgical Care Initiative, FSCI), are described in detail below.
In 2004, after being introduced to the recently started ACS NSQIP at the national ACS meeting, a member of a community hospital in Tennessee initiated a process that resulted in a collaborative that included hospitals, payors and the Tennessee Hospital Association. The collaborative would be controlled by a leadership committee that comprised four surgeons appointed by the local society chapter, along with hospital CEOs and a member of the Tennessee Hospital Association. The proposal included funding for the participation of eight hospitals, estimated to be US$2 550 000 for 3 years. The money covered half the expense of the SCR, salary support for the surgeon champions, and administrative costs. This example highlights many of the key components for building a successful program—surgical leaders taking a role, supportive administration, and collaboration with other hospitals.
A strikingly different collaborative was set up in Florida. The Florida Hospital Association (FHA) was aware of the high surgical mortality demonstrated by the Dartmouth Atlas project (which identified disparities in access to and utilisation of health care) in their state.33 The FHA, along with the payor, BlueCross BlueShiled, collaborated to generate a financial incentive for hospital participation. A new version of the program collecting only four outcomes was developed, thus lowering costs. Currently, 64 hospitals are participating in the FSCI and the participation of 39 more is pending. This example demonstrates additional features that help encourage participation: individuals at the state level and hospital administration taking a lead, a flexible program designed to fit the needs of the collaborative, and the role of the local payor in incentivising hospital participation.
Costs
The costs of participation vary depending on what type of program the hospital joins. The annual administrative fee varies by hospital size and level of participation, salary of the SCR, and optional bonus payments to support the surgical champion or quality improvement team. This fee ranges from US$10 000 (rural and hospitals that deal with <2000 cases/year) to US$25 000 (>2000 cases). Hospitals can lower their costs by participating in a collaborative.
The salary for the SCR comprises the bulk of the expense of participation. Previously, the clinical reviewer had to be a registered nurse (RN), but because of issues such as nursing shortages, individuals such as licensed vocational nurses, with medical training but without advanced nursing degrees, have been successfully employed as SCRs. All reviewers must pass a credentialing examination annually. Their expenses will vary based on experience, level of training and region, and range from around US$40 000 per year to US$100 000 for an experienced RN. Many hospitals suggest that paying for a surgeon champion (an amount such as US$5000) is helpful for increasing their involvement, although a recent survey of surgical champions (109 respondents) found that 72.5% did not receive salary support compensation.34
The highest total cost of participation is estimated to be US$135 000 annually; however, this estimate is for a large hospital that hires an RN as the reviewer.35 ,36 Many participating hospitals may pay less as they have lower volumes of patients and therefore decide to participate in a smaller program.
Since the overarching goal of ACS NSQIP is to reduce complications, which are costly, the business case for participating is that the expense of the program translates into savings for the hospital. Examples of such savings reported by NSQIP sites are shown in table 4. Pre–post data without control groups are shown and so inference of a causal relationship is limited by the study design.
Published examples of changes in complications and costs following participation in ACS NSQIP
One cost-effectiveness study of ACS NSQIP has been published. Costs and outcomes for 2229 general and vascular surgery cases at one large academic study were assessed. The study compared two time intervals - one 6 months and the other 1 year from inception of the program. Postoperative events declined over time, from 17 to 13 percent. The incremental costs were $832 and $266 per patient for the two time periods, meaning the cost per patient of the program declined after the first 6 months of participation. The incremental cost-effectiveness ratio to avoid 1 postoperative event was $25,471 in the first 6 months, declining to $7,319 in the second time period, meaning that the longer the institution participated in the program, the more cost-effective the program became.37
Discussion
Although no randomised trials have assessed the use of outcomes measurements and reporting in surgery, the strength of the evidence that doing so improves operative mortality and morbidity is moderate or even high, given the strong theoretical rationale why it should work, the evidence that outcome reporting has likely improved surgical outcomes in other settings (eg, the New York state CSRS), the numerous reports from ACS NSQIP sites describing the implementation of quality and safety initiatives following identification of high outlier status, and the ensuing, sometimes dramatic, improvements in those outcomes. There is a great deal of experience on how to implement ACS NSQIP as it has been rolled out in more than 400 hospitals. Some of the key components of ACS NSQIP (collecting complications data, sharing models of O/E results, multi-site data collection systems across institutions that provide results back to the sites for benchmarking, contexts for learning and sharing tools that appear to be effective across sites) are similar to those of other successful patient safety practices such as the Michigan Keystone ICU Project to reduce catheter-related bloodstream infections.38 Despite ACS NSQIP and the Keystone ICU Project having started with different original ‘interventions’ (the feedback of procedure-specific surgical outcome data to surgeons and a checklist of processes to reduce infections), the fact that current versions of the interventions include many similar components suggests the implementation of certain types of practices across hospitals is generalisable.
ACS NSQIP provides hospitals and providers with usable clinical data that are otherwise not available to them. Currently, all hospitals use administrative data to some degree to assess quality through the CMS Hospital Compare program or the Surgical Care Improvement Project (SCIP). These data lack clinical information and are limited by the variables reported for claims. More importantly, the correlation between administrative data and actual complications or diagnoses is inadequate. ACS NSQIP uses detailed clinical data to highlight areas where improvements are needed.
The greatest benefit has been seen in the larger hospitals in the procedures with higher complication rates. Whether the above improvements will transfer to low-risk but common procedures, such as outpatient procedures, is unclear. Most of the early adopters have been large academically affiliated hospitals. How successfully and widely the program can be implemented at smaller hospitals remains to be seen. ACS NSQIP has been flexible in terms of changing to fit the needs of those participating. The program specifics have been adapted for variable hospital sizes and large versus small collaboratives, and the program has provided vastly different modules for specific procedures, and even a new program for a specific surgical population—ACS NSQIP Pediatric for paediatric surgery reporting.
NSQIP was developed specifically for surgery. Like other surgical outcomes reporting systems, the conceptual model behind ACS NSQIP works best in situations where outcomes are measureable within a short time frame after the relevant care has been delivered, and there are reasonable means to adjust for case-mix differences. This model may not generalise to all types of health care, such as primary care, chronic care etc, but may translate well to those sharing similar clinical properties, like treatment provided in intensive care units.
A limitation of ACS NSQIP, or any outcomes-based quality improvement program, is that knowing outcomes does not necessarily provide the answer to producing better outcomes, rather, it requires the unit to know itself and identify the defect. ACS NSQIP does provide educational tools, such as guidelines and best practices, but it still requires leadership at the local level to lead the charge. Lastly, as process improvement is needed to sustain better outcomes, it is also important to measure and benchmark process compliance, which ACS NSQIP does not currently do.
Acknowledgments
The author thanks West Los Angeles Veterans Affairs, UCLA Medical Center and Olive View-UCLA Medical Center. Paul Shekelle provided suggestions on the format and presentation of the data and advice on the manuscript.
References
Footnotes
-
Funding This study was funded by the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services (contract no. HHSA-290-2007-100621). The Agency for Healthcare Research and Quality reviewed contract deliverables to ensure adherence to contract requirements and quality, and a copyright release was obtained from the Agency for Healthcare Research and Quality before submission of this manuscript. The author of this study is responsible for its content. Statements in the study should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.
-
Competing interests None.
-
Provenance and peer review Not commissioned; externally peer reviewed.