Background Periodic external accreditation visits aiming to determine whether trauma centres are fulfilling the criteria for optimal care are part of most trauma systems. However, despite the growing trend towards accreditation of trauma centres, its impact on patient outcomes remains unclear. In addition, a recent systematic review found inconsistent results on the association between accreditation and patient outcomes, mostly due to the lack of robust controls. We aim to address these gaps by assessing the impact of trauma centre accreditation on patient outcomes, specifically in-hospital mortality and complications, using an interrupted time series (ITS) design.
Methods We included all major trauma admissions to five level I and four level II trauma centres in Quebec, Canada between 2008 and 2017. In order to perform ITS, we first obtained monthly and quarterly estimates of the proportions of in-hospital mortality and complications, respectively, for level I and level II centres. Prognostic scores were used to standardise these proportions to account for changes in patient case mix and segmented regressions with autocorrelated errors were used to estimate changes in levels and trends in both outcomes following accreditation.
Results There were 51 035 admissions, including 20 165 for major trauma during the study period. After accounting for changes in patient case mix and secular trend in studied outcomes, we globally did not observe an association between accreditation and patient outcomes. However, associations were heterogeneous across centres. For example, in a level II centre with worsening preaccreditation outcomes, accreditation led to −9.08 (95% CI −13.29 to −4.87) and −9.60 (95% CI −15.77 to −3.43) percentage point reductions in mortality and complications, respectively.
Conclusion Accreditation seemed to be beneficial for centres that were experiencing a decrease in performance preceding accreditation.
- audit and feedback
- health services research
- quality improvement
Data availability statement
Data may be obtained from a third party and are not publicly available. Data may be obtained through a request to the Quebec’s health insurance board (Régie de l’assurance maladie du Québec, RAMQ).
Statistics from Altmetric.com
Trauma systems, which are an organised and multidisciplinary response to injury along the continuum from prehospital care to rehabilitation, have led to important reductions in injury mortality in many high-income countries.1 2 Essential to trauma systems are trauma centres, which are designated by states or provinces according to levels of care (levels I–V for adults and I or II for paediatric centres).3 Trauma centres are acute care hospitals where resources are prioritised to ensure that injured patients receive appropriate and timely care.4 5 Injury organisations, including the American College of Surgeons (ACS) and the Trauma Association of Canada, have established trauma facility standards.3 These criteria have been used to develop a trauma centre accreditation process, which aims to determine whether trauma centres are fulfilling the criteria for optimal care.
Accreditation covers broad aspects of trauma care including a centre’s organisation chart, transfer agreements, emergency and operating room protocols, intensive care unit and medical imaging.6 It generally requires a centre to submit a prereview questionnaire and to complete an on-site visit by an experienced peer review team.3 Advocates of accreditation believe that it allows for standardisation of personnel, equipment as well as stronger hospital commitment to trauma care.7 Opponents, however, highlight the required mobilisation of human and financial resources and the possibility that improvements in care are transitory.8–10 A recent systematic review synthesising the evidence of trauma centres accreditation found that it was imprecisely associated with reductions in mortality and the occurrence of complications.11 That review, however, highlighted methodological concerns that might bias observed associations, including the inappropriate selection of control centres in cross-sectional studies, inadequate control for underlying trends in outcomes in pre-post studies and the lack of adjustment for centre and patient-level potential confounders. In addition, the external validity was limited, with all published studies from the USA, where accreditation is mostly voluntary.
The accreditation process has become common practice in North America, under the rationale that it verifies trauma centres’ abilities to deliver appropriate levels of care.5 12 13 Accordingly, we hypothesised that accreditation should lead to an improvement in patient outcomes. This study aims to fill important gaps in the current literature by using an interrupted time series (ITS) approach, which can account for underlying trends in outcomes, to assess the impact of trauma centre accreditation on in-hospital mortality and major complications.
The Quebec Trauma system consists of 3 adult level I, 2 paediatric level I, 5 adult level II and 49 lower level centres.14 Accreditation is mandatory in Quebec, unlike other Canadian provinces and the USA where it is voluntary. In Quebec, accreditation is performed by the l’Institut National d’Excellence en Sante et Service Sociaux.15
Since the establishment of a trauma system in 1991, four cycles of accreditation have been conducted. During the accreditation visit, a committee of external experts verify adherence to criteria based on recommendations from the ACS Committee on Trauma.3 16 Each centre is evaluated according to its level of designation. Following the accreditation visit, a centre can have one of the following results: unconditional accreditation, in which case the certificate lasts approximately 5 years; provisional accreditation, in which case a new site visit would be performed within 18 months; and accreditation postponed. The latter can result in a downgrading of trauma centre designation status. Due to data availability, we assessed the third cycle of accreditation, which was conducted between January 2012 and March 2015. Therefore, while accreditation is used throughout the text, we are assessing the impact of reaccreditation. In addition to the accreditation, some centres also experienced cointerventions 1 year after the visit. Specifically, they were visited to maintain their status as specialised centres for spinal cord injury or neurotrauma.
Our population consisted of major trauma patients, defined as those with an Injury Severity Score (ISS) ≥12 admitted between April 2008 and March 2017 in all three level I, four level II adult and two paediatric trauma centres of the province of Quebec, Canada.17 One level II centre was excluded from the analysis because we lacked information on the chronology of different cointerventions. The focus on major trauma patients is because they are the primary target of trauma systems, and if accreditation leads to better patient outcomes, that effect will be stronger for these patients. All included centres obtained unconditional accreditation following the visit.
We used data from the Quebec trauma registry, which is subject to validation procedures and contains information on all patients admitted to trauma centres.18 Linkage with hospital discharge data has shown that the trauma registry captures more than 85% of severe trauma.19 Patients dead on arrival, and patients aged 65 years or more with isolated orthopaedic fractures due to a fall were excluded from the study population.20 These patients were excluded because their injury is oftentimes the result of chronic conditions (eg, osteoporosis) that do not require treatment in designated trauma centres.
Our outcomes of interest were in-hospital mortality, defined as any death occurring between arrival in the emergency department and discharge, and major complications, defined as the occurrence of any of the following during the hospitalisation: acquired respiratory distress syndrome, cardiac arrest, myocardial infarction, pneumonia, pulmonary embolism, renal failure, respiratory failure, sepsis, stroke and death.21–23 Death is generally considered as a complication in the trauma literature, and it is a competing risk for non-fatal complications.
Design and statistical methods
ITS is a quasiexperimental design, which accounts for unobservable or unmeasured variables that are fixed over time, for underlying trends in outcomes and regression to the mean.24–27 When properly conducted, it can provide results concordant with those of a cluster-randomised trial.28 However, it requires the presence of regular repeated measurements of an outcome of interest, which represents the time series. We therefore aggregated patient data monthly for level I and quarterly for level II centres. This choice was guided by the compromise between having more time units to properly account for the underlying trends in the data and having enough patients per time unit.
To obtain valid estimates of the impact of trauma centre accreditation, we needed to account for possible changes in the composition of patients admitted to centres (case mix) throughout the study period. In fact, due to the ageing population, the mechanism of injury and number of comorbidities could change with time. In addition, previous studies in a voluntary context have observed significant changes in the ISS and proportion of transfer-in patients following accreditation.29–31 We used prognostic scores for standardisation. A prognostic score describes a subject’s risk of an outcome given its covariate pattern.32 33 Using a sample of ‘control patients’, that is, patients treated during the preaccreditation periods, a pooled logistic regression model was performed: , with representing the outcome of interest for a patient i in centre c , and a vector of patient prognostic factors including age, sex, number of comorbidities, systolic blood pressure, Glasgow Coma Scale and pulse measured on arrival at the emergency department, the body region of the most severe injury, mechanism of injury, ISS and transfer-in from another acute care hospital. The coefficients were then used to estimate the probability of the respective outcome or the prognostic score, for all patients during the entire study period based on their observed covariates . This score was then used to obtain standardised aggregated proportions of each outcome. These proportions represent for each centre, the probabilities of in-hospital mortality and major complications if the same group of patients were treated at each time point.
To obtain estimates of the impact of accreditation on level and trend changes, linear regressions with autocorrelated errors to account for the serial correlation were used34: with . is the monthly or quarterly proportion of the outcome at the time unit u, is the baseline level of the outcome, is coded 1–108 for level I or 1–36 for level II centres, and its coefficient is the baseline trend. The dummy variable indicates whether each time point occurred before or after the accreditation (0 for all time prior and 1 for all time after). The coefficient is the change in the level of associated with the preparation for accreditation. represents the number of time units since accreditation (0 for all time until the intervention; 1,2,3…for subsequent time points), and its coefficient the change in the trend of the studied outcome. is the autoregressive component that comprised , which is the autoregressive parameter for lag p, and , the random error.
Seasonality and non-linearity in trends were investigated and modelled by incorporating autocorrelated error terms at a given seasonal lag and splines or quadratic terms.34 Our focus was on the accreditation process per se, rather than just having the certificate. Therefore, data from the 3 months preceding the accreditation visit were excluded from our analyses, in order to capture the preparation effect. Analyses were performed by centre because of the presence of specific cointerventions in some centres and the complexity of accounting for centre-specific non-linear trends, that is, specifying the underlying trend to be non-linear for centre 1 while being linear for centre 2, for example. This is critical because it is the extrapolation of the preperiod trend that serves as the counterfactual in ITS designs. More details on the model specifications of our statistical approach are presented in online supplemental appendix 1.
Nevertheless, to provide an estimate of the impact of accreditation across all centres stratified by level, we used a generalised estimating equation (GEE) model with robust SEs to account for the clustering of time units (ie, months or quarters) within centres and fixed effects for each centre to account for unmeasured time-fixed differences between centres.35
It is not unusual for a centre to be asked to improve its recording of data on comorbidities or complications following an accreditation visit. In sensitivity analyses, we therefore investigated potential biases introduced, for example, by an increase in reporting of complications (which could lead to differential measurement error in the outcome)36 and comorbidities (residual confounding)37 due to changes that may have occurred in coding practices following accreditation. The formulas used for these analyses are described in online supplemental appendix 2.
Segmented regressions are unable to model changes in variation and/or correlation of outcomes following an intervention. In addition, the time at which accreditation initially affects the outcomes may occur earlier than 3 months before the visit (due to the preparation for the accreditation visit).38–40 We applied a robust interrupted time series (robust-ITS) that overcomes these limitations,41 in order to investigate the possibility that accreditation may have led to change in variability and dependency of studied outcomes. However, only centre 2 was used, because the software currently available for robust-ITS only allows for monthly time series and cannot handle cointerventions.42
Multiple imputation with chained equations was used to impute missing data.43 Covariates with missing data included the Glasgow Coma Scale score (11.6%), number of comorbidities (2.2%), systolic blood pressure (2.2%), pulse (2.2%) and age (0.6%). Rubin’s rules were used to combine estimates across imputed data sets and to obtain 95% CIs.44 Analyses were conducted using SAS software V.9.4 and an RShiny toolbox.42
There were 51 035 admissions, including 20 165 for major trauma during the study period. The number of major trauma admissions remained stable throughout the study period, with an annual average of 539, 40 and 137 admissions, respectively, for adult level I, paediatric level I and adult level II centres. The proportion of patients transferred from another hospital was stable. However, there was an increase in the mean age of admitted patients, comorbidities, the proportion of falls and thoracic-abdominal injuries in all centres (online supplemental appendix 3-eTable 1). Among the major trauma patients, 12.90%, 5.73% and 12.77% experienced in-hospital death and 28.46%, 11.19% and 21.33% experienced major complications, respectively, for adult level I, paediatric level I and adult level II centres.
Figure 1 displays the crude and standardised monthly probabilities of mortality for the three adult level I centres. In each centre, we observed a decrease in the mean level and monthly trends of mortality following accreditation. However, the 95% CIs were wide, and we lacked the precision to conclude that accreditation had a consistent beneficial impact on patient outcomes. Combining the three centres led to similar results (table 1).
In adult level II centres, data were aggregated by quarter due to the smaller sample sizes. Across centres, we observed substantial variation in performance (figure 2). For centre 5, which exhibited a strong preaccreditation increase in mortality, accreditation was associated with a 9.08 percentage point reduction (95% CI −13.29 to −4.87) in mortality. We also observed a 0.41 percentage point reduction (95% CI −0.71 to −0.10) in the quarterly trend in centre 4. Combining all level II centres, observed associations were no longer present, given that centres 5 and 7 had a smaller volume of patients (around 800 admissions for each centre during the study period) compared with centres 4 and 6, which each recorded more than 1300 admissions (table 1).
Due to the low number of deaths in paediatric centres (n=41), we used a weighted GEE model combining the two centres. Results did not show an association between accreditation and in-hospital mortality (table 1).
Figure 3 displays the crude and standardised monthly probabilities of major complications in adult level I centres. Trend in complications was non-linear. Accreditation was associated with a 0.25 percentage point decrease (95% CI −0.35 to −0.15) in the monthly trend for centre 2. When combining the three centres together, accreditation was associated with a decrease in the monthly trend of complications (table 2).
Among adult level II centres, for centre 5, which experienced a 0.60 (95% CI 0.12 to 1.08) percentage point increase in the quarterly preaccreditation trend, accreditation was associated with a 9.60 percentage point reduction (95% CI −15.77 to −3.43) in the level and a 0.63 percentage point reduction (95% CI −1.24 to –0.02) for the quarterly trend in complications (figure 4). When we further adjusted for the outlier point two quarters before the accreditation visits in those centres, we observed a smaller 5.68 percentage point reduction (95% CI −11.64 to 0.28). We did not observe an association between accreditation and complications when combining all level II centres together (table 2).
Accreditation was not associated either with a change in levels or trends of complications in paediatric centres, after combining the two paediatric centres due to the low number of major complications (n=80) (table 2).
Table 3 presents corrected estimates and 95% CIs for the average change in outcomes following accreditation in centre 5 (centre for which accreditation seems to have the greatest observed association), over a range of different values of sensitivity parameters (RRUY and RRAU). The columns represent the inverse of the largest relative risk of the effect of one or more unmeasured confounders U on the outcome (RRUY) and the rows the inverse of the largest relative risk of accreditation and U (RRAU). Our interest relies on the threshold from which the CIs around the estimates include the null value. Table 3 indicates that if following accreditation, there is a 40% increase in reports of comorbidities only due to changes in coding practices (RRAU=0.7), patients who were more likely to have their comorbidities under-reported should have at least twice the risk of death compared with other patients (RRUY=0.5) to explain away the observed associations. In the case of complications, if RRAU=0.7, then patients who were more susceptible to have their comorbidities under-reported should have at least a 65% higher risk of complications compared with other patients (RRUY=0.6) to explain away the observed association.
Applying a sensitivity analysis formula for differential measurement errors in continuous outcomes (online supplemental appendix 2),36 we estimated that an increase in the reporting of complications following accreditation could only amplify the observed associations, while a decrease in the report of complications would reduce the observed association. The former is more plausible because some centres have been invited to improve their recording of complications after the accreditation visit. Therefore, our results are likely an underestimate of any true effect.
Finally, applying robust-ITS to assess the impact of accreditation for centre 2 yielded higher estimates of change in level compared with our main analysis, mostly because the change point in the series was identified as occurring 5 months before the visit (online supplemental appendix 4-eTable 2). In the case of complications, a 2.84 percentage point reduction (95% CI −6.19 to 0.51) in level and a 0.22 percentage point reduction (95% CI −0.33 to −0.11) in the monthly trend were observed. A change in data dependency following accreditation was also observed; there was a negative first-order autocorrelation in the preperiod not present in the postperiod.
After accounting for changes in patient case mix and secular trend in studied outcomes, our study did not find consistent evidence of a beneficial impact of accreditation on in-hospital mortality or complications in severely injured patients. However, for centre 5 which experienced a preaccreditation increase in the levels of studied outcomes, accreditation was associated with a decrease in levels (due to the preparation for the visit) and trends. Similar results were found for centre 2, where a decrease in the monthly trend of major complications was observed following accreditation. These associations were robust to moderate levels of residual confounding and differential measurement error, potentially due to changes in coding practices following accreditation.
Previous studies looking at the impact of accreditation on mortality and complications found inconsistent results.11 Accreditation was imprecisely associated with decreased mortality, except among critically injured patients (ISS >24) (OR 1.17; 95% CI 1.05 to 1.30).45–50 Some studies suggested an association between accreditation and reductions in the occurrence of complications, which was stronger among older adults (OR 0.40; 95% CI 0.27 to 0.60) and paediatric critically injured patients (OR 0.23; 95% CI 0.12 to 0.47).47 48 Other studies did not observe any association between accreditation and patient outcomes.51 52 However, all prior studies were either cross-sectional or pre-post and conducted in the USA where accreditation is mostly voluntary. Cross-sectional studies do not distinguish between centres that failed during the accreditation process from those that never applied among their control centres, and pre-post studies cannot account for the underlying trend in the measured outcomes before accreditation, which can bias estimates in either direction.53
Our study has some caveats that should be considered when interpreting our results. Other events capable of influencing studied outcomes may have occurred at the same time as accreditation, introducing bias. In addition, changes in coding practices following accreditation may also have biased our results by introducing errors in the measurement of patient characteristics and outcomes. However, our sensitivity analyses suggested that the magnitude of these errors would have to be strong to completely explain our observed effects. In addition, we adjusted for known major cointerventions that occurred in some centres. We also adjusted for seasonality which can bias the results if the accreditation occurs around a seasonal changing point. The Ljung-Box test of each ITS model indicated that residuals could be considered as white noise and thus our models were correctly accounting for trends, seasonality and any other cycles present in the series.54
In sensitivity analyses, we also applied a robust-ITS model that overcomes limitations of segmented regressions for centre 2. We observed slightly higher estimates of change in level and trends compared with our main analysis, mostly because the change point in the series was identified as occurring 5 months before the visit. In addition, there was a negative autocorrelation in the preperiod (higher for major complications) suggesting that outcomes taken close together in time are likely to be dissimilar, while in the postaccreditation period there was less evidence for presence of autocorrelations.55
Our study might be underpowered, even with several time points. Recent simulations demonstrated that in addition to the number of time points, other factors such as the sample size per time point, expected effect size, location of intervention in the time series and preintervention trends need to be considered to denote an ITS analysis as sufficiently powered.56 However, there was no way to increase our sample size, given that we include all trauma patients satisfying inclusion criteria. Our outcomes were percentages and restricted to lie between 0 and 100. This has important consequences given that ceiling or floor effects can bias results.41 57
Estimates of accreditation impact stratified by levels are weighted average, with weights being a function of both centre size and variances. Given that the accreditation visit dates were different by centre, these estimates might be a poor summary of the average centre-specific effects.58 59In addition, with heterogeneous treatment effects, linear regressions with period and group fixed effects can yield a negative average effect while all specific effects are positive.60 Although performing analyses by centre may have allowed for a better specification of the underlying functional form and facilitated verification of the consistency assumption necessary for making causal inference, it potentially induced inferential problems due to multiple testing.61 However, in sensitivity analyses using a single model with interaction between centres and time, accreditation yielded similar results, almost identical for centres with linear underlying trends (online supplemental appendix 5).
These caveats considered, our results do not consistently support the hypothesis that accreditation decreases in-hospital mortality and major complications. Our hypothesis was based on the fact that accreditation aims to ensure the standardisation of the human and material resources within the centre and/or better adherence to evidence-based clinical processes of care. However, its impacts on patient outcomes in a mature trauma system might be less evident if centres have already achieved a plateau in their performance.38 62 63 This is supported by the fact that we do observe some associations within centres experiencing a deterioration in their performance before the accreditation visit. While a changing trend from positive to negative or flat can be considered a successful outcome of accreditation, the lack of deviation from an already flat trend does not necessarily constitute a failure given that other outcomes such as organisational culture, staff recruitment and retention and patient-reported measures could also be impacted.
We presented a comprehensive assessment of mandatory trauma centre accreditation in a mature trauma system. Our study fills a gap in the literature given that previous studies were limited in their internal validity, since they lacked a design to identify the effect of accreditation, and their external validity, since the prior literature was derived from the USA where accreditation is mostly voluntary. Accreditation seems to be beneficial for centres experiencing a decrease in performance in the months preceding the visit. However, further studies looking at clinical processes of care and other outcomes such as patient or health staff satisfaction are needed to improve our understanding of the impact of accreditation.
Data availability statement
Data may be obtained from a third party and are not publicly available. Data may be obtained through a request to the Quebec’s health insurance board (Régie de l’assurance maladie du Québec, RAMQ).
Patient consent for publication
The authors would like to thank Melanie Berube, PhD, Amina Belcaid and Xavier Neveu for their help in the design and interpretation of this study.
Contributors BB, LM, HC and AN contributed to the development of research objectives. BB conducted data analysis and drafted the manuscript. BB, AN, ES, LM and HC revised the manuscript and approved the final version.
Funding Funds for this project are covered by the Fonds de Recherche du Québec-Santé (FRQS) PhD Scholarship (BB) and a Canadian Institute of Health Research (CIHR) Foundation grant (FRN 353374 for LM and FRN 148467 for AN).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.