Article Text
PDF
Abstract
Background Ecological fallacy refers to an erroneous inference about individuals on the basis of findings for the group to which those individuals belong. Suppose analysis of a large database shows that hospitals with a high proportion of long length of stay (LOS) patients also have higher than average in-hospital mortality. This may prompt efforts to reduce mortality among patients with long LOS. But patients with long LOS may not be the ones at higher risk of death. It may be that hospitals with higher mortality (regardless of LOS) also have more long LOS patients—either because of quality problems on both counts or because of unaccounted differences in case mix. To provide more insight how the ecological fallacy influences the evaluation of hospital performance indicators, we assessed whether hospital-level associations between in-hospital mortality, readmission and long LOS reflect patient-level associations.
Methods Patient admissions from the Dutch National Medical Registration (2007–2012) for specific diseases (stroke, colorectal carcinoma, heart failure, acute myocardial infarction and hip/knee replacements in patients with osteoarthritis) were analysed, as well as all admissions. Logistic regression analysis was used to assess patient-level associations. Pearson correlation coefficients were used to quantify hospital-level associations.
Results Overall, we observed 2.2% in-hospital mortality, 8.1% readmissions and a mean LOS of 5.9 days among 8 478 884 admissions in 95 hospitals. Of the 10 disease-specific associations tested, 2 were reversed at hospital-level, 3 were consistent and 5 were only significant at either hospital-level or patient-level. A reversed association was found for stroke: patients with long LOS had 58% lower in-hospital mortality (OR 0.42 (95% CI 0.40 to 0.44)), whereas the hospital-level association was reversed (r=0.30, p<0.01). Similar negative patient-level associations were found for each hospital, but LOS varied across hospitals, thereby resulting in a positive hospital-level association. A similar effect was found for long LOS and readmission in patients with heart failure.
Conclusions Hospital-level associations did not reflect the same patient-level associations in 7 of 10 associations, and were even reversed in 2 associations. Ecological fallacy thus potentially influences interpretation of hospital performance when patient-level associations are not taken into account.
- mortality (standardized Mortality Ratios)
- quality measurement
- healthcare quality improvement
- continuous quality improvement
Statistics from Altmetric.com
- mortality (standardized Mortality Ratios)
- quality measurement
- healthcare quality improvement
- continuous quality improvement
Introduction
Many indicators are collected in hospitals to monitor quality of care and to compare performance against other hospitals. Dutch hospitals for example annually have to collect data on at least 3026 variables for quality measurements.1 Indicators are used for different purposes, such as quality improvements in hospitals, facilitation of patients’ choices for hospitals and healthcare insurer’s decisions for purchasing health services. Commonly used quality indicators include in-hospital mortality, 30-day readmissions and long length of stay (LOS). Long LOS and readmissions have been shown to be partly due to complications during admission and after discharge, respectively.2–8
Previous studies examined the relationship between indicators to see whether they may affect the interpretation of each individual indicator. For example hospitals with shorter LOS may have higher readmission rates. One explanation for this association is that patients are discharged too early and therefore need to be readmitted. Bottle et al 9 found that patients with heart failure (HF) and a longer LOS had less chance on readmission for HF, with same-day discharges having the highest risk. Kaboli et al 10 on the other hand found for all medical diagnoses combined that patients with longer LOS have higher readmission rates. In addition there is a trend towards fast-track surgery for specific diagnoses, for example, decreasing LOS after total hip or knee arthroplasty. Williams et al 11 and Clement et al 12 found that patients with a hip/knee arthroplasty and an increased LOS had an increased risk for readmission, but other studies showed that a decreased LOS over time was not associated with a change in readmissions.13 14 Results from previous studies are thus inconsistent.
One of the explanations for inconsistent results is that most studies assessed associations at either patient or hospital-level, but hospital-level and patient-level associations may differ due to ecological fallacy. Ecological fallacy is the false assumption that inferences made at hospital-level would apply to individual patients.15 When a reversed association is found at the patient-level, this means that the association found at the hospital-level does not reflect the association at the patient-level and is thus not correctly interpreted.16–18 This is important as it may influence the focus for quality improvement initiatives. Suppose analysis of a large database shows that hospitals with a high proportion of long LOS patients also have higher than average in-hospital mortality. This may prompt efforts to reduce mortality among patients with long LOS. But patients with long LOS may not be the ones at higher risk of death. It may be that hospitals with higher mortality (regardless of LOS) also have more long LOS patients—either because of quality problems on both counts or because of unaccounted for differences in case mix, resulting in the wrong patient group being targeted to reduce mortality. Studying patient-level and hospital-level associations at the same time and linking these provides more insight into whether ecological fallacy is at play, and if so why it takes place and how we can use this information to improve quality of care. Therefore, we aimed to assess whether hospital-level associations between in-hospital mortality, readmission (acute, 30 days) and long LOS reflect patient-level associations to provide more insight into the extent to which ecological fallacy influences the evaluation of hospital performance indicators.
Methods
Study population
We used routinely collected administrative admission data of the Dutch National Medical Registration (LMR) from 2007 to 2012. The LMR contains data of approximately 88% of the hospital admissions in 2007 to 76% in 2012,19 including medical data such as diagnosis and surgical procedures, as well as patient-specific data such as age and hospital stay. Each patient is assigned an anonymous unique patient identifier so that multiple admissions within one patient in the same or different hospitals could be identified.20
In all hospitals, coders retrospectively assigned one primary diagnosis code according to International Classification of Diseases, Clinical Modification (ICD-9-CM) to each admission based on the discharge letter and other available information in the medical records of patients, and secondary diagnosis codes if applicable. We included clinical admissions with a primary diagnosis code. Patients only receiving daycare preceding tests or treatments for a number of hours were excluded. Not all hospitals coded and supplied their admission data each month and every year. Therefore, we excluded admissions from hospitals with incomplete follow-up, defined as months without any coded admissions, particularly relevant for readmissions within 30 days. We also excluded admissions from hospitals without a previous month of measurements, because it was unknown whether these patients had an admission within the previous 30 days, so that we did not know whether it was an index admission or a readmission. In 2012, 45% of the procedures were missing and therefore procedures in 2012 were excluded.
Patient admissions for specific diseases were considered, as well as all admissions. The Clinical Classifications Software (CCS) for ICD-9-CM was used to classify admissions by type of condition. Individual procedure codes (CvV) were used to classify admissions into procedure groups. We selected patient groups for which we expected different patterns in indicators: stroke (CCS 109) with high mortality rates and long LOS, colorectal carcinoma (CCS 14 and 15) with long LOS, HF (CCS 108) with high readmission rates, acute myocardial infarction (AMI) (CCS 100) with relatively high mortality rates, and total hip/knee arthroplasty (THA/TKA) in patients with osteoarthritis (CCS 203) with high readmission rates.
Definitions
We studied the associations at hospital-level and at patient-level between three indicators:
in-hospital mortality: defined as death in hospital during the index admission
acute readmission: an emergency readmission within 30 days after discharge
long LOS: defined as a LOS in the upper quartile LOS of the specific diagnosis (CCS group) or procedure group (for THA/TKA); the cut-off in number of days was chosen such that the percentage of patients would be as close to 25% as possible.
The following variables were included to adjust for differences in case mix: 5-year age groups, sex, socioeconomic status based on the postal area of the patient’s address (six categories), year of admission, diagnosis or procedure group (for THA/TKA), method of admission (acute/not acute), transferred from other hospital, urgent admission in previous month (yes/no), and the Charlson comorbidity score to correct for comorbidities based on the secondary diagnosis codes.21
Statistical analysis
We used logistic regression analysis to assess whether patients with long LOS had different likelihood of readmission and in-hospital mortality (patient-level association), both for all admissions and for disease-specific admissions. Long LOS was used as independent variable, and in-hospital mortality or readmission as dependent variables in two separate models, which were adjusted for case mix and hospital (fixed effect). All case mix variables defined above were included, as well as statistical interactions between age and Charlson comorbidity score, and between method of admission and transfer based on previous findings.22 Age groups with fewer than 10 events were iteratively combined with the immediately older group.
To investigate hospital-level associations, we first calculated standardised in-hospital mortality, readmission and long LOS by dividing the observed numbers by the expected numbers × 100 per hospital. The observed numbers were the number of patients who died in each hospital, who were readmitted or with a long LOS. The expected numbers were the sum of all patients’ expected probabilities for in-hospital mortality, readmission or long LOS within a patient diagnosis or procedure group. The patients’ expected probabilities were calculated using logistic regression analysis with all the above-defined case mix variables included, as well as statistical interactions between age and Charlson comorbidity score, and between method of admission and transfer. Hospital was not included in the models used to calculate the expected probabilities. The next step was to estimate the hospital-level associations between the standardised long LOS and standardised mortality and standardised long LOS and standardised readmission using Pearson correlation coefficients, for all admissions as well as disease-specific admissions. We defined the strength of the correlation as follows: ≤0.35 weak correlation, >0.35–0.67 moderate correlation and >0.67–1 strong correlation (same for negative correlations).23
Both patient-level and hospital-level associations were estimated for the period 2007–2012, but also for each year and for 3-year periods (2007–2009 and 2010–2012), to assess whether the associations were stable over time or varied, for instance, as a result of national policies or interventions to improve quality of care. We also performed sensitivity analyses using different definitions of readmission and long LOS:
all readmissions within 30 days after discharge rather than only acute readmissions, as this is used in many Dutch quality indicators
acute readmission for the same CCS group to separate a readmission for the same diagnosis from a readmission for comorbidity or complications
long LOS as a LOS in the upper decile LOS.
All analyses were executed using the software package SPSS (IBM SPSS Statistics V.20.0).
Results
Population
Based on the inclusion criteria 8 478 884 admissions (96%) of the 8 821 686 admissions from 95 hospitals were included. Patient groups had different patterns with respect to age, comorbidity, chances of urgent admissions as well as the outcome indicators (table 1). Of the five selected patient groups, the largest group consisted of HF admissions (n=133 488). The overall in-hospital mortality across all admissions was 2.2%, with the highest in-hospital mortality observed for stroke (14.5%) and lowest after THA/TKA (0.1%). Readmissions occurred in 8.1% of the admissions, with the highest percentage (14.6%) for patients with HF and the lowest percentage for patients with a THA/TKA (3.7%). The LOS was 5.9 days on average (SD 8.8) and longest for patients with colorectal carcinoma (11.2 days, SD 12.0). The number of admissions varied considerably between hospitals, with a median of 83 599 admissions across the 6-year period and an IQR of 50 048–138 284 (table 2). Colorectal carcinoma had the highest median standardised in-hospital mortality rate, 105.6 (IQR 80.3–127.7). The lowest median rate was found for THA/TKA (80.1 (IQR 0.0–100.9)) but also the largest between-hospital variation.
Baseline patient characteristics and indicators (n (%) or mean (SD))
Baseline characteristics and indicators at the hospital-level, median and IQR
Long LOS and in-hospital mortality
Overall, patients with a long LOS had an increased odds of in-hospital mortality after adjusting for hospital and case mix (OR 1.48 (95% CI 1.46 to 1.49)) (table 3). This was reflected in the positive hospital-level correlation (r=0.48, p<0.01), meaning that hospitals with many long LOS patients also have higher in-hospital mortality rates.
Relations between standardised long LOS and standardised in-hospital mortality or standardised readmission at the patient and hospital-levels
However, associations varied for disease-specific admissions (table 3). Of the five patient-level associations tested in disease-specific patient groups, one was reversed at hospital-level, one was consistent, two had significant patient-level associations but were not significantly associated at hospital-level, and one not significantly associated at patient-level had a positive hospital-level association. The reversed association was found for patients with stroke. Patients with stroke with a long LOS have a lower odds to die in-hospital (OR 0.42 (95% CI 0.40 to 0.44)), but hospitals with many long LOS patients also have higher in-hospital mortality rates (r=0.30, p<0.01), thereby not reflecting what was found at the patient-level. Figure 1A, B shows that within each separate hospital we found that patients with a long LOS had a decreased odds of in-hospital mortality, but that the LOS varied across hospitals, resulting in a positive hospital-level correlation (figure 1B). Only for colorectal carcinoma the positive association found at the hospital-level reflects the patient-level association (figure 2A,B).
Opposite relations between in-hospital mortality and long length of stay (LOS) at the patient and hospital-levels for stroke. (A) Shows that within each separate hospital, patients with a long LOS had a decreased odds on in-hospital mortality, but the varying LOS across hospitals results in a positive correlation: hospitals with a longer LOS were also the hospitals with a higher mortality (B).
Same relations between in-hospital mortality and long length of stay (LOS) at the patient and hospital-levels for colorectal carcinoma. (A) Shows that within each separate hospital, we found that patients with a long LOS had an increased odds on in-hospital mortality, which was reflected in (B) showing a positive correlation when hospital averages are used.
Long LOS and readmission
Overall, patients with a long LOS had an increased odds of readmission (OR 1.19 (95% CI 1.18 to 1.20)), but no correlation was found at the hospital-level (r=−0.10, p=0.36).
Again, associations varied between patient groups and of the five patient-level associations tested in disease-specific patient groups; one was reversed at hospital-level, two were consistent and two were not significantly associated at hospital-level, whereas significant patient-level associations were found (table 3). Only for colorectal carcinoma a positive association was found in both patient-level (OR 1.26 (95% CI 1.20 to 1.32)) and hospital-level (r=0.35, p<0.01). A reversed association was found for HF: patients with a long LOS had a higher odds of readmission (OR 1.06 (95% CI 1.02 to 1.09)), whereas at hospital-level a negative correlation was found (r=−0.28, p=0.01), meaning that hospitals with many long LOS patients had a low readmission rate. Again we found within each separate hospital that patients with a long LOS had an increased odds of readmissions, but that the LOS varied across hospitals, resulting in a negative correlation (figure 3A,B).
Opposite relations between readmission and long length of stay (LOS) at the patient and hospital-levels for heart failure. (A) Shows that within each separate hospital, patients with a long LOS had a (slightly) increased odds on readmission, but the varying LOS across hospitals results in a negative correlation: hospitals with a longer LOS were also the hospitals with lower readmission rates (B).
Sensitivity analyses
Evaluating all readmissions rather than acute readmissions only, the positive patient-level association between readmission and long LOS was now reflected in a moderate positive hospital-level correlation (r=0.50, p<0.01) for all patients, which was not found for acute readmission (r=−0.10, p=0.36). Other associations between long LOS and readmission were comparable, except for colorectal carcinoma whose OR and correlation coefficient changed in direction (see online supplementary file). For HF the reversed (negative) correlation at hospital-level became non-significant. Only the hospital-level associations for all admissions and colorectal carcinoma reflect the same patient-level associations. In general though, correlations for different patient groups were similar when evaluating all or only acute readmissions, with most associations only significant at either hospital-level or patient-level, so that patient-level associations were not reflected at hospital-level.
Supplementary file 1
We also evaluated acute readmission for the same CCS group as this may be associated stronger with reduced/inadequate care (see online supplementary file). Overall, patients with a long LOS had a decreased odds for acute readmission for the same CCS group (OR 0.94 (95% CI 0.93 to 0.94)), which is opposite compared with the patient-level association for all acute readmissions, indicating that most of the acute readmissions after a long LOS may have been for comorbidities or complications. Similar results were found for colorectal carcinoma, namely that patients with a long LOS had decreased (rather than an increased) odds for acute readmission for the same CCS group. The significant patient-level association for HF disappeared, indicating that patients with HF were not acutely readmitted for their HF after a long LOS, while patients with AMI with a long LOS now had a decreased odds to be acutely readmitted for their AMI (rather than no association for all acute readmissions). For the other patient groups, the results were in the same direction, but stronger. Hospital-level correlations were comparable as for all acute readmissions, except for colorectal carcinoma where a negative (rather than positive) correlation was found. Only associations for colorectal carcinoma were consistent for both hospital and patient-levels.
Overall, using upper decile LOS instead of upper quartile LOS showed stronger associations. Some patient-level associations now became significant, for example, for upper decile LOS and in-hospital mortality among patients with HF (OR 1.36 (95% CI 1.29 to 1.33)), whereas the significant association disappeared for upper decile LOS and acute readmission in patients with HF (OR 0.99 (95% CI 0.94 to 1.04)). Hospital-level correlations were similar (see online supplementary file). Still reversed associations at patient and hospital-levels were found for stroke, and only associations for colorectal carcinoma and HF (long LOS with mortality) were consistent for both hospital and patient-levels.
Lastly, both patient-level and hospital-level associations were similar when looking at 1-year or 3-year period, and thereby are relatively stable over time (data not shown).
Discussion
This study shows that ecological fallacy is at play, resulting in different associations at hospital and patient-levels. Hospital-level associations did not reflect the same patient-level associations in 7 of the 10 disease-specific associations, and were even reversed in 2 associations. Ecological fallacy thus potentially influences interpretation of hospital performance when patient-level associations are not taken into account. For stroke, patients with a long LOS had a decreased odds of in-hospital mortality, whereas at hospital-level, hospitals with many long LOS patients also had high in-hospital mortality rates. Within each separate hospital, we found that patients with a higher LOS had a decreased odds of in-hospital mortality (negative association), but that the LOS varied across hospitals, thereby resulting in a positive hospital-level association. A similar effect was shown for HF. Patients with a long LOS has an increased odds of readmission, but hospitals with many long LOS had lower readmission rates.
These reversed associations clearly show that we have to be careful in interpreting and drawing conclusions from hospital-level associations. For example if hospitals with many long LOS patients as well as high in-hospital mortality rates for stroke want to reduce their in-hospital mortality while ignoring patient-level associations, they may wrongly conclude that patients with a long LOS should be the target of quality improvement interventions. However, including the patient-level association shows that patients with long LOS had a decreased odds of in-hospital mortality so that they should actually not focus on patients with long LOS to reduce their in-hospital mortality rates but on the short LOS patients. In addition, hospitals with many long LOS patients with HF had lower readmission rates, while at patient-level the opposite was found with long LOS patients having increased odds of readmission. Hospitals may wrongly conclude from hospital-level associations to focus on short LOS patients to improve their readmission rate, whereas in fact they should focus on patients with a longer LOS, looking at the patient-level associations. Furthermore, the indicators may be associated at hospital-level without a patient-level association or vice versa. For example, hospitals with many long LOS patients with AMI also have higher in-hospital AMI mortality rates, while at patient-level no association was found. Based on only the hospital-level association, one may wrongly conclude that these hospitals have complex patients (who have longer LOS and high in-hospital mortality), whereas in fact these indicators were not significantly associated at patient-level so that the performance may be relatively poor on both indicators. In addition, four significant patient-level associations were found but non-significant correlations at hospital-level. This may reflect a lack of power due to small sample sizes at hospital-level,24 meaning that weak correlations are not significant due to the small number (73–95) of hospitals. The direction of two of these non-significant hospital-level associations was opposite of the associations at patient-level, suggesting that we may have found more reversed associations if more hospitals could have been included. Thus ideally analyses at both patient and hospital-levels should be performed for proper interpretation of hospital performance and which patient group should be targeted for improvement.
Comparison with previous studies
Several studies described ecological fallacy both outside and inside healthcare.16 17 25–27 A well-known example outside healthcare was described by Robinson,25 showing that states in the USA with a higher proportion of immigrants also had higher literacy rates, thereby suggesting that immigrants had higher literacy rates, while at the individual level immigrants were less literate than native citizens. This was caused by the fact that immigrants tended to settle in states where the native population was more literate. Recently, a study within healthcare demonstrated that the lack of association between a decreasing annual door-to-balloon time and changes in mortality at the population level should not be interpreted as an indication of the same individual-level relation in patients with ST-elevation myocardial infarction undergoing primary percutaneous coronary intervention (pPCI). Shorter patient-specific door-to-balloon times were consistently associated with lower mortality in each year, but across years patients at higher mortality risks were also undergoing pPCI, thereby resulting in no change in mortality over time.18 Other studies also showed with hypothetical data that process relationships can differ at the patient and hospital-levels, caused by variations in case mix between hospitals17 or difference in confounding variables.16 26 In our study we used existing data and showed that ecological fallacy results in different associations at hospital and patient-levels, thereby affecting evaluation of hospital performance indicators. For stroke, for instance, this was explained by differences in the average LOS across hospitals. This again teaches us that the wrong conclusion could be drawn by assuming that the relationship observed at group level would be the same at the individual level.
The results of our study regarding the associations are partly consistent with associations found in previous studies. Kaboli et al 10 also found for all medical diagnoses combined that patients with longer LOS have higher readmission rates, which is consistent with our overall results at the patient-level. At hospital-level they also found that hospitals that tend to discharge patients sooner than expected also have higher readmission rates, while we did not find a correlation at hospital-level. On the other hand, Bottle et al 9 found that the relation between LOS and readmission was J-shaped, with same-day discharges having a high readmission rate. Williams et al 11 and Clement et al 12 found that hip and/or knee arthroplasty patients with an increased LOS had an increased risk for readmission, consistent with our patient-level results. However, other studies suggested that a decreased LOS over time was not associated with a change in readmissions,13 14 but this was based on aggregated data comparing a decrease over time for LOS with relatively stable readmission rates over the same period. In our study we also did not find a hospital-level association between long LOS and readmission, whereas we did find an association at a patient-level. Thus the associations that seem different between studies may in fact be explained by using patient-level or hospital-level associations. In addition, differences in results between studies may be due to differences in healthcare structures between countries,22 such as discharge policies and the availability of resources for supporting palliative patients and home care services, potentially resulting in lower LOS and lower in-hospital mortality and thereby different associations.
Strengths and limitations
A strength of this study is that we used a large nationwide database, including almost all hospitals in the Netherlands, with a large number of admissions and readmissions in the same or different hospitals. Furthermore, we examined both patient-level and hospital-level associations to gain more insight into how hospital-level associations should be interpreted. In addition, we also included patient groups with different patterns in indicators (eg, high mortality vs low mortality). This provides more insight in patterns within specific conditions.
A limitation of our study is that only data from 2007 to 2012 were available. Since patient-level and hospital-level associations did not differ when looking at 1-year or 3-year period, we do not expect different results using more recent data. In addition, data did not include information on mortality after discharge. This does not influence our results if mortality after discharge follows the same pattern as in-hospital mortality, so that both patients and hospitals with relatively high in-hospital mortality also have relatively high postdischarge mortality. However, hospitals may differ in their admission and discharge policies, for example, when external hospice care is available, which results in a shorter LOS. This also reduces in-hospital mortality rates and may thus affect the observed associations, because these patients cannot be readmitted whereas they were counted as being at risk for readmission in our analysis. Furthermore, the administrative data are well-known to be limited in the ability for case mix adjustments, for example, for severity of the disease.28 Overall more complex and severely ill patients are admitted to academic hospitals and more high-risk interventions are performed with higher changes of mortality, readmission and long LOS compared with general hospitals, which could not be distinguished in our data.
Conclusions
This study showed that we have to be careful in drawing conclusions from hospital-level associations without knowing patient-level associations as ecological fallacy may be at play. Additional patient-level data are needed to be able to evaluate the quality of care in hospitals and to start quality improvement initiatives targeted at the right patient group. Furthermore, many associations were found at either patient or hospital-level which might also result in the wrong conclusion being drawn from positive hospital-level associations being due to complex patients, whereas in fact performance on both indicators was poor. Ecological fallacy thus potentially influences interpretation of hospital performance when patient-level associations are not taken into account.
Acknowledgments
We gratefully acknowledge the input of the members of the study group. In addition to the authors, the study group consist of (in alphabetical order): M de Bruijne, J Kievit, N Klazinga, H Lingsma, M Muris, Peter Wee, A van der Veen, B van Leiden-Vriens and M Weijers.
References
Footnotes
Contributors PJM-vM designed the study. SNH wrote the article and carried out the study. PJM-vM supervised the study and writing of the manuscript. All authors have critically read and modified both the study protocol and previous drafts of the manuscript and have approved the final version. All authors read and approved the final manuscript.
Funding This research project is supported by a grant from the Netherlands Organisation for Health Research and Development (grant number 516022513).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.