Methods

The HSMR compares the actual number of hospital deaths with the expected number for those patients with a primary diagnosis within the set of diagnostic groups that account for 80% of all deaths in hospital nationally.

The national LMR dataset for 2005–2007 was used as the data source for the logistic regression calculations. The HSMRs were calculated for 2005–2007. In the LMR dataset, diagnoses are coded using the International Classification of Diseases, Ninth Revision (ICD-9), and these are converted to 259 Clinical Classification System (CCS) groups developed by the US Agency for Healthcare Research and Quality.10 From these CCS groups, those responsible for 80% of hospital deaths nationally were determined. Day cases (which have very few deaths) and inpatient admissions were included in the analysis. Logistic regression models were fitted for each of the CCS groups separately in order to generate an expected risk of death for each patient. The HSMR is derived from the sum of the observed deaths and expected risks across the CCS groups.

Details of Dutch HSMR calculations

1. The 2005–2007 LMR data were used to form the model. These were data made available by Prismant, with permission of the Dutch Hospital Association (NVZ) and the Dutch Association of Medical Specialists. Seventeen thousand fifty-six ICD-9 codes in the Dutch hospital data were assigned to the 259 US Agency for Healthcare Research and Quality CCS groups.

2. After removing vague or undetermined diagnoses, the 50 CCS groups that give rise to 80% of all deaths in 2005–2007 were determined (table 1). Patients with lengths of stay under one year were used. The 50 CCS diagnoses covering 159 987 deaths were used for the model. The reported HSMR is then calculated using data from 2005 to 2007.

3. The calculation for non-average hospitals, hospitals with a case mix very different from the national average, which were excluded, was done by:

   1. calculating the percentage of expected deaths nationally for each of the diagnostic groups making up the HSMR (leading to 80% of all deaths nationally);

   2. calculating as in (a) for each hospital;

   3. scale up or down the number of expected deaths by a scaling factor (Sf) for each diagnostic group to make the percentage of expected deaths at each hospital the same as the national %;

   4. scale the observed deaths at each hospital by the same scaling factor for each diagnostic group;

   5. use the scaled values of the numbers of observed and expected deaths at each hospital to calculate a “scaled HSMR”;

   6. calculate the difference, D, between the normal (unscaled) HSMR and the scaled HSMR and

   7. for the “average”, or non-specialist, hospitals' D tends to be less than 7.5 for the hospitals that are not specialist hospitals.

4. Patients' age was determined from the date of admission—date of birth age groups used were those for the English Hospital Episode Statistics (HES) (ie, <1 year=1, 1–4 years=2, 5–9 years=3, 10–14 years=4, 15–19 years=4, 20–24=6, 25–29 years=7, 30–34 years=8, 35–39 years=9, 40–44 years=10, 45–49 years=11, 50–54 years=12, 55–59 years=13, 60–64 years=14, 65–69 years=15, 70–74 years=16, 75–79 years=17, 80–84 years=18, 85–89 years=19, 90+ years=20)

5. The number of days of care was coded into length of stay (LOS) categories: 1 day=1; 2–7 days=2; 8–16 days=3; 17–23 days=4; 24–1000 days=5 (but only LOS to 365 was used in the data analysis).

6. The age group, sex, urgency, LOS group, CCS diagnosis, month of admission, social deprivation and year categories were determined for each patient.

7. The source of referral for each patient was coded as 0=own habitat; 1=nursing/elderly home; 2=born in hospital; 21=hospital—academic/top clinical; 22=hospital—general; 23=hospital specialised; 24=other care organisations; 29=hospital—unknown.

The statistical performance of the model was measured by the c statistic (area under the receiver operating characteristic curve) for each SMR and for the hospital level HSMR.11 The c statistic is the probability of assigning a greater risk of death to a randomly selected patient who died compared with a randomly selected patient who survived. A value of 0.5 suggests that the model is no better than random chance in predicting death. A value of 1 suggests perfect discrimination. In general, values above 0.75 suggest good discrimination.

Results

In the 2005–2007 data and for the HSMR CCS groups only, there were 2 363 332 admissions and 90 873 deaths (crude death rate 3.85%). The quality of the data of 15 hospitals did not fulfil the national registration standards in 2007, so we did not include them in a national comparison. Seven out of these 15 hospitals did not fulfil the standard for two or more criteria. Six of the 15 hospitals had more than 5% vague diagnosis, eight hospitals had less than 33% urgent admissions and ten hospitals had a ratio of comorbidity diagnosis to main diagnoses of <0.2. Another six hospitals were excluded because they had a patient population that differed too much from the national average. Four of these hospitals had less than 100 expected deaths in 2007, and finally two more hospitals were excluded because they are non-average hospitals in terms of their case mix. A funnel plot of the HSMRs of the remaining 65 hospitals is shown in figure 1.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1

Funnel plot showing HSMR variation, 2005–2007 in Dutch hospitals (excluding 24 hospitals) with 95% and 99.8% control limits.

Hospitals in figure 1 that lie within the control limits are said to exhibit common cause variation and those outside special cause variation unlikely to be due to natural random variation (in Shewhart's original terminology). Funnel plots provide a simple and easily understandable way to plot institutional comparisons.12 They have been used to plot anonymised mortality rates by surgeon for paediatric cardiac surgery13 and have been promoted as providing a strong visual indication of divergent performance, with the advantage of displaying actual event rates and allowing an informal check of a relationship between outcome and volume of cases.14

Dutch HSMRs differ widely among hospitals. According to this analysis, the chance of death in the hospital with the highest HSMR is 2.3 times the chance of dying in the hospital with the lowest HSMR, after adjusting for available case-mix factors.

The c statistic of the Dutch HSMR model is 0.91, similar to the values found for the other countries. Table 1 shows also the c statistics of all CCS groups: they vary from 0.68 to 0.96.

Significant factors determining the total hospital mortality were: primary diagnosis, age, sex, admission urgency (urgent/not-urgent, equivalent to emergency/elective (planned)), LOS, comorbidity (measured by the Charlson Index),15 area-level social deprivation (from the Dutch Central Office of Statistics), month of admission, type of organisation that made the referral and the CCS subgroup. These factors and their coefficients vary among each CCS group. Table 1 gives the significant factors (p<0.05) for every CCS group.

Discussion

HSMRs have been calculated for The Netherlands in a manner similar to that used in several other countries. Currently, almost every Dutch hospital has asked for their HSMR without any pressure from the government or Healthcare Inspectorate. In addition, more than 50 hospitals have ordered a “Hospital Mortality Profile” over the last two years—a brief report giving the HSMR of a hospital broken down into its constituent diagnostic group SMRs and by age group, urgency and length of stay. The following applications of HSMRs are used in Dutch hospitals:

• With the Hospital Mortality Profile to identify high and low risk “areas” within the hospital. Such a retrospective profile enables more directed intervention for patient safety.

• Dr Foster's RTM—a tool used in 16 hospitals for early warning, continuous monitoring and analysis of their mortality by diagnosis and procedures using the same risk models underpinning the HSMR. Hospitals use this tool to follow their own progress in decreasing patient safety risks.

• Some hospitals use HSMRs in combination with clinical audits. They drill down to the level of the mortality risk of individual patients admitted. By doing so, hospitals can select “unexpected cases”. These are patients who die in the hospital but have a relatively low risk of dying in hospital. These cases are perhaps the most useful for case note review and complication analysis and can aid improvement initiatives.

Our analysis of data completeness found no missing values of the date of admission, date of discharge, age, sex, urgency of admission or postal code (for social deprivation). However, for the recording of secondary diagnosis in particular, we cannot tell whether there is no comorbidity present or if comorbidity has simply not been recorded. Miscoding may also affect the HSMR.

The LMR data use a limited number of clinical variables but for the HSMRs examined in this study, the discrimination of the risk prediction model was very good. A recent UK study concluded that, at least for three common procedures, risk prediction with discrimination comparable with that obtained from clinical databases is possible using routinely collected administrative data.16

Although simplified models of risk prediction might be as effective in predicting outcome as some complex models currently in use,17 18 further improvements to the case-mix model are being evaluated. The numbers of previous admissions within a given time period, which requires the linking of admissions of the same patient, could be of potential use. Other features of the healthcare system that could potentially affect hospital mortality ratios include admission thresholds, the proportion of people in the area dying in hospital, discharge policies or underlying disease rates in the catchment population. It is unclear, however, whether and how one should measure and adjust for these factors.

A relevant discussion is also whether the length of stay and the procedure group are factors that are part of the case mix or determine quality. Both are related to the patient's illness but also to treatment.

Based on experience in other countries, the introduction of HSMRs raises various questions.19–22 Most recently, attention has been focused on the so-called “constant risk fallacy”23 in which some SMRs—for example, for some Charlson scores, differ from the overall HSMR. One paper suggests at least two mechanisms that might contribute: the first involves differential measurement error, and the second involves inconsistent proxy measures of risk.24 Measurement error, including poor coding, will have an impact on HSMRs, and this is the first thing that a hospital should check. The variation in SMRs can be interpreted in two ways, either as bias or as real differences in risk. Either way, further investigation using local data sources and case note reviews rather than more statistical analysis is suggested.

Another often heard query is that the methodology should correct for regional variation in health conditions or in the organisation and performance of healthcare facilities adjacent to the hospital. A multiple regression analysis has been developed for the Dutch HSMRs to find the factors that best explain the variation of HSMRs throughout The Netherlands.25 Depending on the extension of the dataset, further yearly refinements can be made to the models for the yearly releases of the HSMRs and SMRs.

The HSMR for The Netherlands appears to be a statistically robust model that can be used as an indicator for hospital deaths to help Dutch hospitals improve their quality of care. The statistical model is robust enough to include all hospitals with more than about 100 deaths per year, an average case mix and good quality data, varying in size and function, into one analysis. However, random variation and data quality issues need to be considered when interpreting the results. HSMRs can be used to highlight hospitals that have significantly high mortality, which may merit further investigation by the hospitals concerned. Furthermore, the impact of interventions designed to reduce mortality can be tracked using this measure.

The Dutch Ministry of Health26 27 has put HSMR high on its quality agenda and commissioned RIVM (the National Institute for Public Health and the Environment) to use HSMRs as one of the performance indicators in the Dutch Health Care Performance report. In the future, international comparisons might also be possible.