Although several hospital administrators continue to express concern about the effects of hospital standardized mortality ratios (HSMRs) — particularly, negative press — proponents say it’s proving to be an effective quality improvement tool.
Representatives of two regional health authorities that received unflattering scores in each of the first three reports on HSMRs in Canadian hospitals released by the Canadian Institute for Health Information (www.cihi.ca/CIHI-ext-portal/internet/EN/TabbedContent/health+system+performance/quality+of+care+and+outcomes/hsmr/cihi022025), say they have ongoing concerns about how the indicator is represented and calculated.
But at an international meeting of health information experts in Toronto, Ontario, in October 2010, the HSMR was cast in a uniformly positive light.
The HSMR “is one more indicator that allows hospitals to measure and monitor their progress to improve quality of care,” Dr. Eugene Wen, manager of health indicators at CIHI, told delegates to the World Health Organization —Family of International Classifications network, which includes representatives from 30 countries on five continents.
Developed in the United Kingdom during the mid-1990s, the HSMR is a ratio of a hospital’s rate of unexpected deaths to a national average, adjusted for age, sex and diagnoses of patients. CIHI started calculating an HSMR for each Canadian hospital from discharge summaries as of 2004–05. It describes the indicator as “big-dot” — a system-level measure that can be used as a screening tool.
Wen states its real value lies in year-over-year trending for an individual hospital, and in drilling down to analyze deaths in the 65 disease groups covered.
Canada is among the first countries to use the HSMR, although the indicator has since been piloted or tested in the United States, Holland and Australia.
But when the first HSMR report was released, it amounted to shock treatment for some administrators contacted by CMAJ.
“I was surprised,” says John Malcom, chief executive officer of the Cape Breton District Health Authority, which oversees six hospitals. “And quite frankly, I thought I’d be out of a job.”
The results came as a “bit of shock” to Arden Krystal, former executive director of the Burnaby Hospital in British Columbia. “The staff were quite devastated,” she says. “It did not square with what their lived experience had been. We also heard from our patients and donors that it didn’t seem right.”
Although CIHI stresses that the HSMR should be used only to improve quality, not to compare hospitals, even Wen noted during the Toronto session that public release of the data hit the front pages in an often negative context. Wen recalled that former Ontario health minister George Smitherman called it “revolutionary” and talked openly about what could be done to protect patients in hospitals with poor scores.
Malcom and Krystal say negative publicity undermined community confidence in their hospitals. “When information like this goes out to the media, it’s just too juicy for the media not to report it,” Krystal says, adding that the “scorecard” nature of the HSMR lent itself to interhospital comparisons.
Wen says CIHI has tried to discourage such a divisive view. “We didn’t want a ranking, with hospitals fighting each other,” he says. CIHI has since changed the methodology from the original UK measure, in which the national average changed each year and meant that a hospital’s ratio could worsen against an improving national average, even if deaths had decreased.
Now, the baseline is unchanged from 2004–05, so one year can be measured against the previous year. “That way, hospitals don’t have to worry about competing with the hospital across the street,” Wen explains.
Administrators say they have been able to improve their own scores year over year.
Both the Cape Breton District Health Authority, and the Fraser Health Authority (comprising 12 acute care hospitals) say the ratios prompted them to conduct chart reviews of recent deaths to find the causes. Both discovered coding problems.
“Documentation was not as comprehensive as it should have been,” says Krystal, now vice-president of clinical operations at the Fraser Health Authority. “We did find charts where comorbidities were related to the outcome but they weren’t noted because of skimpy documentation. There were some methods implemented to improve discharge summaries.”
In Cape Breton, many deaths that were palliative were not coded as such, says Malcom. “But the coding was a red herring. It was really about having end-of-life discussions.” Malcom says staff were respecting family wishes to withhold from patients the terminal nature of their illness. “We were reflecting community values, but clearly it was unethical to withhold information from a competent patient.”
The authority shifted to a policy of having honest end-of-life discussions. As a consequence, a palliative care unit has been opened on the island and additional hospices are planned.
Malcom says the authority’s HSMR also increased because of a high percentage of people dying in hospital, the product of a lack of care provided at home due to out-migration of young people.
Likewise, Krystal notes that the large number of people of Asian descent in the Fraser Valley affects the HSMR because many people of Asian descent are more comfortable with death in hospital than at home.
Malcom and Krystal are also concerned that the HSMR is not adjusted for population income levels. “As we know, income has a major impact,” says Krystal. “A person with a very low income with one comorbidity may be in much worse health than a wealthy person with the same comorbidity.”
One study indicates that HSMRs can “inaccurately label good hospitals as poor performers and fail to identify many hospitals with true quality problems” and provide “no indication of the underlying quality problems (www.cmaj.ca/cgi/doi/10.1503/cmaj.080010).
Wen “respects the argument” that death may not reflect process issues. But he points out that are two outcomes in health care: death and improved function. As with hockey, a team may have excellent skills, but will not win the game unless it scores, he says. “We believe the process is important, but it needs to be reflected in the outcome.”
Malcom regards the HSMR as a “golf score” to be improved upon, and tries to foster that view in staff.
Dr. Andrew Webb, vice-president of medicine at Fraser Health notes that hospitals have little choice but to use the HSMR. “It’s mandated.” But recent success in lowering the ratio is not an indicator of anything, he adds. “We could turn around and say ‘Aren’t we wonderful?’. Instead we’ve said, ‘We don’t know what this means.’ We don’t know what the factors are that have caused that reduction. ... We take less notice of the HSMR, more notice of the factors that might factor into it, such as nosocomial infection rates and variation in clinical practice between hospitals.”