Each year, the Health Care Financing Administration (HCFA) releases a report comparing hospital mortality rates with predicted rates. Some argue that the HCFA's prediction model does not adequately account for patient severity. We tested this hypothesis by comparing the HCFA's model (replicated as closely as we could) to a second that added a severity measure (the Stage of Principal Diagnosis at Admission, a subscale of the Severity of Illness Index). In our simulation, the HCFA's model had very limited capacity to predict mortality (average R2, 2.5%). Patients grouped according to admission severity had markedly different mortality rates, which the HCFA's model's predictions could not differentiate. The HCFA model also failed to predict large differences in mortality between hospitals with low- and high-severity admissions. Adding severity to the HCFA's model yielded more than an eightfold increase in the R2, to 21.5%, and reduced instances of higher than expected hospital mortality to chance levels. These findings suggest that the HCFA's mortality release needs to be made much more sensitive to admission severity before it can be used to make valid inferences about the quality or effectiveness of hospital care.