This study examines the relationship between outlier status based on adjusted mortality rates and theoretical underlying quality of care in hospitals. We use Monte Carlo stimulation to determine, in the absence of case mix variation, if random variation noise could obscure the signal of differences in underlying rates of quality of care problems. Classification of hospitals as "outliers" is done compared with "true" hospital quality, based on underlying rates for quality of care problems in mortality cases. Predictive error rates with respect to "quality" for both "outlier" and "non-outlier" hospitals are substantial under a variety of patient load and cutoff point choices for determining outlier status. Using overall death rates as an indicator of underlying quality of care problems may lead to substantial predictive error rates, even when adjustment for case mix is excellent. Outlier status should only be used as a screening tool and not as the information provided to the public to make informed choices about hospitals.