Validation of diagnostic codes within medical services claims

https://doi.org/10.1016/S0895-4356(03)00246-4Get rights and content

Abstract

Objectives

Few studies have attempted to validate the diagnostic information contained within medical service claims data, and only a small proportion of these have attempted to do so using the medical chart as a gold standard. The goal of this study is to determine the sensitivity and specificity of medical services claims diagnoses for surveillance of 14 drug disease contraindications used in drug utilization review, the Charlson comorbidity index and the Johns Hopkins Adjusted Care Group Case-Mix profile (ADGs).

Study design and setting

Diagnoses were abstracted from the medical charts of 14,980 patients, and were used as the “gold standard,” against which diagnoses obtained from the administrative database for the same patients were compared.

Results

Conditions associated with drug disease contraindications with the exception of hypertension and chronic obstructive pulmonary disease (COPD) showed a specificity of 90% or higher. Sensitivity of claims data was substantially lower, with glaucoma, hypertension, and diabetes being the most sensitive conditions at 76, 69, and 64%, respectively. Each of the 18 disease conditions contained in the Charlson comorbidity index showed high specificity, but sensitivity was more variable among conditions as well as by coding definitions. Although ADG specificity was also high, the vast majority of ADGs had sensitivities of less than 60%.

Conclusion

The administrative data was found to have diagnoses and conditions that were highly specific but that vary greatly by condition in terms of sensitivity. To appropriately obtain diagnostic profiles, it is recommended that data pertaining to all physician billings be used.

Introduction

Administrative clinical databases are increasingly being used for the purposes of medical research. During the past decade alone, many studies have been published that use hospital discharge claims and physician medical services claims data to examine various health and policy issues. Examples of this literature include studies that investigate health outcomes [1], [2], [3], [4], [5], [6], drug utilization review [7], utilization of services [8], [9], [10], [11], [12], policy evaluation [13], [14], prevalence/incidence and surveillance [15], [16], [17], risk adjustment/health economics [18], [19], [20], and physician profiling and quality of care [21], [22], [23], [24], [25], [26], [27], [28], [29].

Two areas of research where administrative clinical databases are of importance are observational epidemiologic studies and drug utilization reviews. The primary advantages of using administrative data for these purposes are that these data are comprehensive, cost efficient, and free of the usual biases associated with survey methods such as recall bias, nonresponse, and subject attrition [30]. However, the utility of these databases for research differs substantially, particularly as it relates to the comprehensiveness of population information coverage, as well as the source and reason for documentation of diagnostic information.

Hospital claims are frequently used for research, but they are limited to information during periods of hospitalization, capturing diagnostic and treatment information for a very ill population over a defined, and usually brief window of time. The chief advantage of hospital claims data for research is that discharge diagnostic and medical procedure information is recorded by medical archivists, based on a detailed review of the medical chart, increasing the likelihood of accurate documentation.

In contrast, medical services claims data cover the full continuum from ambulatory to hospital-based care, providing information on almost all contacts with physicians in the health care system. As medical services data are created as a by-product of claims for physician reimbursement in a fee-for-service billing system, almost all services provided by fee-for-service physicians will be recorded [31], [32]. However, only the procedure code (e.g., major assessment visit; closed reduction of femur fracture), which is linked to the level of reimbursement, is carefully audited. Diagnostic information that is recorded on each medical service claim, indicating the reason for the medical service, is not typically validated, as these data are not usually linked to remuneration.

There is increasing interest in using medical services claims data for medical research as they capture many of the populations of interest for epidemiologic studies, particularly those that receive the majority of care in the ambulatory setting [24], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42]. Indeed, Klabunde showed that the medical services claims diagnoses augmented information recorded in hospital claims, increasing the detected prevalence of Charlson comorbidity index conditions from 10 to 25%, particularly for conditions such as diabetes that tend to be managed on an ambulatory basis [32]. Yet, the accuracy of the diagnosis data remains suspect as it is neither collected for the purpose of clinical research nor is it directly linked to remuneration. This unproven accuracy limits the potential utility of these databases for research.

Few studies have attempted to validate the diagnoses recorded in medical service claims, and only a small proportion of these have attempted to do so using the medical chart as a gold standard. A number of studies have compared patient self-reported health problems with diagnoses recorded within medical claims and showed modest concordance [30], [43]. However, patient self-report likely underestimates the accuracy of diagnostic codes in claims data because patients are not necessarily aware of all diagnoses recorded by their physicians [44]. Direct comparisons between chart documented diagnoses by physicians and diagnostic data recorded in medical services claims is generally associated with higher degrees of concordance [45], [46], but investigation has been limited to a select number of conditions [47], or a small number of patients and physicians [46]. Furthermore, there has been no direct chart-based validation of medical services claims diagnoses for commonly used methods of comorbidity assessment or case-mix adjustment [35]. To increase the potential utility of medical services claims for research, priority needs to be placed on the validation of frequently used measures of diagnosis-based risk adjustment, as these measures are essential for unbiased comparisons in observational studies.

The Charlson index is one of the most frequently used comorbidity measures [35]. The index is based on diagnoses of 18 disease conditions, and scores are weighted by the relative risk of mortality [48]. Although developed and validated as a hospital-based measure of mortality risk [48], it is increasingly used for ambulatory populations [33]. A recent study, which compared Charlson index values based on hospital relative to medical services claims diagnostic data suggested that medical services diagnostic data yielded higher estimates of mortality risk for some conditions, while risk estimates were equivalent or lower for others [32]. The possibility that the discrepancy in estimated mortality risks is related to coding inaccuracies in medical claims diagnoses has not been assessed.

Case-mix adjustment is essential for unbiased comparisons in quality of care and outcome studies [37], [39], [49]. The Johns Hopkins adjusted clinical groups systems is an approach that is used to predict health care utilization and costs based on groupings of medical service claims diagnoses into 32 homogeneous classes and subclasses [50]. Although the Johns Hopkins system has been shown to predict up to 30% of future physician use and costs, variation in predictive capacity has been shown between studies and populations [37], [38]. To date, there has been no direct chart-based validation of the accuracy of diagnostic information used to classify patients into ambulatory diagnostic groups, and this assessment is particularly timely given the interest in using diagnosis-based case-mix adjustment to formulate equitable reimbursement policies for physicians paid on a per capita basis [24], [39], [51].

Finally, the demand for better methods of drug utilization review has highlighted the need to validate diagnostic information in medical services claims data so that it could be used in the surveillance of current practices [52]. Many potentially preventable admissions for drug-related illness are related to drug-disease contraindications [53], yet in ambulatory settings there is no systematically collected source of validated disease information that could be used for drug and disease utilization review. Medical services claims diagnostic codes provide a potentially viable option by which drug-disease utilization can be conducted.

In this study, data collected for a cohort of elderly patients enrolled in a clinical trial were used to assess the validity of medical services claims diagnoses in relationship to diagnoses recorded in the medical chart. These data were used to determine the sensitivity and specificity of medical services claims diagnoses for surveillance of 14 drug disease contraindications used in drug utilization review, the Charlson comorbidity index [48], and the Johns Hopkins Adjusted Ambulatory Care Group (ACG) Case-Mix profile [54].

Section snippets

Context

The validity of medical service claims diagnostic data was assessed in Quebec, where a universal health insurance program covers the costs of medical and hospital care for all provincial residents. Similar to other Canadian provinces [55], a provincial health insurance agency administers the universal health plan, which includes the registration of provincial beneficiaries and payment of physicians who provide services to Quebec beneficiaries. Services provided outside of the province or

Results

In the year prior to the start of the MOXXI study, 631,488 fee-for-service claims were retrieved for the 14,980 patients, of which 163,129 (26%) were claims submitted by the MOXXI primary care physicians. Overall 70% of the claims contained valid operational ICD-9 codes; both for all physicians and the subset of claims submitted by MOXXI physicians (Table 2). The most frequent invalid code was V999, a default code that is commonly inserted in computerized billing software when no diagnostic

Discussion

Data within medical services claims files represent a potentially rich resource for health service and epidemiologic research. However, the assembly of populations for study, and the adjustment for differences in case-mix between comparison groups, depends to a great extent on the validity of diagnostic information in these administrative databases. The relative paucity of scientific investigation concerning the validity of diagnostic data within medical services claims data is, in part,

Acknowledgements

This study was funded by the Medical Research Council of Canada, Fonds de la recherche en santé du Québec. We are extremely thankful to the medical staff at the Royal Victoria Hospital for their helpful assistance in identifying procedure codes within the RAMQ database, Dr. Michael Edwardes for his helpful comments and advice, and of course, to the MOXXI research team, without whom this research would not have been possible.

References (72)

  • N.P Roos et al.

    Using administrative data to predict important health outcomes: entry to hospital, nursing home, and death

    Med Care

    (1988)
  • S.G Leader et al.

    Myocardial infarction in newly diagnosed hypertensive Medicaid patients free of coronary heart disease and treated with calcium channel blockers

    Am J Med

    (1997)
  • J.B Mitchell et al.

    Using physician claims to identify postoperative complications of carotid endarterectomy

    Health Serv Res

    (1996)
  • W.W Young et al.

    Dissemination of clinical results. Mastectomy versus lumpectomy and radiation therapy

    Med Care

    (1996)
  • L.L White et al.

    Antimicrobials prescribed for otitis media in a pediatric Medicaid population

    Am J Health Syst Pharm

    (1995)
  • J Weiner et al.

    Ambulatory care practice variation within a medicaid program

    Health Serv Res

    (1996)
  • Herz EJ, Chawla AJ, Sredl K. Changes in the receipt of age-appropriate immunizations among young children under...
  • M.L Rosenbach et al.

    Use of electroconvulsive therapy in the Medicare population between 1987 and 1992

    Psychiatr Serv

    (1997)
  • W.P Welch et al.

    A detailed comparison of physician services for the elderly in the United States and Canada

    JAMA

    (1996)
  • D.G Ives et al.

    Mammography and pap smear use by older rural women

    Public Health Rep

    (1996)
  • P Grootendorst

    Health care policy evaluation using longitudinal insurance claims data: an application of the panel ToBit Estimator

    Health Econ

    (1997)
  • N.I Gavin et al.

    The use of EPSDT and other health care services by children enrolled in Medicaid: the impact of OBRA'89

    Milbank Q

    (1998)
  • G Maskarinec

    Diabetes in Hawaii: estimating prevalence from insurance claims data

    Am J Public Health

    (1997)
  • D.C Lestina et al.

    Creating injury episodes using medical claims data

    J Trauma

    (1998)
  • M.S Dighe et al.

    Economic evaluation of Medi-Cal epilepsy patients on monotherapy or polytherapy in 1994–95

    AHSR.FHSR Annu Meet Abstr Book

    (1996)
  • J Xuan et al.

    The economic burden of congestive heart failure in a managed care population

    Am J Managed Care

    (2000)
  • J.P Weiner et al.

    Risk-adjusted Medicare capitation rates using ambulatory and inpatient diagnoses

    Health Care Financ Rev

    (1996)
  • D.W Garnick et al.

    Focus on quality: profiling physicians' practice patterns

    J Ambul Care Manage

    (1994)
  • S.T Parente et al.

    Developing a quality improvement database using health insurance data: a guided tour with application to Medicare's National Claims History file

    Am J Med Qual

    (1995)
  • J.P Weiner et al.

    Variation in office-based quality: a claims-based profile of care provided to Medicare patients with diabetes

    JAMA

    (1995)
  • S.T Parente et al.

    Profiling resource use by primary-care practices: managed medical implications

    Health Care Financ Rev

    (1996)
  • R.J Ozminkowski et al.

    Profiling primary care physicians for a new managed care network

    Health Serv Manage Res

    (1997)
  • N.R Powe et al.

    Systemwide provider performance in a Medicaid program. Profiling the care of patients with chronic illnesses

    Med Care

    (1996)
  • B Starfield et al.

    Costs vs quality in different types of primary care settings

    JAMA

    (1994)
  • N Dhanani et al.

    The effect of physician specialty on resource use and outcomes

    AHSR.FHSR Annu Meet Abstr Book

    (1996)
  • W Chang et al.

    Apply case mix adjustment in physician performance profiling using large administrative database

    AHSR. FHSR Annu Meet Abstr Book

    (1996)
  • Cited by (375)

    • Sex-specific medication trajectories in older adults newly diagnosed with diabetes

      2023, Exploratory Research in Clinical and Social Pharmacy
    • Effectiveness and Safety of Direct Oral Anticoagulants Among Octogenarians with Venous Thromboembolism: An International Multidatabase Cohort Study

      2023, American Journal of Medicine
      Citation Excerpt :

      In Québec, medical services are covered for all residents, but the Public Prescription Drug Insurance Plan covers all residents aged ≥65 years, those without private drug insurance plans, and recipients of financial assistance.15 The quality of RAMQ data is high.16,17 The MED-ÉCHO includes records of all hospitalizations in Québec with date of admission and discharge, inpatient diagnoses (coded using ICD-10-CA) and procedures.

    View all citing articles on Scopus
    View full text