ADMINISTRATIVE DATA BASED PATIENT SAFETY RESEARCH

We conducted an extensive literature review aimed at identifying all empirical research in patient safety or medical errors that used administrative data. Our review started with the IOM report,¹ the review performed by the University of California at San Francisco-Stanford Evidence-Based Practice Center under contract with AHRQ,⁸ Iezzoni’s review of administrative data based research on quality of care,⁹ and our previous research.^10,11 We then carried out a systematic search of Medline and AHRQ grant databases from 1966 to 2002 using the following search algorithm ((patient safety OR medical error* OR medical-errors* OR adverse event* OR complication* OR iatrogenic* OR nosocomial) AND (administrative data* OR insurance-claim-review* OR claims data or ICD-9-CM)).

Use of administrative data in quality and safety assessment

Administrative data, also called claims data, are by-products of administering and reimbursing healthcare services. Government payers (such as Medicare, Medicaid and Veterans Affairs) and private insurance companies regularly maintain a large amount of administrative data concentrated primarily on acute hospital admissions and, increasingly, on outpatient care, nursing homes, home care, and hospice programs. The core data elements of an administrative data system are admission date, discharge date and status, primary and varying numbers of secondary International Classification of Diseases, 9th Revision, Clinical Modification (ICD-9-CM) diagnoses, procedures, and external causes of injury, and some demographic variables. These data are often available as compiled research databases from federal agencies, state health departments, health plans, and private data institutions. For example, AHRQ Healthcare Cost and Utilization Project (HCUP), a partnership of federal government and over 30 participating states, compiles uniform hospital discharge records for research purposes (see http://www.ahrq.gov/data/hcup/ for more details).

Overall, there has been limited use of administrative data in quality and safety research. Roos and Brazauskas in 1990¹² proposed screening claims data for adverse events and to guide subsequent medical record reviews to determine whether a quality problem existed. Leatherman et al in 1991¹³ described a quality screening and management program developed at the United Healthcare Corporation that used claims data to explore incidence rates, adverse events, and other outcomes measures. Riley et al in 1993¹⁴ used ICD-9-CM codes in Medicare claims to identify readmission to hospital for adverse events following selected procedures.

The work of Iezzoni and colleagues on CSP in the early 1990s,^5,15,16 supported by AHRQ, was the first systematic exploration of the value of administrative data in quality and patient safety research. The CSP relied on ICD-9-CM codes to identify 27 potentially preventable in-hospital complications such as postoperative pneumonia, hemorrhage, medication incidents, and wound infection. Iezzoni et al¹⁶ found that patients with complications were significantly older, more likely to have comorbid conditions, more likely to die, and were higher in charges and lengths of stay than other patients. They also found that hospital complication rates generally were correlated across clinical areas, but not correlated with hospital mortality rates.¹⁵ Higher relative rates of complications were associated with larger hospitals, availability of major teaching facilities, and provision of open heart surgery, as well as with coding more diagnoses per case.¹⁵ Such findings, along with findings from other studies,^5,17–19 suggested that the CSP had certain clinical validity for research use, but also cast some doubts on the usefulness of CSP as a tool for provider level quality assessment.

In the mid 1990s AHRQ developed a set of administrative data based quality indicators as a companion tool for HCUP, named HCUP QIs.⁶ The 33 original QIs included several measures of avoidable adverse events and complications. Given the substantial nationwide interest in quality of care and a lack of quality assessment tools, the QIs were soon found in empirical studies. For example, Needleman and colleagues²⁰ at Harvard University and Kovner and colleagues²¹ at AHRQ used some of these QIs to assess the association between inadequate nurse staffing and high rates of complications in hospitalized patients.

The IOM’s call to develop medical error reporting systems three years ago¹ prompted renewed interest and vigor in developing a tool specifically designed for patient safety research that takes advantage of the large volume of existing claims data. Researchers at AHRQ¹⁰ compiled a list of potential administrative data measures from CSP, HCUP QI and other published works, and from hand searching ICD-9-CM codes for complications, adverse events, medical negligence, and iatrogenic conditions. For each potential indicator, ICD-9-CM code inclusion and exclusion criteria were created to identify appropriate risk pools and to minimize ambiguity as to whether an event was a true error or an unpreventable complication. This list of potential codes was grouped into 13 measures based on clinical cohesiveness of the codes. Analyses using this algorithm revealed significant safety incidences and associated adverse patient outcomes in both hospitalized adults¹⁰ and children.¹¹

Realizing the potential value of administrative data based measures to screen for patient safety events, AHRQ contracted with the Evidence-Based Practice Center (EPC) at the University of California San Francisco (UCSF) and Stanford University to further expand, test, and refine these measures as well as to improve the evidence behind their use with extensive literature reviews and broad clinical consensus panels. The final product of this joint effort is the AHRQ patient safety indicators (PSIs).

AHRQ PATIENT SAFETY INDICATORS (PSIS)

The UCSF-Stanford team developed AHRQ PSIs by a five step process^8,22:

• they reviewed the literature to develop a list of candidate indicators in addition to the initial PSIs developed at AHRQ and collected information about their performance;

• they formed several panels of clinician experts to solicit their judgment of clinical sensibility and suggest revisions to the candidate indicators;

• they consulted ICD-9-CM coding experts to ensure that the definition of each indicator reflected the intended clinical situation;

• they conducted empirical analysis of the promising indicators using HCUP data; and

• they produced the software and documentation for public release at AHRQ.

The PSIs include 20 indicators with reasonable face and construct validity, specificity, and potential for fostering quality improvement. Seven of the PSIs are recommended to be area based PSIs to capture complications/adverse events occurring in an area as opposed to within an institution. The PSI software calculates raw rates, risk adjusted rates derived by applying the average case mix of a baseline file that reflects a large proportion of the US hospitalized population in patient age, sex, diagnosis related groups (DRGs) and co-morbidities, and smoothed rates that dampen random fluctuations over time.⁷ Thirty co-morbidity categories²³ are automatically generated by the software and used as risk adjusters together with variables available in most administrative data systems. Table 1 describes the definitions of the numerators, denominators, and key exclusions for the 20 PSIs, and table 2 provides the findings from applying the PSI to the 7.45 million discharges in the HCUP Nationwide Inpatient Sample for the year 2000. Note that each PSI has a unique risk pool determined by its denominator definition and exclusion criteria. Table 3 presents unadjusted length of stay, charges, and in-hospital mortality for patients with and without PSI events. Tables 2 and 3 show substantial numbers of patient safety events with tangible impacts on patient outcomes in terms of increased length of stay, increased likelihood of in-hospital death, and increased charges for patients experiencing a PSI event as opposed to those that do not. Taken together, these tables clearly point to a significant potential role for administrative data in patient safety efforts.

CHALLENGES OF ADMINISTRATIVE DATA BASED PATIENT SAFETY RESEARCH

Any discussion of patient safety research using administrative data should recognize some data limitations and understand how such limitations play in their analysis. In particular, we focus on how these issues relate to the PSI and/or are addressed by the PSI.

Problems with ICD-9-CM coding

There are many concerns over ICD-9-CM coding with regard to patient safety research. First, we can only find events for which there are corresponding ICD-9-CM codes. A small number of standard codes and E codes appear to identify medical errors. For example, ICD-9-CM codes 9984, 9987, and E8710-E8719 can be used to record a foreign body left after a procedure. A coder should in theory code both the standard ICD-9-CM code and the E code. These codes, including E codes that are specifically designed to record injuries, in no way capture any significant percentage of the entire universe of medical errors that can occur. Secondly, there may be a substantial amount of coding errors due to misunderstanding of codes, or errors by physicians and coders, or miscommunications between them. An IOM study in 1977 found that agreement on the principal diagnosis between hospital reports and IOM reabstraction was only 65.2%.²⁴ Thirdly, coding is very likely to be incomplete because of limited slots for coding secondary diagnoses and other reasons. Fourthly, assignment of ICD-9-CM codes is variable because of the absence of precise clinical definitions and context. Iezzoni and colleagues⁹ found that the mean number of diagnoses coded in 441 California hospitals ranged from 2.5 to 11.7, and this variation explained part of the differences between high and low mortality hospitals. Some of the variation may be driven by financial reasons, such as in “DRG creep” where hospitals choose codes with higher Medicare pay schedules.^25–27 Finally, diagnoses are not dated in current administrative data systems, making it difficult to determine whether a secondary diagnosis occurred before admission (a co-morbid disease) or during the stay in hospital (a complication or medical error).^9,18,28 Overall, these limitations were not amenable to be proactively addressed in developing the PSIs.

Inadequate reliability and validity in identifying medical errors

Administrative data have been shown to have low sensitivity but fair specificity in identifying quality gaps. Bates and colleagues²⁹ found that, while medical record review results in many false positives, administrative data were able to identify only half of patients with adverse events but had a fair specificity of 74%. Iezzoni and colleagues conducted several validity studies on CSP. One study³⁰ reported that physician reviewers confirmed CSP flagged complications in 68.4% of surgical and 27.2% of medical cases. Another study¹⁸ found that 89% of surgical cases and 84% of medical cases had their CSP trigger codes corroborated by review of the medical records. A third one¹⁹ indicates that objective clinical criteria or physicians’ notes supported the coded diagnosis in 70% to over 90% of most CSP flagged conditions. Focusing on specific adverse events for a specific patient population, as is built into the PSIs, improves specificity appreciably. Romano et al³¹ showed that specificity for postoperative complications after diskectomy can be as high as 98%. No attempts have been made to identify the validity and reliability of AHRQ PSIs.

Lack of clinical details for risk analysis and risk adjustment

Lack of clinical details is a major limitation of claims data.³² Of special concern is severity of illness that affects patient outcomes and conceivably affects the likelihood of medical errors. Analysis of outcomes and risk factors associated with medical errors are limited to variables available from administrative data.

AHRQ PSIs and other similar tools usually identify a relatively homogeneous risk pool for each PSI which not only reduces misclassification but also alleviates variation in risk factors.^5,6 Coding co-morbidities using ICD-9-CM codes represents another major effort built into the PSI for risk adjustment.^23,33–36 Iezzoni³⁷ provided an excellent review of several claims based systems measuring severity. The performance of these systems depends substantially on complete coding of diagnoses.³⁸

Analytical issues

The large size of the administrative data and the relative rarity of safety events requires special consideration in statistical analysis. The sheer size of the administrative data can give the illusion of great precision and power.³⁹ Given the standard errors for cases with obstetric trauma without instrumentation and their risk pool (table 3), as an example, a difference of 0.014 days in length of stay in hospital between the two groups is statistically significant (p<0.05). Such differences are often of little clinical significance. Coupled with missing important confounding variables and difficulty in choosing correct statistical models that fit the data, clinically insignificant but statistically significant results could lead to biased inferences and erroneous conclusions.

Matched case-control analysis appears to be a method particularly applicable to administrative data based analysis where cases are rare and controls are plenty. Classen et al⁴⁰ and Bates et al⁴¹ matched cases of adverse drug reactions (ADR) with controls without ADR on DRG, co-morbidity, severity, and demographic characteristics to estimate the excess costs, mortality, and length of stay attributable to ADR. Jensen et al⁴² matched cases of hospital acquired Staphylococcus aureus infections in Denmark hospitals to patients with the same primary diagnosis at admission to identify risk factors among unmatched factors such as age, anemia, etc. Bates et al⁴³ matched patients with ADR with patients from the same hospital unit and similar pre-event length of stay to study risk factors for ADR. Matching retains only cases and controls with similar covariates. By matching cases and controls to the same hospitals, researchers could focus on patient level factors without concerns over hospital coding practices and hospital effects.

POTENTIAL APPLICATIONS OF ADMINISTRATIVE DATA IN PATIENT SAFETY RESEARCH

Patient safety indicators as a screening tool

First and foremost, PSIs are considered indicators, not definitive measures, of patient safety concerns.^10,22 As with the CSP,¹⁷ the intention of these indicators is to provide a useful screening tool to highlight areas in which quality should be investigated in greater depth. PSIs enable institutions to quickly and easily identify a manageable number of medical records for closer scrutiny. Using administrative data to screen cases for chart review has also been proposed by Roos and Brazauskas in 1990 and by Silber and colleagues more recently.^12,44 For example, in cases with a foreign body left in after surgery (table 2), 7.45 million medical records have to be extracted to uncover 536 cases. Screening the claims with PSIs would quickly identify such rare events and associated medical records could be abstracted for in-depth analysis. This approach has great potential in enhancing the design of medical record based patient safety research, but its use has yet to be widely adopted.

Epidemiological study in patient safety

Administrative data are valuable in epidemiological studies of the incidence and consequences and factors associated with medical injuries. Our earlier studies^10,11 and those of Romano et al²² revealed substantial incidence rates and provided some insights into the outcomes and risk factors associated with medical errors. Our ongoing analysis of the 2000 data suggests that medical errors identified in table 2, excluding death in low mortality DRGs and failure to rescue where patients with errors all died during hospitalization, account for a total of 2.4 million extra days in hospital, $9263 million extra charges, and 32 591 attributable deaths in the US per year. It is also possible to identify certain risk factors such as nurse staffing.^20,21

Public reporting

At a time when no reliable reporting system exists, applying PSIs to administrative data could reveal the overall incidences and trends and provide useful benchmarks at the local, state, regional, and national levels for tracking progress. However, such use must be made with care. Coding differences across institutions,⁹ lack of robust risk adjustment,^37,45 relative rarity of safety events, and many other reasons make it uncertain that differences between PSI rates reflect true differences in quality.^46,31 Because of these limitations, public reporting of PSI rates for institutions and regions may raise contentions over technicalities rather than facilitate quality improvement. Developers of PSIs and similar administrative data based systems in general express caution with regard to the use of the indicators for public reporting at an institutional level.^10,22

CONCLUSION

We have highlighted generic and specific concerns regarding administrative data and their use, in particular, for patient safety research. Despite the known limitations, the lack of tools for patient safety today makes administrative data based tools like the AHRQ PSIs appealing. AHRQ PSIs could be useful to identify potential patient safety problems that merit further investigation. With proper methodology, administrative data can provide valuable insights into the incidences, adverse impacts, and risks of medical errors. In addition, PSI rates could serve as useful monitors at local, state and national levels and as benchmarks for tracking progress in patient safety. Further research will be needed to establish whether, and under what circumstances, these indicators are valid measures of safety related hospital performance for comparative purposes. Most paramount in this effort is work at explicitly examining cases flagged by PSIs using chart review. Some of this work is already ongoing.

Future growth in electronic health data will make tools like the PSI more useful. Ongoing refinement of ICD-9-CM and, eventually, ICD-10-CM should introduce more data elements and may allow clearer distinction of complications from conditions present at admission and increase the specificity of codes. Iezzoni¹⁷ predicted that the definition, content, and scope of administrative data would change dramatically in the near future, and inclusion of clinical information from both clinicians and patients in administrative data would make exciting new possibilities.

In 2001 the Department of Health and Human Services (HHS) of the AHRQ launched a $50 million initiative aimed at improving patient safety, focusing primarily on medical error data or reporting systems.⁴⁷ The fruition of this investment will hopefully evolve into real time, user friendly, nationwide error reporting systems. It is conceivable that any eventual reporting system would involve triangulation between current administrative data, chart review, and self-reports in order to maximize the amount of information available with respect to medical errors. At present, the value of administrative data—with its large scale, uniformity, and regularity—should be fully harvested in our campaign against medical errors.