Objective: Electronic medical records seldom integrate performance indicators into daily operations. Assessing quality indicators traditionally requires resource intensive chart reviews of small samples. We sought to use an electronic medical record to assess use of β-adrenergic antagonist medications (β-blockers) following myocardial infarction, to compare a standardized manual assessment with assessment using electronic medical records, and to discuss potential for future integration of performance indicators into electronic records.
Design: Cross-sectional data analysis.
Setting: An urban academic medical center.
Participants: US Medicare beneficiaries 65 years of age or older, admitted to hospital with myocardial infarction between 1995 and 1999.
Measurements and main results: Manual chart review was compared with a computer driven assessment of electronic records. Administration of β-blockers and cases excluded from use of β-blockers were measured, based on Medicare criteria. Among 4490 older adults, 391 (4%) of 9018 hospital admissions contained codes for myocardial infarction. In 323 (83%) of the 391 hospital admissions, criteria for excluding β-blockers were met; 235 (60%) were excluded due to heart failure. Of 68 hospital admissions for myocardial infarction that did not meet exclusion criteria, physicians prescribed β-blockers in 49 (72%) on admission and 42 (62%) at discharge. Compared with manual chart review, electronic review had a sensitivity of 83–100% and led to fewer false negative findings.
Conclusions: An electronic medical records system can be used instead of chart review to measure use of β-blockers after myocardial infarction. This should lead to integration of real time automated performance measurement into electronic medical records.
Statistics from Altmetric.com
- AMI, acute myocardial infarction
- CHF, chronic heart failure
- COPD, chronic obstructive pulmonary disease
- EMR, electronic medical record
- QIO, Quality Improvement Organizations
- electronic medical records
- acute myocardial infarction
- performance measures
- AMI, acute myocardial infarction
- CHF, chronic heart failure
- COPD, chronic obstructive pulmonary disease
- EMR, electronic medical record
- QIO, Quality Improvement Organizations
- electronic medical records
- acute myocardial infarction
- performance measures
In many countries objective assessments of quality of medical care show deficiencies in numerous areas such as preventive care and care of cardiovascular diseases, diabetes mellitus, and mental illness. Acute myocardial infarction (AMI) has received special attention, especially due to available interventions that prolong survival.1,2 In a comparison of data from Australia, Canada, New Zealand, England, and the United States, Hussey and colleagues reported that 30 day in-hospital case fatality rates for AMI are highest in Canada where they exceed the rate in Australia by about 50%, regardless of age.3 In the US, major organizations involved with accreditation4–6 and national quality assurance7 include among important measures of quality of health care the use of β-adrenergic receptor antagonists (β-blockers) in treating AMI. A recently published report from South Wales, UK showed that only 58% of patients admitted for AMI were discharged on β-blockers.8 Although a fraction of the other 42% met exclusion criteria, some exclusion criteria are questionable and have been debated. In any case, suboptimal quality of care9 poses challenges for all health systems.
To develop programs and policies to improve health care, medical institutions and clinicians need timely and accurate data about the quality of care that they are providing. Much of the available data comes in the form of reports generated from research rather than from routines in daily practice. In many countries assessment of performance occurs by manual chart review. In the US, for example, government contracted Quality Improvement Organizations (QIOs) help to assess and promote quality for many patients such as nearly all those over 65 years of age who qualify for Medicare national health insurance (box 1). The QIOs assess quality indicators by requesting a sample of individual paper charts from health institutions and manually reviewing each chart in the sample. In a recent 3 year period about 8000 charts per state have been reviewed in a national assessment of two samples (4000 charts per state per sample), and many QIOs perform additional reviews to increase sample size for feedback to hospitals. Although the QIO reimburses hospitals for copying medical records for these audits, a hospital’s expenditure for copying often exceeds the reimbursement provided. The total cost of copying and reviewing 8000 charts manually is estimated at more than US$10 million. With such a burden, the degree to which an individual chart can be evaluated is limited10 and national assessments of state level indicators may be possible only every few years.11 Two thirds of US hospitals paid by the two largest government insurance programs—Medicare and Medicaid—do not report any quality measures.12 As a result, applying performance data at the point of care—a major goal of efforts at improving quality—is often impossible.
Box 1 US Medicare
Federal health insurance program created under the Social Security Act of 1965, originally to ensure access to health care for retired older Americans.
Covers nearly all older Americans and now also certain younger people with disabilities or end-stage renal disease.
Covers hospital care, home health care, hospice care, and up to 100 days in a skilled nursing facility without additional premiums; supplemental coverage for outpatient care, physicians’ services, and durable medical equipment can be purchased separately.
Program defines several measures of quality, including prescription of β-blockers following acute myocardial infarction (AMI).
Contracts with Quality Improvement Organizations for aspects of administration, as well as assessment and improvement of quality of care.
Uses manual chart reviews of national samples to assess quality.
The need for better management of health information9,13 prompts new methods of collecting and using data about performance. Computerization of medical records offers many advantages not only for care of individual patients but for care of populations. For example, interactive electronic medical records (EMR) can prompt clinicians automatically with reminders about care and flags of potentially adverse outcomes. These forms of active decision support have also been proved to change physicians’ practices and improve health outcomes.14–22 Leaders of medical organizations who receive information from these systems can then also take proactive approaches in the timely development of programs to improve quality of care,23 including providing financial or other incentives based directly on performance.24–26 The government of the UK has already implemented a national plan to reward performance27 and, in the US, the Medicare Prescription Drug, Improvement and Modernization Act of 2003 increases payments to hospitals that publicly report data about quality.
The problems associated with assessing quality of health care on a large scale beg for more affordable and smarter uses of EMRs. Institutions with computer based approaches to quality improvement still provide a minority of health care. Finding inexpensive, efficient, and accurate ways for individual medical institutions to assess their own quality indicators may lower governmental and institutional costs and lead to more local programs that can directly improve quality of care.28 The US Secretary of the Department of Health and Human Services has recently commissioned the design of a freely distributable standardized model of an EMR.29 In the meantime, vendors that create and sell commercial EMRs compete in a narrow marketplace by providing divergent, proprietary products. Many vendors would welcome but lack standardized approaches to automating performance measures, while a clear solution to the problem requires direct integration of performance measures into these products. We sought to use an EMR to assess the use of β-adrenergic antagonist medications (β-blockers) following myocardial infarction, to compare a standardized manual assessment with assessment using EMR, and discuss the potential for future integration of performance indicators into electronic records.
Electronic medical records (EMRs)
The main analysis was conducted using EMRs of the Regenstrief Medical Records System (RMRS).30–32 RMRS registers and stores data for all patients with inpatient or outpatient encounters in Wishard Health Services, an urban public medical institution of Marion County, Indiana. It captures inpatient and outpatient registrations, scheduling, clinicians’ orders, diagnoses, notes, documentation of allergies, laboratory results, electrocardiograms, radiology reports and images, and charges. Like many pharmacy systems nationwide, information about all medications ordered or dispensed through the pharmacies is recorded electronically. Medications prescribed but not administered at discharge are also recorded in this system. Adjacent to approximately half of the hospital’s beds are machines that automatically capture blood pressure and pulse.
Defining cases of AMI and a subgroup eligible to receive β-blockers
After a patient’s discharge, hospital personnel identify major events, procedures, and diagnoses, along with appropriate diagnostic codes from the International Classification of Diseases (ICD), 9th Revision, Clinical Modification,33,34 which is universal to all hospitals. In this study only cases of AMI were reviewed. To identify patients discharged with AMI we screened the ICD codes of all Medicare covered hospital admissions between January 1995 and December 1999. AMI was defined to occur if a primary or secondary discharge ICD code of 410.x (AMI), 410.x0 (episode unspecified), or 410.x1 (initial episode) was identified. Excluded were hospital admissions for patients under 65 years of age and those with a discharge ICD code of 410.x2 (subsequent episode) or 411.0 (post-infarction syndrome).
The Centers for Medicare & Medicaid Services (CMS) defines two measures that reflect the use of β-blockers in treating Medicare beneficiaries admitted to hospital with AMI: administration within 24 hours of arrival at the hospital and prescription at discharge. Patients eligible to receive β-blockers—that is, those in the denominator—are defined as Medicare beneficiaries of any age admitted to an acute care hospital with AMI (discharge ICD code 410.xx) and discharged alive. Excluded from the denominator are patients who have received initial treatment for AMI and are readmitted for further observation, evaluation, or additional treatment within 8 weeks (ICD code 410.x2), and patients with any of the following which the Medicare program has considered a contraindication to β-blockers: allergy to β-blocker, chronic heart failure (CHF), shock, chronic obstructive pulmonary disease (COPD), asthma, or peripheral vascular disease (ICD codes 440.21, 440.22, 440.23, or 440.24). According to CMS criteria, a pulse of less than 60 on admission, when the patient was not on a β-blocker prior to admission, excludes patients from the admission indicator; a last pulse of less than 50 excludes patients from the discharge indicator. Similarly, systolic blood pressure of less than 100 mm Hg on arrival excludes patients from the admission indicator and last systolic blood pressure in the same range excludes patients from the discharge indicator. Also excluded from the denominator are patients enrolled in a Medicare+Choice plan and patients transferred from another acute care hospital or emergency department. The numerator includes patients in the denominator who are prescribed a β-blocker within 24 hours of arrival at the hospital (for the admission indicator) or upon discharge (for the discharge indicator).
Using the EMR to assess performance
We sought a process of using electronic data that would match the QIO’s manual process as closely as possible. To assess use of β-blockers and to match the QIO’s inclusion and exclusion criteria, we used electronic data pertaining to vital signs, discharge ICD codes, laboratory test results, administration of drugs in the hospital, and discharge orders. Because an exact time of admission is not recorded in the data system, we used the dispensation of β-blockers on the day of admission or the following day as a proxy for dispensation within 24 hours of the admitting time.
We extracted the relevant factors from the data system, including information about demographic data, drugs, diagnoses, and vital signs for the period of interest. We searched the pharmacy data files for occurrences of all β-blockers (acebutolol, atenolol, bisoprolol, carvedilol, esmolol, labetalol, metoprolol, nadolol, penbutolol, pindolol, propranolol, sotalol, and timolol) administered non-topically. We included all available prescription drug data pertaining to eligible participants, separating inpatient from outpatient data according to whether dates of drug administration were included in periods of hospitalization.
Manual chart review by the QIO
To provide a measure of comparison against a known “gold standard”, we manually reviewed a sample of paper records. We sought to simulate a usual manual review by the QIO. To review paper charts and assess the use of β-blockers after AMI, the QIO typically requests a sample of charts from a hospital in their region. The hospital is responsible for copying and delivering the charts to the QIO, as described above. At Wishard Memorial Hospital, providing a complete record of the encounter on paper for the QIO requires special printing of some electronic data, such as laboratory test results, which are not routinely stored in paper form. QIO staff then review seven indicators of quality of care including use of β-blockers in AMI.
We used a computer generated random number sequence to identify, for manual review, a random sample of 170 hospitalizations (just over 40% of all cases) for AMI. The QIO requested the charts from the hospital whose staff copied the charts and delivered them to the QIO. Upon receipt of records, two QIO staff members proceeded to review the charts for use of β-blockers and for any contraindications, according to the schematic shown in fig 1. As shown, duplicate reviews of many charts were included to assess reliability between the two reviewers. Although most charts were reviewed only for use of β-blockers, a subsample was also reviewed for the seven standard quality indicators in case reviewing more indicators led to different results for β-blockers than when only the β-blockers were assessed. The number of charts selected for review allowed us to detect meaningful differences between reviewers and types of review. One hundred and forty of the charts underwent primary review for the two indicators pertaining to β-blockers.
Analysis of participants and comparison of manual and electronic methods
The basic characteristics of patients hospitalized for AMI were reported. To assess whether prescription was associated with patients’ characteristics, we compared cases in which β-blockers were prescribed with those in which they were not prescribed (regardless of eligibility), with both hospitalization and participant as the unit of analysis. χ2 or t tests were used to compare participants’ characteristics according to whether β-blockers were prescribed. To compare manual and electronic methods among the subgroup that underwent manual review we created a series of 2×2 tables indicating whether the methods showed that β-blockers were prescribed. Using manual review as a standard, we assessed the sensitivity and specificity of the electronic method. A kappa statistic was calculated for each comparison.
The university’s institutional review board approved the study.
Cases of AMI and a subgroup eligible to receive β-blockers
We identified 15 610 Medicare covered hospital admissions during 1995–99 corresponding to 7251 patients. Among 4490 adults aged 65 years or more, 391 (4%) of their 9018 hospital admissions contained a primary or secondary discharge ICD code for AMI.
Electronic chart review
Table 1 shows the demographic characteristics of the participants, whether they had CHF or COPD, and whether they received β-blockers according to electronic review. Electronic review of all cases of AMI revealed that physicians prescribed β-blockers in 259 (66%) of the 391 hospital admissions at any time during the admission. Ignoring exclusion criteria, β-blockers were prescribed at admission in 218 (56%) of the 391 hospitalizations for AMI and at discharge in 176 (45%). Compared with those who did not receive β-blockers during their first admission for AMI, those who did were significantly less likely (p<0.05) to die in hospital or have COPD.
Figure 2 shows the reasons for exclusions noted in 323 (83%) of the 391 cases of AMI. Of the 68 (19+49) hospital admissions with no Medicare exclusion criteria, physicians prescribed β-blockers in 49 (72%; 13% higher than the national median) on admission and 42 (62%) at discharge (data not shown). CHF accounted for exclusion of 235 (60%) of the cases. According to Medicare criteria, participants were not eligible to receive β-blockers in 169 (218 – 49; 78%) of the hospital admissions in which β-blockers were given on admission anyway.
Manual chart review
Of 140 charts that underwent primary analysis in comparing manual with electronic review, 27 were not reviewed. Of these, 25 (18% of all 140 cases) had an unconfirmed diagnosis of AMI on manual review; the other two had died, were discharged, or transferred on the date of admission. Of the 113 cases remaining, one was not reviewed for the admission indicator and 25 were not reviewed for the discharge indicator because of death (N = 22) or transfer to another acute care facility. Of the 113 cases reviewed, 78 (70%) of 112 (18+94) met the exclusion criteria for the admission indicator and 52 (59%) of 88 (21+67) were excluded from the discharge indicator. Of cases eligible for the indicator, a β-blocker was given to 28 (82%) of 34 (2+32) on admission and to 32 (89%) of 36 (3+33) at discharge. The two QIO reviewers always agreed independently about whether β-blockers should be administered or met the exclusion criteria.
Manual versus electronic review
Cases reviewed using manual and electronic methods are compared in table 2 using manual review as the standard for comparison. The sensitivity of electronic review ranged from 83% to 100% and the specificity from 17% to 75%. When electronic review indicated that an exclusion criterion was present, manual review often did not (19 cases on admission; 24 at discharge).
Re-review of discrepancies between methods
Cases in which the QIO review indicated exclusion but RMRS indicated none were reviewed individually. For exclusion on admission there were 22 disagreements between QIO and RMRS. The reasons given by the QIO for exclusion of three cases were bifascicular block on electrocardiogram, history of COPD, and shock on arrival. Of the remaining 19 disagreements, diagnosis of CHF was a cause of disagreement in 10. In two cases, re-review acknowledged an initial QIO error due to overlooking a diagnosis of CHF, asthma, or shock. For exclusion from the discharge indicator there were 33 disagreements between QIO and RMRS. The reasons given by the QIO for exclusion of nine cases were hypotension or shock, bifascicular block on electrocardiogram, history of COPD, or CHF. Of the remaining 24 disagreements, CHF was a cause of disagreement in 18. In eight cases, re-review acknowledged an initial QIO error due to erroneous assessment of pulse rate or diagnosis of CHF.
Validity of ICD code for CHF
Of the 391 hospital admissions for AMI, an ICD code for CHF was present in the index admission in 209 (53%) and a code for CHF was present any time during the study period in an additional 26 (7%). Because this large number of hospital admissions met Medicare’s exclusion criteria for administration of β-blockers, we assessed validity of the ICD code for CHF by comparing ICD codes with echocardiography results documented during hospital admission for AMI (table 3). Of 146 hospital admissions for AMI in which echocardiography was performed and an ICD code for CHF was present, 111 (15+96; 76%) had an echocardiogram consistent with systolic heart failure, defined as a cardiologist’s recording of left ventricular systolic dysfunction or fractional shortening of 0.2 or less. Fractional shortening is a measure made from a two-dimensional echocardiogram and represents the percentage of shortening of the diameter of the left ventricle during the cardiac cycle. The fractional shortening is calculated as the difference between left ventricular end diastolic and end systolic dimensions, divided by the end diastolic dimension. It may be considered the two-dimensional analogue of the ejection fraction.
This study of AMI among older adults in an urban public hospital over 5 years reveals for the first time that assessment of use of β-blockers using an electronic medical record is comparable to manual review, has high sensitivity, and has advantages over manual methods. Electronic review also caught the prescriptions of several hospitalized patients that were not detected by manual review, raising the possibility of improved accuracy through the use of valid electronic systems. Since the institution’s EMR captures information about all medications ordered or dispensed through the pharmacies, a manual process for identifying routine administration of β-blockers could not be better than the electronic review. The manual method used here might even overestimate an average of similar methods used elsewhere, since paper charts in our hospital contain not only individual drug orders but also printed summaries which may facilitate manual chart review.
Despite the recognized therapeutic benefit of β-blockers, including decreased morbidity and mortality in patients with CHF35–37 and decreased mortality,1,2 costs,38 magnitude of infarction, and incidence of complications and morbidity in patients with AMI,39 these drugs are underused internationally.40–42 According to a study by Jencks et al11 in which a systematic random sample of 850 or fewer records per US state were reviewed, US Medicare beneficiaries with AMI received β-blockers within 24 hours in only about 64% of 1998–99 hospital admissions (judged by median state). Use of critical pathways in treating AMI has led to disappointing results,43 and even a multifaceted intervention incorporating guideline oriented tools had modest results.44 Our study results are consistent with the notion that, regardless of the criteria that Medicare has considered exclusive, overall death rates are lower when β-blockers are administered. In the US, prescribing has been especially low for female,45 indigent, or African-American patients46 and by non-internists and physicians in rural, southern, or western US regions.47,48 Krumholz, Bates, and colleagues have highlighted the need to use informatics and real-time decision support to improve quality of care in this area.49,50 Computers can be exploited more aggressively to improve the quality of health care in hospitals and clinics.
Integrating performance measurement into EMRs could reduce errors and limitations in reporting that occur through human review. We identified many instances in which reviewers made errors in reviewing charts, primarily by overlooking diagnoses or related inclusion or exclusion criteria. CHF, one of US Medicare’s exclusion criteria for administration of β-blockers, is a prime example, especially since half of the 101 hospital admissions without any diagnosis of CHF but with echocardiographic testing showed echocardiographic results consistent with CHF. It is worth pointing out that the converse situation—that is, the diagnosis of CHF without echocardiographic evidence of systolic dysfunction—might usually be explained by diastolic dysfunction not easily appreciated by echocardiography. In any case, the related errors occurring through manual review of these records may be caused by fatigue, inattention, difficult to read handwriting in charts, buried information, inconsistent construction of charts, or missing data. Electronic assessment of performance indicators may overcome difficulties with accuracy or inconsistency found among reviews by QIOs.51
A primary advantage of electronic review is more accurate, systematic, and efficient ability to identify chronic disease by easily including cumulative assessments of archived data. In the case of CHF as an exclusion criterion for prescription of β-blockers, Medicare defines CHF based only on the index hospital admission. This is sensible from the standpoint of a manual review. CHF, however, was often coded in an earlier hospital admission instead of the index admission. The ability of EMR to detect a prior diagnosis is evident in its low specificity, which is misleading since it found CHF in many cases where the QIO did not; in other words, most of these cases represent false negatives (undetected CHF) under manual review. Thus, the information based power of automated systems will itself narrow the gap between factors that are measurable and those that are targeted by medical evidence, leading to more accurate assessments of performance. Answering clinical questions in more than superficial ways, of course, requires rich data systems.
Rapid developments and changes in clinical guidelines demand easily modifiable information systems. Medicare guidelines indicating that patients with CHF should not receive β-blockers were created some 20 years ago and remain “valid” with respect to the performance indicator for β-blockers, despite many recent years of medical evidence to the contrary. Using Medicare guidelines, most participants in this study were ultimately excluded due to presence of CHF, regardless of whether electronic or manual review is performed. A review from all acute care hospitals in Alabama, Connecticut, Iowa, and Wisconsin also showed that 70% of patients with AMI in 1992–3 met exclusion criteria for administration of β-blockers.52 Many of these participants may have had only mild left ventricular systolic failure—when β-blockers may not pose a threat. Indeed, others have also shown that patients with AMI and relative contraindications to these drugs, including those with mild asthma or COPD,53 are still likely to benefit from β-blocker therapy.54–57 Automated assessments of performance measurement can more readily facilitate changes to clinical guidelines.
Real time performance improvement can benefit from timely and accurate feedback about previous performance.
Manual copying and auditing of paper charts is expensive and time consuming.
Electronic review is directly comparable to manual chart review in assessing prescription of β-blockers following acute myocardial infarction.
Electronic review can efficiently summarize medications, chronic diseases, and exclusion criteria for certain performance indicators, especially as documented from previous medical encounters.
ICD coding in AMI is especially prone to error, probably due to “rule out” cases. Iezzoni noted that AMI did not occur in up to 42% of hospital admissions with ICD codes that indicated AMI; almost a third of cases had suffered the infarction 5–8 weeks before admission.58,59 Others have found the false positive codes to occur in only 12% of cases.60 In our study, chart review suggested that AMI may not have occurred in 18% of hospital admissions with ICD codes indicating AMI. Although manual review was used as the standard with which electronic review was compared, this study makes clear that each method has unique advantages—although the electronic method appears to have the edge. When rich clinical electronic data—such as those reflecting bedside vital signs—are available, electronic performance assessment will become more accurate through use of robust diagnostic rules that can automatically cross check and verify diagnoses using multiple criteria. With an appropriately designed data system we expect that electronic review would serve as a desirable and valid proxy for manual review.
Although we examined all cases of AMI in the study period, the sample is small and manual chart reviews may yield different results elsewhere. Nevertheless, development of an automated system has produced transportable guidelines for data handling. We have assessed only a single performance indicator rather than a large series, to provide an example of how computers can be used to design automated features that can lead to timely feedback about population based performance.
In conclusion, EMRs may be used in place of expensive, time consuming manual reviews to assess the use of β-blockers in patients admitted to hospital with AMI. We have provided evidence suggesting that appropriately designed computer systems can generate data and reports that should serve as a new gold standard in efforts to measure and improve performance. Compared with manual methods, electronic systems may more readily identify prescribed medications and the presence of some chronic diseases such as CHF. The hospital whose records were studied is now securely providing continuous, real time, electronic data about performance directly to CMS through the Internet in a pilot demonstration of a National Health Information Infrastructure.61 All countries can consider the importance of creating regional and national networks for exchanging real time digital data about quality of care. The next steps should involve broader study of computer based performance measures, adaptation of existing guidelines in developing portable modules for integration into disparate EMRs, and studying the uses of real time, population based performance feedback in clinical practice.
The analyses upon which this publication is based were performed under Contract Number 500-99-IN03 entitled “Utilization and Quality Control Peer Review Organization for the State of Indiana” sponsored by the Centers for Medicare & Medicaid Services, Department of Health and Human Services. The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the US Government.
This work has been supported by Regenstrief Institute Inc, Health Care Excel Inc, and Centers for Medicare and Medicaid Services. Dr Weiner is supported by grant number 5K23AG020088-02 from the National Institute on Aging (US). Dr Callahan is supported by NIA grant number AG 00868.
The authors have no conflicts of interest in this work.
The authors assume full responsibility for the accuracy and completeness of the ideas presented. This article is a direct result of the Health Care Quality Improvement Program initiated by the Centers for Medicare & Medicaid Services which has encouraged identification of quality improvement projects derived from analysis of patterns of care, and therefore required no special funding on the part of this contractor. Ideas and contributions to the author concerning experience in engaging with issues presented are welcomed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.