Article Text

Download PDFPDF

Computerized surveillance of adverse drug events in hospital patients
  1. D C Classen1,
  2. S L Pestotnik1,
  3. R S Evans1,
  4. J P Burke1
  1. 1Department of Clinical Epidemiology, LDS Hospital, Salt Lake City, Utah, and the Division of Infectious Diseases, Medicine, University of Utah School of Medicine, Salt Lake City, USA
  1. Correspondence to:
 Dr D C Classen
 Department of Clinical Epidemiology, LDS Hospital, Eighth Avenue and C Street, Salt Lake City, UT 84143, USA
  1. J B Battles2
  1. 2Agency for Healthcare Research and Quality (AHRQ), Center for Quality Improvement and Patient Safety (CQuIPS), Rockville, MD 20850, USA;


    Objective: To develop a new method to improve the detection and characterization of adverse drug events (ADEs) in hospital patients.

    Design: Prospective study of all patients admitted to our hospital over an 18 month period.

    Setting: LDS Hospital, Salt Lake City, Utah, a 520-bed tertiary care center affiliated with the University of Utah School of Medicine, Salt Lake City.

    Patients: We developed a computerized ADE monitor, and computer programs were written using an integrated hospital information system to allow for multiple source detection of potential ADEs occurring in hospital patients. Signals of potential ADEs, both voluntary and automated, included sudden medication stop orders, antidote ordering, and certain abnormal laboratory values. Each day a list of all potential ADEs from these sources was generated, and a pharmacist reviewed the medical records of all patients with possible ADEs for accuracy and causality. Verified ADEs were characterized as mild, moderate, or severe and as type A (dose-dependent or predictable) or type B (idiosyncratic or allergic) reactions, and causality was further measured using a standardized scoring method.

    Outcome measure: The number and characterization of ADEs detected.

    Results: Over 18 months we monitored 36 653 hospitalized patients. There were 731 verified ADEs identified in 648 patients, 701 ADEs were characterized as moderate or severe, and 664 were classified as type A reactions. During this same period only nine ADEs were identified using traditional detection methods. Physicians, pharmacists, and nurses voluntarily reported 92 of the 731 ADEs detected using this automated system. The other 631 ADEs were detected from automated signals, the most common of which were diphenhydramine hydrochloride and naloxone hydrochloride use, high serum drug levels, leukopenia, and the use of phytonadione and antidiarrheals. The most common symptoms and signs were pruritus, nausea and/or vomiting, rash, and confusion-lethargy. The most common drug classes involved were analgesics, anti-infectives, and cardiovascular agents.

    Conclusion: We believe that screening for ADEs with a computerized hospital information system offers a potential method for improving the detection and characterization of these events in hospital patients.

    Statistics from

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    As many as 30% of hospitalized patients may experience an adverse drug event (ADE) during their hospital stay, according to current estimates.13 Moreover, fatal ADEs are expected in approximately 0.31% of hospitalized patients (60 000 to 140 000 patients annually) in the United States.3–5 Adverse drug events lead to 2–5% of all hospital admissions each year,5,6 and one recent report found that complications from drug therapy were the most common adverse events in hospitalized patients.7

    The exact costs attributed to ADEs are unknown, but it has been suggested that ADEs can prolong hospital stays and add to health care expenditures.3,8–10 Studies have indicated that hospitalized patients who are exposed to more than 16 different drugs during their hospitalization have a 40% probability of experiencing an ADE.11 Patients who have experienced a true ADE are two to three times more likely to experience another subsequent event than patients who have not had an ADE.11 In addition, hospitalized patients are often elderly and have underlying co-morbidities that impair their ability to distribute, metabolize, and excrete drugs, and these elderly patients are more likely to experience toxic reactions. Clearly, hospitalized patients have multiple risk factors predisposing them to ADEs.

    For these and other reasons, ADE detection and reporting systems have been advocated. The need for hospitals to assume a more active role in ADE surveillance has been addressed both nationally and internationally. The World Health Organization,12 the US Food and Drug Administration (FDA),13 and the Joint Commission on Accreditation of Healthcare Organizations14 have all addressed this need. The Joint Commission on Accreditation of Healthcare Organizations has required that hospitals have an ongoing drug surveillance program that is designed to monitor and evaluate the effects of drugs (both beneficial and harmful) and to continually improve the use of drugs to provide appropriate, safe, and effective drug therapies. The detection, reporting, and prevention of ADEs are but a few important aspects of this mission. With increasing competition and regulatory pressures to continually improve the provision of health care, health care institutions must improve the detection of ADEs in order to prevent these negative patient outcomes. A recent discussion15 summarized the available information on generic screening in quality assessment and found that ADEs were one of the medical clinical outcomes that should be used as a generic screen to develop a database for the continual improvement of patient care and clinical performance. Before programs to prevent ADEs and before databases of ADEs can be developed for quality assessment, hospitals must improve their ability to detect ADEs in a timely manner. At present most hospitals use surveillance systems that rely on voluntary reporting, but such systems are potentially cumbersome and less effective than prospective surveillance.16

    It has become increasingly evident that voluntary spontaneous reporting systems have resulted in low reporting rates of ADEs in the United States.17 With the increasing availability of computerized hospital information systems and their attendant clinical databases, it will soon be feasible to develop automated, non-voluntary surveillance systems to concurrently monitor all hospital patients for the occurrence of ADEs. We have previously developed computerized methods to enhance voluntary reporting of ADEs and to detect automated signals of potential ADEs. In this report we describe the operation of this system and its evaluation in an 18 month study period. The goal of this project was to determine the magnitude of enhanced detection and characterization of ADEs, with this method as a basis for efforts to prevent ADEs.


    The LDS Hospital, Salt Lake City, Utah, is a 520-bed acute care referral center that serves as a teaching facility for the University of Utah School of Medicine, Salt Lake City. The hospital provides a wide range of clinical services, except for a general pediatric service. A hospital information system, known as HELP, was used in this study. The HELP system has been clinically operational at the hospital for more than 15 years. The hallmark of this system is a computerized medical record that contains an integrated patient database drawn from numerous sources, including pharmacy, laboratory, surgery, and radiology. In addition to the integrated patient database, an interactive modular knowledge base is employed which continually analyzes information contained in the computerized medical record.18 We previously have used this hospital information system and its attendant knowledge base to develop several interactive computer programs to monitor hospital-acquired infections, duration of prophylactic antibiotic use, inappropriate use of therapeutic antibiotics, and drug exposures in hospital patients; it has also been used to perform extensive drug use evaluation programs.19

    Traditionally, the method of reporting ADEs at LDS Hospital has been a voluntary system that relied on a written incident report, generated either by the physician, the pharmacist, or the nurse who recognized and detected the event. These voluntary ADE incident reports were reviewed by the Pharmacy and Therapeutics Committee and, when appropriate, were submitted to the FDA. Our experience with this voluntary reporting mechanism was dismal at best, with approximately 10 to 20 ADEs detected and reported annually at LDS Hospital.

    The World Health Organization’s definition of an ADE was used for this investigation: an ADE is one that is ‘‘noxious and unintended and occurs at doses used in man for prophylaxis, diagnosis, therapy, or modification of physiologic functions.’’20 Furthermore, for the purposes of this investigation, this definition excluded therapeutic failures, poisonings, and intentional overdoses.20 Using this definition, computer programs were written to develop an ADE monitor for LDS Hospital. This program had two components. The first allowed for enhanced voluntary reporting through physician, nurse, and pharmacist entry of potential ADEs at all computer terminals throughout the hospital. Once activated, this program allowed easy entry of patient name, type of ADE symptoms, and reporter identification. For the second component, automated methods for detection were developed. These methods used algorithms created within the knowledge base for automated detection of potential ADEs through the use of various signals, including discontinuation of medications, decreases in dosages, ordering of known antidotes, ordering of specific laboratory tests (such as drug levels), Clostridium difficile toxin assays, and specific laboratory test abnormalities (such as elevated eosinophil counts, elevated serum potassium levels in the setting of potassium supplementation, and low white blood cell counts).21

    Each day at a predetermined time, all potential ADEs in the previous 24 hour period were summarized and printed in the daily ADE report. This service was provided Monday through Friday of each week; weekend information was assessed and evaluated on Mondays. The average time involved in providing this service was about 2 person hours each day. The information in the report included patient name, location, attending physician, hospital service, current and past drug profile, including starting and stopping times for all drugs, and the ‘‘signal’’ for detection of the possible ADE. A clinical pharmacist (SLP) used the report to supervise the review of all potential ADEs. Medical charts were reviewed, physicians and nurses were contacted, and patients were interviewed each day to determine causality. To promote consistency and to standardize the assessment of the relationship between the suspected drug and the ADE, the Naranjo algorithm was used to estimate the probability of an ADE.22 The Naranjo algorithm is a simple questionnaire that can easily be used at the bedside to perform causality assessment of ADEs.23 The algorithm consists of 10 weighted questions that yield the following associations between total score and causal relationship: (1) less than 0 points equals doubtful; (2) 1 through 4 points equals possible; (3) 5 through 8 points equals probable; and (4) 9 or more points equals definite.

    Once causality was assigned, the ADE was characterized by severity as mild (self-limited), moderate (requiring treatment), or severe (life-threatening, disabling, or markedly prolonging hospitalization). The ADEs were further classified by mechanism as type A or type B reactions. Type A reactions, which typically produce 70–80% of all ADEs, are related to a drug’s pharmacological characteristics and are usually dose-dependent, predictable, and preventable. Type A reactions are rarely life-threatening, while type B reactions are idiosyncratic or allergic in nature and are not dose-dependent or related to a drug’s pharmacological characteristics. Type B reactions are usually the most serious and potentially life-threatening of all ADEs and are rarely predictable or avoidable. Anaphylaxis is a classic type B reaction.24

    Once the ADE was verified, the prescribing physician was notified. All verified ADEs were permanently stored in the computerized medical record. Information stored on each ADE included the offending drug, clinical manifestation of the ADE, time of drug administration, time of ADE, time of drug discontinuation, source of the ADE signal, Naranjo algorithm score, and severity and mechanism (type A or B) classifications. In addition, monthly reports were sent to the Pharmacy and Therapeutics Committee for review and, when appropriate, were submitted to the FDA.

    All statistical analyses were done by the χ2 method.


    During the study period from 1 May 1989 to 31 October 1990 there were 36 653 patients who were admitted to LDS Hospital and who were concurrently monitored for the occurrence of ADEs. Sixty-one percent of the patients were female, and the average age was 48 years (range 0 to 101 years}. Surgical and surgical subspecialty hospital admissions accounted for 64% of all patients. Medical and obstetrics-gynecology hospital admissions accounted for 22% and 14% of all patients, respectively. A total of 557 860 drug exposures occurred in this patient population during the study period.

    During the study period, 731 ADEs were detected and verified among 648 patients. Therefore, among the 36 653 total patients, 731 ADEs occurred, for an overall ADE rate of 1.67%. During this same period, a total of nine ADEs were reported by the traditional voluntary incident report method. Adverse drug events occurred in 438 female patients and 210 male patients; the average age of patients who experienced an ADE was 57 years (range 12 to 92 years) and the average length of stay was 13 days vs 5 days in patients who did not experience an ADE (p<0.05). Patients who experienced an ADE received an average of 33 drug exposures vs only 13 drug exposures among patients who did not experience an ADE (p<0.05). The first detected ADE occurred after an average of 15 drug exposures and after an average of 6 days in the hospital. Two hundred and four patients experienced their first ADE while in an intensive care unit setting. Analysis of the distribution of ADEs among hospital services revealed that 480 were patients on the surgical services, while 251 were patients on the medical services when they experienced their first ADE. There were 52 hospital admissions due to an ADE. Age stratification revealed that there was an increasing ADE rate with each decade of age and that the ADE rate for patients who were 60 years or younger was 1.4%, while patients who were older than 60 years experienced an ADE at the rate of 3.3% (p<0.0001, table 1).

    Table 1

     Rate of adverse drug events (ADEs) and mean number of drug exposures in relation to population at risk

    Among the 731 ADEs detected, 101 were classified as severe, 600 were moderate, and 30 were mild. The majority (664) of ADEs were type A reactions; only 67 were classified as type B reactions. Of the type B reactions, eight were severe and 59 were moderate. Causality assessment using the Naranjo algorithm (average score 9.2) revealed that definite causality was established in 450 (61.6%) of the ADEs, probable causality in 276 (37.7%), and possible causality in five (0.7%). There was no case of doubtful causality.

    Once an ADE was verified, the prescribing physician was notified of the findings and the event was permanently stored in the patient file. A typical ADE alert, which illustrates the usefulness of automated detection of non-voluntary signals in conjunction with computer decision support, is as follows: a patient was identified as having received an antidiarrheal medication concurrent with an antibiotic. The prescribing physician was contacted, the case was reviewed, and the decision was made to put a hold on the antibiotic and order a C difficile toxin assay. The test was positive, the antibiotic and the antidiarrheal medicine were stopped, and the diarrhea resolved uneventfully without further complications. The event was permanently archived in this patient’s computerized medical record, and future ordering of the offending antibiotic agent will result in an appropriate computer generated warning.

    The most common signals of confirmed ADEs were generated by antidote use (41.3%), therapeutic drugs for ADEs (16.9%), drug levels (16.5%), and personnel reporting (12.1%). Diphenhydramine hydrochloride and naloxone hydrochloride accounted for the majority of antidotes that generated signals, and they represented 32.7% and 8.6% of all signals, respectively (table 2). Among therapeutic agents used to treat ADEs, the most common signals were phytonadione (7.0%), antidiarrheals (6.7%), and sodium polystyrene sulfonate (1.2%). Eosinophilia (>5% eosinophils on differential) signaled only three ADEs and epinephrine use signaled four ADEs. Of the 731 ADEs recorded, 90 were detected by the enhanced voluntary component (nurses reported the majority) and 641 were detected through automated methods.

    Table 2

     Signals of adverse drug events (N = 731)

    Among the 731 ADEs detected, 1398 different clinical manifestations were elicited from patients. Pruritus was the most common, occurring 12.1% of the time. Less frequently observed were nausea and/or vomiting (9.7%), rash (9.0%), confusion-lethargy (7.5%), dysrhythmia (5.7%), bradycardia (4.5%), hypotension (5.3%), diarrhea (4.8%), and bradypnea (3.7%). Other clinical manifestations accounted for the remaining 37.5% of the clinical manifestations observed. Fever was observed in three patients and eosinophilia was noted in three.

    Agents associated with the 731 ADEs tended to cluster in a few therapeutic classes (table 3). Analgesics and narcotics represented 31.0% of all ADEs, while antibiotics were found to be the cause of 23.3%. Cardiovascular agents and anticoagulants accounted for 19.4% and 9.3% of the ADEs, respectively. All other classes individually accounted for 2% or less of all the ADEs. Among analgesics, morphine sulfate, meperidine hydrochloride, and the combination product of acetaminophen and oxycodone hydrochloride were the most common offenders. Cefazolin, vancomycin hydrochloride, and the combination product of imipenem and cilastatin sodium were the most common antibiotics that caused ADEs. Warfarin, heparin, and streptokinase were the most frequent anticoagulant-thrombolytic-type agents that caused ADEs; digoxin, lidocaine, and procainamide hydrochloride were associated with ADEs most commonly in the cardiovascular area. The most common drug classes used at LDS Hospital during this period were slightly different; they included analgesics, fluid and electrolyte supplements, gastrointestinal drugs, psychotherapeutic agents, anti-infectives, and cardiovascular agents.

    Table 3

     Therapeutic classes causing adverse drug events (N = 733)*


    With the development of a computerized ADE monitor, we have increased by over 60-fold the detection and reporting of ADEs at our hospital. The resulting rate of ADE detection at our institution ranks among the highest reported by hospitals with enhanced ADE detection systems.3–5 This supports the observation that ADEs are markedly under-reported in hospitalized patients.16,17 In addition, more than 95% of the ADEs detected were moderate or severe, thus requiring a change in drug therapy and/or a prolongation of hospital stay. Mild ADEs were not easily detected by this method, which could be viewed as a limitation of this concurrent monitoring approach. However, to further delineate the frequency of mild ADEs in this population would require the prospective study of all 36 653 patients admitted to the hospital during the study period—clearly, an arduous and expensive task. This program was specifically developed to target those events that required changes in therapy and/or prolonged patient hospitalizations, not to screen for mild and self-limited ADEs. In addition, no attempt was made to measure the sensitivity or specificity of this method for the detection of ADEs. Calculation of sensitivity and specificity of this method was not possible because of the lack of a clear ‘‘gold standard’’ for comparison.

    Patients experiencing ADEs in this study tended to be older, have longer hospital stays, and experience more drug exposures. However, in contrast to previous studies, the occurrence of one ADE did not markedly increase the risk of a second ADE. More than 50% of ADEs occurred in patients older than 60 years, and the majority of severe reactions occurred in those patients in this same age group. Several studies have observed that the elderly are at a greater risk of developing ADEs because of age-related alterations in drug pharmacokinetics, co-morbidities, and the greater risk of multiple drug exposures.25–27 With the exception of analgesics, the leading drug classes associated with ADEs in this study included drugs commonly used in the elderly (e.g. antibiotics, cardiovascular agents, anticoagulants, and bronchodilators). In contrast to previous studies, fever and eosinophilia were distinctly uncommon occurrences among the ADEs characterized in this population; the explanation for this is unclear, although it may be partially age related.

    However, caution should be exercised in interpreting or extrapolating our, as well as others’, data with regard to the effect of aging and the occurrence of ADEs. The relationship between aging and the risk of ADEs is more complex than the mere chronological characteristics of the patient.28 Individual physiological and functional patient characteristics, as well as polypharmacy, are often more important in determining the risk of an ADE than age. Furthermore, this study was designed to develop methods to enhance the detection, recognition, and reporting of ADEs in a hospital population and not to investigate the relationship between aging and the risk of ADEs.

    Many organizations, including the Joint Commission on Accreditation of Healthcare Organizations, the World Health Organization, and the FDA, have called for better methods to detect, characterize, and report ADEs in hospitalized patients. Several methods have been used to detect ADEs in hospital populations, including voluntary reporting, concurrent monitoring, and prospective study. The pioneering work of the Boston Collaborative Drug Surveillance Project has delineated the role of prospective monitoring of all patients receiving drugs for the occurrence of ADEs.29 This method is quite sensitive in detecting ADEs, although it detects many ADEs that are minor and self-limited; in addition, it is expensive and probably not cost-effective in most hospitals. The FDA has created a large, spontaneous reporting system for ADEs that entirely relies on voluntary reports. Similarly, most hospitals use various forms of voluntary detection and reporting. Unfortunately, many of these programs are logistically cumbersome and often physician-dependent, thus tending to discourage most ADE reporting. Although potentially useful for the detection of severe and unknown effects of new compounds, these systems are not well suited to the hospital environment.30,31 Several reports have detailed efforts to improve these voluntary ADE reporting systems in hospitals.32–34 However, even with education, voluntary reporting still heavily depends on the time and enthusiasm of physicians and nurses, clearly a precious commodity.

    Concurrent monitoring of drug use in hospitals offers another approach for ADE detection. Several studies have emphasized the value of concurrent monitoring as a mechanism for the detection of ADEs in hospitals.35,36 In one study,35 a concurrent method was considerably more sensitive than a voluntary method in the detection of ADEs. This report used screening of medication orders and certain laboratory values for identification of possible ADEs. However, the concurrent monitoring of all patients for ADEs can be time- and labor-intensive.

    Computers have been used extensively for pharmacy databases and less frequently as a mechanism to detect and prevent potential drug interactions and adverse effects.37,38 In one report,39 personal computers were used to monitor ADEs. However, this system used voluntary report forms that were hand entered into a personal computer after submission, thus failing to add any increased efficiency of detection or characterization of ADEs. Obviously, the development of large integrated hospital databases offers considerable advantages in the detection, characterization, and reporting of ADEs in hospitalized patients. The concurrent monitoring system for ADEs described herein incorporates the efficiency of an electronic medical record for detection of ADEs with the judgment of a specialized clinical pharmacist for the formulation of causality. Indeed, this effort required less than 2 person hours each day.

    Although computerization of voluntary ADE reporting offers added efficiencies, the present study suggests that automated detection systems can contribute significantly greater potential to the enhanced detection of ADEs. More than 85% of the ADEs documented in this report were detected by automated screening of various signals. In addition, over 75% of verified ADEs in this study did not have the causative agent stopped until study personnel informed the physician of the ongoing ADE. This finding is similar to a previous report40 that outlined the lack of physician understanding of ongoing ADEs. The timely and aggressive feedback of ADE information to physicians in this study may have prevented development of more severe manifestations of ADEs. In addition, verified ADEs were permanently stored in the electronic medical records of all ADE patients. This permits the automatic alerting of new prescription orders for the offending drug in present or future hospitalizations of each patient. Future studies may elucidate the role that this mechanism will have in the prevention of ADEs in these patients.

    In summary, we have developed a computerized surveillance system for ADEs that has markedly increased their detection in hospitalized patients. This approach has merged the advantages of a hospital information system with the judgment and experience of a clinical pharmacist to create a new method for the enhanced detection of moderate and severe ADEs in hospitalized patients through concurrent surveillance. We found that automated detection methods were more effective in detecting ADEs than an enhanced voluntary method. In addition, this study has also detected ADEs in a timely manner, allowing for early cessation of the causative agent and potential prevention of more serious ADE manifestations. Further study is needed to define the most effective strategies for the prevention of ADEs in hospital patients.



    Any discussion of patient safety usually begins with citations from the report of the Institute of Medicine (IOM) “ToErr is Human1 released in 1999. However, there were core articles that preceded To Err is Human which were used to form the basis of this landmark study. The paper by Classen et al2 published in JAMA in 1991 is one of those articles. Classen and colleagues described how adverse drug events could be identified in an integrated hospital information system using a computer program to detect a variety of indicators of harm associated with adverse drug events.

    At the time the article was written in 1991, most of the attention in patient safety was being directed to determining the rate of adverse medical events, and adverse drug events in particular. Classen et al compared the rate of detection using the computerized protocols with the integrated hospital information system (now referred to as the electric health record (EHR)) with traditional spontaneous event reporting. The use of the EHR to detect events produced a nearly tenfold increase in the number of events identified. Similar results using computerized record systems were reported by others in subsequent years.3,4 The use of protocols and algorithms to identify patient harm associated with clinical care has continued to mature. These protocols have become known as triggers which can be used as tools with either conventional medical records or with the EHR.5

    During the early days of patient safety a good deal of effort was spent on trying to determine which form of identification of events was most effective. However, it is now being recognized that there is a need to use multiple methods for the detection of harm including spontaneous event reporting, triggers from records, and patient safety indicators using administrative data.6 The IOM in its most recent report “Patient Safety: Achieving a New Standard of Care7 has recommended using multiple approaches for the identification of harm. One of the factors limiting the acceptance and use of the methods outlined by Classen and colleagues has been the availability of computerized health records systems in most institutions. The IOM has also called for the development of standard triggers to be used as part of new EHR systems.

    With the growing emphasis being placed on the use of health information technology (HIT) solutions to patient safety, there is a need to deploy common sets of triggers that can be built directly into EHR systems. National agencies such as AHRQ in the US and the NPSA in the UK should begin to develop universal triggers for the detection of harm that can be used by any vendor of EHR systems.

    Classen and colleagues gave us the way forward in 1991; it is up to us today to fully implement the computerized surveillance systems in every healthcare institution worldwide. With today’s emphasis on HIT and EHR systems we cannot lose the opportunity to build such systems into our daily practice, just as was done in Utah in 1991.



    • * This is a reprint of a paper that appeared in JAMA, 1991, Volume 266, pages 2847–51.

    • This study was supported in part by grant HS06028 from the Agency for Health Care Policy and Research, Bethesda, Md.

    • We appreciate the tireless and enthusiastic help of Sheron B Bass, RN, for assistance in the identification and monitoring of patients experiencing adverse drug events.


    • The opinions and assertions contained herein are the private views of the author and are not to be construed as official or as reflecting the views of the Agency for Healthcare Research and Quality.

    Linked Articles

    • Quality Line
      BMJ Publishing Group Ltd