Medications are the most frequent cause of adverse events, and such injuries are called adverse drug events (ADEs).^1,2 ADEs are common in most clinical settings including adult inpatients with a reported incidence of 6.5%,³ adult outpatients with an incidence of 27.4%,⁴ and pediatric inpatients with a reported incidence of 2.3%.⁵ These ADEs have substantial consequences including hospital admissions, prolonged hospital stay, additional resource utilization, and time away from work, as well as lower patient satisfaction.^6,7

Some ADEs are caused by errors called medication errors.² Medication errors are much more frequent than ADEs but only a small minority actually cause ADEs.² In one inpatient study the frequency of medication errors was 5.3 per 100 medication orders, much higher than the ADE rate of 0.25 per 100 orders.² Another recent report from 36 hospitals which evaluated the administration stage in particular showed that 19% of medication administrations contained an error.⁸

The goal of research in ADEs and medication errors is to reduce the likelihood of harm related to medications. To do this it is essential to be able to describe the epidemiology of these problems. Building on previous work, we have developed over the last decade a methodology for identifying and classifying medication safety issues which we present here so that others may use these methods to investigate ADEs and medication errors in their own settings. We also describe the strengths and limitations of this approach, including assessment of reliability.⁹

The methodology includes:

• definitions of incidents (ADEs, medication errors, and other drug related terms);

• general processes for identifying them;

• case identification methods (practice data review, self-reports from health professionals, and patient surveys);

• comparison of methods;

• methods for classifying incidents; and

• tools for validating the findings.

METHODOLOGICAL APPROACH

Definition of incidents

An incident includes any irregularity in the process of medication use. It might represent an ADE, potential ADE, medication error, or none of these—it is essentially a “catch all” term for what to call something before it has been classified. An incident can occur at any point in the medication use process (ordering, transcribing, dispensing, administrating, and monitoring). There are several ways to categorize incidents: actual (ADEs) v potential; preventable v non-preventable; ameliorable v non-ameliorable; and error v non-error. These categories may overlap and it is important to understand their relationship with each other (fig 1).

A medication error can occur at any step of the medication use process. Some ADEs are associated with medication errors and all potential ADEs are medication errors (fig 1). Minor errors that have little or no potential for harm are not considered potential ADEs—for example, a dose of non-critical medication such as docusate is given several hours late—but are considered to be medication errors. If the incident has the potential to harm a patient—for example, a dose of critical medication such as an intravenous antibiotic is not given—it is considered both a medication error and a potential ADE.

A potential ADE is a medication error with the potential to cause an injury but which does not actually cause any injury, either because of specific circumstances, chance, or because the error is intercepted and corrected—for example, an order is written for an overdose of medication but the error is intercepted by the pharmacist.

An ADE is an injury due to a medication—for example, cough due to angiotensin converting enzyme (ACE) inhibitors is considered an ADE. ADEs may or may not result from medication errors—for example, cough due to an ACE inhibitor in a patient without a history of ACE inhibitor induced cough is not the result of a medication error, while a medication error has occurred if the patient has a prior history of ACE inhibitor induced cough.

A preventable ADE is an injury that is the result of an error at any stage in the medication use—for example, a coma due to an overdose of a sedative. A non-preventable ADE is an injury due to a medication where there is no error in the medication process—for example, an allergic reaction in a patient not previously known to be allergic to the medication. These are also known as adverse drug reactions, or non-preventable reactions due to side effects or allergic reactions. An ameliorable ADE is an injury of which the severity or duration could have been substantially reduced if different actions had been taken—for example, sexual dysfunction lasting for several months while taking a selective serotonin reuptake inhibitor. A non-ameliorable ADE is an injury in which there is no current reasonable way to reduce the severity or duration—for example, bradycardia after the first usual dose of β blocker.

General process for identifying incidents

Our group has used three methods to collect incidents: (1) collecting practice data, (2) soliciting any incidents from health professionals, and (3) surveying patients for drug related events (fig 2). Other methods such as direct observation have also been used and this approach in particular is highly effective for detecting administration errors.⁸ Practice data sources include charts, laboratory, and prescription data, as well as administrative data.

In the inpatient setting, the Institute for Healthcare Improvement has suggested that the following sections require particular attention: discharge summary (may include ADEs), procedure notes (narrative sections for ADEs), physician progress notes (changes in plan of care related to effects of medications), laboratory reports (pertinent laboratory results), physician orders (pertinent medications), and nursing/multidisciplinary progress notes (symptoms related to ADEs).¹⁰ These data sources may be used individually or in combination to identify possible incidents. Trained research pharmacists, nurses, assistants, or physicians as well as pharmacy externs and medical students can usually do the review of charts, laboratory, and prescription data manually. This review can also be automated using rule sets to extract data from electronic medical records (EMR) or computerized physician order entry (CPOE) in order to target possible incidents more efficiently.¹¹

Screening of administrative data is usually based on ICD-9 coding associated with ADEs and medication errors such as poisoning or urticaria, although it can also include drug and laboratory data if available. This screening is usually computerized. The Agency for Healthcare Research and Quality (AHRQ) has developed patient safety indicators which are based on administrative data and use ICD-9 coding.¹² All patient safety indicators are associated with injuries from medical care, but not always with ADEs and medication errors. For example, the indicators include iatrogenic pneumothorax which is not related to ADEs or medication errors. On the other hand, complications of anesthesia include an overdose of anesthetic medications which is a medication error. Because the indicators and the calculation programs on the commercial statistical packages for administrative data are publicly provided, they can be used as a tool for screening. Overall, however, the positive predictive value of ICD-9 codes for ADEs is very low (about 2%) and is much lower for medication errors.¹³

Regardless of the method of identification, incidents are then generally presented to two independent reviewers who independently categorize them into ADEs, potential ADEs, medication errors, or exclusions. At the same time the reviewers classify the incidents according to a variety of parameters such as preventability and severity.

Case identification

Data review

In reviewing charts, laboratories, and prescription data, the reviewing process will differ somewhat between facilities with an EMR or CPOE system and those without them. At sites without an EMR or CPOE system the research nurses, pharmacists, or research assistants review charts, laboratory results, and prescriptions and identify any possible incident during the study period. Prescription data can be collected by screening carbon copies of prescriptions.

With an EMR or CPOE system reviewers can also review all the documents and data on the computer, but automatic extraction using computer programs and text searching or natural language processing can detect any keywords such as “allergy” or “falls”, specific laboratory values such as potassium levels of 6.5 mEq/l, or drug names such as an angiotensin II receptor blocker.¹¹ These concepts can be linked to form rules—for example, heparin and low hematocrit. A positive yield from a rule is called a trigger. The use of triggers is much more practical and less labor intensive than conventional chart reviewing because the triggers can be automated by computer and the reviewer’s search is much more focused. Gurwitz and colleagues¹⁴ used computer based triggers and other means to find 1523 ADEs from 27 617 patients; the computer based triggers found 66% of ADEs. Triggers can also be used to monitor for ADEs and medication errors in daily practice once their discrimination ability is established.¹⁵ Specific drugs, combinations of drugs, symptoms, ICD-9 related diagnoses, and laboratory results can all be used as triggers. These triggers from an EMR or CPOE are then reviewed by the research nurses, pharmacists, or research assistants and assessed as incidents or not. Written text or other information in the chart can be used in this evaluation. Honigman and colleagues¹⁶ transformed EMR data such as physician progress notes or laboratory results to text data and cleaned them for spelling, syntax, abbreviation, and inconsistency. The text data were then screened for possible incidents using a computer program.

When reviewing practice data the general rule to determine if an incident is related to a medication is to look for any irregularity in the patient’s condition such as change in mental status, sudden drop in blood pressure, sudden drop in oxygen saturation, new rash, or new diarrhea, and then to consider whether it might be related to a medication (table 1). If a likely relationship is present and these symptoms cannot be accounted for by underlying diseases or other plausible reasons, they are considered incidents. Certain actions among health professionals can be clues in identifying the incidents—for example, changes or clarification to a physician’s order, time changes on the medication administration record, outdated narcotic orders, late administration of drugs, order for Clostridium difficile toxin test or culture, and new allergy listed in the medication record (table 1). Many specific diagnoses associated with ADEs and medication errors can also be reviewed to identify incidents. The diagnoses are usually defined by ICD-9 codes, and extraction of the ICD-9 codes is usually done from administrative data (table 2).

Medication prescribing can also be evaluated to identify incidents. Prescribing certain drugs implies the occurrence of an incident (table 3)—for example, when diphenhydramine is given it is often to treat a reaction to another medication. Combinations of drugs should also be evaluated for incidents including drug–drug interactions that are frequently observed in practice (table 4).

Combinations of drugs and other factors also provide important clues for incidents (tables 5 and 6). The symptoms or diagnoses may or may not have a previously demonstrated association with the listed drugs. If plausible reasons other than medication use appear to have caused symptoms or diagnoses, they would be considered as events independent of drugs and are excluded. Otherwise, they are considered incidents. Some combinations of drugs and miscellaneous factors are also considered (table 7). For example, women of childbearing age, elderly persons, patients on dialysis, and pregnant women should not take certain specific drugs. Acetaminophen (paracematol) is a frequently prescribed analgesic both alone and in combination with other analgesics. Because patients can receive multiple forms of this agent resulting in an excessive daily dose, the cumulative dose should be assessed for possible excessive dosing. Patients who ingest non-steroidal anti-inflammatory drugs (NSAIDs) or selective cyclooxygenase (COX II) inhibitors may develop gastrointestinal problems, and those patients who undergo oesophagogastroduodenoscopy often have these problems because of these drugs.

Laboratory triggers are useful for detecting incidents, especially when the review process is assisted by computerized rules.¹¹ Some abnormal laboratory values themselves, even when there are no symptoms, are considered possible incidents because of the likelihood they have been caused by a medication. In the reviewing process all cases of hepatotoxicity (raised transaminase or bilirubin) and renal dysfunction (raised creatinine or BUN) that cannot be associated with another clinical issue and are associated with medication use are generally considered as incidents. In addition, some specific abnormal laboratories or inappropriate monitoring of medications are incorporated into the computerized rules to detect incidents (table 8).

Reports

There are two main types of self-report from health professionals: those from physicians, nurses, pharmacists, or other health professionals who become aware of any ADE, potential ADE, or medication error (sometimes called an “incident report”), and those generated by research assistants, nurses, or patient safety officers who visit wards or clinics to solicit any possible incident and record it.³ The process of using self-reports from health professionals is especially useful for identification of incidents in inpatient settings.³ Investigation of iatrogenic disorders such as ADEs and medication errors always provokes concerns about liability among health professionals and such concerns may inhibit the self-report.¹⁷ It is thus important to educate health professionals about the purpose of the study, which is to clarify the treatable factors associated with ADEs and medication errors and not to reprimand the responsible individuals. A non-punitive culture is important to maximize data capture at the study sites.

Patient surveys

Because incidents may not be recorded in a patient’s medical record, particularly in the outpatient setting, survey methods to elicit these types of incidents from patients are especially important. These include direct interviews in the office or patient’s home, mail surveys, telephone surveys, and web based surveys. We conducted a telephone survey in outpatients to collect the incidents which patients experienced.⁴ Patients were sent a letter describing the study and requesting their participation in a telephone survey. Within 10–14 days after an index visit, research assistants asked patients who agreed to participate about specific symptoms (table 5). If symptoms were present, structured questions followed about the timing and action taken. A similar telephone interview was conducted again 3 months later. All possible incidents collected by self-reports or surveys were sent to and screened by research pharmacists or assistants. Those unlikely to be associated with drugs were excluded and the rest were sent to the physician reviewers.

Comparison of methods

The above mentioned four methods (chart reviews, computer based triggers, self-reports, and patient surveys) are complementary. In the inpatient setting, computer based triggers detected 45% of all ADEs, chart reviews detected 65%, and self-reports 4%.¹¹ The overlap of chart reviewing and computer based triggers was 12% while that of chart reviewing, computer based triggers, and self-reports was only 1%. On the other hand, patient surveys identified 92% of ADEs in the outpatient setting and chart reviewing identified 28%; the overlap was only 19%.⁴ Thus, the proportion of overlap is typically minor and combinations of these methods can be useful in research to determine the true underlying rate of these events.

Methods for classifying incidents

In classifying incidents the reviewers consider the timing of findings (symptoms, abnormal laboratories, diagnoses), whether the physician in charge or the patient attributed the findings to the drug, and the strength of published data based on the relationship between the findings and the drug.

Once possible incidents are deemed actual incidents, they are classified according to the following categories: if a medication error is present, if a potential ADE is present, or if an ADE without an error is present. In addition, severity, preventability, ameliorability, the level of disability, the stage in the medication use process at which the error occurred, and the category of healthcare personnel responsible for the error can be classified (figs 2 and 3). The reviewers assess preventability or ameliorability on the basis of the physician’s presumed knowledge at the time the drug was prescribed. If insufficient information is available the reviewers assume that the physician’s decision was correct. When the incident is preventable or ameliorable, the reviewers specify the type of error and how it might be prevented. What is preventable may change over time because many ADEs judged non-preventable or non-ameliorable in the past might become preventable or ameliorable with new approaches such as genetic testing. We consider some laboratory abnormalities to be ADEs even if there are no symptomatic problems—for example, extreme hyperkalemia without arrhythmia or markedly raised INR without apparent bleeding (table 8). If an error is associated with the abnormal laboratory result—for example, failure to check a creatinine level within a 12 month period while the patient is receiving a medication known to increase creatinine levels—the incident would be considered a medication error.

Each ADE and potential ADE is classified as fatal, life threatening, serious, or significant (table 9). If a medication error is present, the stage of the process where the error occurred and the person responsible for the error considered are: ordering (physician, nurse practitioner, or physician assistant); transcribing (a secretary or nurse); dispensing (pharmacist); administration (nurse, pharmacist, or patients); and monitoring (physicians or patients).

Confidence about the classification of incidents is rated on a scale such as a 6 point scale (1 = little or no confidence; 2 = slight to moderate confidence; 3 = <50% confidence but a close call; 4 = >50% confidence but a close call; 5 = strong confidence; and 6 = virtually certain). Exclusion is made by a cut off point selected a priori and we considered 50% or more as indicating an actual incident.⁴

The rating method developed by Naranjo et al¹⁸ is another reliable way to estimate the confidence about the classification. This rating method assigns point values of from −1 to +2 to each of the following 10 attributes: (1) previous conclusive report on the reaction, (2) ADE appeared after the suspected drug was administered, (3) adverse reaction improved when the drug was discontinued or a specific antagonist was administered, (4) adverse reaction reappeared when the drug was readministered, (5) alternative causes other than the drug could cause the reaction, (6) reaction appeared when a placebo was given, (7) the drug was detected in the blood or other fluids in concentration known to be toxic, (8) severity of reaction changed when the dose was changed, (9) patient had a similar reaction to the same or similar drugs in any previous exposure, and (10) ADE was confirmed by any objective evidence. Each possible incident with the summed score is categorized as a definite, probable, possible, or doubtful case.¹⁸

Training reviewers and assessment of reliability

Research nurses, pharmacists, and research assistants usually take part in the case identification step and physician reviewers in the case classification step. The standard training of reviewers is therefore crucial for reliability of the methods. We have developed training manuals with definitions of incidents, description of the case identification process, case examples, and practice exercises with regard to ADEs and other drug related incidents. Reviewers are trained in advance using the manual and are also trained by pilot reviewing using the actual practice data. The results of the exercise and pilot reviewing are evaluated in terms of accuracy and inter-reviewer variability and the reviewers receive feedback. This process has been well received and understood by reviewers in training and has produced consistent results.

Inter-rater reliability for reviewer judgments is calculated using percentage of agreement and the kappa (κ) statistic.¹⁹ The percentage of agreement is calculated by dividing the number of agreed cases by the total cases. κ is calculated from (Po − Pc)/(1 − Pc) where Po  =  proportion of observed agreement and Pc  =  proportion of agreement expected by chance and ranges from −1 (complete disagreement) to +1 (perfect agreement). Differences between the two reviewers’ judgments about the decisions and classification are usually resolved by discussion. If consensus cannot be reached, a third reviewer evaluates the incident.

DISCUSSION

Our approach used in previous and ongoing studies on ADEs involves three steps: case identification, classification of incidents, and assessment of reliability. Other identification approaches can be used depending upon the target population, the area of the medication use process being evaluated, and the data available. For example, for detecting a small proportion of ADEs inexpensively using administrative data, the patient safety indicators developed by the AHRQ represent one option.¹² For detecting medication administration errors Barker et al⁸ have shown that direct observation can detect large numbers of errors and is highly reliable in the inpatient and long term care settings. This approach can also be used for detecting dispensing errors.

Reliability of classification is the crucial factor for the science of research collecting information about “soft” outcomes such as ADEs and errors. Many have raised concerns about the reliability of the process. In the process of identifying adverse clinical outcomes Sanazaro and Mills⁹ suggested several concerns about screening methods in medical quality assessment including (1) screening could have a high error rate and high false positive rate up to 95%, even by specially trained personnel; (2) physicians substantially vary in conducting peer reviewing; and (3) peer review is more likely to ascribe adverse outcomes to medical care if the outcomes are serious. These criticisms are an important issue in research in healthcare quality and should be considered carefully. A common way to address such criticism is assessment of inter-rater agreement using kappa statistics. The κ value indicates the extent to which the observational probability of agreement is in excess of the probability of agreement hypothetically expected under the baseline constraints. A κ score of 0.4–0.6 has been considered as moderate agreement, 0.6–0.8 as substantial, and 0.8–1.0 as almost perfect.¹⁹ In our studies in inpatients the κ score for the presence of ADEs was 0.98, exclusion was 0.81, preventability was 0.92, fatal or life threatening v serious or significant was 0.37, and significant v fatal or life threatening or serious was 0.32.³ Gandhi et al⁴ obtained κ scores for exclusion of 0.89, severity 0.72, and preventability 0.70 in an outpatient setting. These figures suggest that the approach we have used is generally reliable.

We have reported the incidence of ADEs, potential ADEs, and medication errors in a variety of clinical settings.^3–5,20 For example, the incidence of ADEs was 6.5 per 100 adult admissions³ and 2.3 per 100 pediatric admissions.⁵ In nursing homes ADEs were found at a rate of 1.89 per 100 resident-months,²⁰ while the incidence of ADEs among outpatients was 27.4 per 100 adult patients.⁴ The incidence of potential ADEs was also reported to be 5.5 per 100 adult admissions,³ 10 per 100 pediatric admissions,⁵ and 0.65 per 100 nursing home resident-months.²⁰ ADEs and potential ADEs are common in any setting but vary substantially in incidence, and the causes of errors and ADEs vary greatly by setting. Adult inpatients tend to receive a larger number of drugs than pediatric inpatients or nursing home residents. In addition, outpatients are more likely to be exposed to drugs for a longer interval than inpatients, and administration is obviously under patient control.

The differences in the methods used varied primarily because different approaches work better in specific settings and for specific purposes. Generally, prescription review is better for identifying medication errors while patient surveys are better for identifying outpatient ADEs. Chart review can identify both medication errors and ADEs. In inpatients or nursing homes incidents were found by reviewing records and self-reports from health professionals because no other approach was practical, although surrogate or family interviews would be of interest.

These research methodologies can be used to identify and prioritize strategies for preventing medication errors and preventing or ameliorating ADEs in daily practice. One approach that has become particularly attractive is computerized prescribing. CPOE associated with decision support has been shown to reduce medication error rates by up to 81%.^21,22 The triggers and text screening based on computerized systems can be a foundation for the development of an ADE monitor.¹⁵ The possible incidents detected through such triggers are then evaluated by a drug safety pharmacist for actual or potential ADEs on a daily basis. The drug safety pharmacist can make clinical recommendations to physicians or pharmacists working in conjunction with physicians. Silverman et al¹⁵ reported that 78% of such recommendations based on real time ADE monitoring were accepted and resulted in changes in medication orders. Schiff and colleagues²³ have also suggested a framework for multiple ways to link laboratory and pharmacy data. Such linkages can be divided into drug selection (laboratory based indications or contraindications), drug dosing (renal or hepatic, blood level guided adjustment), laboratory monitoring (laboratory signals of toxicity, baseline, and ongoing monitoring), laboratory result interpretation (drug interfering with test), and other quality improvement (surveillance for unrecognized toxicity, monitoring clinician response delays). Because information technology progresses quickly, it would be straightforward to use new devices as tools for patient safety.²⁴

Although we have used state of the art methods for identifying ADEs, potential ADEs, and medication errors in non-experimental settings with available resources at this time, the methodology described here has several limitations and should be further refined. We believe the future improvement of patient safety research will focus on the computerized approach. Jha et al¹¹ reported that computerized monitoring could detect 45% of ADEs and 42% of potential ADEs in an inpatient setting. However, the efficacy of computerized monitoring was not satisfactory and its positive predictive value was only 17%. Thus, the remaining 83% of alerts were not associated with ADEs, potential ADEs, or medication errors and might result in unnecessary physician actions or resource utilization. Continuous refinement of computerized monitoring rules based on research findings is necessary.

In conclusion, a multidimensional case identification process followed by physician review provides the current best estimates of rates of ADEs, potential ADEs, and medication errors, although this approach is expensive and using all the approaches is probably only practical in the research setting. Each incident can be assessed for preventability, ameliorability, disability, severity, staging, and responsible persons. The classification scheme has acceptable reliability and feasibility. The use of computerized ADE screening and monitoring systems as well as CPOE and EMR systems will undoubtedly improve the efficacy of ADE identification in addition to real time patient safety, but there is still substantial room for improvement of these applications.