Adverse events (AEs) occur with alarming frequency in health care.¹ These events, or injuries resulting from medical care, represent incalculable loss both to affected patients and to institutions and clinicians attempting to provide safe and effective care. In order for institutions to learn from these events, they must be identified, disclosed, and examined.² Inadequate information about the occurrence of adverse events has been identified as one of the major obstacles to achieving meaningful improvements in patient safety.^3,4

Healthcare providers continue to search for an accurate and reliable method to identify and measure AEs in hospitalized patients. The methodology developed for use in the Harvard Medical Practice Study^5,6 has become a paradigm for AE detection methodologies, and the practice of conducting medical record review based on screening criteria has become a staple of virtually all subsequent approaches.

The Harvard Medical Practice Study approach, however, was developed for the purpose of conducting large scale research by retrospective medical record review. Our intention was to create a simple and practical adaptation in which data would be collected during patients’ stay in hospital, thereby permitting a more complete investigation in real time than can be accomplished with retrospective review. Furthermore, with this methodology, reviewers have the opportunity to intervene to prevent or mitigate harm in some patients. We wanted these data to be useful for patient safety improvement purposes and to support the rapidly evolving investment in electronic medical data. Our method of “active surveillance” is based on the well established model used to identify healthcare associated infections.⁷

In this paper we describe the active surveillance methodology and its use of electronic triggers, derived from abnormal laboratory values and designated medications, as signals of AEs in hospitalized patients.

METHODS

Setting

The study was conducted at a 725 bed academic medical center in Chicago, Illinois. Information systems were used to generate lists of all patients with selected triggers (described below) from both the laboratory and pharmacy databases. The study was approved by the Northwestern University Institutional Review Board.

Definitions and outcome measures

Our objective in this project was to identify AEs and medical errors which occur during hospital admission. An adverse event was defined as any injury caused by medical management rather than by the underlying disease or condition of the patient.^1,5Medicalerror was defined as the failure of a planned action to be completed as intended or the use of a wrong plan to achieve an aim.^1,8 A medical error that resulted in patient harm was referred to as a preventable AE. Events were identified as preventable AEs whether they resulted from a distinct identifiable error or whether no distinct error could be found but the event was clearly preventable—for example, a bloodstream infection. A medical error which did not cause harm could be intercepted (if it was caught before reaching the patient) or non-intercepted (if it reached the patient and had the capacity to cause harm, but failed to do so).⁹Active surveillance is medical record review performed concurrently while the patient is in hospital rather than retrospectively following discharge. An intervention was defined as an action taken following identification of an error or a potentially harmful situation, to prevent harm to the patient or to minimize any harm that may have already occurred.

Trigger methodology

Based on previously published investigations^10–12 and a prior pilot project at this institution,¹³ a list of triggers was adapted for use in the study (table 1). The triggers consist of certain abnormal laboratory values, high risk medications, and medications used as antidotes, all known to be associated with the occurrence of AEs. We revised previously published triggers^10–12 to target preventable events by excluding some triggers and including certain medications considered to be at high risk of causing patient harm if used incorrectly (such as dalteparin, enoxaparin, and warfarin, excluding prophylactic doses).¹⁴ All events were classified as being triggered if they were directly related to one of the pre-selected abnormal laboratory values or medication triggers, or non-triggered if they were not related to one of the triggers but were discovered incidentally upon review of the medical record.

Sample selection

Medical records were selected for review based on the identification of patients who met the criteria for at least one trigger. Electronic lists were obtained daily from the clinical laboratory (for abnormal laboratory results, positive cultures, and fresh frozen plasma) and the pharmacy (for medications). Neonates were excluded from the study and no surveillance was conducted while patients were in the operating rooms, labor and delivery, or the emergency department. No pediatric medical records were reviewed as our institution has no pediatric inpatient units other than the neonatal intensive care unit. Only patients hospitalized for at least 48 hours and still in the hospital at the time of medical record review were included.

Each weekday, medical records were assigned for review by a study case manager. The case manager is a nurse on the research team who compiled a daily list of records for review based on the electronic laboratory and pharmacy reports. Assignments were made based on our goal of reviewing 300 medical records over a period of 3 months. Medical records were assigned according to the following algorithm. All cases with two or more triggers were assigned first. Cases were then assigned sequentially, one from each of the three following categories, on a rotating basis: cases from the medication list, cases from the abnormal laboratory value list, and cases in which the patient had a positive blood culture. Approximately 8% of the records eligible for review were assigned for review over the course of the study.

Data collection and rating procedures

Medical record review was performed by trained patient safety researchers grouped into teams composed of one registered nurse and one pharmacist. A schedule was established to ensure that surveillance was performed each weekday; triggers from the weekend were included in cases reviewed on Monday.

When performing surveillance, all components of the medical record were reviewed to fully assess the context of all identified triggers and to identify non-triggered events (events not directly related to one of the study triggers but discovered during record review). As medical records were undergoing a transition from paper to electronic at the time of the project, surveillance required a review of both paper records (physicians’ progress notes, the medication administration record, and nursing flow sheets) and of the electronic medical record (laboratory results, radiology results, and some nursing documentation). Once the trigger was assessed, the researchers continued to review the complete medical record for any additional events. Events that occurred in the outpatient setting but were identified during the patient’s hospitalization were recorded and documented as being outpatient events. All relevant information was documented on a standardized data collection form. Investigators discussed events with the clinicians providing direct patient care when additional information was needed or when a situation was discovered which required immediate action to prevent or mitigate harm.

Surveillance of each medical record was performed using a flow diagram based on the methodology used in the Harvard Medical Practice Study¹⁵ (fig 1). The National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) scale (table 2) was used for severity ratings.¹⁶ The flow diagram assisted the investigators in answering the following questions:

• Was there harm? (Yes/no)

• If there was harm, rate your confidence that it was caused by medical management. (Six point scale, dichotomized for analysis)

• If there was harm and it was likely caused by medical management, was it preventable? (Four point scale, dichotomized for analysis)

• If there was harm and it was likely caused by medical management, rate the severity of the harm. (Five point scale, NCC MERP categories E–I)

• If there was no harm, was there an error? (Yes/no)

• If there was an error but no actual harm, rate whether the error reached the patient. (Three point scale, NCC MERP categories B–D)

Any laboratory value in the critical range for our laboratory was classified as harm.¹⁷ Additionally, all adverse events were categorized by type of event based on a taxonomy used for quality assessment of incidents reported through the hospital’s voluntary reporting system. A summary of the event classification taxonomy is shown in table 3.

The two members of each research team reached consensus on all event ratings, then presented events with harm to one of two board certified internists who independently performed all ratings. For any rating on which the physician reviewer and researchers did not agree, the second physician conducted an independent review. Had there been instances of continuing disagreement, resolution would have been achieved through discussion between the two physician reviewers; however, no such instances arose.

Data analysis

Data analysis was performed using Microsoft Excel (Version 2002), Microsoft Access (Version 2002), and SPSS (Version 11.5, Chicago, IL). Categorical variables were analyzed using the χ² test or Fisher’s exact test, as appropriate. Analysis of continuous variables was performed using the Wilcoxon rank sum test. Two tailed tests were used and a p value of ⩽0.05 was considered statistically significant.

RESULTS

From June to September 2003 a total of 327 medical records of hospitalized patients were reviewed. The mean (SD) age of patients with an AE was 60.6 (17.9) years (range 17–97), while the mean (SD) age of all inpatients during the study period was 48.7 (18.9) years (range 10–103) (p<0.001). The mean length of stay for patients with an AE was 20.0 (18.1) days (range 2–101) compared with the mean length of stay for all inpatients during the same time frame of 4.84 (6.48) days (range 1–108) (p<0.001). Most records reviewed were of patients on the general medicine service (42%), followed by hematology (11%), neurosurgery (7%), and surgical transplantation (6%).

A total of 493 triggers were investigated, representing 8% of the total number of triggers received over the study period. Overall, 44% of the reviewed laboratory triggers and 25% of the reviewed medication triggers identified AEs. Of all triggers, an elevated international normalized ratio (INR) (96%) and positive blood culture (92%) had the highest yields for AEs. These were followed by abnormal prolonged partial thromboplastin time (PTT) (59%), glucose (57%), and digoxin level (56%; table 1).

At least one AE or medical error was identified in 243 records (74% of all records reviewed). Overall we identified 462 events, 70 (15%) of which were categorized as outpatient events (occurring before the patient’s admission to the hospital). Of the 392 inpatient events, 254 were AEs (caused by medical management rather than by the underlying disease or condition of the patient); of these, 163 (64%) were preventable (medical errors that resulted in patient harm). We found 138 medical errors that did not result in patient harm.

Of the 163 preventable adverse events, 101 were directly related to one of the triggers and 62 were non-triggered. Non-triggered events were more likely to be severe (29% NCC MERP categories G–I) than were triggered events (6% categories G–I; p<0.001).

In assessing the severity of the AEs, it was found that 85% resulted in temporary harm and required medical intervention or initial or prolonged hospitalization (NCC MERP categories E and F), 4% resulted in permanent harm, 10% required intervention to sustain life, and 1% may have contributed to the patient’s death.

All AEs were categorized to understand better the hospital systems which may have contributed to the event (table 3). These categorizations allowed for a systematic review of the types of events identified, and for an analysis of the preventability and severity of harm associated with each type. These data were used to further evaluate hospital systems and to set priorities for process improvement projects. The mean time to complete the review of an entire medical record was 35 minutes for charts not requiring physician review and 45 minutes for charts in which physician review was necessary.

DISCUSSION

Our trigger tool methodology was beneficial in targeting medical records to identify AEs, with 74% of the records containing at least one AE or medical error. Currently, there is no standard methodology that effectively identifies AEs in hospitalized patients, although the design of a useful tool—either manual or electronic—has been an ongoing effort.^18–20 By using components of the Harvard Medical Practice Study^5,6,15 and trigger tool methodology,^10–12 we were able to develop a process similar to the epidemiological surveillance model widely used in infection control.⁷ This model was chosen as a basis for our approach because of similarities in research objectives and focus.

It is known that events reported through voluntary incident reporting systems represent only a small fraction of all AEs and errors that occur.^4,21–24 The reporting that does occur is sporadic and influenced by unpredictable factors.²³ Our methodology was designed to detect events unlikely to be captured through our voluntary spontaneous incident reporting system or our retrospective administrative data mining efforts, such as the Agency for Healthcare Research and Quality Patient Safety Indicators. Without active surveillance, these events would not have been brought to light and made available for analysis.

Interventions to prevent or ameliorate harm

Active surveillance was an effective means both to capture previously unrecognized events and to reduce harm directly. The primary advantage of this approach is that it permits investigation of AEs during a patient’s stay in hospital at a time when care providers are available to supply important information that may not be documented in the medical record, and when the potential exists for errors to be corrected before patient harm occurs. Whenever a researcher discovered a situation having the potential to cause patient harm or causing ongoing harm, he/she would intervene by bringing the situation to the attention of the appropriate care provider. Interventions to prevent or mitigate patient harm at the time of record review were made in 47 (14%) of the 327 medical records reviewed.

Interventions were most frequently related to the incorrect prescribing of medications and to inconsistent or incorrect documentation of clinical information. For example, one intervention involved informing a nurse that an order to discontinue a patient’s vancomycin had been written 3 days previously, while the patient continued to receive the medication. Another involved a patient admitted to the hospital with a supratherapeutic INR. The physician ordered warfarin in the same dose as the patient reported taking before admission. The researcher discussed the order with the prescribing physician and the warfarin was put on hold. Although it is likely that some of the prescribing errors would have been detected by clinical pharmacists had they not been first discovered by the study’s researchers, the documentation errors were found only because a pair of well trained eyes were poring over the patient’s record; it is unlikely that they would have been otherwise recognized and corrected. Given the number of interventions made, the preventive and ameliorative potential of bringing ongoing discrepancies or errors to the attention of direct care givers is significant. Furthermore, researchers found that their comprehension of the etiology of errors and of the contribution of hospital systems to AEs and medical errors was significantly enhanced through communication with front line caregivers.

Use of the electronic medical record

During the time of the study the medical record was not fully electronic and did not permit text searches of progress notes or discharge summaries. However, electronic data from the laboratory and pharmacy—combined with up to date admission, discharge, and transfer data—permitted the accurate and efficient assignment of medical records for review.

Recent advances in the development and implementation of the electronic medical record suggest that detection of AEs using this technology will improve the efficacy and efficiency of AE detection. Recent reports^20,25,26 describe the preliminary development of a system of free text searches for trigger words (a precursor to natural language processing) to screen electronically stored discharge summaries for the presence of AEs. While the specificity of this method is low, it holds promise as a productive use of electronic information. The use of automation to identify medical records with a high likelihood of containing an AE could significantly reduce the time and cost associated with these methodologies.²⁵

A computerized AE detection tool that can identify a wide range of AEs would be an important advance in patient safety, and could be used by hospitals for the routine detection of AEs on an ongoing basis and in real time.²² Prospective detection of preventable AEs in medicine has been shown to be more effective than retrospective detection,²⁷ and the use of electronic triggers can facilitate this process. This approach has been recommended with a call for data collection that is concurrent rather than retrospective.³

Extension of knowledge beyond the scope of events identified through voluntary incident reporting

The importance of having a valid and thorough understanding of the types of AEs that occur within an institution cannot be overestimated. Yet many institutions rely solely on the information obtained through voluntary incident reporting systems as a basis for quality improvement efforts. The use of these systems has limited utility; providers frequently do not voluntarily report AEs and errors because they do not recognize that an AE has occurred, they do not have the time, they are concerned about litigation, or they are worried about their reputation.^23,28 Thus, certain types of incidents are significantly under-reported in voluntary reporting systems.^23,29 Incident reporting will only be improved when providers understand the types of incidents that need to be reported, when they work in a non-punitive culture, and when reporting is quick and easy to accomplish. While voluntary incident reporting is likely to remain an important part of institutions’ ongoing quality operations, it is unlikely that it alone will ever provide a thorough picture of all AEs and medical errors that occur within any healthcare institution. Supplementing this information with events detected by active surveillance will provide a more complete portrait of the types of events that occur and of the actions which must be taken to prevent their recurrence.

Application in the acute care setting

Extending the use of the data collected through this methodology to patient safety efforts can be achieved in various ways. We found the most useful information to enhance patient safety came from an assessment of the event classifications and their related preventability and severity ratings. For example, six events related to medication allergies which caused harm were identified but none of these events was found to be preventable. Conversely, five of six events in which there was patient harm due to a premature discharge or transfer were preventable. This type of analysis of the event categories allows for prioritization of events by focusing on processes and systems that might be contributing to the incidence of preventable AEs, and for examination of the clinical practices which surround the occurrence of these preventable events. Data collected during our surveillance have led to the expansion of roles for our glucose monitoring and anticoagulation teams.

In addition to examining trends and patterns in types of AEs, we have been able to identify clinical issues which represent a low volume but are high risk. One example is the identification of a small number of patients with critically elevated partial PTT who were receiving lepirudin or argatroban. These events led us to question the safe prescribing and monitoring of these agents and prompted us to conduct a study to assess dosing guidelines.³⁰ The findings of this project have resulted in improvements in the dosing and monitoring of these low volume but high risk medications.

Non-triggered events

An important feature of this methodology is the extension of medical record review to capture events beyond those related to triggers. This is valuable as non-triggered events were more likely to be severe than triggered events. The greatest number of the severe non-triggered events were preventable surgical or procedural complications, the types of events which made up about one half of the AEs identified in the Harvard Medical Practice Study.^5,6 These types of events, plus others not detected with the exclusive use of triggers (including errors in diagnosis and in clinical management) represent an important cohort of events that can lead to patient harm and that are essential to assess in any patient safety improvement effort.³¹ In order to identify and target those systems most likely to cause severe patient harm, it is necessary to look beyond the scope of events that can be identified solely through the use of laboratory and medication related triggers.

Limitations of the study

This study has several limitations. It is well documented that interrater reliability in the evaluation of AEs detected through medical record review is poor to moderate.³² In this study there were a small number of investigators and two physicians who independently rated all events; all of these individuals were trained to apply the more conservative rating consistently whenever they had any doubt as to whether an event was caused by medical management or was preventable. Furthermore, while interrater reliability was not directly evaluated in this study, we have shown high interrater reliability among the same group of investigators in a previously published study.³³ It is clear that reliability can be improved when a homogeneous approach is taken.

The low reliability associated with medical record review may be further exacerbated by the use of implicit rather than explicit assessment methods.³⁴ It has been suggested that, while explicit assessment may be more reliable than implicit methods, it may miss more errors.³⁵ Furthermore, implicit assessment allows the reviewer to observe the event in its full context and to apply judgment based on clinical expertise, thus adding validity to these complex assessments.

The timing of the chart review conducted in this study represents both an advantage and a limitation. Concurrent review of medical records during patients’ stay in hospital affords researchers the opportunity to obtain first hand, “real time” information on details surrounding an AE or medical error from providers responsible for the care of the patient at the time the event took place. It also permits intervention when potentially harmful situations are discovered, so that harm can be prevented, mitigated, or shortened in duration. On the other hand, events that occur during a patient’s admission that would be apparent with retrospective review might be missed if they are not apparent or have not yet occurred on the day of concurrent review. Furthermore, with concurrent review, assessment of harm is performed based on the patient’s condition on the day of review, without knowledge of the patient’s condition at the time of discharge from the hospital. Severity ratings based on an assessment of the patient’s condition at the time of discharge would in some cases permit a more accurate representation of patient harm.

Another limitation of this methodology is the cost of personnel required to perform active surveillance. Although it is expected that review time will diminish as investigators gain experience, the average review times of 35–45 minutes must be taken into account in determining whether this methodological approach is practical for any particular institution. Finally, the use of our current triggers favors the identification of adverse drug events; AEs not related to medications are likely to be underestimated. An ideal methodology would facilitate the detection of all types of AEs and would be as effective in identifying non-medication related events (such as diagnostic and clinical management errors) as it is in finding medication related errors.

To enhance the usefulness of our active surveillance model, it is our intention to continue to refine the electronic triggers, retaining those with the highest yields. It will also be necessary to add additional triggers capable of signaling AEs other than those targeted by the present triggers. We are particularly interested in exploring further the more severe non-triggered events such as those associated with diagnosis, procedures, and surgery. Such triggers might include risk adjusted length of stay, unexpected transfer to a higher level of care, and unplanned readmission to the hospital within a specified period of time following discharge. Furthermore, as the electronic medical record becomes more advanced, text searches or natural language processing may become valuable tools.

As hospital care systems improve, it is expected that the insights from surveillance and other event identification systems will be incorporated into electronic clinical alerts and practice changes through such tools as standard order sets. We anticipate that computerized prescriber order entry and sophisticated electronic pharmacy and medical record systems will ultimately be helpful in addressing system failures in areas such as anticoagulation, electrolyte and glucose management, moderate sedation, and unrecognized gradual clinical deterioration (such as renal function, vital signs). As these relatively common failures are addressed with advanced tools, the AEs which are sought and identified in surveillance systems are likely to become more complex and challenging to understand. Over time, active surveillance will therefore become even more essential to building a full and comprehensive understanding of AEs in the clinical context.

In summary, the availability of a valid and reliable system for the detection of AEs in hospitalized patients can arm healthcare providers with valuable information to better understand and improve the systems that affect the delivery of safe care in their institutions. This methodology permits a systematic approach to the identification and assessment of AEs and provides a unique opportunity to intervene and prevent patient harm. Additionally, the knowledge gained from this project will provide the basis for design enhancements that will further improve the efficiency and effectiveness of AE detection.