Although the true prevalence and incidence of adverse events associated with health care remains unknown, estimates in the United States and internationally suggest the magnitude of this problem is substantial and represents a grave danger to the safety of patients treated at all levels of the healthcare system. In 1999 the Institute of Medicine report “To Err Is Human” estimated that “more than 1 million preventable adverse events occur each year in the United States, of which 44 000 to 98 000 are fatal”.^1,2 Although these numbers have been challenged in the literature,^3–5 some agree that these numbers represent the lower bound or the “tip of the iceberg”.^4,6 If the estimates of adverse events were not alarming enough, the estimated numbers of close calls could total as many as 4 million,² or higher.^7,8,9,10

Researchers have used a variety of data sources to describe and analyze adverse events within the general population and the Veterans Health Administration.^6,11–35 Data sources have included surveys, healthcare providers’ reports, direct observation, medical records or charts, incident reports, malpractice claims, tort claims, risk management files, and specific quality improvement and quality assurance files from specialties within unique settings.^{11–13,15–36} To improve patient safety by reducing adverse events, the accuracy of these data sources are extremely important since they may serve as a tool for assessing risk exposure and improve patient safety. But “underreporting of adverse events is estimated to range from 50% to 90% annually”,⁷ suggesting that these data sources are highly inaccurate in assessing risk exposure.^7,37 Reasons for underreporting are multifaceted; they are based on whether a reporting system is voluntary or mandatory, confidential and/or anonymous, and if it implies a culture of blame (that is, the personal approach) or a culture of patient safety (that is, the systems approach).^1,2,38–48 Reporting of adverse events has many barriers that exist no matter whose perspective is taken—be it the patient, the risk managers, the organization, the legal world, or the government.^7,37,49–53

We wanted to learn about the relationship between reporting adverse events and tort claims. Our rationale was that adverse events with tort claims—particularly those resulting in settlement or payment—would be highly reported. We therefore looked at the patient incident reporting system and tort claims within the Veterans Health Administration between the fiscal years 1993 and 2000. The patient incident reporting system was established to obtain comprehensive data on adverse events that affect patients and to act as a harbinger for risk management. During the period of time under investigation, the patient incident reporting system was mandatory as described in the literature^14,34 and “require[d] each VA [Veterans Affairs] medical center to report and investigate categories of incidents … [therefore making] a move from a case-by-case evaluation to an epidemiologic evaluation of the rate of incidents”.³⁴ Since it was required, one might hypothesize that the mandatory patient incident reporting system should encompass 100% of the adverse events associated with tort claims that occurred in the Veterans Health Administration over the defined time period; however, it is acknowledged that, in the final analysis, with a self-reporting system all reporting is voluntary.⁵⁴ It should also be noted that the system that we evaluate here preceded the reporting system currently used to guide patient safety root cause analyses and action plans.⁵⁵

The main objective of this paper was to gain an understanding of the thoroughness of reporting of adverse events in a mandatory reporting system. We accomplished this by examining the relationship between tort claims outcomes and patient incident reports and by comparing the characteristics of patient incident reports for those with and those without tort claims.

METHODS

Data and variables

Two datasets were used to analyze the relationship between tort claims and patient incident reports (PIRs). The primary sources of data consisted of a unique identifier dataset, a tort claim dataset, and a PIR dataset. The unique identifier dataset contains 10 793 social security number-record number matches. The tort claim dataset has previously been described and extensively analyzed.¹⁵ Although this dataset consists of 11 066 tort claims closed between 1 October 1988 and 30 September 2000, we limited our analysis to the fiscal years 1993–2000 (N = 8260) so that tort claims would exactly overlap the patient incident reports. A claim settled was a settlement that was reached before the claimant filed a lawsuit. A denied claim was a claim found not to be deemed substantive by local or regional counsel. A suit dismissed was a claim that proceeded to federal court and was dismissed. Suit defendant was a claim that developed into a lawsuit and was resolved in favour of the Veterans Health Administration. Suit plaintiff was a claim that proceeded to trail and was found in favour of the plaintiff. A suit settled was a claim that developed into a lawsuit but was than settled out of court. The tort claim process in the Veterans Health Administration is detailed in box 1. The variables within this tort claim dataset are: a record number, four different dates (the date of the incident, of filing the claim, and both the date and the fiscal year of closing the claim), the location of the incident, the outcome of the claim, and the amount paid, if any. Tort claim outcomes were classified as: claim settled, denied, suit dismissed, suit defendant, suit plaintiff, and suit settled.¹⁵ Tort claim payment is defined as a claim resulting in a settlement or in judgment for the plaintiff.

The patient incident report (PIRs) dataset encompasses 33 286 PIRs obtained from the Veterans Health Administration’s patient incident reporting system. Our analysis is limited to 32 207 patient incident reports that occurred between the fiscal years 1993–2000 since the patient incident reporting system became mandatory in 1993. This dataset provides seven pieces of information: a social security number, the type of incident, the date of the incident, the action the medical center took following the incident, the station number where the PIR was completed, its status, and the severity level of the incident. The status of the PIR is defined as: closed, open, or deleted. The severity level of the incident was reported as: death, major injury (requiring medical intervention or resulting in loss of function), minor injury (not requiring medical intervention or resulting in loss of function), or no injury or disability.

To ensure the highest levels of confidentiality and to comply with the requirements and regulations of both the Health Insurance Portability and Accountability Act (HIPAA) and the Veterans Health Administration regarding research using health data containing identifying information, we removed all unique identifiers once the final dataset was developed. Institutional review board approval was obtained from the Veterans Administration in White River Junction, Vermont and Dartmouth Medical School, Hanover, New Hampshire.

Final dataset and variables

Social security numbers were used to merge datasets and records that included both a tort claim and a PIR were classified into three categories. Related incidents are those where recorded dates of the incident from both the tort claim and the PIR were within 1 month of each other. Non-related incidents had recorded days of incidents that were not within 1 month of each other. A PIR-tort claim match was classified as not sure if the date of the incident was present in the PIR but was not in the tort claim. Where duplicate PIRs were found, we assigned only one to the tort claim with the closest date of incident.

Analysis of data

Most of the descriptive statistics were obtained using tabulations for categorical variables and summations for continuous variables performed in STATA version 7.0. Microsoft Excel was used to calculate total payments for both all and within claim outcome categories and to perform simple percentages.

The characteristics of patient incident reports were summarized by dividing patient incident reports into two groups: those with and those without tort claims. Crude odds ratios and their 95% confidence intervals were calculated for tort claims outcome, tort claim payment, and tort claim payment as a function of having a patient incident report for a specific type of incident. To calculate the odds ratios and 95% confidence intervals for tort claims outcomes, they were split into three categories: claim settled, denied, or resulted in a lawsuit (fig 1). High frequency incidents are those with more than 400 patient incident reports.

To perform two sample t tests with unequal variances, we conducted log transformations on claim payments from both the original tort claim dataset and the final dataset to normalize their distributions since they were both right skewed. Log transformations and the two sample t tests with unequal variances were performed using STATA version 7.0, but we present mean (5th and 95th percentiles) for dollar payments, and also present the log transformed data where comparisons were made for the calculation of the 95% confidence intervals around the mean difference.

For non-related matches the time period between the dates of incidents listed in the PIR and tort claim was analyzed based on the hypothesis that veterans would be more likely to file a tort claim to express their dissatisfaction.

RESULTS

A summary of the tort claims made in fiscal years 1993–2000 is shown in fig 1. From the original tort claim dataset there were 8260 tort claims (mean payment $140 065). Once the merging of these datasets was completed, 575 numerical tort claim-PIR matches emerged (mean payment $145 312). Of the 575 numerical matches, 59.7% were related claims (343/575), 29.7% were non-related claims (171/575), and 3.0% were matches classified as not sure (17/575). Of the 343 related matches, the majority of tort claims were settled claims (43.4%) while the remaining proportion of tort claims was roughly divided between being denied (29.5%) and those that resulted in a lawsuit (27.1%). There were 229 tort claims payments among the related matches (mean payment $152 239) representing 8.2% of all tort claims payments and 2.7% of all tort claims. These related tort claims payments consisted of 149 settled claims (mean payment $89 792), 77 lawsuits ending in settlements (mean payment $254 924), and three payments awarded to the plaintiff (mean payment $618 170).

The characteristics of the PIRs are summarized in table 1. Among the related PIR-tort claim matches, 368 types of incidents and level of severity were recorded amounting to 1.1% of all PIRs filed. Of all the 31 839 PIRs that did not have a tort claim, 37% (N = 11 781) involved incidents resulting in major severity while 16% (N = 5089) involved deaths. In contrast, among the related 368 matching PIRs, 52.2% (N = 192) of incidents resulted in major severity and 38.3% (N = 141) resulted in death. For the eight high frequency incidents, the percentage of torts with related PIRs versus PIRs without tort claims ranged from 0.09% for suicide attempts to 2.59% for injuries not otherwise listed. Over the 8 years of the study (1993–2000) we found no trends in the number of patient incidents reported (fig 2).

The point estimates of the odds ratios and their 95% confidence intervals for the relationship between PIRs and tort claim characteristics are shown in fig 3. Tort claims were most likely to be settled when a related PIR was present (OR 2.71; 95% CI 2.16 to 3.40), slightly more likely to have developed into a lawsuit (OR 1.42; 95% CI 1.10 to 1.82), and least likely for those denied (OR 0.31; 95% CI 0.24 to 0.40). When a matching PIR was present, tort claim payments were much more likely (OR 3.62; 95% CI 2.87 to 4.60). Tort claim payments were more likely for medication errors (OR 8.37; 95% CI 2.06 to 73.25) and least likely for suicides (OR 0.25; 95% CI 0.11 to 0.55). Furthermore, tort claim payments were 100% (5/5) among transfusion errors (a low frequency type of incident) that had a related PIR.

The t test results are shown in table 2. There were significant differences between the log transformed mean payments in the final dataset and the tort claim dataset for all claims ($152 239 v $140 065; p<0.001), for claims settled ($89 792 v $85 682; p<0.001), and for suits settled ($254 924 v $206 107; p = 0.047), and were substantially higher, but not statistically significant so, for payments resulting from lawsuit judgments in favor of the plaintiff ($618 170 v $480 802; p = 0.35). Analysis of the payments with 5th and 95th percentiles shows the large variation in payments made for adverse events.

The mean time period for all non-related matches (N = 171) was 421 days and the median was 177 days. 115 (67.3%) had negative time periods, indicating that the date of the incident listed in the PIR followed the tort claim listed date of incident.

DISCUSSION

Ours is the first study to examine the relationship between a mandatory patient incident reporting system and tort claims within a US healthcare system that provides comprehensive care throughout the nation. While only very few tort claims had a related PIR, if a PIR was present the tort claim was more likely to result in a payment and the payment was likely to be higher.

Our primary finding that only 4.15% of tort claims were reported in the patient incident reporting system is consistent with large scale underreporting of adverse events. Although underuse of incident reporting systems has been documented elsewhere,^18,56 our finding within a mandatory reporting system suggests that all reporting systems—even the mandatory ones—are voluntary. Our finding suggests that administrators who expect that they will increase the quality or volume of data collection by making an adverse event reporting system mandatory may be mistaken. Other methods to enhance the quality and volume of adverse event reporting such as confidentiality or anonymity may be more effective.

Despite the incompleteness of reporting, we found evidence that reports of adverse events are associated with increased likelihood of tort claims, increased likelihood of tort claim payment, and higher tort claim payments. This suggests that the data collected within adverse event reporting systems may not be representative of all adverse events: they may be inherently biased towards the collection of events that have the worst outcomes. There are several possible explanations for this bias; events with the worst outcomes may stimulate reporting activities or be harder to justify not having been reported. Regardless of the reasons, our findings suggest that use of an adverse events reporting system for risk management may be limited and may overstate risks.

Limitations of the study

Our study has several limitations. Firstly, we used existing datasets that were incomplete. Although this was a limitation, these datasets are the same as those used by risk managers and the Veterans Health Administration to assess their risk exposure in addition to other systems such as root cause analyses. Because of the retrospective nature of the study, hindsight bias and implicit judgments might have limited our findings. According to other studies, these two limitations will, at best, produce a lower estimate of the prevalence of adverse events.^4,6,36

Secondly, the period of time may have been too short for some of the incidents to develop into a settled tort claim. As previously reported, a time lag exists between when a tort claim is filed and its resolution. Within the Veterans Health Administration claims that were denied or settled were resolved, on average, in just over a year, whereas those that went to court took about twice as long to resolve.¹⁵ We witnessed a similar time lag in this analysis.

Thirdly, and most important, because our study was conducted within the Veterans Health Administration, these results may not be generalizable to the general population. The Veterans Health Administration differs from the general population in significant ways. It serves mainly men who are more socioeconomically disadvantaged,¹⁵ may have fewer options of switching providers,⁵⁷ and may be less willing to sue and/or more willing to accept lower payments.^13,15,16

Implications of the findings

Despite these limitations, our findings suggest that researchers and administrators should be cautious in their analyses of adverse event reporting systems. While efforts to improve reporting of adverse events may be enhanced by removing some of the barriers to reporting, establishment of a mandatory system will not achieve complete reporting. Any reporting system is voluntary and is dependent on the reporter to complete the report. Efforts to improve reporting should therefore focus on addressing vulnerabilities of the system that thwart reporting. We found evidence that there is a bias toward reporting adverse events with the worst outcomes—a bias that may overstate risk exposure. However, within a systems based culture of safety, reporting systems may become more effective at detecting adverse events and allowing risk management and quality improvement efforts to be conducted in a more cost effective manner.