The new paper is not a study

Though the paper by Makary and Daniel was widely cited as ‘a study’, it presented no new data nor did it use formal methods to synthesise the data it used from previous studies. The authors simply took the arithmetic average of four estimates since the publication of the IOM report, including one from HealthGrades,5 a for-profit company that markets quality and safety ratings, a report from the US Office of the Inspector General (OIG)6 and two peer-reviewed articles (table 1).7 ,8 The paper did not apply any established methodology for quantitative synthesis nor did it include a discussion either of the intrinsic limitations of the studies used or of the errors associated with the extrapolation process. To bolster their claims, Makary and Daniel did highlight the agreement between their estimates and that of a similar analysis published a few years ago by James.9 The apparent consensus is not, however, surprising, since they use mostly the same studies (listed in table 1, together with a more recent analysis commissioned by the Leapfrog group10).

Issues with the studies on which estimates of deaths due to medical error are based

Some of the widely quoted estimates of deaths due to medical error, including the IOM estimates,1 Makary and Daniel4 and James,9 are based on studies that in fact did not set out to estimate the rate of mortality linked to medical error. Instead, these primary studies sought to measure the prevalence of harm from medical care (ie, adverse events).

Consistent with their primary purpose, these studies included no methodology for making judgements about the degree to which adverse events played a role in any deaths that subsequently ensued. For instance, a patient admitted to the intensive care unit with multisystem organ failure from sepsis might develop a drug rash from an antibiotic to which he has exhibited a past allergic reaction. This patient has certainly experienced a preventable adverse event. But, if the patient eventually dies of progressive organ dysfunction a week after the antibiotic was changed, the medical error probably did not cause the death. An error that has occurred close to a death is not a sufficient basis for concluding that the error is the cause of death. Yet these studies do not have an explicit methodology for handing this situation—for distinguishing deaths where error is the primary cause from deaths where errors occurred but did not cause a fatal outcome.

A further problem with the basing estimates on studies that use adverse event and trigger tools of the type used by Makary and Daniel (and in the similar review by James9) is that they typically involve very small numbers of deaths. For instance, one study used a trigger tool approach to review 100 charts per quarter from each of 10 hospitals in North Carolina from January 2002 to December 2007.7 This study sought to detect any decline in adverse events that might have occurred as a result of patient safety efforts. In passing, the authors report that 14 adverse events were judged to have ‘caused or contributed to a patient's death’. These 14 deaths represented 0.6% of the total patients in the study. Similarly, one US government report included three preventable deaths;11 another reported 12.6 One of the widely quoted peer-reviewed studies identified nine deaths.8 Any extrapolation that generalises from so few deaths (14 or fewer) to so many (200 000–400 0004 ,9) surely warrants substantial scepticism.

The need for scrutiny is particularly important because when studies are designed specifically to identify preventable deaths, they typically report low rates. Studies that have reviewed inpatient deaths and asked physician reviewers to judge preventability have reported proportions under 5%, typically in the range of 1%–3%.12–15 The largest and most recent of these studies13 reported that trained medical reviewers judged 3.6% of deaths to have at least a 50% probability of being avoidable.

Epidemiological errors

While most of the studies used by Makary and Daniel did not have as their primary purpose the estimation of deaths due to medical errors, the estimate from HealthGrades did. This estimate, and a more recent one from the Leapfrog group10 (table 1), uses a methodology that depends on combining the frequency of hospital-acquired conditions (HACs), such as central line bloodstream infections and pressure ulcers, with estimates for the mortality attributable to these HACs. Some of these HACs include the PSIs produced by the US Agency for Healthcare Research and Quality.16

Using this methodology, HealthGrades estimated 389 576 deaths due to preventable adverse events per year in the USA (circa 2000–2002). From 2000 to 2010, the annual rate of inpatient deaths in the USA ranged from 715 000 to 776 000,17 meaning that HealthGrades proposed that over 50% of all inpatient deaths are preventable. The more recent Leapfrog produces a somewhat lower estimate of avoidable deaths in US hospitals each year—206 021,10 with the lower number reflecting deliberate avoidance of double-counting deaths in patients who developed more than one HAC. Preventable conditions such as pressure ulcers, thromboembolism and healthcare-acquired infections occur, of course. They occur far more often than they should. They are profoundly distressing for patients and their families. But, saying that they account for 30%–50% of inpatient deaths flies in the face of clinical experience. It is likely instead that many patients die with, rather than of, these conditions.

Why the fuss? What harm could come from estimating so many deaths due to medical error?

Around 700 000 deaths occur in US hospitals annually.17 Makary's and Daniel's estimate that over 250 000 of those are preventable implies that around a third (or more) of inpatient deaths result from medical error or preventable adverse events. If, as studies that have actually studied preventable deaths have concluded, 3.6% is the more correct rate, then it means something like 25 000 deaths might be averted each year by eliminating medical error—a far cry from 251 454. Every life avoidably lost is a tragedy. No one is disputing the need to improve safety. But does this 10-fold difference in the death rate matter?

One could argue, as many did in the wake of the IOM estimate, that we should avoid picking apart estimates of mortality rates for medical error, since they draw attention to a much-neglected issue. This argument lies at the heart of one of the central tensions in the field of patient safety since its inception: the tendency to call for action on the basis of limited evidence.24–26 The urge to do so may arise from the desire to make up for lost time—the many decades of exclusive focus in medical research on discovering new tests and treatments while neglecting the basic duty not to harm patients. But, it is an urge that must be kept in check.

First, patient safety needs to establish its scientific credentials. It does the field no favours if the basic epidemiological facts cannot be trusted. For a start, it means that the metrics of progress will be constantly disputed.27 After 16 years of sustained attention to improve patient safety and other aspects of healthcare quality, it is disappointing to find that some are claiming that the problem is fivefold bigger than previously announced. We need reliable measures of progress over time, just as any field does. If ‘anything goes’ as far as the metrics are concerned, we have no hope of demonstrating that all the investment and effort in patient safety are worth it—thus, discouraging further investment.

Second, the narrow focus on preventable death risks distracting attention from the many harmful consequences of failures to manage risks adequately in healthcare that do not result in death.2 ,3 ,12 ,28–30 Just as most deaths do not involve medical error, most medical errors do not produce death—but they can still produce substantial morbidity, costs, distress and enduring suffering. Highlighting preventable deaths as the focus of patient safety efforts risks drawing resources away from many safety problems and many settings of care—including most non-hospital environments—where death is not the most relevant outcome. For instance, medication safety is universally regarded as one of the largest categories of safety problems; yet drug errors, though very common, do not usually result in fatal outcomes.31 Most pressure ulcers do not result in death, but they are a painful and miserable experience for patients. Does this mean that medication safety and pressure ulcers should not receive attention? Of course not. But, this is the risk of repeatedly focusing the attention of the public and policymakers on death as the sole outcome of interest.