Use of cumulative mortality data in patients with acute myocardial infarction for early detection of variation in clinical practice: observational study

Introduction

The UK Department of Health has recently published a national service framework for coronary artery disease, which aims to coordinate efforts to improve medical care of patients by the implementation of minimal standards of care and audit.1 Many risk models have been derived in an attempt to reliably compare care at different centres. It is self evident that if a doctor or hospital is responsible for a high proportion of sick patients then simply assessing crude mortality will result in an unfavourable comparison of care. Most of the risk-adjusted models that have been derived suffer from overcomplexity, exclude substantial proportions of patients with higher risk profiles, use administrative data without clinical details, or are applicable only to selected subgroups. 2 3 We have developed a model based on three objective admission clinical characteristics—age, heart rate, and systolic blood pressure. We have validated our model with subgroups of our patients with acute myocardial infarction, and it seems to accurately reflect patient outcome.4^–6

Scoring systems, such as the Parsonnet score,7 have been used to assess individual surgeons' performance by comparing actual and expected death rates. Lovegrove et al have described the variable life-adjusted display,8 a graphical technique based on the cumulative sum (cusum) method 9 10 that incorporates information of estimated risks for each individual case. Cusum charts record successive cases horizontally, with plots ascending by single units for each outcome event (such as death) reached. The variable life-adjusted display incorporates outcome information, simultaneously accounting for prior estimated risk in each case and plotting the difference in cumulative expected and observed mortality. Sherlaw-Johnson et al recently described a method for estimating variability for such displays in order to assess the probability that observed deviations occurred as a result of chance rather than because of a difference in quality of care.11

In this study we extend the use of variable life-adjusted displays in combination with a simple, accurate, and validated risk model for assessing care of patients with acute myocardial infarction and suggest indications for more detailed review of patient care.

Subjects and methods

Results

The variable life-adjusted displays (figs 1 and 2) show the number of cases on the x axis and the cumulative difference between the predicted and observed deaths on the y axis. Figure 1 shows the results for hospital A, the centre with the highest standardised mortality ratio. This small hospital had a series of only 30 patients with acute myocardial infarction. The observed and expected mortalities were 44.2% and 27.5% respectively (standardised mortality ratio 1.61). The risk model would predict eight deaths, whereas the actual number was 13. The display reveals minimal variation from that predicted for the first 15 patients followed by a run of unexpectedly high mortality.

Fig 1

Variable life-adjusted display of a small cottage hospital for a series of 30 consecutive patients with acute myocardial infarction. Predicted mortality was calculated by a validated risk score based on admission characteristics of age, systolic blood pressure, and heart rate

• Download figure
• Open in new tab
• Download powerpoint

Fig 2

Variable life-adjusted display of a large teaching hospital and regional cardiac centre for a series of 188 consecutive patients with acute myocardial infarction. Predicted mortality was calculated by a simple risk score based on admission characteristics of age, systolic blood pressure, and heart rate

• Download figure
• Open in new tab
• Download powerpoint

Figure 2 shows the for hospital B, which admitted 188 patients with acute myocardial infarction during the study period. The plot terminates with 3.5 extra “lives” above expected (observed number of deaths 42, expected 45.5; standardised mortality ratio 0.92). Between case numbers 35 and 135 the plot ascends from a cumulative (expected minus observed) net lives gained of −4 to 7.

Discussion

Using just patients' age, blood pressure, and heart rate to adjust for case mix, we derived a continuous plot to compare observed and expected outcome for patients with acute myocardial infarction.

The main purpose of audit is to provide high quality data analyses with which to monitor the required standards and to allow clinicians to evaluate the performance of their units. Block audit, however, is retrospective and does not allow immediate identification, and thus rectification, of potential problems. With application of a simple risk-adjusted model to populations of patients with acute myocardial infarction, we have produced variable life-adjusted displays. Such plots have become increasingly integrated into surgical care, allowing immediate monitoring of performance for individual surgeons or units over time, but they are not yet used in routine audit of medical patients.

Recently, Sherlaw-Johnson et al showed that prediction intervals can be derived for plots of standardised mortality ratio, with the preoperative case mix taken into account.11 We did not calculate prediction intervals for our populations because we consider that the usefulness of the graphical displays is enhanced by being qualitative rather than quantitative. We do not consider that achievement of a P value less than 0.05 is helpful in the context of continuous, real time monitoring, nor should it provide evidence that is not in keeping with other qualitative indicators. For example, if hospital B (fig 2) had a major shortage of beds in its coronary care unit beds that coincided with the period of increased deaths, actions might be taken even without a significant P value. Although an apparent run of poor results may not represent a real change, a failure to act on the qualitative trend observed until a significant P value had been achieved could result in the needless loss of life.

Our display method of monitoring performance or outcome over time can be used as an early warning system to allow more detailed audit to be performed.12 For example, the sharp descent of the plot in the second half of figure 1 highlights the possibility of poor performance. Conversely, in figure 2 the graph ascends from a cumulative net lives saved of −4 to 7 in the space of 100 consecutive cases, suggesting that any possible problem in the hospital was identified and corrected. Such periods of poor or above average performance are liable to be missed with block audit, especially when the overall figures are not significantly different from those forecast. As stated by de Mol when commenting on the original description of the variable life-adjusted display,13 differences in care measured by patient outcome are common. A judgment of whether this performance is above or below average could be based on predefined limits in order to adequately identify important outliers as well as permitting more cost effective and logical use of audit time. However, we would further suggest that rigorous application of a P value (which is itself subject to errors in the context of multiple observations) may mean that early qualitative trends towards harm remain unheeded. Solely relying on quantification of variability in performance rather than a qualitative assessment is therefore laden with shortcomings.14 A hypothetical example of this limitation is that 16 of 160 patients would have to die after routine cardiac surgery (10% mortality) before a statistically significant doubling of deaths could be detected with 90% chance. 15 16 Common sense suggests that such a trend should be identified and corrected long before the problem was so far progressed, especially if other strong qualitative indicators were present.

Any risk model can become less accurate over time because of progressive improvements in care. The Parsonnet scoring system, derived in 1989, has consistently been shown to predict risk strata that correlate with the observed mortality in adult acquired heart disease. However, it equally consistently overestimates the risk of contemporary coronary surgery by up to 100%. 7 17 18 In order to avoid this problem if our model were to be in general use, we would recommend that patients' heart rate and blood pressure on admission should be added to the core dataset of the national service framework for coronary artery disease. If hospitals were then to provide the data required for the risk model to a centralised database the model could be continually updated and made available on the internet. This would reduce the likelihood of overestimating risk in patients with acute myocardial infarction.

As stated in the national service framework documentation, good quality data can provide a powerful engine for change and improvement. The method we describe constitutes an improvement over simple clinical intuition and has a potentially important role in indicating the need for more detailed audit. In the interests of patient safety, we suggest that there is a need for continuous, qualitative observations of deviations from expected outcomes.