Elsevier

The Lancet

Volume 362, Issue 9382, 9 August 2003, Pages 485-491
The Lancet

Public Health
Following Shipman: a pilot system for monitoring mortality rates in primary care

https://doi.org/10.1016/S0140-6736(03)14077-9Get rights and content

Summary

As part of the investigations into the crimes of Harold Shipman, it has become clear that there is little monitoring of deaths in general practice. By use of data on annual deaths at family physician and practice level for five English health authorities for 1993–99, we investigate whether cumulative sum charts (a type of statistical process control chart) could be used to create a workable monitoring system. On such charts, thresholds for deaths can be set, which, if crossed, may indicate a potential problem. We chose thresholds based on empirical calculations of the probabilities of false and successful detection after allowing for multiple testing over physicians or practices. We also statistically adjusted the charts for extra-Poisson varition due to unmeasured case mix. Of 1009 family physicians, 33 (including Shipman) crossed the alarm threshold designed to detect a 2 SD increase in standardised mortality, with 97% successful detection and a 5% false-alarm rate. Poor data quality, plus factors such as the proportion of patients treated by these physicians in nursing homes or hospices are likely explanations for most of these additional alarms. If used appropriately, such charts represent a useful tool for monitoring deaths in primary care. However, improvement in data quality is essential.

Section snippets

Statistical Issues

A monitoring system must be designed to quickly detect unusual variations in the underlying mortality rate in any unit. To design such a system requires that expected or acceptable mortality rates be defined, corresponding to what is termed the in-control process in control-chart studies. Criteria must also be decided to define when the observed mortality is sufficiently different from that expected to warrant special attention, corresponding to the mortality rates being out of control. Units

Data linkage

The current death registration process does not record the deceased's family physician. Only the name of the certifying doctor is recorded. To obtain deaths by practice or family physicians, we extracted 7 years of mortality data (1993–99) from the statutory death register held by the Office for National Statistics. We linked the data with general practices' lists of patients held on five English health authority information systems (including that in which Shipman's practice was situated, West

Construction of control charts

A common statistical assumption is that mortality counts follow a Poisson distribution with mean given by the indirectly standardised expected count. However, since we were able to make only limited adjustment of the expected (in-control) mortality counts for case mix, there are probably many unmeasured risk factors leading to variation in the observed data other than that due to random Poisson variation. We obtained estimates of the amount of extra-Poisson variation (overdispersion) in the

Application of charts

The Office for National Statistics provided the NHSIA with 281 777 mortality records from the five pilot areas for the years 1993–99. The success of linking mortality data to registers of lists of patients varied over time and between health authorities. Overall, 92% of mortality records were linked with data on list of patients. The proportion of mortality records successfully linked in all but the West Pennine health authority was similar, with the match rate for the other four health

How charts compare

Various methods for health-care surveillance have been proposed,6, 27, 28 but none seems to have dealt effectively with the monitoring of multiple units over multiple points in time. Nor has the difficulty of how to adjust SPC charts for the inevitable overdispersion in routinely collected health outcome data been previously considered. We have shown the usefulness of log-likelihood ratio cumulative sums for monitoring mortality in primary care with use of real data for a large number of units

Interpretation of charts

Patients' mortality is clearly highly variable at family physician and practice levels. This variability is much greater than would be expected by chance alone. Key explanations for this finding include inadequate adjustment for case-mix and poor data quality yielded from the pilot exercise to link primary care populations and mortality at family physician level. The limited success of the latter was largely due to missing NHS numbers on the Office for National Statistics mortality records for

Conclusions

We envisage cumulative sum charts being used as a governance tool for monitoring performance since they enable a first-pass analysis of the data and can highlight units with unusual outcomes. We caution however, that the charts cannot by themselves shed light on the reasons for apparent poor performance. Methods of operation at a local level will be required that enable the clinical explanations for outliers to be readily identified. Even in this context, use of cumulative sum charts, or any

References (30)

  • WhiteheadJ

    The design and analysis of sequential clinical trials

    (1983)
  • Marshall EC, Best NG, Bottle A, Aylin P. Statistical issues in the prospective monitoring of health outcomes at...
  • SonessonC et al.

    A review and discussion of statistical issues in public health monitoring

    J R Stat Soc C

    (2003)
  • VanbrackleL et al.

    A study of the average run length characteristics of the national notifiable disease surveillance system

    Stat Med

    (1999)
  • Terje LieR et al.

    A sequential procedure for surveillance of Down's syndrome

    Stat Med

    (1993)
  • Cited by (69)

    View all citing articles on Scopus
    View full text