You are here

  • Home
  • Archive
  • Volume 20, Issue 6
  • Exponentially weighted moving average charts to compare observed and expected values for monitoring risk-adjusted hospital indicators

Article Text

Article menu

Download PDFPDF

Original research
Exponentially weighted moving average charts to compare observed and expected values for monitoring risk-adjusted hospital indicators
  1. D A Cook1,
  2. M Coory2,
  3. R A Webster3
  1. 1Princess Alexandra Hospital, Queensland Health, Queensland, Australia
  2. 2School of Population Health, Mayne Medical Centre, University of Queensland, Queensland, Australia
  3. 3Statistical Analysis Unit, Health Statistics Centre, Queensland Health, Queensland, Australia
  1. Correspondence to Dr David A Cook, Intensive Care Unit, Princess Alexandra Hospital, 199 Ipswich Road, Woolloongabba, Queensland 4102, Australia; david_cook{at}health.qld.gov.au

Abstract

Objective To introduce a new type of risk-adjusted (RA) exponentially weighted moving average (EWMA) chart and to compare it to a commonly used type of variable life adjusted display chart for analysis of patient outcomes.

Data Routine inpatient data on mortality following admission for acute myocardial infarction, from all public and private hospitals in Queensland, Australia.

Methods The RA-EWMA plots the EWMA of the observed and predicted values. Predicted values were obtained from a logistic regression model for all hospitals in Queensland. The EWMA of the predicted values is a moving centre line, reflecting current patient case mix at a particular hospital. Thresholds around this moving centre line provide a scale by which to assess the importance of trends in the EWMA of the observed values.

Results The RA-EWMA chart can be designed to have equivalent performance, in terms of average run lengths, as variable life adjusted display chart. The advantages of the RA-EWMA are that it communicates information about the current level of an indicator in a direct and understandable way, and it explicitly displays information about the current patient case mix. Also, because it is not reset, the RA-EWMA is a more natural chart to use in health, where it is exceedingly rare to stop or dramatically and abruptly alter a process of care.

Conclusion The RA-EWMA chart is a direct and intuitive way to display information about an indicator while accounting for differences in case mix.

  • Control charts
  • risk adjustment
  • EWMA
  • RA-EWMA
  • health care quality
  • statistical process control
  • patient outcomes

https://doi.org/10.1136/bmjqs.2008.031831

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Risk-adjusted (RA) control chart schemes are used to sequentially analyse healthcare outcomes where the probability of an adverse event varies with differences in patients' risk profiles. The RA cumulative sum (CUSUM) chart1 is effective in detecting sustained shifts in outcome rate2 and tests sequences of the logarithm of sequential probability ratio tests.3 Unfortunately, this statistic does not have an intuitive or physical interpretation.

    Control charts of cumulative counts of the expected minus observed deaths4 5 are more intuitive than RA-CUSUMs. The variable life adjusted display (VLAD) statistics have been interpreted as “net lives saved”.2 None of the control limits that have been proposed for such charts are directly interpretable in terms of run-length performance.2

    A solution to this problem is a surveillance scheme that combines the VLAD and RA-CUSUM.6 The VLAD plot indicates the trends, and the RA-CUSUM2 signals that the trends are potentially important. Another approach is an EWMA statistic of the observed less expected mortality counts, using Bayesian principles to set an alarm threshold.7

    In this article, we introduce a simple and explicit RA-EWMA scheme that displays the observed and predicted values for an indicator, such as in-hospital mortality, rather than combining them into a plot of the expected less observed outcomes. We provide the details of its design and compare its performance in monitoring the mortality of acute myocardial infarction (AMI) patients admitted to hospital with that of the combined VLAD and RA-CUSUM chart.6

    Methods

    Data and risk model

    Routine administrative data were collected from 13 350 patients admitted for AMI who presented at 116 public or private hospitals in Queensland (2003–2007). The ICD-10-AM codes8 and inclusion criteria used to define the patient population are published on the Queensland Quality Measurement Program website.9

    Our logistic regression model used data for all Queensland hospitals (2003–2004) to estimate the risk-factor parameters for 30-day, in-hospital mortality following admission for AMI. The risk factors (5-year age groups, sex, dysrhythmias, congestive heart failure, hypertension, diabetes, chronic renal failure, dementia, stroke and malignancy) were identified in other studies10 11 and confirmed as predicting short-term mortality for AMI. The Queensland Quality Measurement Program obtained c-statistic values of 0.77 to 0.82 for its similar models.

    RA-EWMA chart

    This article extends EWMA chart methods12 by describing a RA-EWMA that plots EWMA(observed) and EWMA(predicted).EWMA(observed)j=λyj+(1−λ)EWMA(observed)j−1EWMA(predicted)j=λpj+(1−λ)EWMA(predicted)j−1where yj: 1 if died, 0 if survived, indexed j. pj: estimated probability of death, indexed j. λ∈(0,1]: the smoothing parameter.

    Under in-control conditions and for event rates in the range 5–95%, the distributions of EWMA(observed) and EWMA(predicted) are sufficiently similar and are robustly normal and symmetrical. By modifying the formula to compute the variance of a standard EWMA12, the variance of the jth observation is given byVar(EWMAj)=λ2∑k=1j[(1−λ)2(j−k)VarYk]where Var Yk is estimated as pk(1−pk).

    The standard formula12 is used to calculated the upper and lower control limits asCLj=EWMAjpj±Lλ∑k=1j[(1−λ)2(j−k)pk(1−pk)]where the width parameter L the number of standard deviations of EWMA(observed), is chosen so that the “in-control”, average-run-length (ARL0) is suitably long while the “out-of-control”, average run length (ARL1) is suitably short (table 1). The EWMA(observed)0 is the starting estimate of mortality rate and the variance of EWMA(observed)0 is assumed to be zero. If historical data are available, a “run-in period” can be used, providing a starting mortality estimate and an estimate of its variance.

    View this table:
    Table 1

    In-control average run lengths (ARL0) and out-of-control average run lengths (ARL1) for various odds ratios (OR) reflecting mortality higher than the combined-hospital average, λ=0.01

    For a moving average filter, like the RA-EWMA, λ is chosen to effectively attenuate noise in the data and smooth an erratic but unbiased risk model, but to allow the passage of trends without distortion.13 In figure 1, we construct EWMA plots with varying values of λ. For maximum smoothing (λ=0.005), we can discern the trends in the observed outcomes; however, the plot of the expected outcomes barely deviates from its starting value. For λ=0.01, the RA-EWMA trends are clear for observed outcomes and discernable for the expected probabilities. For values of λ>0.01, the EWMA of the observed outcomes becomes more difficult to interpret. We chose λ=0.01 for this application. Our experience with RA monitoring in intensive care is that a choice of λ between 0.005 and 0.020 is needed so the distributions of the observed and predicted EWMA plots are comparable.

    Figure 1
    Figure 1

    Risk-adjusted exponentially weighted moving average (RA-EWMA) plots for different values of λ, in-hospital acute myocardial infarction (AMI) mortality, Hospital A, 2003–2007.

    We estimated ARLs from the start of monitoring to the cross of the upper threshold by simulation using the predicted and observed values from the years 2003 and 2004. Simulations were run 10 000 times for various shifts in the mortality rate (table 1). Based on these simulations, an acceptable trade-off between ARL0 and ARL1 for this application was L=2.07, which gave ARL0=1000 and an acceptable ARL1 for various shifts. For example, a shift in the mortality rate corresponding to an odds ratio of 2.0 gave ARL1=50. Thus, a false alarm would be expected within about 1000 AMI cases if the odds of death were unchanged. However, if the odds of death doubled, then 50 AMI would be admitted, on average, before EWMA(observed) would be greater than the upper threshold of EWMA(predicted).

    Note that the RA EWMA is not reset.

    Variable life adjusted display

    The design procedure for VLAD charts, used in this article, is based on a scheme proposed by Sherlaw-Johnson.6 Because some results in this study suggested that the observed mortality rate continued to be higher than expected after the RA-CUSUM component of the VLAD had signalled and was reset, we added a fast initial reactive feature14 using start and resetting values of h/2. This is recommended when there is a possibility that a process is not fully in control at the start of monitoring.15 We set h=3.3 based on simulations with ARL0=1000 and ARL1=58 (for OR 2.0), similar to those for RA-EWMA, with λ=0.01 and L=2.07 (table 1).

    Results

    The RA-EWMA and VLAD control charts for four Queensland hospitals are shown in figure 2. These were chosen to illustrate the features of the RA-EWMA.

    Figure 2
    Figure 2

    Risk-adjusted exponentially weighted moving average (RA-EWMA) and variable life adjusted display (VLAD) charts, in-hospital acute myocardial infarction (AMI) mortality, four selected hospitals, Queensland, 2003–2007.

    Based on the RA-EWMA chart, Hospital A was in control for almost all of the 5-year period, except for the final month (December 2007). There were no signals on the VLAD chart.

    EWMA(predicted) for Hospital B was consistently above the average mortality rate for all Queensland hospitals (12.8%). This suggests that Hospital B saw patients at higher risk of death than the average and the thresholds around EMWA(predicted) are shifted upwards with EWMA(predicted). Notwithstanding case mix, EWMA(observed) indicates that the observed mortality rate was higher than that predicted over the 5-year period.

    The continuing downward trend in the sum of the predicted–observed values of the VLAD indicated that the observed mortality was higher than expected. However, the RA-CUSUM component of the VLAD only signalled twice in 23 months that the mortality rate was significantly greater than expected. In Hospital B's circumstances, it appears to be more difficult to interpret the VLAD plot as continuous higher than expected mortality compared with the RA-EWMA, which clearly indicates an ongoing lack of improvement.

    Hospital C had predicted values that were lower than the State average. This might be because Hospital C truly treats AMI patients who are less likely to die than the State average. The RA-EWMA chart still signalled, indicating higher than predicted, risk-adjusted mortality. From October 2007 to December 2007, the EWMA(observed) was just above the pooled hospital average of 12.8%; however, because the thresholds are shifted downwards with EWMA(predicted), the observed mortality rate is potentially excessive. The EWMA(predicted) can be thought of as providing a moving centre line that communicates differences in patient case mix.

    Hospital D had predicted values that were higher than the State average. The EWMA(observed) indicated that the mortality rate was lower than the State average for much of the 5-year period. The VLAD chart signalled once.

    Discussion

    EWMA charts are widely used in industry; however, few applications of RA and EWMA in health exist outside of the critical care application.16 17 Benneyan used a RA-EWMA for surgical site infections, standardised for the National Nosocomial Infection Surveillance risk categories.18 Grigg and Spiegelhalther7 described a RA-EWMA scheme to monitor hospital outcomes that differ from ours in several respects. Their chart addresses the “underlying risk for a baseline patient treated”. Their Bayesian perspective uses an underlying baseline risk with a statistical (β) distribution where the distribution's parameters are updated after each observation. The β parameters are derived using constant variance and their RA-EWMA as the estimated mean. The chart plots the RA-EWMA and the bounds of the 95% credible interval (CI) of the distribution of the underlying baseline risk and signals when the CI does not include some trigger value, such as the initial value of the baseline risk.

    Our RA-EWMA explicitly compares EWMA(observed) and EWMA(predicted). Thresholds around the EWMA(predicted), which functions as a moving centre line,16 provide a scale by which to assess the importance of discrepancies between the simultaneous estimates of predicted mortality rate and the observed mortality rate.

    The RA-EWMA chart has advantages over other risk-adjusted charts more commonly used in health, such as the VLAD. The RA-EWMA is simple to design and the parameter choice is straightforward. The vertical axis units are simply mortality. The VLAD y-axis is the cumulative difference between the predicted and observed deaths, and the RA-CUSUM units have no direct interpretation.

    The RA-EWMA displays separate plots of the expected and observed mortality rates that provide a visual display of their differences over time. The explicit display of predicted values allows users at a particular hospital to check the face validity of the risk-adjustment model. For example, if a large academic referral hospital had predicted mortality that was lower than the State average, then users might question the validity of the risk adjustment.19

    One of the benefits claimed for control charts is that the accumulation of evidence occurs over time. This is expunged when the charts like the VLAD and RA-CUSUM are reset. To some extent, this is countered by using a rapid response reset (eg, h/2). Resetting is neither necessary nor desirable with the RA-EWMA, which makes it more suitable for health applications where it is extraordinarily rare to suddenly suspend or profoundly alter the process of care, unlike applications in engineering where a machine can be quickly stopped and recalibrated.

    A disadvantage of the RA-EWMA is that it can be slow to react to an increase in mortality if the EWMA(observed) is close to the threshold (inertia problem),20 although Lucas and Saccucci point out that this is not a common occurrence.20 21 Nevertheless, we guard against inertia by using the RA-EWMA as part of a co-ordinated safety and data analysis structure that includes analysis of index events (mortality review, sentinel events and near misses) and process reviews.

    Routine inpatient data used in this study have limitations of data quality and control of confounding. Consequently, the analyses are best viewed as a starting point for learning about variations in outcomes. They are not hypotheses tests of good or bad quality care but more closely resemble exploratory data analysis.2 Their role is to help understand how a system is performing, diagnose problems and develop theories for possible solutions, using a framework such as the pyramid model of investigation.22 23

    Conclusion

    This article presents a new type of EWMA chart. It has a moving centre line with thresholds to help scale trends in EWMA(observed). It has an intuitive, direct interpretation. It is not reset and is a more natural chart to use in health where it is rare to stop or abruptly alter a process of care. The disadvantage is that it could be slow to identify an abrupt change in outcomes and, consequently, it should not be used in isolation.

    References

    1. ↵
      1. Steiner S,
      2. Cook R,
      3. Farewell V,
      4. et al
      . Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 2000;1:441–52.
      OpenUrlAbstract
    2. ↵
      1. Woodall W
      . The use of control charts in health-care and public-health surveillance. J Qual Technol 2006;38:89–104.
      OpenUrl
    3. ↵
      1. Wald A
      . Sequential analysis. New York: Wiley, 1947.
    4. ↵
      1. Lovegrove J,
      2. Valencia O,
      3. Treasure T,
      4. et al
      . Monitoring the results of cardiac surgery by variable life-adjusted display. Lancet 1997;350:1128–30.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Poloniecki J,
      2. Valencia O,
      3. Littlejohns P
      . Cumulative risk adjusted mortality chart for detecting changes: observational study of heart surgery. BMJ 1998;316:1697–700.
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Sherlaw-Johnson C
      . A method for detecting runs of good and bad clinical outcomes on variable life-adjusted display (VLAD) charts. Health Care Manag Sci 2005;8:61–5.
      OpenUrlCrossRefPubMed
    7. ↵
      1. Grigg O,
      2. Spiegelhalter D
      . A simple risk-adjusted exponentially weighted moving average. J Am Stat Assoc 2007;102:140–52.
      OpenUrlCrossRefWeb of Science
    8. ↵
      National Centre for Classification in Health. International Statistical Classification of Diseases and Related Health Problems, tenth revision, Australian modification (ICD-10-AM), 2009. http://nis-web.fhs.usyd.edu.au/ncch_new/home.aspx.
    9. ↵
      Clinical Practice Improvement Centre. Variable life adjusted display (VLAD), Queensland Health Quality and Safety Program, 2008. http://www.health.qld.gov.au/quality/vlad.asp.
    10. ↵
      1. Alter D,
      2. Naylor C,
      3. Austin P,
      4. et al
      . Effects of socioeconomic status on access to invasive cardiac procedures and on mortality after acute myocardial infarction. N Engl J Med 1999;341:1359–67.
      OpenUrlCrossRefPubMedWeb of Science
    11. ↵
      1. Naylor C,
      2. Slaughter P
      1. Tu J,
      2. Austin P,
      3. Naylor C,
      4. et al
      . Acute myocardial infarction outcomes in Ontario. In: Naylor C, Slaughter P, eds. Cardiovascular health and services in Ontario: an ICES Atlas. Toronto: Institute for Clinical Evaluative Sciences, 1999:83–100.
    12. ↵
      1. Montgomery D
      . Introduction to statistical quality control. New York: John Wiley & Sons, Inc., 2001.
    13. ↵
      1. Brockwell P,
      2. Davis R
      . Introduction to time series and forecasting. New York: Springer, 1996.
    14. ↵
      1. Lucas J,
      2. Crosier R
      . Fast initial response for CUSUM quality control schemes: give your CUSUM a headstart. Technometrics 1982;24:199–205.
      OpenUrlCrossRefWeb of Science
    15. ↵
      1. Grigg O,
      2. Farewell V,
      3. Spiegelhalter D
      . Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Meth Med Res 2003;12:147–70.
      OpenUrlAbstract/FREE Full Text
    16. ↵
      1. Cook D
      . The development of risk adjusted control charts and machine learning models to monitor the mortality rate of intensive care patients. PhD Thesis: University of Queensland, 2004.
    17. ↵
      1. Cook D,
      2. Duke G,
      3. Hart G,
      4. et al
      . Review of the application of risk-adjusted charts to analyse mortality outcomes in critical care. Crit Care Resusc 2008;10:239–51.
      OpenUrlPubMed
    18. ↵
      1. Benneyan J
      . Discussion of “The use of control charts in health-care and public-health surveillance”. J Qual Technol 2006;38:113–23.
      OpenUrl
    19. ↵
      1. Scott I,
      2. Thomson P,
      3. Narasimhan S
      . Comparing risk-prediction methods using administrative or clinical data in assessing excess in-hospital mortality in patients with acute myocardial infarction. Med J Aust 2008;188:332–6.
      OpenUrlPubMedWeb of Science
    20. ↵
      1. Woodall W,
      2. Maragah H
      . Discussion on Lucas and Saccucci: Exponentially weighted moving average control schemes; properties and enhancements. Technometrics 1990;32:17–18.
      OpenUrlCrossRefWeb of Science
    21. ↵
      1. Lucas J,
      2. Saccucci M
      . Exponentially weighted moving average control schemes: properties and enhancements. Technometrics 1990;32:1–12.
      OpenUrlCrossRefWeb of Science
    22. ↵
      1. Lilford R,
      2. Mohammed M,
      3. Spiegelhalter D,
      4. et al
      . Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet 2004;363:1147–54.
      OpenUrlCrossRefPubMedWeb of Science
    23. ↵
      1. Duckett S,
      2. Coory M,
      3. Sketcher-Baker K
      . Identifying variations in quality of care in Queensland hospitals. Med J Aust 2007;187:571–5.
      OpenUrlPubMedWeb of Science

    Footnotes

    • Competing interests None.

    • Provenance and peer review Not commissioned; externally peer reviewed.