Power computations in time series analyses for traffic safety interventions

Abstract

The evaluation of traffic safety interventions or other policies that can affect road safety often requires the collection of administrative time series data, such as monthly motor vehicle collision data that may be difficult and/or expensive to collect. Furthermore, since policy decisions may be based on the results found from the intervention analysis of the policy, it is important to ensure that the statistical tests have enough power, that is, that we have collected enough time series data both before and after the intervention so that a meaningful change in the series will likely be detected. In this short paper, we present a simple methodology for doing this. It is expected that the methodology presented will be useful for sample size determination in a wide variety of traffic safety intervention analysis applications. Our method is illustrated with a proposed traffic safety study that was funded by NIH.

Introduction

Intervention analysis (Box and Tiao, 1976, Cook and Campbell, 1979, Chapter 6; Hipel and McLeod, 1994, Chapter 19) provides a statistical method for quantifying the effect on known interventions on a time series. It has been one of the most commonly used statistical procedures to evaluate traffic safety interventions or other policies that can affect road safety (e.g. Abdel-Aty and Abdelwahab, 2004, Blose and Holder, 1987, Elder et al., 2004, Elder et al., 2002, Gruenewald and Ponicki, 1995, Hagge and Romanowicz, 1996, Hingson et al., 2000, Holder and Wagenaar, 1994, Reeder et al., 1999, Holder et al., 2000, Langley et al., 1996, Mayhew et al., 2001, Murry et al., 1993, Nathens et al., 2000, Vernon et al., 2004, Voas et al., 1997). Indeed, various applied researchers have written specifically on the value of using time series analyses in intervention or evaluation research (Biglan et al., 2000, Gruenewald, 1997, Rehm and Gmel, 2001).

Often applied researchers are under pressure to evaluate the impact of a traffic safety intervention soon after its implementation. In these time series analyses, sample size may be limited. For example, Vingilis and Salutin (1980) evaluated the impact of a spot-check enforcement program called R.I.D.E. (Reduce Impaired Driving in Etobicoke). This program was a 1-year pilot project and sponsors of the program wanted an evaluation of the impact at the end of 1 year. The evaluation used a mixed-methods, multi-measures evaluation methodology. Among their measures was the time series analysis of collisions for 2 years prior to the R.I.D.E. program and for the 1-year intervention, for a total of 36 data points – 24 months pre-intervention and 12 months post-intervention. The results of the time series analyses found a near significant downward trend at p ≈ 10% for alcohol-related collisions for the intervention area. The study concluded: “The ultimate change in alcohol-related accidents might not be in evidence because of contaminations mentioned previously or because of the need for a longer time period” (Vingilis and Salutin, 1980, p. 274). This study is an example of how difficult it is to interpret a near significant finding as one cannot discern whether the program had little impact or whether statistical power was at issue. Thus, it is important to know whether a proposed time series analysis has a chance of detecting a meaningful change in the system.

Power is the statistical term used for the probability that a test will reject the null hypothesis of no change at level α for a prescribed change. As the statistical literature contained no power computation methods for use with time series analyses, a method was developed for computing the power function for the general case of intervention analysis (McLeod and Vingilis, 2005). When planning a future study only limited information is usually available and so the approach described in the next section is often useful for determining approximate sample sizes, that is, of the lengths of the pre-intervention and post-intervention series needed to ensure that the analysis can detect meaningful change caused by the intervention.

It is important to understand that it is only statistically meaningful if the power computations are carried out before the data are analyzed (Lenth, 2001, Verrill and Durst, 2005). The article of Verrill and Durst (2005) demonstrates that as the sample size increases, the power rapidly approaches one. This result underlines the importance of power computations in planning the collection of data.

Van Belle (2002, Chapter 2) gives advice and rules of thumb for sample size computations for non-time series statistical tests and Lenth (2006) provides an online power calculator for many sorts of statistical tests. Our online calculator (McLeod, 2007) implements the method described in this note.