Significance Tests

Dr. Simon Müller

Event studies are oftentimes the first step in a sequence of analyses that aims at identifying the determinants of stock market responses to distinct event types. Event studies yield as an outcome abnormal returns (ARs), which are cumulated over time to cumulative abnormal returns (CARs) and then 'averaged' - in the case of so called sample studies - over several observations of identical events to AARs and CAARs - where the second 'A' stands for 'average'. These event study results are then oftentimes used as dependent variables in regression analyses.

Besides this type of analysis, event studies are traditionally also performed to specify if the abnormal effects pertaining to individual events or samples of events are significantly different from zero, and thus not the result of pure chance. This assessment wil be made by hypothesis testing.

Following general principles of inferential statistics, the null hypothesis ($H_0$) thus maintains that there are no abnormal returns within the event window, whereas the alternative hypothesis ($H_1$) suggests the presence of ARs within the event window. Formally, the testing framework reads as follows:

\begin{equation}H_0: μ = 0 \end{equation}

\begin{equation}H_1: μ \neq 0 \end{equation}

Event studies may imply a hierarchy of calculations, with ARs being compounded to CARs, which can again be 'averaged' to CAAR in cross-sectional studies (sometimes also called 'sample studies'). In long-run Event Studies instead of CAR the Buy-and-Hold abnormal return (BHAR) is widely use. BHAR can alos be 'averaged' to ABHAR in cross-sectional studies. There is a need for significance testing at each of these levels. μ in the above mentioned equations may thus represent ARs, CARs, BHARs, AARs, CAARs, and ABHARs. Let's shortly revisit these six different forms of abnormal return calculations, as presented in the introduction and long-run event-studies:

and $S^2_{AR_{i,t}}$ is the forecast-error corrected standard deviation defined in EQ $\ref{EQ:FESD}$.

The literature on event study test statistics is very rich, as is the range of significance tests. Generally, significance tests can be grouped in parametric and nonparametric tests (NPTs). Parametric tests assume that individual firm's abnormal returns are normally distributed, whereas nonparametric tests do not rely on any such assumptions. In research, scholars commonly complement a parametric test with a nonparametric tests to verify that the research findings are not due to eg. an outlier (see Schipper and Smith (1983) for an example). Table 1 provides an overview and links to the formulas of the different test statistics.

Parametric test statistics ground on the classic t-test. Yet, scholars have further developed the test to correct for the t-test's prediction error. The most widely used of these 'scaled' tests are those developed by Patell (1976) and Boehmer, Musumeci and Poulsen (1991). Among the nonparametric tests, the rank-test of Corrado (1989), and the sign-based of Cowan (1992) are very popular. EST provides these test statistics (soon) in its analysis results reports.

Why different test statistics are needed

The choice of test statistic should be informed by the research setting and the statistical issues the analyzed data holds. Specifically, event-date clustering poses a problem leading to (1) cross-sectional correlation of abnormal returns, and (2) distortions from event-induced volatility changes. Cross-sectional correlation arises when sample studies focus on (an) event(s) which happened for multiple firms at the same day(s). Event-induced volatility changes, instead, is a phenomenon common to many event types (e.g., M&A transactions) that becomes problematic when events are clustered. As consequence, both issues introduce a downward bias in the standard deviation and thus overstate the t-statistic, leading to an over-rejection of the null hypothesis.

Comparison of test statistics

There have been several attempts to address these statistical issues. Patell (1976, 1979), for example, tried to overcome the t-test's proneness to event-induced volatility by standardizing the event window's ARs. He used the dispersion of the estimation interval's ARs to limit the impact of stocks with high return standard deviations. Yet, the test too often rejects the true null hypothesis, particularly when samples are characterized by non-normal returns, low prices or little liquidity. Also, the test has been found to be still affected by event-induced volatility changes (Campbell and Wasley, 1993; Cowan and Sergeant, 1996; Maynes and Rumsey, 1993, Kolari and Pynnonen, 2010). Boehmer, Musumeci and Poulsen (1991) resolved this latter issue and developed a test statistic robust against volatility-changing events. Furthermore, the simulation study of Kolari and Pynnonen (2010) indicates an over-rejection of the null hypothesis for both the Patell and the BMP test, if cross-sectional correlation is ignored. Kolari and Pynnonen (2010) developed an adjusted version for both test statistics that accounts for cross-sectional correlation.

The nonparametric rank test of Corrado and Zivney (1992) (RANK) applies re-standardized event window returns and has proven robust against induced volatility and cross-correlation. Sign tests are another category of tests. One advantage the tests’ authors stress over the common t-test is that they are apt to also identify small levels of abnormal returns. Moreover, scholars have recommend the used of nonparametric sign and rank tests for applications that require robustness against non-normally distributed data. Past research (e.g. Fama, 1976) has argued that daily return distributions are more fat-tailed (exhibit very large skewness or kurtosis) than normal distributions, what suggests the use of nonparametric tests.

Several authors have further advanced the sign and ranked tests pioneered by Cowan (1992) and Corrado and Zivney (1992). Campbell and Wasley (1993), for example, improved the RANK test by introducing an incremental bias into the standard error for longer CARs, creating the Campbell-Wasley test statistic (CUM-RANK). Another NPT is the generalized rank test (GRANK) test with a Student t-distribution with T-2 degrees of freedom (T is the number of observations). It seems that GRANK is one of the most powerful instruments for both shorter and longer CAR-windows.

The Cowan (1992) sign test (SIGN) is also used for testing CARs by comparing the share of positive ARs close to an event to the proportion from a normal period. SIGN's null hypothesis includes the possibility of asymmetric return distribution. Because this test considers only the sign of the difference between abnormal returns, associated volatility does not influence in any way its rejection rates. Thus, in the presence of induced volatility scholars recommend the use of BMP, GRANK, SIGN.

Most studies have shown that if the focus is only on single day ARs, the means of all tests stick close to zero. In the case of longer event windows, however, the mean values deviate from zero. Compared to their nonparametric counterparts, the Patell and the BMP-tests produce means that deviate quite fast from zero, whereas the standard deviations of all tests gravitate towards zero. For longer event windows, academics recommend nonparametric over parametric tests.

Therefore, the main idea is that in case of longer event-windows, the conclusions on the tests power should be very carefully drawn because of the many over- or under-rejections of the null hypothesis. Overall, comparing the different test statistics yields the following insights (see Table 2 for further details):

Parametric tests based on scaled abnormal returns perform better than those based on non-standardized returns

Generally, nonparametric tests tend to be more powerful than parametric tests

The generalized rank test (GRANK) is one of the most powerful test for both shorter CAR-windows and longer periods

Insights about strenghts and weaknesses were compiled from Kolari and Pynnonen (2011)

Formulas, acronyms, and the decision rule applicable to all test statistics

Let $L_1 = T_1 - T_0 + 1$ the estimation window length with $T_0$ as the 'earliest' day of the estimation window, and $T_1$ the 'latest' day of the estimation window relative to the event day and $L_2 = T_2 - T_1$ the event window length with $T_2$ as the 'latest day' of the event window relative to the event day. Define $N$ as the sample size (i.e. number of events / observations); $S_{AR_i}$ represent the standard deviation as produced by the regression analysis over the estimation window according to the following formula

The Patell test is a widely used test statistic in Event Studies. In the first step Patell (1976, 1979) suggested to standardize each $AR_i$ before calculating the test statistic by the forecast-error corrected standard deviation.

with $\bar{R}_{m}$ as the mean of the market returns in the estimation window. $SAR_{i,t}$ is distributed as a t-distribution with ${M_i-2}$ degrees of freedom under the Null. Test statistic for testing $H_0: AAR = 0$ is then given by

$$z_{Patell, t} = \frac{ASAR_t}{S_{ASAR_t}},$$

where $ASAR_t$ is the sum over the sample of the standardized abnormal returns

Kolari and Pynnönen (2010) propose a modification to the Patell-test to account for cross-correlation of the abnormal returns. Using the standardized abnormal returns ($SAR_{i,t}$) defined as in (EQ: $\ref{eq:sar}$), and defining $\bar r$as the average of the sample cross-correlation of the estimation period abnormal returns, the test statistic for $H_0: AAR = 0$ of the adjusted Patell-test is

where $z_{patell, t}$ is the Patell test statistic. It is easily seen that if the correlation $\overline r$ is zero, the adjusted test statistic reduces to the original Patell test statistic. Assuming the square-root rule holds for the standard deviation of different return periods, this test can be used when considering Cumulated Abnormal Returns ($H_0: CAAR = 0$):

$$z_{Patell}=z_{Patell} \sqrt{\frac{1}{1 + (N - 1) \overline r}}.$$

[6] Standardized Cross-Sectional or BMP Test (Abbr.: StdCSect Z)

Similarly, Boehmer, Musumeci and Poulsen (1991) proposed a standardized cross-sectional method which is robust to the variance induced by the event. Test statistics on day $t$ ($H_0: AAR = 0$) in the event window is given by

$$z_{BMP, t}= \frac{ASAR_t}{\sqrt{N}S_{ASAR_t}},$$

with $ASAR_t$ defined as for Patell-test [2] and with standard deviation

and $SCAR_i = \frac{CAR_i}{S_{CAR_i}}$. $S_{CAR_i}$ is the forecast error corrected standard deviation from Mikkelson and Partch (1988). The Mikkelson and Partch correction adjusts for each firm the test statistic for serial correlation in the returns. The correction terms are

where $L_i$ is the count of non-missing return values in the event window and $M_i$ is the count of non-missing return values in the estimation window for firm $i$. $\bar{R}_{m}$ is the mean of the market returns in the estimation window, see e.g. Patell Test.

Kolari and Pynnönen (2010) propose a modification to the BMP-test to account for cross-correlation of the abnormal returns. Using the standardized abnormal returns ($SAR_{i,t}$) defined as in the previous section, and defining $\bar r$as the average of the sample cross-correlation of the estimation period abnormal returns, the test statistic for $H_0: AAR = 0$ of the adjusted BMP-test is

where $z_{bmp, t}$ is the BMP test statistic. It is easily seen that if the correlation $\overline r$ is zero, the adjusted test statistic reduces to the original BMP test statistic. Assuming the square-root rule holds for the standard deviation of different return periods, this test can be used when considering Cumulated Abnormal Returns ($H_0: CAAR = 0$):

which is asymptotically standard normal distributed. For a further discussion on skewness transformation we refer to Hall (1992) and for further discussion on unbiased estimation of the second and third moment we refer to Cramer (1961) or Rimoldini (2013).

In a first step, the Corrado's (1989) rank test transforms abnormal returns into ranks. Ranking is done for all abnormal returns of both the event and the estimation period. If ranks are tied, the midrank is used. For adjusting on missing values Corrado and Zyvney (1992) suggested a standardization of the ranks by the number of non-missing values $M_i$ plus 1

$$K_{i, t}=\frac{rank(AR_{i, t})}{1 + M_i + L_i} $$,

where $L_i$ refers to the number of non-missing (i.e., matched) returns in event window. The rank statistic for testing on a single day ($H_0: AAR = 0$) is then given by

$$t_{rank, t} = \frac{\overline{K}_t - 0.5}{S_{\overline{K}}},$$

where $\overline{K}_t = \frac{1}{N_t}\sum\limits_{i=1}^{N_t}K_{i, t}$, $N_t$ is the number of non-missing returns across firms, and

where $\overline{K}_{T_1, T_2} = \frac{1}{L_2} \sum\limits_{t=T_1 + 1}^{T_2}\overline{K}_t$ is the mean rank across firms and time in event window. By adjusting the last day in the event window $T_2$, one can get a series of test statistics as definded by Campell and Wasley (1993).

Note 1: The adjustment for event induced variance as done by Campell and Wasley (1993) is omitted here and may be implemented in a future version. In such a case, we recommend the GRANK-T or GRANK-Z test.

[11] Generalized Rank T Test (Abbr.: Generalized Rank T)

In following steps we assume for sake of simplicity that there are no missing values is estimation and event window for each firm. In order to account for possible event-induced volatility, the GRANK test squeezes the whole event window into one observation, the so-called 'cumulative event day'. First, define the standardized cumulative abnormal returns of firm $i$ in the event window

$$SCAR_{i}=\frac{CAR_{i}}{S_{CAR_{i}}},$$

where $S_{CAR_{i}}$ is the standard deviation of the prediction errors in the cumulative abnormal returns of firm $i$, namely

The standardized CAR value $SCAR_{i}$ has an expectation of zero and approximately unit variance. To account for event-induced volatility $S_{CAR_{i}}$ is re-standardized by the cross-sectional standard deviation

which converges under null hypothesis fast to the standard normal distribution as the firms $N$ increase.

[13] Sign Test (Abbr.: not available in our API)

This sign test has been proposed by Cowan (1991) and builds on the ratio of positive cumulative abnormal returns $\hat{p}$ present in the event window. Under the null hypothesis, this ratio should not significantly differ from 0.5.

Under the Null Hypothesis of no abnormal returns, the number of stocks with positive abnormal cumulative returns ($CAR$) is expected to be in line with the fraction $\hat{p}$ of positive $CAR$ from the estimation period. When the number of positive $CAR$ is significantly higher than the number expected from the estimated fraction, it is suggested to reject the Null Hypothesis.

where $\varphi_{i,t}$ is $1$ if the sign is positive and $0$ otherwise. The Generalized sign test statistic ($H_0: CAAR = 0$) is

$$z_{gsign}=\frac{(w-N\hat{p})}{\sqrt{N\hat{p}(1-\hat{p})}},$$

where $w$ is the number of stocks with positive cumulative abnormal returns during the event period. For the test statistic, a normal approximation of the binomial distribution with the parameters $\hat{p}$ and $N$, is used.

Dr. Simon Müller studied mathematics and technical mechanics at the University of Stuttgart, Germany. He holds a Ph. D. in mathematics from the University of Stuttgart. After his Ph.D. thesis Simon worked as a Postdoc at the Dr. Margarete Fischer-Bosch Institute of Clinical Pharmacology in Stuttgart. Since 2012 he works as an independent statistical consultant. He is an expert R programmer and has working knowledge on statistic software SAS Base/Stat/Graph, and SPSS Statistics.