Table of Contents

Comparison of the Mantel-Haenszel Method in Different Software

The Mantel-Haenszel method is an approach for fitting meta-analytic fixed-effects models when dealing with studies providing data in the form of 2x2 tables or in the form of event counts (i.e., person-time data) for two groups (Mantel & Haenszel, 1959). The method is particularly advantageous when aggregating a large number of studies with small sample sizes (the so-called sparse data or increasing strata case).

The method is available in the metafor package via the rma.mh() function. By default, the results obtained may differ slightly from those obtained via the metan function in Stata (for more details, see Harris et al., 2008; Sterne, 2009), the Review Manager (RevMan) from the Cochrane Collaboration, or Comprehensive Meta-Analysis (CMA). The reason for such discrepancies is explained further below using an illustrative dataset from a meta-analysis comparing the risk of catheter-related bloodstream infection (CRBSI) when using anti-infective-treated versus standard catheters in the acute care setting (Niel-Weise et al., 2007).

Data Preparation

The data to be used for this example are stored in the dataset dat.nielweise2007:

Variables ai and ci indicate the number of CRBSIs in patients receiving an anti-infective or a standard catheter, respectively, while n1i and n2i indicate the total number of patients in the respective groups. Note that the number of infections was quite low in many studies, with zero cases observed in several of the treatment groups. Also, no cases (infections) were observed in either group in the Yucel (2004) study.

Mantel-Haenszel Method

An analysis of these data using the Mantel-Haenszel method can be carried out with:

Therefore, the odds ratio is estimated to be .299 (with 95% CI: 0.193 to 0.462). In other words, the odds of an infection are estimated to be approximately 70% lower (i.e., $(1 - .299) \times 100%$) in patients receiving an anti-infective-treated catheter instead of a standard catheter. The overall effect is clearly statistically significant (with both the Wald-type z-test and the Cochran-Mantel-Haenszel chi-square test in close agreement). The Q-test for heterogeneity is not significant ($Q(16) = 16.86, p = .39$), although Tarone's test is suggestive of potential heterogeneity.

Results from Stata

The same analysis run in Stata using the metan command (with default settings) yields the following results:

Note that the estimated overall odds ratio (and corresponding CI) is slightly different than the one obtained earlier. Also, the z-test of the overall effect and the chi-square test for heterogeneity are slightly different.

Results from RevMan

After entering the same data into the Review Manager and running the analogous analysis yields the following results:

These results match what is reported by Stata and are again slightly different compared to the results obtained with metafor.

Results from CMA

Finally, the figure below shows the results from Comprehensive Meta-Analysis (CMA). These results match those obtained with Stata and RevMan and differ slightly from those obtained with metafor.

Reason for the Difference

The results differ because studies with zero cases in either group are handled by default in a different way in metafor compared to Stata, RevMan, and CMA. To understand this better, note that the Mantel-Haenszel method itself does not require the calculation of the observed outcomes of the individual studies (in the present example, the observed (log) odds ratios of the $k$ studies) and instead directly makes use of the 2×2 table counts. Zero cells are not a problem (except in some extreme cases, such as when there are zero cases in one or both groups across all of the 2×2 tables). Therefore, it is unnecessary to add some constant to the cell counts of a study with zero cases in either group. However, both Stata, RevMan, and CMA apply an adjustment (often called a continuity correction) to the cell counts in such studies (but studies with zero cases in both groups are dropped/excluded from the method). In particular, 1/2 is added to each of the cells of the 2×2 table in such studies before applying the Mantel-Haenszel method.

By default, metafor uses the same adjustment when calculating the observed outcomes (the observed log odds ratios) of the $k$ studies (here, zero cells can be problematic, so adding a constant value to the cell counts ensures that all $k$ values can be calculated). Also, similarly, studies with zero cases in both groups are automatically dropped/excluded. However, when applying the Mantel-Haenszel method, no adjustment to the cell counts is made, since this is not necessary (and in fact can increase the bias in the Mantel-Haenszel method – see Bradburn et al., 2007).

We can, however, adjust the settings, so that metafor also applies the cell count adjustment, not only when calculating the observed outcomes, but also when carrying out the computations of the Mantel-Haenszel method. For this, we have to adjust the defaults of the add, to, and drop00 arguments (see the documentation of the escalc() and rma.mh() functions for further details). In particular, we could use:

These are the exact same results as obtained with Stata, RevMan, and CMA. However, the results of Bradburn et al. (2007) suggest that the 1/2 adjustment should only be used with caution when applying the Mantel-Haenszel method. Also, alternative correction factors could be considered, which may actually lead to more accurate results (see Sweeting et al., 2004). Finally, the findings by Bradburn et al. (2007) suggest that Peto's method (as implemented in the rma.peto() function) can actually give the least biased results and may be preferrable when events are rare (as long as treatment and control groups are of approximately equal size within trials and the true odds ratio underlying the studies is not very large).