Meta-Analysis

You can summarize results from multiple studies with Stata's new meta-analysis suite. Use random-effects, fixed-effects, or common-effect meta-analysis to combine individual results and compute overall effect size. Forest plots allow you to visualize the results. With subgroup analysis or meta-regression, you can explore heterogeneity of studies. And you can evaluate publication bias using funnel plots and the trim-and-fill method.
Meta-analysis helps aggregate the information, often overwhelming, from many studies in a principled way into one unified final conclusion or provides the reason why such a conclusion cannot be reached.

Example dataset: Effects of teacher expectancy on pupil IQ

To demonstrate the meta suite, we use the famous example from
Raudenbush of the meta-analysis of 19 studies that evaluated the
effects of teacher expectancy on pupil IQ. In their original study,
Rosenthal and Jacobson discovered the so-called Pygmalion effect, in which
expectations of teachers affected outcomes of their students.

The goal of the experiment was to investigate whether the identification of
the randomly selected group of students (experimental group) to teachers as
"likely to show dramatic intellectual growth'' would influence teachers'
expectations for these students. The authors found a statistically significant
effect between the experimental and control groups with respect to students'
IQ scores.

Later studies attempted to replicate the results, but many did not find the
hypothesized effect. Raudenbush
suspected that the Pygmalion effect
might be mitigated by how long the teachers had worked with the students
before the experiment.

We load the dataset and describe some of its variables below.

. describe studylbl stdmdiff se week1

storage display value

variable name type format label variable label

studylbl str26 %26s Study label

stdmdiff double %9.0g Standardized difference in means

se double %10.0g Standard error of stdmdiff

week1 byte %9.0g catweek1 Prior teacher-student contact > 1 week

stdmdiff and se record precomputed effect sizes, standardized
mean differences between the experimental and control groups, and their
standard errors.

studylbl contains study labels, which include the authors and
publication years.

weeks records the number of weeks of prior contact between the teacher
and the students. week1 is the dichotomized version of weeks,
which records the high-contact (week1=1) and low-contact
(week1=0) groups.

Prepare your data for meta-analysis

Declaring the meta-analysis data is the first step of your meta-analysis in
Stata. During this step, you specify the main information needed for
meta-analysis such as the study-specific effect sizes and their standard
errors. You declare this information once by using either
meta set or meta esize, and it is then used by all
meta commands. The declaration step helps minimize potential mistakes and typing; see

Let's declare our pupil IQ data. Our dataset contains
already calculated effect sizes (stdmdiff) and their standard errors
(se), so we use meta set for
declaration. If you have study-specific summary data and want to compute
effect sizes.

. meta set stdmdiff se, studylabel(studylbl) eslabel(Std. Mean Diff.)

Meta-analysis setting information

Study information

No. of studies: 19

Study label: studylbl

Study size: N/A

Effect size

Type: Generic

Label: Std. Mean Diff.

Variable: stdmdiff

Precision

Std. Err.: se

CI: [_meta_cil, _meta_ciu]

CI level: 95%

Model and method

Model: Random-effects

Method: REML

We also specified how we want the meta commands to label studies and
effect sizes in the output.

After the declaration, you can use meta query or meta update to
describe or update your current meta settings at any point of your
meta-analysis.

Both commands include the information about study-specific effect sizes and
their CIs, in addition to the estimate of the overall effect size and its CI.
For instance, from the plot, the estimated overall standardized mean
difference is 0.08 with a 95% CI of [-0.02, 0.18].

Various heterogeneity measures and tests are also reported; we explore them below in
Heterogeneity.

Heterogeneity

In meta-analysis, heterogeneity occurs when variation between the study
effect sizes cannot be explained by sampling variability alone.
meta summarize and meta forestplot
report basic heterogeneity measures and the homogeneity test to assess
the presence of heterogeneity.

When there are study-level covariates, also known as moderators, that may
explain some of the between-study variability, heterogeneity can be explored
further via subgroup analysis and, more generally, via meta-regression.
Subgroup analysis is used with categorical moderators, and meta-regression
is used when at least one of the moderators is continuous.

Summary measures and homogeneity test

Consider the forest plot we produced in Meta-analysis summary.

The between-study variation of the effect sizes is evident from the forest
plot. The reported heterogeneity statistics indicate the presence of
heterogeneity in these data. For instance, I² is estimated to be 41.84%,
which, according to Higgins et al. (2003), indicates the presence of
"medium heterogeneity".

The test of homogeneity of study-specific effect sizes is also rejected, with
a chi-squared test statistic of 35.83 and a p-value of 0.01.

Subgroup analysis

Subgroup analysis is used when study effect
sizes are expected to be more homogeneous within certain groups. The grouping
variables can be specified in option subgroup() supported by
meta summarize and meta forestplot.

In Summary measures and homogeneity test, we established the presence of
heterogeneity between the study results. As we said in Example dataset:
Effects of teacher expectancy on pupil IQ, it was suspected that the amount of
contact between the teachers and students before the experiment may explain
some of the between-study variability.

Let's first consider the binary variable week1 that divides the
studies into the high-contact (week1=1) and low-contact
(week1=0) groups. (Below Meta-regression, we explore the impact of
continuous weeks on the effect sizes.)

For categorical variables, we can perform subgroup analysis—separate
meta-analysis for each group—to explore heterogeneity between the groups.

. meta forestplot, subgroup(week1)

Effect-size label: Std. Mean Diff.

Effect size: stdmdiff

Std. Err.: se

Study label: studylbl

Option subgroup() can be used with meta forestplot and meta
summarize to perform subgroup analysis. In our example, we specified only
one grouping variable, week1, but you can include more, provided
you have a sufficient number of studies per group.

After stratifying on the contact group, the results appear to be more
homogeneous, particularly within the high-contact group (> 1 week). The
test of no differences between the groups, reported at the bottom of the graph, is rejected
with a chi-squared test statistic of 14.77 and a p-value less than
0.01.

Meta-regression

Meta-regression is often used to explore heterogeneity induced by the
relationship between moderators and study effect sizes. Moderators may include
a mixture of continuous and categorical variables. In Stata, you perform
meta-regression by using meta regress.

Continuing with our heterogeneity analysis, let's use meta-regression to explore
the relationship between study-specific effect sizes and the amount of prior
teacher–student contact (weeks).

There is a statistically significant negative relationship between the
magnitudes of the effect sizes and the number of weeks of prior contact: the
more time teachers spent with students before the experiment, the smaller the
estimated effect size.

After accounting for weeks, we find that the remaining between-study residual
heterogeneity is roughly 30%.

Postestimation: Bubble plot

Continuing with Meta-regression, we can produce a bubble plot
after meta-regression with one continuous covariate to explore the
relationship between the effect sizes and the covariate.

. estat bubbleplot

The standardized mean difference decreases as the number of weeks of prior
teacher–student contact increases. There are also several outlying studies in
the region where weeks is less than roughly 3 weeks. The size of the
bubbles represents the precision of the studies. Some of the outlying studies
also appear to be among the more precise studies.

Small-study effects and publication bias

The term "small-study effects'' refers to situations where the effects of
smaller studies differ systematically from the effects of larger studies. For
instance, smaller studies may report larger effect sizes than larger studies.
Two common reasons for the presence of small-study effects are between-study
heterogeneity and publication bias.

Publication bias arises when the decision of whether to publish a study's
results depends on the significance of the obtained results. Often, smaller
studies with nonsignificant findings are suppressed from publication. This
may lead to a biased sample of studies in a meta-analysis, which is often
collected from the published studies.

The meta suite provides three commands you can use to explore
small-study effects and publication bias.

Meta funnelplot produces standard and contour-enhanced funnel plots,
which can be used to explore small-study effects and publication bias visually.

Meta bias provides several statistical tests for small-study
effects, also known as tests for funnel-plot asymmetry.

Standard and contour-enhanced funnel plots

To demonstrate, let's produce a funnel plot for the pupil IQ data.

. meta funnelplot

Effect-size label: Std. Mean Diff.

Effect size: stdmdiff

Std. Err.: se

Model: Common-effect

Method: Inverse-variance

In the absence of publication bias and, more generally, small-study effects,
the funnel plot should resemble a symmetric inverted funnel. In our example,
it appears that a few points (studies) are missing in the lower left portion
of the funnel plot, which makes it look asymmetric.

Recall, however, that in our earlier heterogeneity analysis,
we established the presence of between-study variability. Thus,
this may be one of the reasons for the asymmetry of the funnel plot.

Contour-enhanced funnel plots are often used to explore whether the
funnel-plot asymmetry is due to publication bias or perhaps some other
factors. Let's add the 1%, 5%, and 10% significance contours to our funnel
plot.

. meta funnelplot, contours(1 5 10)

Effect-size label: Std. Mean Diff.

Effect size: stdmdiff

Std. Err.: se

Model: Common-effect

Method: Inverse-variance

Based on the contour-enhanced funnel plot, it appears that we are missing
a few smaller studies that fall both in the significant and nonsignificant
regions of the funnel plot. Under publication bias, we are likely to see
missing smaller studies only in the nonsignificant regions. So, perhaps, the
funnel-plot asymmetry in our example is due to some other reason such as
heterogeneity.

In fact, the meta-analysis literature recommends that the heterogeneity be
addressed before the exploration of the publication bias. For instance, in our
example, we can produce funnel plots separately for each contact group.

. meta funnelplot, by(week1)

Effect-size label: Std. Mean Diff.

Effect size: stdmdiff

Std. Err.: se

Model: Common-effect

Method: Inverse-variance

Within each contact group, funnel plots look more symmetric.

Tests for funnel-plot asymmetry

Continuing with Standard and contour-enhanced funnel plots,
we can use one of the statistical tests to test formally
for the funnel-plot asymmetry. These are also known as tests for small-study
effects.

Let's use the Egger regression-based test to test for the funnel-plot
asymmetry in the pupil IQ data.

. meta bias, egger

Effect-size label: Std. Mean Diff.

Effect size: stdmdiff

Std. Err.: se

Regression-based Egger test for small-study effects

Random-effects model

Method: REML

H0: beta1 = 0; no small-study effects

beta1 = 1.83

SE of beta1 = 0.724

z = 2.53

Prob > |z| = 0.0115

The null hypothesis of no small-study effects or, equivalently, of the
symmetry of the funnel plot is rejected at the 5% significance level with a
z statistic of 2.53 and a p-value of 0.0115.

But, if we account for the between-study heterogeneity due to week1,
the results of the test are no longer statistically significant.

. meta bias week1, egger

Effect-size label: Std. Mean Diff.

Effect size: stdmdiff

Std. Err.: se

Regression-based Egger test for small-study effects

Random-effects model

Method: REML

Moderators: week1

H0: beta1 = 0; no small-study effects

beta1 = 0.30

SE of beta1 = 0.729

z = 0.41

Prob > |z| = 0.6839

Trim-and-fill analysis

In the presence of publication bias, it is useful to explore its impact on
the meta-analysis results. One way to do this is to perform trim-and-fill
analysis.

In Standard and contour-enhanced funnel plots and Tests for funnel-plot asymmetry,
we detected the asymmetry of the funnel plot but commented that this may be
because of heterogeneity rather than publication bias. In fact, the
contour-enhanced funnel plot suggested that the asymmetry is likely not
because of publication bias. But, for the purpose of this demonstration, let's
go ahead and pretend that the observed asymmetry in the funnel plot is induced
by publication bias and that we want to explore its impact on our
meta-analysis results.

meta trimfill estimated the number of studies missing presumably due to
publication bias to be 3, imputed the omitted studies, and reported additional
results using both the observed and imputed studies. With the imputed
studies, the overall effect-size estimate is reduced from 0.084 to 0.028 with
a wider 95% CI.

We also specified option funnel to produce the funnel plot that
includes the omitted studies. The imputed studies make the funnel plot look
more symmetric and identify the areas where studies are missing.

Given the presence of heterogeneity, however, we should have addressed it
first before the trim-and-fill analysis. For instance, we could have run
meta trimfill separately for low-contact and high-contact groups.

Cumulative meta-analysis

In Meta-regression, we established that there is a negative association between
the magnitudes of effect sizes and the amount of prior teacher–student
contact (weeks). We can perform cumulative meta-analysis to explore the
trend in the effect sizes as a function of weeks. We display the results
as a forest plot.

. meta forestplot, cumulative(weeks)

Effect-size label: Std. Mean Diff.

Effect size: stdmdiff

Std. Err.: se

Study label: studylbl

We specified weeks in meta forestplot's option
cumulative() to perform cumulative meta-analysis with weeks as
the ordering variable. This option is also supported by meta summarize.

The studies are first ordered with respect to weeks, from smallest to
largest amount of contact. Then, separate meta-analyses are performed by
adding one study at a time. That is, the first result of the cumulative forest
plot corresponds to the effect size and its CI from the first study. The
second result corresponds to the overall effect size and its CI from the
meta-analysis of the first two studies. And so on. The last result
corresponds to the standard meta-analysis using all studies.

As the number of weeks increases, the overall standardized mean difference
and its significance (p-value) decreases.