Value of sample size for computation of the Bayesian information criterion (BIC) in multilevel modeling

Abstract

The Bayesian information criterion (BIC) can be useful for model selection within multilevel-modeling studies. However, the formula for the BIC requires a value for sample size, which is unclear in multilevel models, since sample size is observed for at least two levels. In the present study, we used simulated data to evaluate the rate of false positives and the power when the level 1 sample size, the effective sample size, and the level 2 sample size were used as the sample size value, under various levels of sample size and intraclass correlation coefficient values. The results indicated that the appropriate value for sample size depends on the model and test being conducted. On the basis of the scenarios investigated, we recommend using a BIC that has different penalty terms for each level of the model, based on the number of fixed effects at each level and the level-based sample sizes.

Notes

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Tabulated output for proportions of ranks for the true model

Table 1

Proportions of ranks for the true model (overall average)

BIC Type

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

N_tot

.42

.39

.10

.09

N_icc

.47

.37

.09

.07

N_grp

.54

.33

.07

.05

Table 2

Proportions of ranks for the true model (average by level 1 fixed effects)

BIC Type

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

L1_FE = 0

L1_FE = .1

N_tot

.48

.36

.15

.01

.29

.29

.16

.26

N_icc

.52

.30

.15

.02

.35

.32

.13

.20

N_grp

.56

.26

.14

.03

.44

.32

.10

.14

L1_FE = .2

L1_FE = .3

N_tot

.44

.42

.07

.07

.47

.48

.03

.02

N_icc

.49

.40

.05

.06

.52

.44

.02

.02

N_grp

.57

.36

.03

.04

.59

.39

.01

.01

Table 3

Proportions of ranks for the true model (average by level 2 fixed effects)

BIC Type

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

L2_FE = 0

L2_FE = .1

N_tot

.82

.13

.04

.01

.08

.60

.17

.15

N_icc

.84

.12

.04

.01

.12

.61

.15

.12

N_grp

.81

.13

.05

.01

.21

.57

.12

.09

L2_FE = .2

L2_FE = .3

N_tot

.29

.47

.12

.12

.48

.34

.08

.09

N_icc

.37

.44

.10

.10

.57

.30

.07

.07

N_grp

.48

.38

.08

.07

.65

.25

.05

.05

Table 4

Proportions of ranks for the true model (average by ICC)

BIC Type

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

ICC = .1

ICC = .2

N_tot

.55

.30

.08

.07

.45

.36

.10

.09

N_icc

.58

.29

.07

.07

.50

.34

.08

.07

N_grp

.67

.24

.05

.04

.58

.30

.07

.05

ICC = .3

ICC = .5

N_tot

.38

.41

.11

.10

.29

.47

.13

.11

N_icc

.45

.39

.09

.08

.37

.45

.11

.08

N_grp

.51

.36

.08

.06

.40

.43

.10

.07

Table 5

Proportions of ranks for the true model (average by number of groups)

BIC Type

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

# Groups = 10

# Groups = 20

N_tot

.24

.38

.17

.20

.32

.42

.13

.13

N_icc

.28

.40

.15

.16

.37

.41

.11

.10

N_grp

.38

.38

.12

.12

.45

.38

.09

.07

# Groups = 35

# Groups = 50

N_tot

.41

.41

.10

.07

.49

.39

.07

.05

N_icc

.48

.39

.08

.06

.55

.35

.06

.04

N_grp

.54

.35

.07

.04

.60

.32

.05

.03

# Groups = 100

N_tot

.63

.32

.03

.02

N_icc

.68

.27

.03

.01

N_grp

.72

.24

.03

.01

Table 6

Proportions of ranks for the true model (average by group size)

BIC Type

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

Rank True = 1

Rank True = 2

Rank True = 3

Rank True = 4

Group Size = 5

Group Size = 15

N_tot

.32

.32

.16

.20

.42

.37

.11

.10

N_icc

.34

.32

.15

.18

.46

.36

.10

.08

N_grp

.39

.33

.14

.14

.53

.34

.08

.06

Group Size = 25

Group Size = 35

N_tot

.44

.40

.09

.07

.46

.42

.08

.05

N_icc

.50

.37

.07

.05

.52

.38

.06

.03

N_grp

.57

.34

.06

.03

.59

.33

.05

.02

Group Size = 45

N_tot

.46

.43

.07

.04

N_icc

.53

.39

.06

.02

N_grp

.60

.34

.05

.02

Tabulated output for tests of level 1 fixed effects

Table 7

Proportions rejecting H0: L1_FE = 0 (overall average)

BIC Type

L1_FE = 0

L1_FE = .1

L1_FE = .2

L1_FE = .3

N_tot

.014

.536

.874

.961

N_icc

.034

.623

.900

.968

N_grp

.071

.703

.926

.978

Table 8

Proportions rejecting H0: L1_FE = 0 (average by ICC)

BIC Type

ICC = .1

ICC = .2

ICC = .3

ICC = .5

ICC = .1

ICC = .2

ICC = .3

ICC = .5

L1_FE = 0

L1_FE = .1

N_tot

.014

.014

.014

.014

.542

.535

.534

.533

N_icc

.024

.030

.037

.045

.593

.615

.631

.655

N_grp

.073

.069

.072

.070

.708

.702

.704

.698

L1_FE = .2

L1_FE = .3

N_tot

.878

.875

.872

.870

.964

.959

.961

.959

N_icc

.891

.898

.904

.906

.967

.966

.969

.971

N_grp

.929

.926

.928

.921

.980

.979

.978

.976

Table 9

Proportions rejecting H0: L1_FE = 0 (average by number of groups)

BIC Type

# Groups = 10

# Groups = 20

# Groups = 35

# Groups = 50

# Groups = 100

# Groups = 10

# Groups = 20

# Groups = 35

# Groups = 50

# Groups = 100

L1_FE = 0

L1_FE = .1

N_tot

.025

.016

.012

.009

.007

.218

.380

.565

.682

.834

N_icc

.058

.042

.029

.023

.017

.348

.507

.656

.747

.859

N_grp

.130

.086

.061

.047

.033

.493

.615

.728

.795

.884

L1_FE = .2

L1_FE = .3

N_tot

.689

.837

.911

.941

.991

.877

.947

.984

.996

1.000

N_icc

.765

.865

.925

.951

.994

.899

.957

.988

.997

1.000

N_grp

.831

.897

.942

.964

.996

.930

.970

.992

.998

1.000

Table 10

Proportions rejecting H0: L1_FE = 0 (average by group size)

BIC Type

Group Size = 5

Group Size = 15

Group Size = 25

Group Size = 35

Group Size = 45

Group Size = 5

Group Size = 15

Group Size = 25

Group Size = 35

Group Size = 45

L1_FE = 0

L1_FE = .1

N_tot

.027

.015

.011

.009

.008

.180

.431

.594

.701

.774

N_icc

.040

.035

.032

.032

.031

.221

.533

.701

.800

.862

N_grp

.072

.072

.073

.068

.071

.294

.636

.791

.873

.920

L1_FE = .2

L1_FE = .3

N_tot

.580

.872

.948

.978

.991

.829

.978

.997

1.000

1.000

N_icc

.628

.911

.973

.990

.997

.855

.987

.999

1.000

1.000

N_grp

.700

.947

.987

.996

.999

.896

.995

1.000

1.000

1.000

Tabulated output for tests of level 2 fixed effects

Table 11

Proportions rejecting H0: L2_FE = 0 (overall average)

BIC Type

L2_FE = 0

L2_FE = .1

L2_FE = .2

L2_FE = .3

N_tot

.021

.091

.327

.550

N_icc

.037

.138

.408

.635

N_grp

.088

.237

.526

.726

Table 12

Proportions rejecting H0: L2_FE = 0 (average by ICC)

BIC Type

ICC = .1

ICC = .2

ICC = .3

ICC = .5

ICC = .1

ICC = .2

ICC = .3

ICC = .5

L2_FE = 0

L2_FE = .1

N_tot

.021

.019

.020

.022

.171

.093

.064

.038

N_icc

.024

.029

.040

.054

.207

.141

.111

.092

N_grp

.086

.086

.091

.090

.368

.250

.190

.138

L2_FE = .2

L2_FE = .3

N_tot

.574

.380

.245

.107

.805

.644

.496

.256

N_icc

.613

.458

.346

.214

.826

.709

.604

.401

N_grp

.762

.599

.464

.279

.910

.816

.702

.475

Table 13

Proportions rejecting H0: L2_FE = 0 (average by number of groups)

BIC Type

# Groups = 10

# Groups = 20

# Groups = 35

# Groups = 50

# Groups = 100

# Groups = 10

# Groups = 20

# Groups = 35

# Groups = 50

# Groups = 100

L2_FE = 0

L2_FE = .1

N_tot

.047

.024

.013

.011

.008

.068

.060

.075

.090

.164

N_icc

.070

.045

.029

.024

.016

.099

.097

.119

.142

.233

N_grp

.184

.106

.067

.052

.033

.234

.192

.204

.232

.322

L2_FE = .2

L2_FE = .3

N_tot

.138

.192

.293

.394

.615

.259

.390

.561

.675

.866

N_icc

.185

.266

.384

.495

.709

.330

.498

.664

.764

.918

N_grp

.362

.411

.503

.592

.762

.510

.624

.742

.816

.937

Table 14

Proportions rejecting H0: L2_FE = 0 (average by group size)

BIC Type

Group Size = 5

Group Size = 15

Group Size = 25

Group Size = 35

Group Size = 45

Group Size = 5

Group Size = 15

Group Size = 25

Group Size = 35

Group Size = 45

L2_FE = 0

L2_FE = .1

N_tot

.034

.022

.018

.014

.014

.101

.093

.093

.086

.083

N_icc

.042

.039

.035

.033

.036

.118

.136

.144

.145

.146

N_grp

.082

.093

.087

.089

.091

.189

.233

.247

.258

.257

L2_FE = .2

L2_FE = .3

N_tot

.304

.342

.332

.329

.326

.537

.562

.554

.551

.547

N_icc

.338

.408

.424

.434

.435

.572

.634

.650

.654

.665

N_grp

.429

.525

.550

.559

.566

.652

.727

.743

.747

.759

Score equations and concentration rates for the random intercepts model

We work with the model Yij = B0 + B1Xij + B2Zj + uj + eij, where the uj are independent and identically distributed Gaussians with mean 0 and variance τ2 and are independent of eij, which are Gaussians with mean 0 and variance σ2. Here we assume that there are J groups with j = 1, … , J and that i = 1, … , nj for group j. The score equation for estimating B = (B0, B1, B2)T is given by

This is the exactly the equation (without asymptotic approximation) for B1 that is obtained when the design points have been mean centered by group (\( {X}_{ij}\mapsto {X}_{ij}-\overline{X_j} \)) before the analysis is performed. We note that, because this transformation depends on j, the MLE is not invariant to it. However, the MLE for B1 computed without the transformation converges to that computed with the transformation.

The asymptotic concentration rates for asymptotic variances are described by the growth rates of the diagonal terms. The diagonal term corresponding to B0 grows at a rate that is bounded above by J/τ2. Similarly, the diagonal term corresponding to B2 grows at a rate that is bounded above by \( \sum {Z}_j^2/{\tau}^2 \). Assuming that \( \sum {Z}_j^2/J \) and the MLE for τ2 converge in probability to finite, positive numbers as J increases, the appropriate penalty for B0 and B2 in the BIC is log(J). In contrast, the diagonal term corresponding to B1 is bounded below by \( \sum \frac{n_j}{\sigma^2}\left(\overline{X_j^2}-{\overline{X_j}}^2\right) \) and above by \( \sum \frac{n_j}{\sigma^2}\overline{X_j^2} \). Assuming that \( \sum \left(\overline{X_j^2}-{\overline{X_j}}^2\right)/J \), \( \sum \overline{X_j^2}/J \), and the MLE for σ2 converge in probability to finite, positive numbers as J increases, then the appropriate penalty for B2 in the BIC is \( \mathit{\log}\left({\sum}_{j=1}^J{n}_j\right) \).