Hierarchical two-parameter logistic item response model

Daniel C. Furr

April 21, 2016

1 Model

1.1 Overview

The two-parameter logistic model (2PL) is an item response theory model that includes parameters for both the difficulty and discrimination of items. A hierarchical extension, presented here, models these item parameter pairs as correlated draws from a bivariate normal distribution. This model is similar to the hierarchical three-parameter logistic model proposed by Glas and van der Linden (2003).

\(\Sigma\) is the covariance matrix for \(\log \alpha_i\) and \(\beta_i\)

Priors:

\(\mu_1 \sim \mathrm{N}(0,1)\) is a weakly informative prior for the mean of the log discrimination parameters.

\(\mu_2 \sim \mathrm{N}(0,25)\) is a weakly informative prior for the mean of the difficulty parameters.

Let \(\tau_1^2 = \Sigma_{1,1}\) be the variance of the log discrimination parameters. Then \(\tau \sim \mathrm{Exp}(.1)\) is a weakly informative prior for the standard deviation.

Let \(\tau_2^2 = \Sigma_{2,2}\) be the variance of the difficulty parameters. Then \(\tau_2 \sim \mathrm{Exp}(.1)\) is a weakly informative prior for the standard deviation.

A weakly informative prior is placed on the covariance, \(\Sigma_{1,2} = \Sigma_{2,1}\), shrinking the posterior towards zero. This is described in more detail in the next section.

1.2Stan program

The Stan program for the model is provided below. It differs from the notation above in that it is written in terms of the Cholesky decomposition of the correlation matrix for better efficiency. This matrix is named L_Omega; Omega because it is the correlation rather than covariance matrix, and L because it is a Cholesky decomposition. The vector tau contains the standard deviations that would be the (square root of the) diagonal of the covariance matrix. (The standard deviations are invariant whether or not the the Cholesky decomposition is used.) In the model block, L_Omega is converted to its covariance equivalent L_Sigma, and item parameter pairs xi[i] are sampled from L_Sigma. The first element of vector xi[i] is the log discrimination for item i, and the second is the difficulty for item i.

Weakly informative normal priors are placed on mu, and weakly informative truncated normal priors are placed on tau. The prior placed on L_Omega using lkj_corr_cholesky() is weakly informative and slightly favors a correlation of zero. See the Stan manual for details.

While more efficient, parameters L_Omega and xi are difficult to interpret. To alleviate this inconvenience, item parameters alpha and beta are derived from xi in the transformed parameters block. (There is some redundancy, as beta[i] and xi[i,2] are equal.) Also, L_Omega is converted to a standard correlation matrix, Omega, in the generated quantities block.

2 Simulation

The R code that follows simulates a dataset conforming to the model. The Stan model will be evaluated in terms of its ability to recover the generating values of the parameters when fit to this dataset.

The simulated data consists of 20 items and 1000 persons. The log discriminations have mean 0 and standard deviation 0.25. The difficulties have mean 0 and standard deviation 1. The correlation between the log discrimination residuals and difficulty residuals is 0.3. The simulated dataset is fit with Stan.

Before interpreting the results, it is necessary to check that the chains have converged. Stan provides the \(\hat{R}\) statistic for the model parameters and log posterior. These are provided in the following figure. All values for \(\hat{R}\) should be less than 1.1.

Convergence statistics (\(\hat{R}\)) by parameter for the simulation. All values should be less than 1.1 to infer convergence.

The Stan model is evaluated in terms of its ability to recover the generating values of the parameters. The R code below prepares a plot in which the points indicate the difference between the posterior means and generating values for the parameters of main interest. This difference is referred to as discrepancy. The lines indicate the 95% posterior intervals for the difference. Ideally, (nearly) all the 95% posterior intervals would include zero.

Discrepancies between estimated and generating parameters. Points indicate the difference between the posterior means and generating values for a parameter, and horizontal lines indicate 95% posterior intervals for the difference. Most of the discrepancies are about zero, indicating that Stan successfully recovers the true parameters.

3 Example application

The example data (Wood et al. 2003) are from a grade 12 science assessment. 600 students responded to 32 dichotomously scored items. Non-responses were scored as incorrect, so the data contain no missing values. The scored response matrix is converted to list form and fit with Stan.

To visualize the correlation between item parameters, the values of alpha[i] and beta[i] may be plotted against one another. This is presented in the left side of the figure below. Because the correlation is actually between xi[i,1] (which is equal to beta[i]) and xi[i,2] (which is alpha[i] on the log scale), a scatter plot pf xi[i,1] versus xi[i,2] is given on the right side.

Discrimination versus difficulty parameters for the SAT data. The lefthand plot shows the pairs in terms of alpha and beta, the usual formulation. The righthand plot shows the pairs in terms of xi, as used by the Stan model.