Given the obvious differences in patterns of responses across countries, can I really make cross-country comparisons? I tried, and got statistically significant but weak results. But if my measure isn’t any good, I can’t trust those results.

I did a tiny bit of research this morning to see if there’s anything I can do to adjust for differences in response styles across students and countries on TIMSS math background items. I had already tried weeding out the countries that show extremely skewed response styles but then I didn’t have enough data to run the analysis. And anyway, throwing out a bunch of data isn’t a good solution.

Buckley (2006) suggests a Bayesian approach that estimates a posterior distribution for each student characterized by a location shift and a scale adjustment that represent how a student’s responses relate to his or her actual attitude. For example, a large positive location shift and a reduced scale would typify extreme acquiescence, as the student picked mostly “strongly agree” type items. Buckley also provides a quick-and-dirty linear regression tactic for estimating a student’s latent true score on the measure taking into account extreme or random response styles. I may give that a shot this morning — the class project is due next week so I have some time — and then later explore a Bayesian solution.

It’s so cool to see my two interests — cross-country psychological studies and Bayesian stats — colliding. Seems like a potential dissertation topic.