Can Smoothing Help When Equating with Unrepresentative Small Samples? Research Report. ETS RR-11-09

Puhan, Gautam

Educational Testing Service

The study evaluated the effectiveness of log-linear presmoothing (Holland & Thayer, 1987) on the accuracy of small sample chained equipercentile equatings under two conditions (i.e., using small samples that differed randomly in ability from the target population "versus" using small samples that were distinctly different from the target population). Results showed that equating with small samples (e.g., N less than 50) using either raw or smoothed score distributions can result in a substantial amount of random equating error (although smoothing reduced random equating error). Even with samples sizes of 100, the random equating error was quite large (greater than the difference that matters or DTM) for almost all score points. Moreover, when the small samples were unrepresentative of the target population, which is quite likely for small samples, the amount of equating bias (in addition to random equating error) was considerably large for both the raw and smoothed equatings. It was concluded that although presmoothing helped reduce random equating error, it is unlikely to reduce equating bias caused by using an unrepresentative sample. Other alternatives to the small sample equating problem that focus more on improving data collection than on improving existing equating methods are discussed. Appended are: (1) Score Distribution of Full Sample; and (2) Conditional Standard Errors of Equating. (Contains 3 tables, 7 figures and 2 notes.)