Development and Validation of a Web-based Test Battery

Series:

Testing of second language pragmatics has grown as a research area but still suffers from a tension between construct coverage and practicality. In this book, the authors describe the development and validation of a web-based test of second language pragmatics for learners of English. The test has a sociopragmatic orientation and strives for a broad coverage of the construct by assessing learners’ metapragmatic judgments as well as their ability to co-construct discourse. To ensure practicality, the test is delivered online and is scored partially automatically and partially by human raters. We used the argument-based approach to validation, which showed that the test can support low-stakes decisions about learners’ knowledge of sociopragmatics in English.

Prices

Chapter Price

Extract

We set out to create a test of ESL sociopragmatics, especially targeted at second language learners of English in an Australian setting. From a psychometric point of view, we made our own task more difficult by limiting the proficiency range to mid- and high-proficiency learners, and not including learners without any residence in the target sample. This led to a more homogenous sample, which tends to depress psychometric analyses. However, we found it important to not artificially inflate our coefficients, and administer the test to a sample from the population for which it was designed. Despite this hurdle, our test achieved satisfactory reliability, and our argument-based validation shows that inferences about test takers’ sociopragmatic knowledge can be justifiably made from its scores. Given the experimental nature of most of our tasks and our emphasis on breadth of construct coverage over high reliabilities, we recommend that low-stakes decisions be based on our test.

But what decisions might those be? Using Kane’s (2006) argument-based approach to validity with Chapelle’s (2008) modifications made it clear to us that utilization of a test is crucial for evaluating the argument. This is a tricky problem in a pure research situation such as ours where the test is not being used for real-world purposes, but we would encourage follow-on studies that investigate the value of the test for real-world decisions by particularly strengthening the extrapolation inference. The real-world instantiation of the construct tested and the test’s ability to make predictive statements about the real...

You are not authenticated to view the full text of this chapter or article.

This site requires a subscription or purchase to access the full text of books or journals.