Don’t Know or Don’t Care?

When we examine the results of standardized test scores we typically think we are seeing evidence of what students know. As it turns out, that is only partially right. Test scores capture both what students know as well as their willingness to exert effort to show us what they know.

So, how do Gema, Collin and Ildefonso know that between 32% and 38% of variation in PISA test performance across countries is explained by effort? They used three different methods to measure the influence of effort. First, they took advantage of the fact that the order of questions without the PISA was randomly ordered. They then compared how well students performed on the first set of items relative to the last. Because the order of items was randomized the first and last questions were, on average, of equal difficulty. The decline in getting items correct from the start to the end of the exam is therefore a function of the decline in effort students are willing to exert, not the difficulty of the items.

If you compare performance in the US and Greece (as can be seen in the figure above), students in the two countries do about as well at the beginning of the test. That means that students in Greece and the US know about the same amount of stuff. But students in Greece decline much more rapidly across the test, which means that those students are less willing to exert consistent effort. When we compare PISA results from the US and Greece we wrongly conclude that content instruction in Greece must be much worse. In reality, Greek students know as much as students in the US but simply exert a lot less effort.

Share this:

Related

This entry was posted on Friday, October 28th, 2016 at 10:59 am and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.

Running out of time is also an indication of effort. Imagine that two people are equally able to answer the first 10 questions correctly, but one answers the last 10 at a higher rate than the other. Because the order of items is randomized, the first 10 and the last 10 items are of equal difficulty. So answering fewer of the last 10 is not a function of how much stuff one knows. It’s a function of how much effort one is exerting. Even if one person answers fewer of the last 10 because they run out of time, how quickly one moves through the test given equal ability to answer the first 10 is a matter of effort, not knowledge.

Shouldn’t any study on PISA results indicate why anyone should care what students do on it? Finnish mathematicians had to get Finland to use TIMSS instead of PISA (after Finland hit the top on PISA) to find out how their students were really doing in math.

What if the second person takes twice as long to answer the first 10 questions as the first person. They get the same score on the first 10, but contestant number two finds himself desperately guessing at the end as time runs out.

The findings on non-answers and careless answers don’t align with the theory that the difference in late questions is speed rather than diligence. People with more non-answers or careless answers should have more time to complete the test. That’s not to say it’s impossible speed plays some role, but it’s not going to fully explain the difference and isn’t likely to explain most of it.