6 weeks scoring 1 math question

As a 23-year-old “slacker” with a college degree, Kevin was hired by Pearson to score a sixth-grade math question on Illinois’ state exam. For six weeks, he scored the same question over and over again for eight hours a day.

Students are graded on a four-point scale on three dimensions: the actual answer they gave, the steps they took to get the answer, and the written explanation of why they took those steps. The first two dimensions were straightforward enough for the scorers, but the third one, the explanation, was a nightmare to score.

Comments

Before NCLB, Maryland had their MSPAP.
On the 8th grade science, the students were led through a series of activities and reading to mineral hardness tests and specific gravity. The final question asked the students how would they determine whether a rock was true gold or not. They were given charts with mineral hardness and specific gravity for several minerals.
A few “got” it, but most flailed around.
The best answer was “I’d take it to a jeweler, but not let him take it into the back room.”