Friday, December 17, 2010

Two minutes to go, you're down by 7 marks, who do you want at that desk?

This week I finished about 100 report cards, writing a short paragraph about each student. Often I ended up noting that the student's marks were not an accurate reflection of their ability. For whatever reason, some students consistently underperform on tests, but would be judged as being better at English by any measure except for testing.

Judging students to be better on the basis of a semester's worth of classes, though, is like judging a team or a player to be better after a season's worth of practices and regular season games. The English Premier League just does this, as my Patriots-loving brother noted after a particularly grievous upset loss. The lore of sports, particularly football, where teams can not play more than 20 meaningful games in one year, is built around the idea of "making plays".

"Making plays" is an empty, just-about meaningless statement that is no more informative than Nike's "just do it", or the fabled track coach's advice to "run fast, turn left". Football does reward those who perform well in the post-season. Whether it's a fluke or otherwise, you are good because you perform well in designated high-pressure situations. Consider the 10-6 Giants who upset the 16-0 Patriots: the Giants' miraculous playoff run isn't held against them, rather, it's Exhibit A in their greatness.

The analogy between football and school isn't quite the same, though. Where it breaks down is that football teams have a season that is consists of competing towards an obvious goal. You could argue that the goal of school is to have the best marks possible since that's the only tangible outcome, but school was never an agreed-upon competition in the same way.

As well, students aren't teams. Teams are easy to compare, but players are not. Even in cases where an individual is statistically superior to another (passing yards, grades, whathaveyou), there are other factors at work: the quality of his teammates, the way his team plays, and even the climate in which he plays his games. Clearly, with players, the agreed-upon goal that lets us arbitrate between teams breaks down, because there are still debates about the greatest quarterback ever.

Morever, some of the quarterbacks who won the most Super Bowls generally aren't considered to be the absolute greatest ever (Troy Aikman, Terry Bradshaw), and quarterbacks with the greatest individual statistics aren't always considered the greatest (Vinny Testaverde, Jim Kelly and Drew Bledsoe all rank highly in passing yards).

So, if individual football players, typically quarterbacks, can't be judged by neither individual statistics nor by designated high-stakes events such as Super Bowls, how can we judge them? We use heuristics, much as we judge students and our peers by a mix of daily performance as well as high-stakes performance. We discount Testaverde as a great quarterback despite his individual statistics and discount Aikman as one of the absolute greatest despite being somebody who did very well on tests. Obviously, we have our reasons.

The heuristic sometimes used is the hypothetical "your team is down by a touchdown with 2 minutes to go, who do you want in the huddle?" question seen on message boards or polls. The idea is to strip players on great teams of their supporting cast, and to judge players who accumulated great statistics without great skills in a high-pressure situation. This would be akin to judging students by simply cornering one in the hallway at random and asking them math problems. This is neither a test, nor a meaningless class, but some sort of "real knowledge". Another example would be a peer in high school or university that we felt wasn't that smart, despite high grades, based on a semester's worth of interactions.

This is not a bad way of doing things, though it's not without its ambiguity. Peyton Manning, John Elway, Joe Montana, and Tom Brady could easily be trusted to overcome a touchdown deficit with 2 minutes left, as could second-tier quarterbacks like Donovan McNabb, Steve McNair or Jim Kelly. Similarly, much as tests supposedly only prove that someone is good at tests, someone who is smart when you talk to them is, therefore, only good at seeming smart in conversation.