Hugh Reilly: Pisa knows its tables are warped

HUGH REILLY

Normally, usually, I’m not a big, huge fan of employing tautology to emphasise a point but I admit I smiled when I heard a Hamas spokesman deride UN Mid-East Peace envoy Tony Blair as being “useless, useless, useless”, a phrase eerily echoing Fettes finest’s of “education, education, education”.

To my certain knowledge, Gaza schools do not come under the gaze of Pisa, (Programme for International Student Assessment), the OECD organisation that ranks the education systems of more than 50 developed nations. Currently, the Scottish education system is evaluated by Pisa but No campaigners warn that a positive result in next year’s referendum could lead Scotland being ascribed Third World status, its schools being compared to those of Papua New Guinea, Kampuchea and Mali; for Glasgow teachers tired of being scourged , it’s yet one more overwhelming reason to vote “Yes”.

Rather tardily, Pisa will publish its league table for 2012 in December 2013, a time lag that makes my annual decision to post a Valentine card in March seem recklessly impetuous. For decades, governments around the globe have reformed their education set-ups based on the oracles of Pisa. However, many leaning, sorry, leading academics state that the organisation’s findings are worthless. As a long-time critic of Pisa, I’m glad that education’s emperor makes the Naked Rambler seem a tad overdressed.

Doubts in my mind first arose when Pisa declared Finland, the land with the highest (non-assisted) suicide rate in the northern hemisphere, to be a paragon of learning; if knowledge is power, erudition comes with somewhat bleak baggage. If he had one, Michael Gove, the education minister for England, would have taken it on the chin that his curriculum overhaul has been built on a fault line in the middle of a tornado zone. Gove, whose Caledonian accent makes Scotty from Star Trek sound like a seaweed-grazing citizen of Shetland, has used Pisa league tables to castigate England’s chalkies. Unfortunately, due to the lack of a webcam in his office, one can only imagine how he feels now that top education professionals have proved Pisa to be a straw man.

To paraphrase Andrew Lang, the great Scottish poet, novelist and literary critic, Gove and his ilk used statistics as a drunken man uses lamp-posts … for support rather than illumination. He and others didn’t bother to employ their critical faculties to question the methodologies used to arrive at findings that have had a profound impact in education worldwide. The fact that there is no common test should have alerted international decision makers. Students are examined in three subject areas: reading, mathematics and science.

Unbelievably, in Pisa 2006, although half of the responders were not tested in reading and maths, full rankings in these subjects were produced. One can but assume that David Blaine secured a consultative role with Pisa. Further, only 10 per cent of all students participating in the scheme sat the entire 28 reading questions. Consequently, the data garnered by Pisa inherently possesses the integrity of midweek questionnaires carried out in Princes Street by cash-poor, time-rich media studies students fiercely competing against overly gregarious charity muggers for the attention of the indolent-liege demographic.

There are those who say that league tables hold no value, usually headmasters whose schools occupy the bottom end of any compilation. Others argue that league tables are something of a necessary evil; without them, tossing brickbats capriciously at education establishments would be a somewhat empty experience. However, I find it bizarre that actual scores are not always used. Instead, the good people at Pisa prefer to utilise “plausible values” and “random numbers” to evaluate a country’s performance. Put simply, it is professional guesswork of what a student would have scored if his country had a particular test in place. In terms of producing an accurate, verifiable outcome, it’s akin to wandering through the Sahara desert with a wish-bone-shaped twig and expecting to find an underground reservoir.

Tellingly, experts at Pisa express doubts in their own work, commenting that there is “uncertainty” with regard to rankings of individual nations and that “large variation in single ranking positions is likely”. Instead of promoting Pisa league tables, governments should relegate them to the dustbin of history.