Monday, September 24, 2012

Surprisingly relevant

The big news in blogland today, locally at least, has been the release of the so called National Standards data. I say so called because as best I can tell, they are not actually standardized.

The National Standards data (NSD) are problematic. There's a number of problems with presenting the NSD in the form that have been. For a start they don't actually say anything. An attempt was made to justify the release, claiming that it wasn't about selling paper. Journalists claim to have released the data to drive discussion on National Standards. This defence is weak at best, on one hand claiming that no measure of quality can be extracted from the data released : "Anyone who read the National Standards results as a proxy for quality would be quite foolish". Four paragraphs later they declare that they there are problems with National Standards: "If there are problems with the National Standards - and it's pretty clear that there are". How they reach this conclusion with no data regarding the quality of National Standards is a mystery. I can only surmise that the journalists who did this are either ignorant of what the job of a journalist actually is or that they are incompetent. Publishing data like this with no attempt to interpret it reduces journalism to stenography.

There are two very large problems with releasing data like this. The first is that it can be very easily be used to support unjustified claims. This sort of behaviour has been well demonstrated today with the herald reporting larger class sizes produce better educational results. A claim that just doesnotstand up. The second problem is that most people are going to trust what the herald and say. Some of us will look closer and understand that the claims put forth by employees of the herald pretending to be journalists are complete tosh. Most will not - they're to busy living their lives and still have a soupçon of faith in our media. And because its the journalists who are making up these stories in the first place then the chances of their unjustified claims of larger class sizes being better are unlikely to ever be reported to the public that they have already misled.

It also illustrates the point I was trying to make last week about transparency and process. In some ways the ideal journalist is similar to a scientist. Journalists are bound a lot looser than scientists but both have a responsibility to sort through the raw data and report on what is actually there rather than what is superficially evident. Journalists do this by attempting to be vaguely competent and consulting those with relevant expertise before publishing something. Scientists generally go through the peer-review process. This makes the peer review process, not an arbiter of what is significant and what is not, but an assurance that a basic level of rigour has been used. Both journalists and scientists have, relatively speaking, positions of authority in today's society. If reporters start putting out raw data without even attempting to explain what it means then they are failing as journalists. And I this is why scientist don't release their data directly to the public before they have published - there is a responsibility to dig through the data and be sure about what it says. And we do this because most people do not have the time or the desire to acquire the statistical background necessary to look any deeper than is required for basic day to day operation.