The anonymous blogger at Emil's Demesne* provides a few useful insights on how we might use National Standards data.
For readers outside of New Zealand, school-by-school data on student achievement rates will soon be published. An early version of that data came out this week when Fairfax compiled results from those schools willing to release their data early; the rest will wait until the government publishes the comprehensive dataset. Emil begins:

My own prejudices. I am a policy analyst with nil expertise in education. I support publication of national standards data. I am agnostic as to whether national standards should have been introduced in the first place, but furious at those “public servants” who sought to obstruct implementation of a lawful Government policy: no integrity.

I’m mainly interested in what the standards data can and can’t tell us, and how they might be used to improve education outcomes. The lens I’m trying to look through is how would I tackle this if I were at the Ministry of Education?

Where cross-school comparisons might come in useful is in identifying stand-out schools so that successful and/or innovative practices can be, where appropriate, replicated more widely. Obviously, what works in a school within one particular cultural, social, and economic context won’t necessarily work for a school that’s in a totally different one. You do your data analysis and your case studies, then you devise sensible categories and work within them.

So long as moderation is at least better than hopeless, in time, we’ll also learn quite a bit more than we already know about the impact of social and economic factors on academic attainment during the early years of school. This is important, because it’s these early years that are assumed to matter most. National standards data, for all its flaws, is or can be made rich enough to support meaningful research that will help us improve how we teach our children.

Depending on whether or not I can be bothered, I might write up some stuff on why I don’t like deciles as analytical tools. But not this week.

The post has nice context around why we shouldn't expect standardized testing or strong grade moderation in New Zealand. What should parents be looking at rather than just a school's ranking?

I’d look for signs that schools are using the standards sensibly on a student level. Especially for students not at standard, I’d want to see individualized learning plans, with achievable benchmarks/milestones. Ideally, these plans would be designed in and as a collaboration between teacher, student, and caregiver. Give the student a sense of direction and ownership: here’s what we want you to be able to do, here’s our plan for getting you there, and here’s how you’ll be able to feel your own progress along the way.

Over time, I’d look for signs that schools are using standards, in conjunction with the learning plans, to do some value-added appraisal of teachers. I would incorporate this formally into professional development structures. I accept that there’s only so much a teacher can be reasonably expected to do for kids who turn up hungry, have caregivers with significant reading difficulties, or who switch between schools a lot. (These are, incidentally, things that are thought to correlate pretty tightly with decile.) Placing appropriate weight on factors like this is something that I think standards will be able to do over time, even if they’re fairly messy.

I think Emil is overlooking the fact that there was already a comprehensive set of achievement measures (STAR, PAT, asTTle tests) in place beforehand. He seems to be under the impression that all of this is somehow new.

I’m aware of PAT, STAR, NumPA etc. The National Standards are something quite different. The material on TKI makes this clear enough; there’s no need for me regurgitate the details.

Teachers and principals are public servants. I put the scare-quotes in to convey contempt for those who, in seeking to obstruct lawful government policy, commit what I regard as a severe ethical breach.

Carol, you refer to PAT, STAR and asTTlE tests, but these are all clearly aimed at teachers. The publication of National Standards seem to be an attempt to provide useful information for parents.

No one seems to be doing anything about the lousy educational outcomes NZ is getting for the bottom quartile of our kids, who are leaving school with minimal capabilities. This educational failure has been going on for many many years. PISA studies show that not only does NZ have a relatively large gap between top and bottom placed students, but also that the gap between top and bottom within a school is also relatively high in NZ.

National standards may be useful once we have a few years of data – I don’t think a snapshot of one year’s data has much value, but if the publication of this data helps prompt discussion about what schools do and in some way leads to improvements for our bottom quartile children, many of whom (lets say it) are Maori and Pasifika, it will have done a lot of good.

Carol Stewart 885 days ago

Au contraire, Possum. School reports are squarely aimed at parents and for years have contained age-normalised information on stanine levels in reading and so on. I have never found them in the least bit difficult to understand.
I agree with you about the tail end of underachievement being a problem for many years, but it’s not like National Standards have suddenly cast light on this. I have been involved with boards of trustees (who consider achievement data) in the past, and I remember hearing about exactly the problems you describe at least six years ago. I suppose it could be argued that National Standards might add urgency to the political will to address these problems but I remain to be convinced.

I’ve found it useful. We’re in the ballot for Ilam school; it seems to have better numbers on reading and maths than on writing. We’re likely then better advised to reinforce writing at home all else equal.

Carol Stewart 884 days ago

Eric, on the subject of national standards in writing:

My lad, who is year 8, received the following assessments of his writing ability in a short time.

A ‘below expectations’ from one teacher.
An ‘at expectations’ from the next teacher (having just changed cities and schools).
And a Distinction in the international ICAS exams for writing administered by the University of New South Wales. He was one of only two kids at a large, high-achieving intermediate school to get this grade.

So, which do I believe?

Carol Stewart 884 days ago

I should add that the first two teacher assessments were official national standards results.

So you can probably see why I am less than convinced about their value.

Michael Edmonds 883 days ago

National standards simply assesses how a child is performing relative to others (although Carol’s comments seem to suggest the usefulness of the information is less than adequate).
The recent release of details has pointed out things educators have known for years – girls are performing better than boys, Maori and Polynesians have lower than success rates. Not particularly much use.

We can assess students as many different ways that we like but NOTHING will change in terms of improved performance until we actually start investing in the LEARNING side of teaching (instead of focusing on ASSESSMENT).

But then it is so much easier to repeatedly identify a problem than actually do something about it

possum 882 days ago

Michael, that’s not a very scientific post.

You make statements based on the data from National Standards. However you obscure the point that the distribution curves of achievement for boys and girls, etc are just slightly offset curves whereby a slighly higher proportion of girls beats any given level of achievement than boys. You seem to imply that all girls are doing better than all boys.

Then you firmly and EMPHATICALLY state a conjecture that is not at all based on either the data or your statements.

If the item on today’s Morning Report is accurate, then National Standards are not particularly reliable as a basis for inter-school comparisons: 49% of teachers scoring ‘writing’ items wrongly (against the same items scored by experts), & 39% for maths. With a tendency to score more highly than the experts.

Michael Edmonds 882 days ago

Possum,

Sorry, that post could have been clearer – the comment that girls are doing better than boys is one that has been repeated for years and which commentators are repeating based on the National Standards data.
However, as we are being pedantic let’s just clear up a couple of things in your post.
My post was note based on the national standards data but on the comments being made about the national standards, including the commonly heard “girls are doing better than boys”
The suggestion that I am implying that “all girls are doing better than all boys” is your interpretation of what I posted. I certainly did not use the word “all” at all.
Also you accuse me of obscuring information, which implies intent. There was no intention to obscure anything on my part, just a somewhat hasty post.
Could you please also clarify what the conjecture I emphatically stated was.
If it was my comment that repeatedly identifying a problem will not solve it or that improved performance will come from investing in teaching and learning then I am happy to debate those points further.

@Alison: We definitely need to know more about whether that’s white noise that washes out on average within schools or whether there’s by-school skew.

possum 882 days ago

Hi Michael. Yes, it was my interpretation of what you wrote and I admit I was a bit surprised you wrote it. And no, you didn’t write “all” – I inferred it. Thanks for the clarification. I’m certainly guity of hasty posts on occasion.

The conjecture was the bit that started “NOTHING will change in terms of improved performance….” which seemed totally unrelated to what you had written before, but read like a conclusion. I’m not qualified to debate the point but I sort of agree with you; because working to assessment results would be somewhat like driving a car forward while looking in the reversing mirror.

But anecdotally, some teachers are better than others and some schools are better than others. If we can identify some of the factors that result in relatively better schools (the best performing decile n schools say, it would be stupid to compare decile 1 against decile 10) and apply these factors to the relatively poorly performing schools of decile n, we’d likely see somewhat better achievement results overall.

Blogroll

About SciBlogs

Sciblogs is the biggest blog network of scientists in New Zealand, an online forum for discussion of everything from clinical health to climate change. Our Scibloggers are either practising scientists or have been writing on science-related issues for some time. They welcome your feedback!