Herald repeats misinformation

James Dann is a Christchurch local body politician and blogs on the Christchurch rebuild etc. His latest blog post is on the Nielsen survey for CERA which has upset a few people because in fact it found not everyone was unhappy.

The key to this survey is in the methodology. It was undertaken between August 29 and October 15 last year. 2381 people responded to the survey. 1156 of them were from Christchurch City, 618 from Selwyn District and 607 from Waimakariri District. So that means that of the people who did respond, more than half (1225) AREN’T ACTUALLY FROM CHRISTCHURCH. I mean no disrespect to the people of Rolleston, Lincoln, Rangiora, Kaiapoi etc. I am sure they have had a hard time. But they don’t have the same issues that people who actually live in the city do. They haven’t had to deal with a lack of services, portaloos, red zoning, TC3 land, roadworks to the same extent that people who live in the city do. …

Yet the survey then merges these three population groups, and continues to extrapolate from the combined population for another 100 pages. If you get to the end of the report, you’ll find the populations of Christchurch, Selwyn and Waimakariri. They are 289,000, 30,000 and 36,000 respectively. More than 80% of the adult population lives in Christchurch, and yet more than 50% of respondents come from less than 20% of the population, a section of the population that the survey shows have a more positive response in this survey.

He is correct that the raw sample is not reflective of the overall population. That 80% live in Christchurch but only half the responses were from Christchurch people.

But what he has either deliberately or accidentally omitted, is that the results are weighted so that the results do precisely match the area, gender and age of the local population. Nielsen say in their report:

The sample design over-sampled residents of the two districts with smaller populations to ensure that the sample size within each district was sufficient to allow reliable and robust analysis.

At the analysis stage, the data was adjusted by a process called weighting. This process adjusts for discrepancies between the profile of people who completed the survey and the known profile of residents of greater Christchurch.

Population statistics are obtained from Statistics New Zealand data and is based on the latest population projections.

Weighting increases the influence of some observations and reduces the influence of others. So, for example, while 618 or 26% of completed interviews came from Selwyn District, the population of Selwyn actually represents about 8% of greater Christchurch. Thus, the data was adjusted so that 8% of any ‘greater Christchurch’ result reported is based on the responses of Selwyn residents.

So the criticism of Dann are entirely invalid, and worse the Herald has repeated them on their website. This is not a matter of opinion. His failure to mention that the results are weighted by area totally invalidate his criticism. The Herald was aware of this also, as one of their editorial staff tweeted his blog post and I pointed out this failing in a response (which they retweeted).

Appendix 5 of the poll results on page 103 details their weighting matrix with precise quotas for area, gender and age. It is a model of transparency.

Incidentally how significant were the differences in responses from those in Christchurch itself, and the two neighbouring districts? Here’s the figures for overall quality of life – good vs poor.

Christchurch City 72% good vs 7% poor

Selwyn 85% good vs 4% poor

Waimakariri 82% good vs 4% poor

Greater Christchurch 74% good vs 7% poor

So the results for Greater Christchurch are almost identical to Christchurch City.

The only question I have is whether Mr Dann knew the survey was weighted and decided not to mention that fact, or whether he just failed to read the results in full.

UPDATE: He has amended his original blog post to now mention the weighting, but note that the version picked up by the Herald doesn’t include this.

UPDATE 2: Dann was a Labour Party candidate at the last local body elections. Yet the Herald didn’t think to mention this. Shocking.

Ross12

Well done DPF.
I am continually amazed at how many people embrace modern technology and use it for blogging , online sites, tweeting , social newtworking etc. but clearly do not understand the speed that it works –ie. if you make an error it is pounced on very quickly and you can be exposed just as quickly. They haven’t latched onto the fact that you have to double or triple check your facts before you blog , tweet etc anything because you will quickly be made to look a fool.
( NB. to his credit this guy has corrected it but …)

It would have been interesting to have had ChCh City results split out by part of town. Wigram, Yaldhurst, Riccarton, & Ilam are effectively a different city when it comes to quake experience.

When we bugged out from South Brighton three days after the quake for a house in Wigram, it felt like we’d moved to another country. I think it was Peter Hyde who’d said there were three cities post-quake: Rescue City, Refugee City, and Shower City. Nobody’s put it better before or after.

You’re right that Dann’s wrong on the weighting.

But I answered that survey. And I remember putting in my street address. And if you look at the Neilson report, they have that as an appendix: street addresses being taken so they could see if there were any differences across different parts of town.

If Lianne Dalziel is not an idiot, she will be doing OIA requests for the data broken down by region. And she will then very likely be able to show that the quake response in the East has been utterly unsatisfactory as viewed by the residents there.

There is potentially massive cross-Christchurch heterogeneity that is masked in the report. They’ve the data to back out regional variation – they requested it specifically for that purpose. And there’s zero reporting of anything broken down by region. Maybe it’s because everybody gave the same answers so regional variation was uninteresting. Or maybe it was a hassle to aggregate up from street addresses to regions. Or maybe things looked completely and utterly horrible in the East, with a lot of us ready to head out with pitchforks and torches.

It is now easy to understand the motivation as to why Dann would spin the survey results to his make a political point that would damage CERA and the government. James is entitled to his opinion but he needs to be up-front about his political affiliations that inform his biases. That the NZ Herald journalists were utterly incurious as to this vital point comes as no surprise. This was typical lefty leaning lazy cut and paste ambush journalism. No wonder the Herald’s parent company in Australia is struggling.