The survey was conducted by ShapeNZ - a subsidiary of the BCSD - and was conducted online. People signed up on the website (with the offer of a cash prize as a sweetener) and answered a bank of questions. Their answers were then "weighted to make it representative of the New Zealand population in terms of age, gender, voting behaviour and personal income". This helps a little, but fundamentally you still have a self-selecting sample rather than a random one. And this could significantly skew the results.

Such a methodology would be rejected by any social scientist (unless there were compelling reasons which made random sampling impossible - and then it would be heavily caveated). But it is regurgitated uncritically by the Herald, and is considered to be a useful guide for policymaking. Sometimes I really do despair...

The important difference here is in sample selection - YouGov selects their sample from a representative pool (and I'm willing to accept that 125,000 people is representative enough, at least for most purposes). ShapeNZ's sample is self-selecting, in exactly the same way that a Stuff or TVNZ online poll's sample is. And it's this that makes their results worthless, no matter how much correction they apply.
Posted by
Idiot/Savant
:
4/16/2007 03:41:00 PM

I/S is right. Some other on-line panels are worthwhile because recruitment is random and independent of the surveyed topics. There is then no reason to expect sample bias.

Having looked at the front of the linked website, it seems to link recruitment to the topic. This subjects the results to an umeasurable sampling bias. Maybe they got lucky and the results are representative! But no-one will ever know for sure from this survey.
Posted by
kiwi_donkey
:
4/16/2007 10:14:00 PM