Scottish referendum: is the SNP's question fair?

Alex Salmond has published his favoured wording for a referendum question on independence. Is it loaded?
Polly Curtis, with your help, finds out. Get in touch below the line, email your views to
polly.curtis@guardian.co.uk or tweet
@pollycurtis

Photograph: Murdo Macleod for the Guardian

Yesterday, the Scottish National Party published its favoured wording for a referendum question on independence. It is:

Former Labour Chancellor Alistair Darling said: 'The question is loaded. He is inviting people to endorse the separation of a successful independent nation. He is not asking if you want to remain part of the United Kingdom, which I would prefer. 'It is asking for trouble and if he tries to push through unfair wording someone will go to court. It's typical of Salmond who wants to call the shots on the rules, the conduct, the wording and ultimately what the result means.'

Ministers viewed the wording of the question as unfair and liable to encourage a Yes vote – speculating that Mr Salmond wants to pick a fight. One Westminster source said: 'This is all about Salmond turning around and saying: "Look at all those dreadful people in London trying to tell us what to do."'

Is the SNP's question fair?

The Today programme put the question to Professor Robert Cialdini of Arizona State University, author of Influence: The Psychology of Persuasion (they tracked down an American academic to speak on this to ensure there was no bias). He said:

I think it's loaded and biased because it sends people down a particular cognitive chute designed to locate agreements rather than disagreements. It's called a one-sided question or a loaded question... [pollsters] for a long time have warned us against those sorts of questions.

There's a very simple fix to de-biasing those sorts of questions. Instead of saying how much do you agree with this policy or option the survey takers simply have to say how much do you agree or disagree... That produces an even handed and unbiased approach.

He described a study in which they asked two slightly different questions and got very different results. The first was: "If there's a serious fuel shortage this winter do you think there should be a law to require people to lower the heat in their homes." That got 38% support. The second added a balancing phrase "would you oppose such a law" and 29% supported idea.

Cialdini concluded:

A 9% swing is enough to change many many elections. To do it with a simple balancing phrase is the most economical and efficient way to do it. We simply say do you agree or disagree with this policy. That balances it.

I've just been speaking with John Curtice, professor of politics at the University of Strathclyde and a well respected expert in electoral behaviour and systems. He said that the wording was a huge improvement on previous suggestions made by the SNP, which had been attempting to circumvent the constitutional restraints which prevent the Scottish government from holding a referendum without Westminster's approval (Reality check looked at that issue here). A previous white paper suggested the wording:

The Scottish government should negotiate a settlement with the government of the UK so that Scotland becomes an independent state.

Curtice said:

It's a lot more straight forward than the previous questions because they are no longer trying to come up with one that will fall within the Referendum Act. If you wish to transfer powers you have to get the agreement of both institutions so previously the questions tried to be non-binding. They've given up and are now saying they will ask a straight forward question. That's the most significant thing that happened yesterday.

I don't think we should be surprised they've come up with that questioning. They could have chosen: do you want Scotland to remain part of the UK or become an independent country? But all things being equal you prefer to be campaigning for a yes rather than than a no. It would have been fairer to ask 'should Scotland become independent' rather than 'do you agree'. In everyday conversation people are inclined to say they agree than don't agree.

If I was writing the question I would prefer to ask 'do you agree or disagree" or simply 'should Scotland become independent?' Some would argue that you should give people the two options: independence or remaining in the union and let them decide. But others say that would over complicate it.

We should ensure the question is as simple and as straight forward as possible but I'm not sure it's going to be a huge issue. People will inevitably be engaged in this one. Mori surveys show that 50% say they will definitely vote in a general election – in a referendum 70% say they would vote. People aren't desperately engaged by current arguments about process, but they do care about the issue. What this will come down to is the quality of the campaigns.

Summary

The experts I heard from don't believe that the SNP's question is the fairest phrasing on the question. Asking people to agree with something is more likely to elicit a positive response than asking them to disagree. The emerging "no" campaign would prefer to campaign on a question that asks people whether they would like Scotland to remain in the union – their favoured outcome. For the same reasons, this also wouldn't be the fairest option. The fairest way would be to give the two options – independence or remaining in the union - and let people decide. However, John Curtice said that the phrasing of the question in this referendum is unlikely to have a very profound impact. Phrasing is far more important on issues that people don't understand the ins and outs of, such as the AV referendum. With the Scottish referendum on independence people are likely to be well versed in the debate.

Martin Boon, research director for social and government polling at ICM Research has got in touch and had this to day:

Salmond's proposed referendum question is good for its simplicity – he's right on that, but it is framed within the context of positivity. The premise of the question is defined by its singularity: it does not include "disagree" (our polling questions on this would - and did actually, going back to pre-devolution days) and if you are a voter who does not want independence the question demands that you actively disagree with the proposition. This is cheeky really – it suggests that "we all agree, don't you?" Obviously, some people find it harder to disagree on something than agree with the "rest of them who must like it".

In short, if this were asked in the context of an opinion poll, I would expect to receive some condemnation for it being deemed as imbalanced, loaded and unfair. I would refuse to ask it for a media organisation as it stands (if they came to me with it and refused usual ICM revision), but it would only require very minimal amending to make it fair. On the other hand, now that this question is in the public domain, there is no problem testing it to determine or predict an outcome based on a snapshot poll (defence being that this IS the question so obviously we have to ask it this way – pretty much what happened in the AV referendum).

Some interesting comments below the line too, mostly sticking to the question. Thanks for that. Here are a selection that jumped out at me:

Do you think the UK should stay in the European Community (Common Market)?

The equivalent phrasing would be ...

Do you think Scotland should remain part of the United Kingdom?

I'd be interested to hear what the experts would say about the phrasing of the first question. The government of the day wished to remain part of the EEC. Had they wanted to leave would they have asked ...

Do you think the UK should leave European Community (Common Market)?

... therefore retaining 'yes' as the response that gets them the outcome they desire?

It seems to me a question starting 'do you agree' also containing the word 'should' is always going to be a loaded question. Not sure whether it would make much of a difference though in the context of the Scottish referendum. However, those on here arguing that there isn't a bias because of the wording would presumably be just as happy with 'Do you agree that Scotland should remain in the United Kingdom'.

A better way would be a tick box choice on the lines of:

Which statement do you agree with:Scotland remains in the United KingdomScotland becomes independent

And why haven't you asked Alistair Darling or anyone from Labour why it was ok for them to ask the Scottish people whether they agree or disagree in their referendum...was that not biased as well ?

Hardly a fair and balanced representation of views you've aired now is it ?

The reason I didn't was that I was looking for independent advice on this – academic or specialist adjudication. There are clearly political arguments that can be made on both sides, as comments in the thread below suggest. But the point of this was to try and get an independent view. If either side wants to make a case, as @AyeWeCan2014 has been on Twitter, do get in touch. I haven't yet seen any evidence that would change the summary I wrote. Do get in touch if you think I've missed something.