What are our true ‘constitutional preferences’?

Last week I discussed the recent BBC/ICM poll’s findings on ‘constitutional preferences’. Because of significant question wording changes, I urged caution about apparently growing support for abolition of the National Assembly. I also suggested that the specific constitutional preference question used in this annual poll since 2012 might, compared to other plausible question wordings, somewhat inflate support for such abolition.

I want to explore these issues further here by looking across all major recent surveys that have included constitutional preference-type questions. Before getting into the details, though, a few cautionary words about what survey questions on constitutional preferences might be expected to tell us.

First, data from such questions share all the flaws inherent to survey research – ‘margins of error’ and other limits innate to sampling, plus the more human problems of declining survey response rates, people sometimes failing to understand questions and so on. Good survey companies try to minimise the problems, but can never eliminate them. Second, public knowledge and understanding of how Wales is governed remains far from perfect. Though most people seem to have reasonably clear broad attitudes, some do not, and relatively few know much about the details. In such circumstances we should expect many survey respondents’ answers to be sensitive to the details of question wording. Third, there is no self-evidently right way to ask about how Wales should be governed: hence, as seen below, the varied wordings employed, all of which might reasonably be critiqued as flawed in some way. None of this makes survey data on Welsh constitutional preference meaningless. But it is imperfect – a mixture, in Nate Silver’s terminology, of ‘signal’ and ‘noise’.

Two main types of constitutional preference question have been asked in surveys since the mid-1990s. The first type offers several specific constitutional options to respondents. One such question was first asked (I believe) in a 1997-post referendum academic survey: it gives a standard introduction (“Which of these statements comes closest to your view?”) and the following response choices:

Wales should become independent, separate from the UK and the European Union

Wales should become independent, separate from the UK but part of the European Union

Wales should remain part of the UK, with its own elected parliament which has law-making and taxation powers

Wales should remain part of the UK, with its own elected assembly which has limited law-making powers only

Wales should remain part of the UK without an elected assembly

Don’t know

This question was used in most major academic surveys in Wales between 1997-2011, most recently the 2011 Welsh Election Study. Variations on this format – offering slightly different response options – were used in various other polls across this period. Among the more recent are this one from the 2011 BBC/ICM poll:

Wales should become independent, separate from the UK and the European Union

Wales should become independent, separate from the UK but part of the European Union

Wales should remain part of the UK with its own Assembly which has full law making powers and some taxation powers

Wales should remain part of the UK with its own Assembly which has full law making powers but no taxation powers

Wales should remain part of the UK with its own elected Assembly which has limited law making powers only (as it has now)

Wales should remain part of the UK and the Assembly should be abolished

None of these

Don’t Know

Another slight variant, with a more limited set of options, was the following (used in a May 2012 Silk Commission/ICM poll):

Wales should become independent, separate from the UK

Wales should remain part of the UK, with its own elected Parliament that has both law making and taxation powers

Wales should remain part of the UK, with its own elected Assembly that has law making but not taxation powers

Wales should remain part of the UK, without an elected Assembly

Don’t know

The second main type of constitutional preference question originated in a 2009 multi-national study, Citizenship after the Nation-State (CANS), which ran parallel surveys across multiple sub-state nations and regions within Europe. Requiring something that could work across multiple contexts, the CANS team developed the following question (Welsh variant shown):

Which of these statements comes closest to your view?

There should be no devolved government in Wales

The National Assembly for Wales should have fewer powers

We should leave things as they are now

The National Assembly for Wales should have more powers

Wales should become independent, separate from the UK

Don’t know

This question has subsequently been used in several other academic studies, including the 2011 Welsh Referendum Study, and most recently in a poll in April 2012. But variants of this question also exist, including that used in BBC/ICM polls since 2012:

Wales should become independent, separate from the UK

The Welsh Assembly should have more powers than it currently has

The powers it has are sufficient and it should remain as it is now

The Welsh Assembly should have fewer powers than it currently has

Wales should remain part of the UK and the Assembly should be abolished

None of these

Don’t Know

And this version, used in a Beaufort/Silk Commission survey in May/June2013:

Wales should become independent from the UK

The National Assembly for Wales should have more powers than it currently has

The powers it currently has are sufficient and it should remain as it is now

The National Assembly for Wales should have fewer powers than it currently has

The National Assembly for Wales should be abolished

Don’t know

There are many other potential influences on surveys findings beyond question wording. These include variations in the samples obtained; genuine changes in attitudes over time; and potential question ordering effects (where cues given by preceding survey questions may influence responses to later ones). But what effects might be expected from these differences in wording? The following hypotheses appear plausible:

1.That support for Independence would be higher when offered as two separate options rather than one

2.That support for Independence would be lower where a question explicitly stated ‘separate from UK’; higher where (as with the Beaufort question) this is omitted

3.That support for full law-making powers would be higher when offered as two separate options rather than one

4.That support for a No Devolution/Abolition option would be higher where this is only option explicitly linked to remaining in the UK

5.That support for No Devolution/Abolition would be lower where abolition is not explicitly linked to remaining in the UK

6.That support for More Powers should go down after the March 2011 referendum, when the status quo changed to give the Assembly greater powers.

So what results have been found? First, a table of findings (%) from the most recent (2011/12) surveys using the first question type:

2011 WES

2011 BBC/ICM

2012 Silk/ICM

Independent (outside EU)

6

4

–

Independent (inside EU or not stated)

7

7

8

Law & Tax Powers

34

35

39

Law Powers only

18

31

Limited Law Powers

28

17

–

No Assembly

18

15

18

None/DK

8

4

4

Now a table of the most recent surveys using the second question type, plus (to test suggestion 6 above) the results from the 2011 Welsh Referendum Study pre-referendum wave of sampling:

2011 WRS

2012 YouGov

2013 Beaufort

2014 BBC/ICM

Independent

9

10

9

5

More Powers

41

30

53

37

As Now

19

31

24

28

Fewer Powers

3

5

3

3

No Devo/Abolition

16

16

9

23

None/DK

13

9

2

5

How do these results relate to the hypotheses suggested above?

1.This hypothesis is supported: the two highest support levels for independence are in surveys that offered two separate independence options, although differences across surveys are mostly quite small.

2.This hypothesis is not confirmed: notably, the level of support for independence in the Beaufort/Silk study was no higher than in the 2011 WRS and 2012 YouGov surveys that used a similar question format but explicitly mentioned separation from the UK.

3.Comparisons of the two 2011 surveys (WES and BBC/ICM) support this hypothesis.

4.This hypothesis is also supported: the 2012 and 2014 BBC/ICM polls are the only ones since 2001 where support for No Devolution/Abolition is above 20%. It seems unlikely this has nothing to do with question wording.

5.This hypothesis is also supported: just as the BBC/ICM wording may facilitate higher reported support for a No Devolution/Abolition option, the Beaufort/Silk wording seems likely to encourage lower such support. The 9% Beaufort found is the joint-lowest level (along with the 2009 CANS study) yet recorded anywhere.

6.This hypothesis is not confirmed. The result of the 2011 referendum has not necessarily led to lower support for more powers.

In short, four out of six hypotheses about the impact of question wording appear to be supported. Question wording will certainly not be the only, or even the main, influence on how people answer constitutional preference questions. But it is very likely that it matters, at least at the margins.

Overall, where do we stand in terms of evidence about how people in Wales want Wales to be governed?

There is no general evidence of either rising or falling opposition to devolution in recent years. It is unfortunate that the two most recent surveys have produced rather ‘extreme’ results. These results are probably due at least in part to question wording. The original CANS question has produced results somewhere in the middle. On all surveys since 2005, on all question wordings except for that used in the BBC/ICM polls since 2012, support for a No Devolution option has been at 20% or below.

Support for independence remains fairly low – at around or below 10%. The 5% in the recent BBC/ICM poll is particularly low, though this could be just a slight outlier.

There is clear and consistent majority support for devolution. This has been found by all variants of constitutional preference questions, as well as in other forms of question, in all surveys conducted from 1999 onwards.

Surveys over the last decade have consistently shown considerable support for more powers: across all variants of these constitutional preference questions, as well as across other question forms.

Where the evidence is somewhat less consistent, however, is on the extent of support for more powers, and where exactly the ‘centre of gravity’ of public opinion lies. The BBC/ICM 2014 poll, and the 2012 Yougov one, would place the median Welsh person (excluding Don’t Knows) clearly in the ‘remain as now’ category; Beaufort would have that person squarely in the ‘more powers’ option.

And perhaps the final message to be drawn from this study of the evidence is that we should always try to avoid placing great weight on the findings of any one question, or any one survey. The results of constitutional preference questions should where possible, be interpreted alongside other questions and/or other sources of evidence. No single survey question can ever yield us clear, unproblematic, and unquestionable truth.

Share this:

Comments

“And perhaps the final message to be drawn from this study of the evidence is that we should always try to avoid placing great weight on the findings of any one question, or any one survey”

And yet……that Beaufort survey was taken in isolation to inform the Silk Commission’s recommendations to the UK Government and they ignored repeated warnings that the survey methodology was likely to result in showing a disproportionately high level of enthusiasm for further devolution.

There is one other influence that you might have looked at, and one that you could possibly experiment with with a small group.

What happens to a respondents concentration and comprehension levels when confronted with a list?
Do respondents avoid extremes if they can help it, particularly under time pressure?

In the case of lists of options do answers become more random with responses gravitating towards, say, the middle 3 of 5 options? Does “Don’t know” become more or less of a feature (after all given 5 options everyone must have a category for their own position?)

If you give just two options do you increase the “Abolition” percentage?:

Are you in favour of devolution of powers to the Welsh Assembly?
Would you like to see the Welsh Assembly abolished?

I would also think that extended surveys that ask questions about National Identity will produce distorted results and, as you know, there is evidence to support that theory.