Polling – how I started, and why it’s a force for good: my Anglia Ruskin lecture

This is the text of a lecture I gave last night at Anglia Ruskin University, explaining why I began political polling, and why (used properly) it can help bridge the gap between the politicians and the people.

Good evening and thank you all very much for coming. It is a great honour for me to have been asked to speak to you on a subject in which I have developed a consuming interest over the last ten years. Indeed I stand before you not just as Anglia Ruskin’s Chancellor but, at least according to the New Statesman, as “the nation’s Pollster-In-Chief”.

Whether or not such an accolade is deserved, I thought I would use this occasion to talk about how I came to be involved in polling in the first place; why I think opinion research is a useful and indeed vital part of the political process; its limitations; and finally, what I think current research tells us about prospects for the general election.

Now would be as good a time as any to deliver the mantra, or what some have started to call my catchphrase, that a poll is a snapshot not a prediction. Indeed you will not be hearing any predictions about the election from me this evening, except this one: I think it is going to be quite exciting.

A decade ago, with a few months to go until the general election of 2005, no such sense of anticipation was in the air. Labour were well ahead in most polls, and few people expected anything other than for Tony Blair to be returned with another comfortable majority.

I was supporting a number of Conservative candidates in marginal seats. They were running positive, energetic, well-organised campaigns – that was part of the deal – but their canvassing returns and general reports showed they were struggling to make headway.

So you can imagine my surprise when the joint party chairman, Lord Saatchi, popped up at the 2004 Conservative Conference and declared that the Tories’ own private polling in marginal seats suggested the party was on course for victory.

This sounded like humbug to me. In fact it sounded so unlikely that I decided to find out for myself what was really going on. I commissioned my first polls in a selection of marginal constituencies and found that, not only were the Tories not storming to victory in these critical seats, they were doing even worse in them than they were doing nationally.

This was not just a matter of making a point. I thought the Conservatives were making a mistake that would have real political consequences. Putting on a brave face and cheering up the troops is one thing, but I suspected that the party I supported was going to delude itself into thinking it was doing better than it really was – and might, as a result, do worse than it would do if it had a realistic grasp of the situation.

For one thing, it would mean the party was kidding itself it could win in seats where it had no real chance, and would campaign heavily in them. In so doing it would not only fail to win these seats, but fail to win seats it could have won if resources had been better targeted.

For another thing, it would mean the Conservative Party continuing to avoid facing up to a question it ought to have asked itself some time before: why did it keep getting trounced at general elections?

In 1992, 14 million people voted Conservative – a tally that remains the highest ever recorded by a party at a British general election. This success, and the fact that it was achieved against expectations and in defiance of most polls, led many in the party to believe that there was a formidable and enduring bloc of Tory voters that shared its values.

As support withered away during that parliament, and the 1997 election confirmed a seismic shift within the electorate, many in the party could not shake their belief that there remained 14 million voters who were, at heart, Conservatives, even if some had been bamboozled into voting for New Labour. These people, so the theory went, would soon realise what a disastrous decision they had made and come flocking back to the Tories.

It remained an article of faith for the Conservatives in opposition that they were somehow more in tune with the people than fashionable Mr Blair, even though the people had elected him with a gigantic majority and were clearly about to do so again. Inherent in the promise articulated by William Hague in 2001 – “come with me and I will give you back your country” – was the assumption that the great mass of voters shared the Tories’ discomfort in the direction Britain seemed to be heading. This assumption was wrong, as the voters themselves confirmed by re-electing the Labour government with a majority of 167 three months later.

Yet this view that the party had been speaking for mainstream opinion had been sincere. Where had it come from? It was in large part the result of the Tories talking and listening only to the people who agreed with them already. During this time, one former Cabinet minister was fond of saying, when presented with the latest opinion research, that the findings did not accord with those of his own focus groups, which he conducted on the doorsteps of his constituency every Saturday morning. It had evidently not occurred to him that the view from his prosperous southern town – one of 165 seats the Tories held out of 659 – had its limitations as a guide to the national mood.

Party activists can reinforce the problem. As I noted when one senior figure was alleged to have described Tory members as “swivel-eyed loons”, Conservative volunteers are decent, patriotic, industrious, generous, tolerant, charitable, open-minded, good-humoured, public-spirited and – if you discount their willingness to deliver leaflets and knock on doors in all weathers – sane. The same is true for members of other parties.

But for all its virtues, the Conservative membership does not look the same as the rest of Britain. This is an inescapable observation, not a criticism. Tory members are, by and large, older and better off than voters as a whole, and their political priorities do not always match those of the wider electorate.

This is not just a problem for the Conservatives, by the way. Tony Blair recalls in his memoirs how Labour activists would carry placards quite literally demanding “no compromise with the electorate”.

The fact is that political enthusiasm is a pretty unusual trait, and members of political parties are, almost by definition, not the same as other people. All parties, but the Tories in particular, still grapple with the question of whether a party should be a vehicle for the views of its members, and how far those members should grin and bear policies designed to broaden the party’s overall appeal.

But this problem had become acute in the Conservative Party in the early years of opposition, and little had changed as it headed towards the 2005 election. If the party was going to win again, the first step was to assemble a definitive body of evidence as to how the voters saw it and why they kept voting for someone else.

I decided to provide that evidence. I expanded my research beyond the marginal seats to look at these questions in detail, and published my findings in June 2005 under the unambiguous title Smell The Coffee: A Wake-Up Call For The Conservative Party.

Smell The Coffee concluded that the Tory brand had become toxic, that the party had become detached, out of touch, stuck in the past, and was, in voters’ eyes, essentially on the side of well-off rather than ordinary people. These themes have since become familiar in political discussion, but this was the first time they had been set out at length and with a mountain of supporting evidence. (Smell The Coffee is still available to download on my website, by the way, and dare I say is worth a read for students of our recent political history).

Above all, Smell The Coffee concluded that the Conservative Party needed to change quite radically before it could expect the voters to put it back in government. David Cameron read and absorbed these findings, basing his leadership election campaign on the need to “Change to Win”, and the rest, as they say, is history. After winning the leadership he appointed me the party’s Deputy Chairman with responsibility for opinion research and target seats, a role in which I served until shortly after the 2010 election.

This tale is an example of what I think is the most important function of opinion polling in politics, which is to be a reality check. It is always easy for any organisation that needs to engage the public – though political parties are the most prone to the mistake, for the reasons I have discussed – to make false assumptions about what people think and thereby misunderstand events.

After the last election I found that Labour supporters were in their own way no less disposed to misread things than their Tory counterparts. I asked swing voters, who had voted Labour in 2005 but not in 2010, why they thought the party had lost. Their three principle reasons were that Gordon Brown had not been a very good Prime Minister; that Labour did not seem to have the right answers on important issues; and that the government had run out of steam.

When I asked people in the Labour movement the same question, their answer was that people had failed to appreciate what Labour had achieved; that credulous voters had been misled by the right-wing media; and that although Labour’s policies were right, they had been communicated badly. Few thought Labour needed to make any major changes, since people would flood back as soon as they saw what horrors the coalition would unleash.

This helps to explain why, with six weeks to go before the election, the polls are as tight as they are – of which more later. It also illustrates that although the reality check provided by polls can be quite stark, that is not to say politicians will always like the reality they see, or do anything about it.

Used wisely, opinion research can help bridge the gap between the political class and the people it is supposed to represent. In an ideal world no such gap would exist, but few would deny that it does. Individual Members of Parliament are probably more in touch with the lives of their constituents than most people would give them credit for. But as a group they have a tendency to talk about their own preoccupations, which they hope voters can be made to care about, rather than what they actually do care about.

Sometimes politicians will claim that these preoccupations are shared by the electorate, and in extreme cases they will really come to believe that they are. Often, people will largely agree with the politician or party about the issue in question – but the issue itself will be relatively unimportant when it comes to deciding how they will vote.

Examples abound. In 2001 the Conservatives invited people to vote Tory to “save the pound”. This was one of the few issues on which they were on the same side as public opinion. But the voters saw that if the pound were ever in danger they would get the chance to save it in a referendum, and used the election to think about other things, like public services and the economy.

In 2005, the Tories tried to make the election all about immigration. Again, it is easy to see why: this was one of the few issues on which the Conservatives were more trusted than Labour. But the fact that the Tories had a huge lead on immigration but were behind on voting intention ought to have signalled that talking more and more about the subject was not going to have the desired effect. Once again, the voters used the election to think about other things……… like public services and the economy.

This is an example of what some in the political world like to describe as “framing”. This is the idea that if you can get people to see an issue or decision in a particular way, they will be more receptive to your version of the answer. We have seen it again in this parliament, particularly in the earlier years, with Labour trying to “frame” the economic debate as being only about growth; by engaging with the argument about the deficit, argued some gurus of the left, Labour would simply be playing into Tory hands.

I am sceptical about this approach. It amounts to saying that if you don’t talk about a problem, voters will come to believe it isn’t there – or that if you say the issue you are most comfortable talking about happens to be the most important issue facing the country, voters will come to believe it really is. I don’t think people are that credulous – a lesson which all parties ought to have learned by now. If an election is an exam, voters will set the question; parties that choose to answer a different question will be marked accordingly. Properly deployed, opinion research can help remind politicians that they are the ones sitting the exam, not the voters.

But the biggest gap between the political class and the rest of the country is not to do with policy so much as the way politics is conducted. Every small setback is portrayed as a disaster; every mistake is a scandal. Parties insist on questioning not just their opponents’ plans but their motives. It is not enough to point out that a policy might have undesirable consequences – it must be claimed that the people proposing it are intent on bringing the country to ruin and reducing your family to a state of penury.

Personal attacks on individual politicians are the ultimate manifestation of this tendency. As I have often pointed out, normal people are completely mystified as to why parties behave like this. Not only do they find it a pretty unedifying spectacle in itself, they wonder why politicians don’t seem to realise how exasperating people find it.

The usual answer is that parties do this sort of thing because it works. I doubt this is as true as is sometimes claimed. As I argued in Minority Verdict, my account of the 2010 general election campaign, people were already pretty fed up with the Labour government and had come to their own conclusion that Gordon Brown was not a particularly good Prime Minister. Their hesitation in voting for change was not over whether they could bear to live without Labour, but whether they could trust the Tory alternative. The Conservatives’ relentless attacks on Mr Brown were therefore not just ineffective but counterproductive. Every new assault was a lost opportunity to explain the Tories’ plans and reassure people about the party’s motives.

Why, then, is negative campaigning, especially the personal kind, so prevalent? I think it comes down to two things. First, attacking the other side is easier, not to say more fun, than putting across your own plans in a succinct and appealing way. And second, politicians and those who work for them love laying into the enemy. They relish the game, and that’s how the game has always been played. Once again, opinion research, if it is heeded, can be a reminder to try not to alienate too many of the spectators.

Language can be another barrier between the political world and everybody else. My research has often found that phrases that are common in the political lexicon can be quite baffling to others. Once in a focus group the subject arose of reducing the size of the state, to which someone asked, “what do you mean? Lopping off Cornwall?” Some people were initially confused about the idea of “free schools”, thinking it simply meant schools you didn’t have to pay to attend. Even basic political concepts like left and right are much less commonly understood than many in the political world would suppose.

More often, what the politician says and what the voter hears are two different things. When the Conservatives talk about their Long Term Economic Plan, for example, this sounds in some quarters like the promise of eternal austerity. And when parties warn of the unintended consequences of voting for their rivals – “vote Farage, get Miliband”, or “vote SNP, get the Tories” – it can easily sound like this: “you people are clearly too dim to understand what you’re doing; once you pull yourself together and start paying attention you’ll be back voting for us”. Not necessarily a winning message.

In these cases research can help ensure politicians are communicating in a way that means something to the audience, but this can be taken to extremes in what I call the search for “magic words”. This is the idea that sceptical voters will become true believers and you will win the day if only you can come up with exactly the right combination of words to describe what you are trying to do. The search for the perfect phrase seems an easy answer to – but is in fact a distraction from – the difficult business of winning voters’ confidence.

The listener does not separate the words from whoever it is that is saying them: to take two examples, the Tories gaining trust on the NHS, or Labour on the public finances, will be more than a matter of arranging the syllables in the right order.

So the reliance on polling to create superficial phrases as a substitute for better understanding what voters think is one of the ways in which opinion research can be misused in politics. Another common mistake is that of over-interpretation.

Nearly two thousand voting intention polls have been published since the last general election. Over time a trend emerges, but a graph plotting every figure from every poll would look more like a child’s painting of a rainbow than sharp lines on a chart. The laws of statistics say that each figure is subject to a margin of error of around 3 per cent, with 95% confidence – which means that nineteen times out of twenty, the numbers will be within 3 per cent of the truth.

Yet every new poll is pored over for its meaning, and explanations are sought for what are usually insignificant changes since the previous one. Some enthusiasts are also inclined dramatically to overestimate the impact of events. To take one real example – and I promise I am not making this up – a Twitter correspondent once asked, after I published some new figures, “ah, but was this poll done before or after the Shadow Communities Secretary attacked Waitrose for giving away free coffee?” Even major set-piece political events like the Budget and the Queen’s Speech usually have next to no impact on public opinion. As Tony Blair rightly observed, the biggest single mistake politicians make is to overestimate the amount of attention people are paying to them.

Some polls should be treated with a good deal of caution – particularly those that seem to show the public agree with the views of whoever commissioned them. These often come from lobby groups, and as the pollster Anthony Wells has pointed out, they often include questions that amount to: “How important is a worthy thing?” “How concerned are you about a nasty thing?” – or , as the blogger Hopi Sen puts it, “Would you like the government to give you a pony?”

Surveys like this sometimes purport to show that people would be much more likely to vote for a party that had policies the lobby group favours. These should also be taken with more than a pinch of salt, especially if the policies in question are on relatively minor issues likely to play little part in a general election.

Many people think a poll can be made to say whatever you want it to say, and a few sceptics regularly question my agenda when I publish new results. If a poll looks good for the Tories, it must be because I’m a Tory. If it looks good for Labour, I must be trying to give the opposition a false sense of security. Once I published, on the same day, results that showed the Conservatives behind nationally but doing well in a local by-election campaign. The conspiracy theorists had a field day with that one.

As so often, the key is transparency. You need not pay much attention to a survey unless you can see all the numbers, the wording of the questions and the order in which they were asked, the sample size, the demographics, how the data were weighted and any other important methodological steps. Only then can you judge whether it really shows what whoever has published it says it shows.

Extreme scepticism is also the correct response to any claims from a political party that their “private polling” shows them to be doing better than the published polls suggest. I described earlier how this practice – what I now call “comfort polling” – prompted me to begin my own research in the first place.

But the practice still goes on to this day. In the last year the Conservatives, Labour, UKIP and the Lib Dems have all been at it. Commendably enough, both UKIP and the Liberal Democrats have recently released private polling data. To nobody’s astonishment, each set of research found the party that commissioned it doing surprisingly well. There is always the suspicion that a party will only publish the most favourable results, keeping the gloomier findings under wraps.

But even when a party’s internal research is published it is worth studying how it was done. There are all sorts of methodological decisions to be made – such as whether to name candidates, whether to weight the sample according to how people voted at the last election, and what to do with people who voted last time but refuse to tell you or say they don’t know how they will vote this time.

All pollsters wrestle with these questions. But in what can only be a strange coincidence, the methodology the Lib Dems used was exactly what you would choose if you wanted to maximise the Lib Dem vote share; and every step of UKIP’s methodology favoured UKIP. What are the odds of that?

Perhaps most interestingly of all, both the Lib Dem and UKIP private polling was conducted by the same company. I wonder which set of results the company thinks is wrong.

But enough of my suspicions. Though political opinion research can be misused in the ways I have described, I think there is also widespread misconception about it. This boils down to the idea that polling inevitably becomes a substitute for principle: politicians will just parrot back to voters whatever they think they want to hear, becoming more and more inauthentic as voters become more and more disengaged.

There is obviously a danger there if our political class is careless enough to fall into that trap. Even if that happened, it would be the politicians’ fault, not the pollsters’. But I don’t think it need happen at all. Sometimes leaders will need to do things that large numbers of people don’t like – in which case it is as well for them to know what they are up against. Sometimes they will not be able to meet popular demands, in which case they need to know what they have to explain. Polling certainly can’t tell you the answers to all problems – as Henry Ford famously said, if he had started out by asking his customers what they wanted, they would have said “a faster horse”. But ultimately, I think Britain will be better governed if our politicians have a better idea of what people really think and why.

As for who is going to be doing the governing forty-five days from now, your guess is as good as mine. (Aren’t you lucky to have the Pollster-In-Chief here with you?) My latest national poll, published yesterday, had Labour and the Conservatives tied on 33%. My constituency polling so far suggests that Labour’s gains from the Tories in England could effectively be neutralised by their losses to the SNP in Scotland, potentially leading to something close to a dead heat.

In terms of sheer unpredictability, this is the most exciting general election I can remember. Unhappily, the voters don’t share my excitement – the reason for this knife-edge state of affairs is that people find the choice so uninspiring.

The fact is that all the established parties have failed the test the voters set them in 2010. The Conservatives needed to show that as well as getting to grips with the economy they were on the side of ordinary people and could be trusted with public services like the NHS – but they score less well on both measures than they did in 2010.

Labour needed to prove that they had learned the right lessons from when they were last in office and can be trusted with the money, but people’s biggest fear about another Labour government is that it would spend and borrow more than the country can afford.

The Lib Dems needed to use what limited influence they had to the best possible effect, but most people struggle to name anything concrete they have achieved – though their heavily localised support may well mean they end up with more MPs than the national polls would imply.

Meanwhile, people are clear where UKIP stand on immigration and Europe but are less sure what they think about anything else, and some worry that unsavoury elements lurk behind the entertaining Mr Farage. And to most people, the Greens still feel like a well-intentioned single-issue fringe party.

That is the choice as many if not most voters see it, certainly the undecided ones. It is often said that younger people are particularly disengaged, at least from party politics. This is not surprising – if the choice looks unappealing to most people, it must be even more so to those who are turning to politics for the first time and do not have old party loyalties to fall back on. Certainly I would expect students to be more wary of direct promises after what happened to tuition fees under the Lib Dems – though interestingly, Nick Clegg’s subsequent apology was not so much for breaking his pledge but for having made such a daft promise in the first place.

At the last election, according to the British Election Study, 62% of 18 to 25 year-olds turned out to vote, compared to 89% of those aged 65 and over. In the poll I published yesterday, just 38% of the 18 to 24 age group said they were certain to vote in the general election, compared to 70% of those aged over 65. People often ask why politicians do so little to engage young people, and why so-called election “bribes” seem to be concentrated on the retired, but I have just given you the answer. Decisions are made by those who show up, and older voters show up in droves.

As for myself, I will carry on polling right up to the day, publishing my regular national poll; the focus groups to tell us more about why people think what they think, rather than just how many people think what; and my research in marginal constituencies throughout the country.

It is these that give the best pointers to the real state of play on the ground: with the national vote share so close the result will be counted in individual seats, and the swing is far from uniform. One effect of these polls is that those inclined to vote tactically will have more information about their own area than ever before: those who want to get rid of Mr Clegg in Sheffield will have to vote Labour, Conservatives who have had enough of Labour in Grimsby will have to vote UKIP, and Labour supporters in Thanet South who do not want to be represented in parliament by Mr Farage will have to hold their noses and vote Tory.

All of which underlines the message I started with, that a poll is a snapshot, not a prediction. There is plenty of room for movement now as voters begin to resolve their dilemmas, and even surveys published very close to polling day will not be able to detect very late movements.

But on Thursday 7th May, one of the parties will get the most votes, and one of the parties – perhaps even the same one – will get the most seats.

Whether that party then ends up in government is another question altogether.

Links

Lord Ashcroft's Website
– Lord Ashcroft’s main website details his biography and work in the areas of business, politics, philanthropy, gallantry, publishing and journalism.

Victoria Cross Heroes
– First published in 2006, Victoria Cross Heroes was written to mark the 150th anniversary of the creation of the VC, along with many heroes awarded Britain and the Commonwealth’s highest decoration for bravery in the face of the enemy.

Special Forces Heroes
– First published in 2008, Special Forces Heroes continued Lord Ashcroft’s celebration of bravery by telling the stories of valour by the recipients of his Special Forces’ medal collection.

George Cross Heroes
– First published in 2010, George Cross Heroes told the remarkable stories behind the awards of the GC, Britain and the Commonwealth’s highest award for civilian bravery.

Heroes of the Skies
– First published in 2012, Heroes of the Skies is the fourth book in Lord Ashcroft’s bravery series and tells the stories of more than 80 British, Commonwealth and other Allied airmen awarded medals for gallantry in the air.

Special Ops Heroes
– First published in 2014, Special Ops Heroes is the fifth book in Lord Ashcroft’s bravery series and recounts the inspirational stories behind more than 50 groups of gallantry and service medals that are part of his Special Forces medals collection.

The Lord Ashcroft Gallery
– The gallery at the Imperial War Museum in London was opened in November 2010 and houses the world’s largest collection of VCs, amassed by Lord Ashcroft, along with VCs and GCs already in the care of the museum.