A few weeks ago the British Prime Minister indicated to the American
President that he would provide support for action against Syria - and
recalled Parliament for approval. The vote went against him, and the
course of history was at least slightly deflected: the military action
did not take place when intended, and the President has himself gone on
to seek broader approval before committing to action, creating a
potentially important precedent for the future in this type of case
(neither Clinton nor Obama sought Congressional approval for air strikes
against Iraq in 1996 or 1998, in Kosovo in 1999 or Libya in 2011).

David
Cameron had expected to win the vote, if narrowly. All sorts of things
might have affected the outcome - a YouGov
opinion poll was widely credited with being prime among them. On August
28th, the day before the vote, the front page of The Sun read “Brits
say no to war in Syria” and citing “the first poll on the new crisis”,
showing a ratio of two to one against missile strikes. The front page
of The Times featured the same poll, saying that it “suggests that
voters overwhelmingly oppose… the use of British missiles against
missile sites inside Syria.” It was also carried in the Daily Mail and
The Independent , including data comparing the case made for the 2003
invasion of Iraq with the case for intervention in Syria.

It's extraordinary that at this stage of the electoral cycle, and after years of a pretty miserable economy with further cuts coming, the opposition party should be less than 10 per cent ahead. Were it not for the continued strength of UKIP, we'd be confident of seeing Cameron remain in Downing Street.

Perhaps it's not so surprising when one considers Ed Miliband's recent interview for The Guardian. In a new effort to relaunch his offensive, he invoked Labour policy on sardines in 1945: “This is a government that banned the import of sardines because they were worried about the balance of payments. It shows a government can be remembered in difficult times for doing great things.”

The point is entirely reasonable and interesting, but the audience for it is tiny. Who outside academia could engage with this?

People
are not good at predicting their own behaviour, so pollsters are wary
of drawing conclusions from questions that ask "what would you do
if...?" The standard voting intention question itself - 'if a general
election
were held tomorrow, which party would you vote for?' - is tricky to
answer: it invites people to use the question as a proxy, to express
approval or (more likely) disapproval for the government today, rather
than make a realistic prediction about their future
choice.

That of course is why oppositions need to be significantly
ahead between elections if they are actually to win at the deciding
moment, when the risk-averse impulse of 'better the devil you know'
comes into full force. That's why we often see a pro-government
swing when the election campaign formally starts. Remember April 2010?
The Conservatives were comfortably ahead in all polls, and everyone
expected a clear win; but YouGov’s 'starting-flag' survey for the Sunday
Times showed a sudden drop that indicated we
were heading for coalition.

There is a widely-held and wrong assumption that only a small section of the electorate is open to being persuaded from their current political voting intention to a different one. A second, also wrong assumption is that these few voters are located along specific parts of a supposed political spectrum, for example where left and right blur into each other, and that the strategy for winning elections is to understand specific narrow band and target it.

Imagine you are going into battle and alongside you in your struggle to survive you can have either mercenaries or regular soldiers - which do you choose? Easy: the ones you can unthinkingly trust. Well, suppose the mercenaries are clearly superior in the craft of fighting; but the regulars are unquestionably, fundamentally, loyal? With mercenaries you are constantly looking over your shoulder to see if they're still on your side or if the enemy has offered them more money; but then again, they may wield the knife so much better. A tough decision.

Last week, YouGov-Cambridge held a conference on restoring trust in banking, and this contrast in the cultures of the mercenary versus the committed regular was used by Dr Tom Simpson to illuminate public attitudes to bankers. Simpson, a philosopher at Sidney Sussex College and, appropriately given his theme, a former Royal Marine who earned the Sword of Honour, suggested that people distrust bankers because they seem like mercenaries, their motivation appears to be primarily financial; there is no natural loyalty to the customer. This applies to the Big Bonus Banker, but not to the person who helps you at the desk (in the YouGov study only 13% trusted investment bankers to tell the truth, bottom of the pile, while 67% trusted ‘staff at my local bank’).

The idea is compelling: if banks feel that the only way to ensure the continued loyalty and effort of their employees is to constantly shower them with huge sums of money, how can customers and citizens trust them? We must constantly be looking over our shoulders, we feel, to be sure such people are really using their special talents on our behalf, and not just serving their own pecuniary interests. That's why the public craves tighter regulation.

This is the essential problem of the banking industry in the court of public opinion, and we can hardly be surprised if politicians are tempted to use - or mis-use - this opportunity to advertise their allegiance to the people: anti-banking legislation is a very tempting reaction.

Politicians themselves are constantly subject to the same litmus test of self-interest versus social solidarity. Are they fighting for us, or for their personal ambition? Are they saying what they believe, or are they more concerned with winning our votes? Do they side with ordinary strugglers in their constituencies, or with those who will advance them in Westminster? Are they authentic, or just pretending? Are they, in other words, mercenaries seeking the highest bid, or regulars, fighting for what they believe? We search for the extrinsic and intrinsic motivations for their behaviour.

In the political tussle for trust, Conservatives find themselves at a clear disadvantage to Labour. The ideas of the left sound so much more people-friendly than the ideas of the right. Labour are on the side of the underdog, Conservatives celebrate success. Even if we are not ourselves underdogs, we grant Labour more trustworthiness for it. They look less like mercenaries.

Conservatives appear to be the natural allies of the bonus-rich banker. But the way to change that image is not to bash the banking industry. The truth is that bankers and Conservatives must follow the same path out of the valley of mistrust.

As the banking industry is now realising, redemption lies on the same path for all good companies: by serving the customer well. There isn't much to be gained by mere message, by the boasted policy; they can best rebuild their reputation for dependability by truly focusing on their clients, who currently feel abused when they try either to save or borrow or just to manage their accounts, but who could creditably be won back with attention to their real needs. That is how to dispel the image of the mercenary.

Conservatives can also win back a good reputation by the same route: by service. They are in power - they can use it to fix things for the citizen-customer. When older people recall how life was different before and after Thatcher, it's not the 'freedom of choice' policies they most readily recall, but their effects: for example, how the months waiting to get a new phone line installed was replaced by same-day service.

If companies succeed by making their offers better or cheaper - and often both together - why shouldn't it work for politicians? Why can't we expect them to perform better at the basic craft of governing? I once saw an election poster abroad with the unexciting slogan, 'Competence guaranteed'. I dismissed it as far too pedestrian, but these days it could sound rather comforting.

Conservatives love to say that we should roll back the frontiers of the state, that we would do better with smaller government. No doubt. But what about making sure whatever size government we do have actually functions well? Politicians love the big ideas, but what about small improvements in how things are run? Conservatives have often achieved management skills at the local level, but sometimes struggle at the national level. Perhaps if you don't really believe in the value of government then you'll never be great at the craft of it. I suspect for voters, competence in running things is taken as a mark of respect. It's something a 'regular' would care about, though a mercenary might not. Social solidarity is best built by noticeable improvements in service.

Two or three unspectacular budgets from now the Conservatives could find themselves with five more years of power. It's usually worth betting on mean reversion (I can reach Olympian levels of dullness on this subject, but the fact is hedge funds the world over make plenty of money that way). Over time, things tend to go back to average: they are rarely quite as bad or as good as they seem. Take the recent Lord Ashcroft poll of the marginals.

This invaluable research showed voting intentions in the seats that actually matter at a level implying a 84 seat majority for Labour. But hang on: a very similar poll YouGov conducted for PoliticsHome in 2008 showed a 145 seat Conservative majority if voting had taken place randomly at that moment, and an exact repeat in 2009 showed 70, but by 2010 the reality was... less than zero.

Voters tend (not always, not inevitably, but usually) to be most anti-government a few years before an election, and then swing back at least a little. Miliband needs a notably bigger lead over the next year to feel on course for victory.

The economy, happily, is also subject to mean reversion. Although most forecasts are dire right now, YouGov's monthly in-depth economic confidence index HEAT (household economic activity tracker composed of eight measures) shows the beginning of an upward trend to more normal levels.

At last weekend's ConservativeHome conference (write-ups here), the stated focus was how to achieve an absolute majority for the Conservatives in 2015. But in truth, no-one believed it; they were really discussing 2020. The fact is that governments only add votes in very rare circumstances, and we are not in those circumstances: Eastleigh confirmed that a) LibDem seats will not easily collapse and b) the strong post-war trend of the two major parties losing vote-share is set to continue; and only Labour could win a majority on less than 40%.

The chart above is the most important of all when considering what happens next. It suggests we face a future of coalitions. The parties have become so similar, and so removed from their traditional base (indeed any base), that it is hard to imagine either of them approaching 50% without taking a radical path. It is more likely they will become practised in the art of the power-sharing deal.

This may after all be no bad thing. The similarities between parties may be the result of a maturing system which produces pragmatic compromises. After all, in spite of so much clamouring for more distinctive manifestos, few propose any strong and credible alternative solutions.

The two biggest obstacles to winning elections are intellectual laziness and physical laziness. It's inherently very hard for the Conservatives to win an outright majority given the current state of the economy, the existing constituency boundaries, the poor condition of the campaigning machine, the trickiness of coalition, the absence of any real of pro-Conservative enthusiasm, and the historic trend against the two-party establishment. And yet pundits still try to offer strategic solutions in a few choice words.

Take this statement: "Tories can't outflank UKIP on the nutty right, they must fight for the centre". Nine out of ten wannabe gurus nod happily at that. But how useful is it really? One classic test is to state the opposite and see if that has any meaning; so: "Tories can outflank UKIP on the nutty right, they must fight for the extremes". If this second statement is obviously daft, then the first statement is obviously pointless. It provides no useful clue on how to win a campaign, it's just about the posturing of the author.

This is where the guru is dangerous to campaigns: truisms are obstacles to genuine analysis and useful action. Right now, the battle of the pundits is focused, absurdly, on defining the exact singular location of the hidden hoard of votes needed for victory: to the left, or to the right? In the centre, or on the flank? Under that bush, or behind that tree? Armchair strategists are having fun.

Politics in Britain today is still about gurus and wannabe gurus; it ought to be about geeks. The edge in today's game is to be had in campaigning by numbers - it's data that will let you maximise your vote, it's data incorporated into your mobilisation machine that could take Cameron or Miliband from mere coalition to an overall majority. Anyone who doubts it should read Sasha Issenberg's excellent account of Obama's big win in bad circumstances, 'Victory Lab: The Secret Science of Winning Campaigns'. Unfortunately, for a few more years, it's likely to stay a secret - at least for practical purposes - in our old-fashioned, creaky political machines.

Obama's campaign was able to secure victory not by some mysterious quality of inspiration or leadership or great political will, but by geeks who understood how to win extra votes with extra information at the micro level. It's not about big ideas, what matters is tiny real-time experiments which gain a few extra votes in ten thousand nooks. I should declare an interest (and also a proud boast): YouGov/Polimetrix, the US arm of my polling firm, played a key role in creating the data that transformed the Democrats' campaigning organisation from 2006, a process described in the must-read sixth chapter of Issenberg's account, 'Geeks versus the Gurus'. Our huge-scale polling combined with DNC datasets helped to create a likely-to-support predictor and likely-to-vote predictor person-by-person across the country. It replaced the absurdly unusable stereotypes of conventional political strategy (like, in the UK, the Worcester-women and the Ford-Focus-Essex-man) with individual case-level media and message tactics.

Most people in politics prefer gurus, not data. Gurus are people with big ideas about what to do - expressed with lots of confidence and almost no genuine evidence. They rely on the kind of clever talk that can never be proved right or wrong. The good ones throw in a few facts and canny observations, the bad ones talk about hunches and what they feel in their gut, but none of them will help you win. In contrast, the geeks are scientists, and most particularly computer scientists armed with sophisticated statistical analysis programmes and mountains of raw information. Instead of having a few big ideas about swing voters and spreading them pointlessly across the airwaves, they ask - and answer - thousands of tiny specific questions: how will this voter in this house watching this obscure gardening programme on this local cable channel react to this or that particular message? Now let's do it, feed the response back into the database and crunch it again for the next prediction. And again, and again.

Sometimes polling is misleading rather than illuminative - especially if one is too desperate to find evidence for one’s own point of view. Consider this question, which ComRes ran last week: “Agree or Disagree: I would have considered voting Conservative at the next election but will definitely not if the Coalition Government legalises same-sex marriage”. 14% of those who had identified themselves as likely Conservative voters agreed. Some have concluded from this that legalising same-sex marriage will have a decisive negative effect on the outcome of the next general election.

It’s possible, and I can’t definitely say whether it will truly happen or not. But it seems so unlikely, so very far from any known voting behaviour, that it staggers me it’s taken seriously by anyone. Ask people what are the most important issues for them at the next election – the economy, the NHS, schools, tax, immigration, or same-sex marriage – and we all know which would come last. But put the question the way ComRes put it and you give the respondent a ‘free hit’ to vent some irritation. Especially when the warm-up question is the suggestive “Agree or Disagree: David Cameron's plan to legalise gay marriage is more to do with trying to make the Conservative Party look trendy and modern than because of his convictions.”

This is no disrespect to ComRes. It’s not unreasonable for them to run the question. It’s not an illegitimate question. Such questions can occasionally shed a little light. Maybe this one allows us to measure the extent of anger among some Conservatives. But no sane person would grant this any kind of predictive value. Rather obviously, the next election will not be won or lost on the issue of gay marriage.

In a few hours, David Cameron will make a historic announcement that he will offer an in/out EU referendum after a period of negotiations for an improved relationship. It looks like a bold move, but it's also the safest bet: we want to feel in charge of our destiny, but we also want to stick together.

At YouGov we recently went back to our database and re-analysed some polling conducted between autumn 2009 and the 2010 election. It was one of those exciting moments you get from working with longitudinal data, when you can see how things really changed: of those who told us six months before the election that the were "absolutely certain" to vote for the party they then preferred, 20% had changed their voting intention by the start of the campaign. This accords well with the 'choice blindness' study I described last week which demonstrated that even strong opinions could be reversed in five minutes without the opinion-holder actually noticing.

So it should come as no surprise that the majority for an 'out' vote that we've been recording in our in/out EU referendum tracker should have suddenly collapsed (as my colleague Peter Kellner noted here from 51-30 to 34-40. That's a 27% swing in favour of EU membership in just eight weeks (from the end of November to last weekend). What happened?

One of the givens of political campaigning is that values are deeper than opinions. This may be true, but unfortunately neither is very deep. Recent research using a ‘magic’ trick demonstrates this very cleverly.

Hall, Johansson and Strandberg (in ‘Lifting the Veil of Morality: Choice Blindness and Attitude Reversals on a Self-Transforming Survey’, 2012) asked volunteers to fill out a paper-based survey presented on a clipboard. A thin film of paper covered one version of a question with an opposite version (click here for pictures and a fuller description). When the volunteer filled in the survey and turned the page, the answered version of the question became automatically removed and a new, contradictory version was revealed on the page. Magic.

So, the respondent saw the questions like “It is more important for a society to protect the personal integrity of its citizens than to promote their welfare”, and then agreed or disagreed on a ten-point scale. On turning the page, the answered question transformed into its opposite: “It is more important for a society to promote the welfare of its citizens than to protect their personal integrity”. Some volunteers got the question in a more concrete, issue-based version, “Large-scale government surveillance of email and Internet traffic ought to be forbidden as a means to combat international crime and terrorism’ – with ‘forbidden’ later turning by magic into ‘permitted’.

In the first of a series of five myths and truths of polling Stephan Shakespeare examines the idea of The Big Swing. Follow Stephan on Twitter.

As we start getting serious about the next general election, we should remind ourselves of some of the basic political realities measured or illustrated by polling. First up, and perhaps the most important for the campaign planner who actually expects to win, is that magical ‘Big Swing’ which at some point lights up even the limpest horse-race, but which is almost always a pure phantom, an in-built illusory effect of our desire for narrative structure.

Remember all that noise about how Romney was closing the gap – nay, winning! Panic in the Obama camp - all hands to the pump! But the race ended exactly as it began, with the President a few points ahead. YouGov’s contention was that Obama was ahead by about the same margin every single day of the election (every one of our many polls had him between 1% and 4% ahead). The only real excitement came from observer over-reaction to error in the polling.

Let’s focus on what was supposed to be the critical moment: the first presidential debate. The polls told us that Romney outperformed Obama. A swing duly followed – Gallup even showed Romney 6% ahead. Republican pundits stoked up the excitement and almost all observers were taking it seriously. But by polling day it was all gone. Two swings? Or none?

"To win outright, Cameron needs a miracle" - so said my YouGov colleague Peter Kellner yesterday. How different to just one year ago. In those over-confident times I had cabinet ministers willing to bet cash they'd achieve a majority in 2015, and even my Labour friends had given up and were strategising for the election after. I was surprised at just how settled the Conservative optimism (and Labour pessimism) then seemed.

Most of us share a fantasy: that really smart well-informed people meeting behind closed doors would make better decisions for the long-term national interest than a bunch of squabbling politicians grubbing for the votes of an unruly mass via our distorting media. How can childish parliamentarians playing to the gallery, booing and hissing in the Chamber, point-scoring on Newsnight then smooth-talking on the Today Programme possibly be an appropriate medium for serious debate and optimal resolution?

Michael Portillo once told me how in Cabinet he would look around the room and wonder if this group of people around the table were really running the country. He imagined there was a secret door leading to another room in Downing Street where "the real cabinet" was meeting. We probably all share a belief that somewhere there are Wise Ones who, if only we could let them get on with it, would sort things out properly. But where are they?

If we can’t have wisdom, at least we want "evidence-based policy". Well, I refer once again to the amazing evidence of Professor Philip Tetlock, an academic at the University of Pennsylvania who made a 20-year study of the forecasts of 284 experts from academia and practitioners, including viewpoints from Marxists to free-marketeers; 28,000 predictions were tested, and the analysis showed that they were only slightly more accurate than random, and worse than basic computer algorithms. It’s an eye-opening study and if you want to check it out for yourself, here’s the Amazon link.

If expert opinion isn’t reliable, what about public opinion? It may not be informed, deliberated or even truly measurable, and yet it dominates decision-making at every level of government: nobody wants to be seen going against what the voters appear to want. So is that a good alternative, can we trust in the "Wisdom of Crowds"? I suspect a scientific test of that proposition would look a lot like Tetlock’s study: slightly better than random.