Whether or not Ukip gets its “political earthquake” in the European polls, it has changed the terrain of the next general election.

Out of his depth? Nigel Farage during flooding on the Somerset levels in February. Photo: Getty

Nigel Farage reveals a lot through the things he chooses not to do and the things he regrets having said. For example, the Ukip leader recently decided he would not stand as a parliamentary candidate in the Newark by-election due on 5 June. He conceded, according to BBC reports, that his party’s “bubble would burst” if he ran and lost.

Farage now denies using the “b” word, for obvious reasons: to concede that his current support is overinflated is to hasten the day that it shrivels. But if Ukip continues to harvest votes on anything like the scale it has done in recent months, it will determine the outcome of the next general election. Farage attracts support from across the political spectrum but causes roughly twice as much damage to the Tories as to Labour. As Ed Miliband’s core vote is bolstered by left-wing defectors from the Liberal Democrat camp, it won’t take much of a Ukip turnout in marginal seats to result in David Cameron’s eviction from office.

For months now, the Conservatives have been braced for a shellacking in the local and European elections, with Ukip the main aggressor. Downing Street’s focus has been on averting panic among Tory MPs in the short term and then, over time, chipping away at Farage’s credibility and respectability. The aim is for angry voters to see Ukip as a tool of midterm protest but not a valid menu item when it comes to choosing a government.

Central to this argument is the warning that Britain could find itself saddled with a pro-European, pro-immigration Labour government, although Tory MPs report limited success with that line. Many Ukip voters are sufficiently enraged with all of Westminster to see no significant difference between the major parties. Downing Street is becoming more aware of the limitations of “Vote Farage; get Miliband” as a message. One No 10 source tells me: “It will have to be more sophisticated than that.” The general election campaign will stress economic dependability. Cameron will be sold as the only candidate who can be relied upon not to poison the recovery with snake-oil policy prescriptions, whether bottled as Farage’s fearmongering nationalism or as Miliband’s wealth-destroying retro-socialism.

Yet the Tories know that they can only begin to push that message once the smoke raised by the European election result clears. This is where the Newark by-election – and Farage’s decision to sit it out – becomes interesting. The seat has a solid Tory majority of 16,000. With different boundaries in the past, it has returned Labour MPs. The present vacancy exists because the incumbent, Patrick Mercer, resigned over egregious breaches of parliamentary rules regarding cash for lobbying. It is the kind of contest that delivers an earth-shaking upset ahead of a general election when there is a prevailing sense that the country is about to jettison the government of the day – as in the run-up to Tony Blair’s 1997 landslide and Gordon Brown’s despatch in 2010. No one in Westminster detects such a mood abroad today. Labour is playing down its chances. The Liberal Democrats will be happy just to finish ahead of the Bus Pass Elvis Party.

Farage looked at the odds and decided that he, too, couldn’t win. He also meditated on Ukip’s handling of two earlier by-elections in this parliament. One was in February 2013 at Eastleigh in Hampshire, where the party came second and scooped the most votes cast on polling day. The Lib Dems held the seat only because they had organised a firewall of postal votes. The other was in February this year, at Wythenshawe and Sale East in Manchester, where Ukip again came second, beaten by Labour’s well-oiled get-out-the-vote machine. The second of these results wasn’t bad for the Faragists but was widely read as an embarrassing setback because, after Eastleigh, the expectations of a breakthrough had run out of control.

The lessons for Ukip’s high command from those episodes were that taking Westminster seats is a whole lot harder than heaping up protest votes in national opinion polls, and that the loss of underdog status poses a significant threat to the party. Once seen as the main challenger to the leading parties, Ukip is subjected to raised expectations of consistency and professionalism that it struggles to meet.

A big problem is finding candidates who have enough character to stand out in a campaign, thus easing the pressure on Farage to be constantly front-of-house, and without looking creepy or betraying views that make wavering supporters flinch. A very senior figure in Ukip once complained to me that it attracted “people who have failed at everything else in life, are feeling angry about it and are looking for someone to blame”. Last year, when there was chatter around Westminster of David Cameron facing a leadership challenge, the party tried hard to “turn” a handful of rebellious Tory MPs but none took the bait.

In any case, Tory defectors do not necessarily make good Ukip advocates. Until last month, the party’s campaigns director was Neil Hamilton, the former Conservative MP best known for losing a safe seat in 1997 thanks to his embroilment in the cash-for-questions scandal. Hamilton was demoted when Farage started struggling to answer reporters’ questions about how an emblem of Westminster sleaze fitted Ukip’s anti-establishment message.

In Newark, the party’s candidate is Roger Helmer, a 70-year-old former Tory MEP whose record of social commentary includes defending a policy to repatriate immigrants, claiming that some rape victims should “share part of the responsibility” for being attacked and sympathising with people who finding homosexuality “abnormal and undesirable”. (Nigel Farage told Jeremy Paxman on 19 May: “He was brought up in a traditional biblical upbringing. He lived as a young man in a country where homosexual behaviour was an imprisonable offence . . . You know social attitudes do change and we shouldn’t demonise people.”)

Ukip’s dilemma is that it relies on votes from people who wish such views enjoyed more currency in mainstream politics, while also feeling the need to dissociate itself from lurid prejudice. It is hard to disentangle the two impulses. Scrutiny of Ukip’s rank and file has flushed out a stream of councillors and activists who deploy the idioms and imagery of the far right – Holocaust denial, Nazi salutes and calls for expulsion of non-white people from Britain. This month, Sanya-Jeet Thandi, a British-born Indian former member of Ukip, published a withering attack on the party for “deliberately attracting the racist vote” and moving in a direction she found “terrifying”.

Nigel Farage’s jovial bluster is no longer sufficient to launder the more sinister views that swirl around him. He turns tetchy when challenged over his distaste for foreign languages spoken on trains. His casual conflations of Romanian nationality and criminal behaviour have prompted hostile comment from previously indulgent Tory-leaning newspapers. Farage seems unsure whether he should be defending the assertion or apologising for it.

It is not yet clear what the impact of any of this will be on Ukip’s poll performance. The party still enjoys some immunity by claiming to be the victim of smears by the “mainstream media” and the “Westminster establishment”. MPs from all parties are torn between the urge to denounce Ukip as a receptacle for xenophobic bile and fear of alienating their own errant voters.

On the Tory side there is growing confidence that Ukip support can only decline as the party acquires a disreputable air. The source of many press reports that embarrass Farage is Conservative Campaign HQ, where researchers scour pamphlets and social media for unsavoury remarks by Ukip candidates. Tory strategists recognise that undermining Farage is a job best undertaken at arm’s length. Too many direct attacks by Cameron would lend Ukip the status of equal adversary and risk reminding Tory dissenters of the times they have felt insulted by their own leader. “We need the racist thing to seep into public consciousness,” says an ally of the Prime Minister. “But it can’t be us saying it.”

Moderate Tories are beginning to see an opportunity to cleanse their own party of toxic associations if the perception of nasty extremism can be made to cling to Farage. Some MPs admit privately to being glad to see the back of some of the defectors from their local constituency organisations – a view reinforced by the abusive letters and threatening emails they get from the new Ukip recruits. “In a bizarre kind of way they’ve done the Tories a favour in the long run by making us look moderate,” says one Conservative who is defending a marginal seat.

Labour has been slow in waking up to that dynamic. At first, the opposition tended to view Farage’s strength as a helpful disruption of Tory support – a family feud on the right that eased Miliband’s path to Downing Street. Then it became clear that Ukip was attracting support from older, working-class voters who felt neglected by Labour in government, especially over immigration policy, but remained culturally immune to voting Tory. At that point, Miliband’s allies conceded that there was a potential hazard down the line but insisted it was not big enough to cost Labour seats in the 2015 general election. Only in recent weeks have aides started voicing concern that Ukip is dragging the whole political debate on to terrain that the Labour leader finds inhospitable.

At the start of the year, senior Labour figures were counting on the result of the European elections to trigger civil strife on the Tory side. Now they are frantically managing down expectations of their own performance, portraying the poll as a false indicator of national allegiances in which the voices of fanatical Eurosceptics will be over-represented – a free hit in which voters will punish all the Westminster parties. (As one strategist puts it: “A f*** you vote in a f*** you election.”)

The results of European ballots show a historically weak correlation with the outcomes of subsequent general elections. William Hague’s Tories won in 1999. A decade earlier the Green Party took 15 per cent of the vote. Politics then reverted to prior trends. Awkwardly for Miliband, that trend looks like a downward drift for Labour. A modest opinion-poll advantage is melting into the margin of error. At this stage of the parliament, an opposition expecting to win the next general election would usually have a commanding lead. Even in the unfamiliar landscape of coalition politics, with Ukip as a wild card, Labour should be the natural choice of people who voted Tory in 2010 and have since regretted it. “You’d expect them to be switching from government to opposition, not to some random third party,” says Ben Page, chief executive of the polling firm Ipsos MORI. “Labour just isn’t getting those switchers.”

Labour’s campaign has not been short of policy. There have been announcements about housing (capping rents; taxing unused properties), wages (beefing up the statutory minimum) and health (guaranteed GP appointments within 48 hours), among others. There have been aggressive assaults on Nick Clegg as the pathetic mascot of millionaire-hugging Tories. Labour has also trained its fire on Ukip, presenting it as a gang of hard-nut Thatcherites, bent on robbing workers of their employee protections and privatising the NHS.

To Labour’s great frustration, those messages have, in media terms, been peripheral to the noisier arguments about EU membership and whether or not a man who doesn’t want to live next door to Romanians is a racist. Miliband believes his focus on cost-of-living issues and fixing an economy that channels more wealth to the wealthy will look more relevant – prescient, even – when attention returns to the general election.

The European poll was always going to be a moment of intense interest in Ukip and a probable high-water mark in its support. Farage’s discomfort when interrogated about race and his difficulty recruiting presentable candidates can only increase when voters are choosing MPs to form a government, rather than lashing out indiscriminately at Westminster. Still, his capacity to harm Cameron’s prospects of re-election remains strong. The working assumption among Labour and Tories alike is that Ukip needs to take only about 7-8 per cent of the vote in 2015 to deprive the Conservatives of a Commons majority, which is a low bar for a party that regularly achieves double that score in the opinion polls.

For Labour, the threat is different but no less severe. Ukip’s reach may be limited by its status as a vehicle for protest votes but that gives it power to define the terms of protest in ways that harm the constitutionally recognised main opposition party. Labour’s priority is to look like a government-in-waiting but part of that image requires also looking like the main destination for people who don’t like the incumbents. Throughout the current parliament, Miliband’s defenders have credited him with courage in challenging the conventional political and economic wisdom; breaking with the old free-market orthodoxies; tackling “powerful vested interests” in the form of Rupert Murdoch, bankers, the big energy companies and (less convincingly) the dominance of Labour Party structures by the trade unions.

If that claim were resonating with voters, at least some of the insurgent energy that is fuelling the Ukip phenomenon would be lighting up Labour’s campaigns. Instead the party is lumped in with the Tories and Lib Dems as part of a shabby establishment stitch-up, with the added baggage of a reputation for economic mismanagement.

It is an old opposition conundrum: how to fashion a message that is dramatic enough to represent a credible alternative to the status quo, yet responsible enough to withstand scrutiny as a potential programme for government. Ukip isn’t bothering with the second part of the equation (which will be its undoing next year), but Farage is hogging the rhetoric of change and upheaval. His incendiary nationalism burns up the oxygen of publicity that Miliband needs to illuminate his milder offer of soft-left populism. That all suits Cameron to the extent that he is in the business of promising security through continuity.

While Labour and Tories have opposing reasons for wanting to see Farage thwarted, the basis for their arguments is the same. Downing Street aides and Miliband advisers both speak of the need to impress upon voters how high the stakes will be in 2015; how the ultimate question is whether Cameron is allowed to continue as Prime Minister – with one side warning that another term of Conservatism would finish off hope of fair rewards for all and the other warning that Labour would guarantee national bankruptcy. What they want, above all, is for the public to view the general election as a two-party race, with the Lib Dems and Ukip as sirens, luring in wasted votes and thereby abetting the real enemy.

How effective that combined effort will be depends on how bloody-minded people are feeling about politics in general. In 2010, the Tories were spooked by Nick Clegg’s sudden emergence from the first televised leaders’ debate as the candidate of political renewal – a mantle Cameron had hoped to wear. An aggressive campaign was launched, supported by Tory-leaning newspapers, to discredit the Lib Dem leader. It did not work, in so far as Cameron still failed to win a majority. Clegg was hobbled by the organisational incapacity of his party to capitalise on a surge of public interest, as much as by press vilification.

The Farage effect is partly analogous to Cleggmania, albeit played out over a longer timescale. Ukip’s appeal is the embittered, pessimistic inversion of the appeal that the Lib Dems once briefly had. Anti-politics is now soaking up the resentment caused by the coalition’s failure to deliver “new politics”. (And Ukip also attracts its share of former Lib Dem voters.) The Conservative newspapers are turning on Farage, which will hinder him, but not necessarily more than he will be hobbled by his own party’s institutional flakiness. Yet he doesn’t need to advance any further to alter the shape of British – or, more precisely, English – politics. That work is already done. Militant hostility to Britain’s membership of the EU and to immigration in general has infected mainstream opinion.

The threat that situation poses to Cameron lies in the electoral arithmetic – Ukip can cost him seats. For Miliband, it is a problem of momentum – Ukip has stolen his
insurgent thunder. The Tories spent too long chasing Ukip’s agenda; Labour spent too long ignoring it. Farage’s bubble will not suddenly burst. More likely, the air will seep out slowly over the coming year, by which time both Cameron’s and Miliband’s prospects of winning a majority may already be blown away.