David Cameron faces schism in the Conservatives. Photograph: Alain Jocard-Pool/Sipa/Rex

Not so long ago, the safer electoral bet was that Ed Miliband could not become prime minister in 2015. He lacked leadership qualities. His personal appeal was underwhelming. He was not palatable to southern middle England. He assumed a leftish take on the economy, more resonant in the nation's seminar rooms than the instincts of aspiring voters.

None of these factors has changed. Yet Mr Miliband now has a better chance of hiring a furniture van bound for Number 10 than he had before the summer – and the reasons are not to do with him or Labour's still middling performance.

They are a reflection of the increasingly schismatic nature of the Conservative party, and a burgeoning disregard for the discipline and teamwork needed to win elections, let alone one as hard fought as the next one. The defection of the idiosyncratic but significant figure of Douglas Carswell to the Faragist ranks was a baleful moment for David Cameron, because it expresses several strategic failures in Number 10 in one dramatic moment.

To hijack the M&S slogan, Mr Carswell was not just any Eurosceptic: he was the Eurosceptic the Tory top team had put most effort into keeping on board. When the PM forged his renegotiation strategy on the EU, one of his closest cabinet allies told me that he had designed it to satisfy the "Carswell test". That meant a position that gave the PM some (if not much) negotiating leverage in Europe, but which the most cerebral of the anti-Brussels backbenchers could back in good faith.

William Hague was therefore right to quip that Mr Carswell is "the only person in British history to leave a political party because he was 100% in agreement with it". What Mr Hague neglected to add was that the Clacton MP has concluded that the PM's renegotiating plan is threadbare and unlikely to produce a figleaf of Eurosceptic credibility, let alone a major shift in Britain's relationship with institutional Europe. That sentiment also informed Chris Kelly's decision to stand down as MP for Dudley South and why Michael Gove, freshly inaugurated as chief whip, cleared his diary for an emergency lunch with Mark Reckless, the Rochester MP who had also been rumoured to be flirting with Ukip.

Most painful for Conservative enforcers was the evidence that Ukip, generally regarded (with some reason) as an organisational disaster, had managed a coup de theatre in attracting Mr Carswell without a leak. Masterminded by Stuart Wheeler, the former Tory donor and now Ukip funder, it demonstrated a killer instinct from Team Nigel that one cabinet source notes "showed that they could actually deliver on all the bluster".

The other damaging insight for Team Cameron is that the sage of Clacton quit for reasons with which many waverers on the right would identify. If German voters, in a political and media culture that is overwhelmingly affirmative about the European Union, are toying with the sceptical party, Alternative für Deutschland, you would have to be a true Euro-believer, or just purblind, not to recognise that a considerable number of Britons feel the same way. Proportionately, Conservatives feel this most of all.

But this has long been the case. The question for a pragmatic Tory leader is how they manage the tension. When it comes to the lure of Ukip, the answer has been pretty badly. I would love a free lunch for every time some Conservative has reassured me that the Farage bubble would burst, that the voters tend to return to the Conservative fold in general elections and that we are approaching "Peak Farage", after which calm will be restored and what Boris Johnson referred to as the "glide path to victory" would be resumed.

The obstacle to this is that the rejection of centre-ground, deal-making, election-focused politics has spread in the past weeks beyond mere Euroscepticism and into the body politic of the right. You can see it in small but telling ways. One has nothing to do with Europe at all – the uproar about the Speaker, John Bercow, and his ill-fated attempt to hire Carol Mills, an Australian official, as clerk of the Commons. It was a vainglorious move, which has ended in an embarrassing backlash. Mr Bercow is ardently disliked by the Tory benches, who regard him as a Labour stooge. But the opprobrium heaped upon him in the Commons at prime minister's questions from the Tory side was a reminder of just how capable the party is these days of getting its priorities in a twist about something which is vastly irrelevant to voters. Do we imagine that the "Bercow test" will swing results in key marginal? Assuredly, we do not.

I understand that Mr Bercow's grating style and heavy-handed interventions annoy the Tory benches. But the fixation on the Speaker (which is indulged in as much by David Cameron and George Osborne as it is by the foot soldiers) is a monumental distraction from an election a mere eight months away. A party that fritters energy on such things looks ill-focused and self-indulgent.

The uppity mood is not restricted to the goings-on in Westminster. I bumped into a relative of a senior Conservative cabinet ministers at a shindig in the shires. Look around this gathering, he said, gesturing to a well-padded gaggle of Conservative Brahmins, including generous donors and constituency chairmen. How many of them, my acquaintance asked, had supported gay marriage? How many had ever wanted the leadership's green-wash, windmills included? How many of them had groaned when Mr Cameron had insisted that he would campaign for a yes vote on the EU in a referendum? No wonder, he concluded, that many of them were flirting with Ukip or simply withholding support from Mr Cameron.

Another way to look at it, I ventured, was that eschewing the present incarnation of mainstream Conservatism was likely to benefit a fringe party, hit mainly Tory seats and aid Mr Miliband. So the wiser option would be to grit teeth, hold a strong line on tax and spending to discomfit Labour, and postpone internal wranglings until after next May. This argument cut no ice, whatsoever.

The thought thus begins to dawn that there might well be a stronger desire to be vindicated on a hotchpotch of convictions and aversions than to win power by facing up to the big, difficult stuff and awkward compromises of governing a modern country. Any leader faced with this mood must feel a chill of despair. And Mr Cameron has the unenviable pincer of the cheery populist Mr Farage to his right and the other cheery populist, Boris Johnson, to both his left and right, depending on the London mayor's target audience on the day.

It seems bizarre that the party should have tired of power after a single period in office. Yet many Conservatives have never felt that the last election was "won" and coalition has frayed already tenuous ties of loyalty to Mr Cameron.

In this fevered mood, missteps matter a lot. Getting rid of Owen Paterson as environment secretary in the summer reshuffle was one of them. True, Mr Paterson had underperformed during the floods. A candidate less likely to win the "I feel your pain" award for empathetic politics is hard to imagine. But he had symbolic heft as a standard bearer of proto-Thatcherite thinking. He too has been playing a bit of footsie with allies of Mr Farage, including Crispin Odey, the affable hedge-funder, who has some Ukip sympathies. Mr Odey lunched not long ago with Mr Paterson, whose office last week explained things in a manner ripe for the Radio 4 News Quiz: "They know each other from their days in the leather trade."

Were he sitting more comfortably, Mr Cameron could shrug off special pleading from the bolshie squierarchy. But he is not. Lord Ashcroft's latest dose of cold reality from the key marginals still gives Labour a 14% lead in more than 30 of the closest Con-Lab constituencies, with the Ukip vote-share share tripling since 2010 election. In his favour, the Tory leader has better leadership polling by a sound margin and the economic argument has never really felt as if it was going Ed's way. Nor, really, has the sense of momentum and alchemy that makes one leader feel like the right one for the times.

The election campaign should bring these things to the fore and Mr Cameron has always been at his best when his back is against the wall. But he urgently needs to restore a broader sense of emotional connection to his party. If Nigel cannot be defeated on populism, nor should Dave seek to outmanoeuvre him on improbable pledges of British isolationism. But the sense of a Tory leadership aloof and out of touch needs to be addressed. He has a only short while to persuade many on his own side to bowl for him with any vigour. If he cannot convince them, it is unlikely that he will convince enough of the rest of us.