The feeling grows that David Cameron is a leader with nothing left to say

Harold Wilson was wrong. A week is just as long in politics as anywhere else. Only the vanity of politicians leads them to imagine that their hourly dramas, epic though they feel from the inside, should stop the clocks for other people.

Westminster careers can certainly end abruptly – Chris Huhne made a speedy transition from cabinet minister to jailbird – but individual scandals don’t have much impact on elections. Only when sleaze comes in clusters, as in the last years of the Major government, do private vices swing public opinion.

The same is true of U-turns. Occasional retreats are ignored or forgiven. Constant pivoting, as practised by the coalition, invites ridicule. The latest panicky policy adjustments are the exemptions for a few needy cases from the “bedroom tax” – a surcharge levied on those in social housing with spare rooms – and the abandonment of minimum alcohol pricing, a temperance measure that many Tories despise as meddlesome.

Each capitulation makes the rest of the government’s agenda look conditional. That irritates MPs who have been dutifully plugging unpopular policies, holding the line that there is no alternative, only to see ministers crumble, thereby proving that there is always the alternative of surrender. “If you push [the coalition] hard enough, it gives,” one usually loyal Cameroon says. “It creates the impression that there’s money behind the sofa and it makes us look incompetent.”

The sense of chronic impermanence in Downing Street also fuels leadership speculation, which is another of those things that is inconsequential in isolated episodes and crippling in aggregate. Voters respond to the scent of defeat wafting from the sweaty body of an anxious party before they even start to evaluate its leader’s record. There is barely a cabinet minister whose name has not now been whispered as a possible successor to David Cameron. No one’s prospects have actually advanced but the signal has been sent that some Tories are giving up.

It still looks like a mistake to replace Cameron when there is no superior candidate available. Coup prophesies, however, can be self-fulfilling. A critical mass of speculation diminishes the leader’s authority until the balance of risks changes and the more hazardous path becomes sticking with an incumbent who has been branded a failure.

That point is still some way off, but Cameron’s allies need to do a better job of defending him. Their weakest argument is the one they deploy most often, which is that opinion polls show the Tory leader to be more popular than his party with swing voters. This only proves that Cameron has failed in his ambition to change perceptions of the Conservatives as a whole. Since that failure cost him a majority in 2010, it is unclear how boasting about it helps.

Cameron’s personal ratings might be more of an asset if he could claim to have tempered his party’s immoderate urges but he hasn’t. Besides, the Liberal Democrats are already in that space, arguing that coalition dilutes the Tories’ pungency. That provokes Conservative backbenchers to demand that Cameron tout his credentials as precisely the kind of Tory he once promised he was not.

The weapon that Downing Street strategists clutch with increasing desperation is the Prime Minister’s undisputed ability not to be Ed Miliband. A lesson that top Cameroons cite from their time in opposition is that the image they crafted for the new leader in 2005 was, with a few nicks and dents, the same one he carried into the 2010 election: the moderniser-in-chief. Even when he was attacked for faking cosmopolitan moderation, the question of character was debated on terms he had chosen.

The same cannot be said for Miliband. The messy circumstances of his 2010 victory in the race to succeed Gordon Brown – delivered by a trade union block vote at the expense of his brother – became the story before he had a chance to write one of his own. Then the complex business of holding the party together took priority over the task of giving definition to the leader.

No 10 calculates that it is now too late for Miliband to control his image and he will fight an election rebutting his enemies’ claims about what he stands for. The rejoinder from Miliband’s team is twofold. First, it doesn’t see Cameron’s record as a case study in victory. Second, its candidate is building a project of more intellectual substance than the mere rebranding exercise that was Tory modernisation. Miliband, his advisers say, is redefining social democracy for the 21st century. “To establish that agenda takes time,” a senior Labour strategist tells me. “It isn’t something you do in the first 100 days and then stick with for the next five years.” In other words, don’t ask for a finished account of what Miliband will offer the country when he hasn’t finished it yet – but time is more on his side than on Cameron’s.

A big factor in the equation is the Fixed-Term Parliaments Act 2011, which seals off the emergency exits from coalition and all but guarantees that polling day is 7 May 2015. By then, Cameron will have had a decade at the helm of his party. Everything that voters could want to know about him will be known, while Miliband still has the capacity to surprise.

That doesn’t mean it isn’t dangerous for the Labour leader still to be so fuzzily defined. But it is no less dangerous for the Tories to serve out the rest of this parliament under a leader with nothing left to say.

Rafael Behr is political columnist at the Guardian and former political editor of the New Statesman