In the summer of 2001, when David Cameron had only been an MP for a matter of weeks, he went to see Iain Duncan Smith in his office at Westminster. In that year’s general election, Tony Blair had just wiped the Conservatives off his shoe for a second time, securing another landslide victory. Dazed Tories wondered what on earth to do next.

When in doubt, the modern Conservative Party tends to have a leadership contest. So it was that IDS, Ken Clarke, Michael Portillo, David Davis and Michael Ancram came to compete for the right to succeed William Hague. IDS, now Work and Pensions Secretary in Cameron’s Cabinet, would eventually draw the short straw, but in July, the leadership election was still ongoing. “Raging” would be too strong a word for such a desultory campaign, taking place against a backdrop guaranteed to depress Tories: Tony Blair was triumphant, the Iron Chancellor, Gordon Brown, said boom and bust had been ended, and it was fashionable to wonder whether the Conservatives could ever win power again.

In search of inspiration, Mr Cameron, like many of his colleagues, toured the offices of the leadership hopefuls, sizing them up in an effort to decide which one to back. There would be various ballots involving MPs before the final choice went out to almost 300,000 party members. (That is not a misprint: relatively recently, the Conservative Party really did have 300,000 members.)

As the highly confident MP for Witney waited to be ushered in to see IDS, a veteran rebel of the wars over Maastricht, he fell into conversation with various others in the outer office. Cameron said he wanted the new leader to be someone who could put a stop to the party’s infighting on subjects such as Europe, in order that it might concentrate on winning again. Why, he asked, can’t we all just get along?

It is in that pragmatic spirit that Mr Cameron has conducted his European policy since he himself became leader more than seven and a half years ago. It turns out to have been a naive hope. Put that question to more excitable Tory MPs – why can’t we all just get along? – and the answer turns out to be: how long have you got?

But the events of the past 48 hours have marked a new low in Mr Cameron’s leadership. In particular, there has been a serious erosion of the Prime Minister’s authority. Mr Cameron did not want a Bill in this parliament on an EU referendum. Then, in a sudden effort to encourage Tory MPs to soft-pedal on an amendment critical of the Queen’s Speech’s failure to mention a referendum, he gave way while on a trip to visit President Obama.

Life is too short to explain fully what the backbenchers’ amendment is about. Suffice to say, the PM hoped that by publishing a Bill, he would calm the situation and assuage the rebels. No such luck. Within hours of No 10 announcing that it, or rather the Tory party, was publishing a draft Bill that would commit the next parliament (a constitutional impossibility) to an in-out referendum, some were pushing for more. The Bill should get Government time, came the demand. Not possible: there’s a Coalition. Have a referendum before the election, they cried. How, with no majority in the Commons? The opinion polls also suggest the public is divided: any such campaign could easily result in a vote to stay in.

While some Tory MPs have gone on record to say they are pleased that the referendum Bill has been published, those who are enthusiastic about staying in the EU (there are about 30) are appalled. Several others I spoke to just seemed to be trying to keep up, hour by hour, with the Prime Minister’s evolving position.

Some Conservatives complain that this is all unfair. This trouble is all got up by the Europhile media – can’t you see that there is total harmony and no Tory split? They act as though saying this repeatedly, with a smile, will somehow make it true. That is Panglossian piffle. I am a Eurosceptic and I know a Tory shambles when I see one. This is definitely a Tory shambles.

The irony, and the tragedy for Mr Cameron, is that before this latest farcical episode of The Muppet Show, he had arrived at a perfectly sensible policy. In January, he declared that if elected in 2015, he would commit himself to renegotiating the terms of British membership of the EU. He would then put the results to the country in an in/out referendum.

This was an entirely logical and coherent position, around which all members of the Tory tribe should have been able to unite. More importantly, the electorate might even like it. Those who believe in getting out could look forward to a vote. Those who want to try renegotiation could be reassured that it would be attempted. Those who want to stay in could go into the election knowing that the issue had been parked, but that they would get the chance to campaign for Brussels. Crucially, this policy meant that voting Tory at the next election would be the only serious way to get a referendum. It also made practical sense, with the euro crisis unresolved.

It had, admittedly, taken Mr Cameron a long time to get to such a sensible position. Initially, he was merely determined that the party should, as he put it, stop “banging on about Europe”, and that his leadership would not be dominated by the rows that have disfigured the Tory party since the fall of Margaret Thatcher.

Mr Cameron set out to change the broken record, hoping that voters who think the Tories weird for their fixation on Europe would take another look. Being David Cameron, he hoped to make Europe go away mainly by relying on his charming personality, and did not think too much about the details.

At first, this got him into terrible difficulty, chiefly over his abandonment of a cast-iron guarantee to hold a referendum on the Lisbon Treaty. (He pointed out, in vain, that the treaty had long since been ratified.) Eventually, with the aid of William Hague, he set out to craft the compromise of renegotiation and then a referendum.

The beauty of this policy was its simplicity. It acknowledged clearly that the EU is important and that the Tories mean to do something about it if they get a majority. But it created space in which they could also talk about other matters, such as welfare, the economy and Labour’s suitability, or otherwise, for office. It also gave Eurosceptics time to construct a case for withdrawal that goes beyond standard-issue rhetoric and baseless assertions.

For a while, it seemed that the overwhelming bulk of the parliamentary party understood this. Yet just four months later, the panic induced by the rise of Ukip seems to have so frightened some MPs that they have moved back to negotiating live on air, on a daily basis, with the Prime Minister. Bizarrely, he seems prepared to go along with this, and to offer concessions rather than holding the line.

The sorry result is that his policy is no longer simple. It is confusing. It has a Bill attached, to be introduced immediately (or perhaps not), and there is a row about how ministers would vote if there was a referendum today, which there won’t be. The matter is being reduced to a traditional Tory squabble, which will baffle many watching voters.

Once again, under pressure, the Conservatives are getting themselves lost in the politics of the student union; of Eurosceptic angels dancing on the heads of pins; of men (almost always men) having arcane arguments about motions, timetables and Private Members’ Bills. They have lost the plot.