When Poland signed the Lisbon treaty on 10 October, the Tory party at once acquired a new hero. This was an unpropitious figure, not a personage who had hitherto loomed large in anyone's consciousness. Nevertheless, Vaclav Klaus, the Czech president, appeared to stand alone between 500 million European citizens and the birth of a vast, monstrous, undemocratic superstate. Like King Leonidas fighting alongside 300 hand-picked Spartans at Thermopylae against the massed hordes of Xerxes, nothing less than western civilisation itself is held by Tories to depend on the heroism of Klaus.

For three weeks, Europe – or at any rate, Conservative Campaign Headquarters – has held its breath. Spies have brought daily reports from the front. But gradually hope has died. Klaus's forces are outnumbered, his resolution worn down. There is talk of treachery. Surrender is expected within days.

Much of Europe seems capable of bearing this calamity with irritating equanimity. Not so the Tories. For many, perhaps most, Conservative activists, the Lisbon treaty poses an existential threat to the British state: we cease to be subjects of the Queen and start to become citizens of Europe. Something ancient and of inordinate value will pass away.

This insight – by no means without truth and merit – poses a grave problem for David Cameron. His supporters will hardly be content for him simply to shrug his shoulders at this disaster. They will want him to take up the banner of freedom and national independence when Klaus lays it down. So David Cameron's response when the Lisbon treaty is passed will pose a classic test of his leadership skills. On the one side, he needs to humour the Eurosceptics, an ever-more formidable force in the Tory party. On the other hand, he knows that, in practice, he can do nothing. This faintly humiliating balancing act is made very much more difficult by concessions that David Cameron has already made on the Lisbon question.

These concessions were not made democratically, as naive spectators might hope and expect. Rather, they were thrashed out as part of the laborious negotiation that eventually led the Murdoch press to switch its allegiance from Gordon Brown's Labour party to David Cameron's Tories.

This wooing was eased by a pledge from Cameron to the readers of the Sun that he would hold a referendum on the Lisbon treaty. "Today," wrote the Tory leader in September 2007 in an article he must bitterly regret, "I will give this cast-iron guarantee: if I become prime minister, a Conservative government will hold a referendum on any EU treaty that emerges from these negotiations." To dramatically emphasise the point, he wrote his personal signature at the bottom. "Small wonder that so many people don't believe a word politicians ever say," added Cameron, "if they break their promises so casually."

This so called cast-iron guarantee sounded all well and good at the time it was made and Lisbon was only a hazy prospect. It's a different matter altogether now that Vaclav Klaus's heroic stand is over and the treaty is about to come into law. It remains open, of course, to David Cameron to honour the promise that he made to Sun readers. He could place the referendum pledge at the heart of the Tory general election manifesto, call and fight an anti-Lisbon campaign in the first 100 days of winning office. Many Tories yearn for him to do just that.

Cameron, however, is by no means mad. Educated at Oxford, he has been trained to spot a lost cause when he sees one. He has no desire to destroy his premiership before he even enters Downing Street, no appetite for the endless squabbles with Europe and Washington a referendum would involve. He knows that he will have too many battles to fight on other fronts – sorting the Afghanistan debacle, dealing with the economic crisis.

So, for the last few weeks, he has been doing his best to oil out of that poorly judged commitment, which was made at the moment of greatest crisis in his leadership, when the Conservatives trailed by 20 points at the polls, Gordon Brown seemed poised to call and win a general election and Cameron's very survival was at stake. The last few weeks, therefore, have seen a series of sordid negotiations between the Conservatives and executives from the Sun in search of a face-saving formula.

The Sun has been in a ferociously strong position. The newspaper is within its rights to remind its readers of Cameron's disastrous pledge back in September 2007 and accuse him of national betrayal. But it has held off. Agreement has been reached. When Klaus succumbs, Cameron will not step into the breach. He will come up with a new guarantee – a pledge that will force the government to hold a referendum on any future European treaty. This is a cop-out and a betrayal, but the Sun is highly unlikely to say so. This is exactly the kind of post-democratic politics which defined, debased, and finally destroyed, the Blair premiership. It is greatly to be feared that a pattern has been set for the future. Nevertheless, the Tory party's alliance with the Sun means that the European problem has been shelved for the time being.

Shelved, but not solved. The relationship between the Conservatives and Europe can be divided into three phases since Edward Heath led Britain into the EEC on 1 January 1973. For 15 years, the Tories were the pro-European party, while many Labour politicians detected a pro-capitalist conspiracy. All this was turned upside down in September 1988, the month when Jacques Delors's famous speech to the TUC conference persuaded Labour politicians that the European Union could be used as a charter for social and economic rights.

Just days later, on 20 September 1988, Margaret Thatcher delivered her famous Bruges speech calling for powers to be returned to nation states from Brussels. Bruges set the seed for the destruction of Thatcher herself and the great Tory civil war over Europe that blazed through the Major premiership and the early years of opposition under William Hague. It only abated (an achievement for which he has never been given credit) when Iain Duncan Smith succeeded in making opposition to the euro unequivocally official party policy.

It was David Cameron, in so many ways a pragmatist, who reopened the European wounds. In the late summer of 2005, his campaign for the Tory leadership was faltering. He could gather very little support and the contest looked like turning into a run-off between the two ambitious right-wingers, Liam Fox and David Davis. Suddenly, in a daring move, brilliantly advised by his ally Michael Gove, Cameron outflanked them both by making a promise his rivals felt unable to make. He promised to take the Tory party out of the EPP centrist coalition in the European Parliament, thus securing the support of core Eurosceptics including William Cash, Douglas Carswell and the talented MEP Dan Hannan. Had Cameron not formed this alliance with Tory Eurosceptics, he would never have become leader

Probably to his credit he kept that promise when he pulled out of the EPP. But ever since he has been paying the price. Labour strategists were quick to realise that the Tory alliance with a new group headed by Polish politician Michal Kaminski made them vulnerable to charges of extremism (the sudden and wholly unexpected Foreign Office reshuffle of ministerial posts between Glenys Kinnock and Chris Bryant can be explained by a decision to make Bryant minister for Europe with a brief to attack the Tories). Miliband has surely been unscrupulous and is open to charges that he has subordinated British foreign policy to naked partisanship in attacking David Cameron's European partners. The Latvian government, for example, has denounced Miliband's mischief-making as "unacceptable" and "misleading". But he has been effective because a fundamental instability over Europe remains at the heart of David Cameron's shadow cabinet.

It contains two very senior figures, William Hague and Kenneth Clarke, one of whom is ardently pro-European and the other who would at heart like Britain out of the EU altogether. Both are crucial to David Cameron's electoral success, but in different ways. Clarke appeals to ordinary, centre-ground voters while no contemporary politician appeals to the Conservative grass roots like Hague. This visceral connection gives him the same status in David Cameron's shadow cabinet as John Prescott used to enjoy under Tony Blair: Hague brings the party with him.

But Hague is more dangerous than Prescott. Intellectually formidable, he is a very live alternative prime minister. More dangerously still, he no longer yearns for power and was only persuaded to return to front-line politics with difficulty. There is very little to stop Hague from resigning and, were he to do so, he could scarcely avoid becoming a very powerful focus of resistance to a Cameron premiership.

Ken Clarke, too, does not much mind whether he becomes a cabinet minister again. During stormy periods, and there are plenty ahead, Hague and Clarke will roll around the political quarterdeck like loose cannons, a hazard to life and limb. It will require superlative management skills if Cameron can survive for long without chucking one of these two heavyweight figures overboard.