David Cameron has become accustomed to going to EU summits to fight battles over the budget, veto treaties, or try to block appointments of federalists.

In December 2011 he famously stood in the way of an EU-wide treaty on fiscal and economic union, delighting Eurosceptics at home but taking his relations with other continental leaders to a historic low. In June this year he was less successful in trying to stop the appointment of the federalist Jean-Claude Juncker as European commission president. After that embarrassing failure, he made clear his dislike of European meetings, sarcastically referring to his next visit to Brussels as "another day in paradise" that he would rather scrub from his diary.

But this weekend is different, and Cameron made clear as much as he arrived in Brussels on Saturday. International crises in Ukraine, and in Syria and Iraq, demand united international responses. They crowd in on every European government, the leaders of which know joint action and solidarity – not grandstanding for domestic audiences – is what is urgently needed.

"We have to address the completely unacceptable situation of having Russian troops on Ukraine soil," said Cameron on arrival in Brussels. "Consequences must follow if that situation continues and we will be discussing that as well today."

The urgency of concerted and coordinated action by EU countries will be further underlined this week when, after Brussels, Cameron will chair a Nato meeting in Newport, south Wales. Barack Obama will want to see the EU acting as one, both in response to Russia's actions in Ukraine and against the threat of Islamist extremism on Europe's borders.

Cameron himself has stressed the need for working together. As the UK government lifted the threat level of a terrorist attack on UK soil to "severe" on Friday, he described the threat of Islamic State (Isis) as greater than that posed by al-Qaida: "We could be facing a terrorist threat on the shores of the Mediterranean and bordering a Nato member." At Saturday's summit he was pressing for the EU to tighten sanctions on Russia and to better coordinate action in hunting jihadis by reviving pan-European plans to allow governments to share their passenger records.

But what is a prime minister to do, if such exhortations to collective endeavour are undermined on the home front by a Eurosceptic insurgency bent on pulling Britain out of the EU?

In a week that saw the Conservative MP for Clacton-on-Sea, Douglas Carswell, defect to Ukip, whose raison d'être is to leave the EU, the prime minister is being tugged in two directions. While the terror threat in the UK has been lifted to "severe", the domestic political threat to Tory unity from Nigel Farage's rampant party warrants a similar rating. For Cameron, navigating between the demands of Brussels and the demands of a disillusioned English coastal town may prove a nigh-on impossible task.

Former Liberal Democrat leader and member of the Commons intelligence and security committee Sir Menzies Campbell said on Saturday that Cameron now had a hugely difficult balance to strike between party and country. "You can only deal effectively with these threats if you do so both with your friends and allies in the European Union and in Nato." Campbell added that, with the terror threat sure to last for years, "another burst of Euroscepticism, while it may encourage some Eurosceptic backbenchers, is not going to help the prime minister as he seeks that necessary cooperation".

As Cameron sat down for talks in Brussels with, among others, his former nemesis Juncker, Nigel Farage was spending the day in Clacton, the seat Carswell held for the Tories but will now fight as Ukip's candidate in a byelection in the autumn. That contest will haunt the Tory party in the runup to and beyond the party conference season – the last before the general election.

On Friday, Farage visited Clacton for the first time, triumphantly showing off his beaming new recruit to voters before retiring for a few pints. "Sorry I didn't hear the phone at first. Too noisy in the pub," Farage told the Observer before cackling with laughter. He described himself as "ebullient". His timing in unveiling Carswell and his jocularity have infuriated Tories, including some hardline Eurosceptics such as Bernard Jenkin, who deplore his action at a time of international crisis. "I think it shows just how irresponsible Ukip is," said Jenkin. Complain as they may, it will be hard to tear the Ukip leader away from the Essex town in the next few weeks. Victory against the Conservatives in the byelection will, he says, "change everything" and be the moment when more Tories and a few from Labour could follow Carswell into Ukip's arms.

"It is very, very important that we win Clacton. We are throwing everything at it," Farage said. The party's youth wing also poured into the town on Saturday armed with fistfuls of leaflets. Jack Duffin, chairman of Young Independence, said: "The response has been fantastic. People are beeping their horns when they pass us. It is brilliant." Farage himself won the Ukip selection for the parliamentary seat of Thanet South last Monday. To him the challenge is clear and simple – to win Westminster seats before and at the next general election. He is confident that, now Carswell is on board, he is on his way.

Farage's joy is met with Conservative panic. The Tory whips have been in overdrive, desperately trying to gauge whether others might be flirting with Ukip. One regarded as a possible defector, Jacob Rees-Mogg, has firmly rejected the suggestion. "This is just silly, really. It's silly season," the 45-year-old North East Somerset MP told the Western Daily Press. "I was born a Tory and I will die a Tory."

He added: "For me, politics is about much more than just one issue or one policy. Ukip's light will wane as we approach the next election, at which the economy will be crucial and whether people feel the recovery has reached them. But no, I have not even thought about defecting to Ukip."

For now, the Tories are confident that there will be no more Carswells. The party high command insists he is a loner, an unguided free spirit with a passionate belief in localism, and that he has made a mistake because the Tories are the only ones who will guarantee a referendum on the UK's membership of the EU. But equally the whips know that there are plenty of Conservative MPs, peers and probably even more grassroots activists who agree with Carswell's views and who do not believe that Cameron can deliver real change in the EU. If, as the next election nears, backbenchers calculate that they stand a better chance with Ukip than Cameron's Tories, no one can rule out another defection.

David Cameron 'another day in paradise' this weekend as he goes to Brussels for crucial international talks. Photograph: Toby Melville/Reuters

Back in January last year, Cameron hoped he had defused his party's Europe problem by promising to hold an in/out referendum, having renegotiated new terms of UK membership. "It is time to settle this European question in British politics," he said then. "I say to the British people: this will be your decision." At first Carswell welcomed Cameron's move, but he became cynical over the next 18 months. As he announced his defection, he unburdened himself. "The problem is that many of those at the top of the Conservative party are simply not on our side. They aren't serious about the change that Britain so desperately needs … Of course they talk the talk before elections. They say what they feel they must say to get our support … but on so many issues – on modernising our politics, on the recall of MPs, on controlling our borders, on less government, on bank reform, on cutting public debt, on an EU referendum – they never actually make it happen."

This weekend – with Cameron in Brussels urging EU action to counter terror threats and the need to tighten sanction on Russia for its invasion of Ukraine – plenty of hardline Eurosceptics are criticising Carswell for defecting. But, despite the grave international context, most would agree with him in private that they now expect Cameron's attempt to renegotiate the UK's terms of membership to deliver very little. Large numbers of backbenchers say they intend to campaign for a no vote if the Tories win the election and a referendum is called. More and more want out of the EU as Labour and the Liberal Democrats argue the community's role in international affairs is increasingly crucial.

Some argue that Cameron underestimated how difficult his task of renegotiating UK membership would be. He hoped the German chancellor, Angela Merkel, would ride to his rescue, and she may still, but the signs are not encouraging. The prime minister has said that at the top of his shopping list will be a demand to extricate the UK from the EU principle of "ever closer union", something other European governments, including the German one, have made clear will not be on the table in any formal sense. Last week Germany announced measures to tackle welfare tourism within the EU, another of Cameron's key demands. But as it did so, Berlin made clear that "free movement of people is an indispensable element of European integration, which we back by all means" – hardly the language to encourage British sceptics.

Cameron finds himself playing EU statesman, in the name of security, while trying to bend the EU to British demands. At one summit, EU leaders are his allies; the next they are his foes. In Brussels on Saturday it escaped no one that Cameron was having to butter up none other than Jean-Claude Juncker, the man he denounced as wrong for the job of commission president, because Juncker will decide what portfolio the UK's Lord Hill and every other nominee will get in the new commission. One Whitehall source said: "The question is whether Juncker will say, 'All is forgiven, David' or whether he won't. I would not expect the UK to get a plumb job if I had to guess."

Cameron is part the committed participant, part the rebel. It is a difficult position to be in, allowing Ukip to deploy its simple mantra that we're better off out. In the latest Opinium/Observer poll, Ukip stands at 16%, with the Tories on 30% and Labour on 36%. Ominously for the Tories, Clacton will ensure Farage and Co are in the news throughout the autumn, while the Ukip leader's campaign in South Thanet will shine a light on him right through to the general election.

The Tory high command dismisses out of hand talk of any pact with Ukip. But the demands will not go away. Robert Oulds, director of the Bruges Group – set up in honour of Margaret Thatcher's 1988 speech in which she warned of an EU superstate – is this weekend calling for "a strategic arrangement between the Conservatives and Ukip whereby Tories will stand aside in a handful of seats in exchange for a free pass in key marginal seats". Oulds said: "The infighting between the two parties who share a similar agenda will only succeed in meaning that neither achieves their objectives. The mutually assured destruction should end."

Meanwhile, Cameron fights on, to limit the role of the EU in national life, on the one hand, while urging it to act more to guarantee international stability and the security of the British people, on the other.

THE QUESTION

How serious is Carswell's defection for the Tories and what should Cameron do?

FRASER NELSON Editor, the Spectator

David Cameron should send for Boris - who, in turn, will have a great opportunity to show that he is prepared to take a risk to come to the aid of the party and actually fight, rather than wait to be anointed king of a safe seat.

Yes, Cameron is now paying the price for treating so much of his core supporters with neglect, bordering on contempt. But Cameron will by now realise that the misnamed "modernising" project served to narrow, rather than broaden, his party's appeal. He needs to unite his tribe behind him (and, yes, against Ukip and Douglas Carswell), which ought to be easy given how close we now are to the election. And he needs to make some basic points more clearly.

There will be two outcomes after polling day: Cameron, and a referendum in 2017, or Ed Miliband, no referendum and five years of François Hollande-style calamity. The polls and bookmakers point to the second outcome, which ought to focus minds (and loyalties).

Three years ago, Cameron hung a Tracey Emin-designed neon sign outside the Terracotta Room of No 10: it says "More passion". The time for such passion has arrived.

PETER WILDING Director, British Influence

Douglas Carswell's defection could herald an SDP 1981 moment, but only if he wins. Clacton demographics have stiffened his byelection sinews.

His victory will put Europe centre stage as a Tory split story, not a Cameron statesman story. Europhobes will harass the prime minister to reveal his repatriation demands now to clarify his line and avoid further defections. Cameron will talk tough on Europe, but he won't fold.

What he should do is tell his party straight that his top reforms are now mainstream, having been agreed by all EU member states in June.

His government should now push to win this reform campaign. Otherwise the PM will continue to lose friends and alienate people.

ISABEL HARDMAN Daily Telegraph columnist

David Cameron might take some comfort from the way his party has turned on the once popular Douglas Carswell for defecting to Ukip. Any more defections won't come immediately.

But the prime minister cannot relax. Carswell's Eurosceptic colleagues are keen to use his departure to extract more details of Cameron's European reform plan.

He will also come under pressure to show that he is at least trying to reunite the right, something which this defection has just made immeasurably more difficult. Cameron had nourished a reasonable hope of a warm relationship with his party this autumn: Carswell has just poured an ice bucket over that.

CHRISTOPHER HOWARTH Senior political analyst, Open Europe

The stakes will have been raised. If Carswell wins, Ukip can then tell voters everywhere that voting Ukip means a genuine chance of electing an MP. This could be catastrophic for the Conservatives and may deprive them of a number of seats, to other parties mostly, but conceivably to Ukip as well.

If Carswell loses, the result will be equally disastrous for Ukip. The Tories can use it to show that, even in a constituency suited to Ukip, with one of their biggest names, they cannot win a seat. So why waste your vote in May 2015 will be their refrain.

For now the initiative is with Ukip, but Cameron can and will (rightly) point out that his party alone represents a real chance of delivering an EU referendum. Add in a reinvigorated EU reform agenda that addresses the concerns of disaffected voters and the Conservatives could start to turn the tables.

MELISSA KITE Spectator columnist

Whenever Ukip pull off a coup, David Cameron's response is always the same. He treated Douglas Carswell like a naughty schoolboy: "It's regrettable when people behave in this way."

Cameron loyalists also struck a condescending note. The consensus was that the Eurosceptic MP had "thrown his toys out of the pram". This suggests they still don't get it. The reason this defection represents such a real and present danger to Cameron is that Tory voters are throwing their toys out of the pram, in seats up and down the country - and I'm told Clacton is most definitely one of them. Carswell spoke for them when he condemned the Cameron "clique" and gave his reason for defecting: "Many of those at the top of the Conservative party are not on our side."

If Carswell has judged the mood right, and wins, it will surely be a game changer. More defectors would follow. Even a close contest will show that Ukip can hit Cameron where it really hurts, and bring the fight to Westminster.

Cameron's plan has always been to hope that troublesome Tory MPs, and voters, will dutifully fall into line on general election day, forgetting broken EU renegotiation pledges, broken pledges not to build on the green belt, and a high-speed rail link that goes through their back gardens. But it is becoming riskier to assume this. Rather than remind traditionalists archly of their duty to be on his side, the prime minister needs to re-engage with the people who put him where he is and show them that he is on their side.