David Cameron was forced to interrupt a pre-referendum trip to Scotland on Thursday as the Conservative leadership scrambled to deal with the surprise defection of a hardline Eurosceptic MP to Ukip, triggering an early byelection.

A major operation by the Conservative whips, launched within minutes of the announcement by Douglas Carswell, ensured that fellow Eurosceptics echoed the prime minister in warning that future defections to Ukip would make an in/out EU referendum less likely by boosting Ed Miliband and Labour.

The whips appeared on Thursday night to have contained the threat of further defections in the immediate future as the former defence secretary Liam Fox, the senior backbencher Bernard Jenkin and the MP Jacob Rees-Mogg – all independent-minded Tories – distanced themselves from Carswell.

Carswell presented the Tories with a dangerous byelection when he threw a hand grenade into Cameron's general election preparations by announcing that he was defecting. He is the first sitting MP since 1982 to resign his seat to allow him to stand for another party. "The problem is that many of those at the top of the Conservative party are simply not on our side," he said. "They aren't serious about the change that Britain so desperately needs."

But Cameron, speaking from Scotland, warned that Carswell's defection would only benefit Labour. The prime minister said: "It is obviously deeply regrettable when things happen like this, when people behave in this way. But it is also counter productive. If you want a referendum on Britain's future in the EU – whether we should stay or go – the only way to get that is to have a Conservative government after the next election. That is what, until very recently, Douglas Carswell himself was saying."

Mark Reckless, a close friend of Carswell's who had been identified by senior Tories as the next possible defector, said on Thursday night that he had no plans to leave the Tories. Reckless told the Guardian: "Douglas is a friend so I don't want to criticise him personally. If people want a vote for an independent Britain they need a Conservative government."

Nigel Farage, the Ukip leader who had personally negotiated Carswell's defection in great secrecy without telling his closest aides until Thursday morning, staged the announcement with a theatrical flourish. Journalists were given short notice to appear at a Westminster press conference.

With cameras crammed into a darkened room at One Great George Street, Farage walked in with Carswell to declare cheerfully that he had brought in the media on false pretences. Carswell then announced, to cheers from Ukip supporters, that he was abandoning the Tories to join Ukip after accusing Cameron of failing to deliver change on Europe.

Carswell made clear that his criticisms went beyond Europe as he criticised what he called the "little clique" of Westminster. He said: "Of course they talk the talk before elections, they say what they feel they must say to get our support when they want our support. But on so many issues – on modernising our politics, on controlling our borders, on less government, on bank reform, on cutting public debt, on an EU referendum – they never actually let it happen."

Farage hailed Carswell's bravery in triggering a byelection as he predicted that other MPs would follow his example. "Oh, this is not my show today, this is Douglas's show today. I don't think it is any secret that there are now a number of MPs sitting on the Conservative benches and indeed some now sitting on the Labour benches who hold Ukip's views very strongly."

The MP for Clacton-on-Sea in Essex held the seat with a majority of 12,068 at the last election, but the Tories admitted they face a difficult byelection, likely to be held on 9 October, in a seat where Ukip topped the poll in the European elections in May. Matthew Goodwin, the author of a book on Ukip, predicted a Carswell victory in Clacton which he described as the "most favourable seat for Ukip in the country".

However, Carswell's declaration that he had decided to abandon the Tories in protest at the failure of Downing Street to open up politics rebounded somewhat when a local farmer in Clacton who had been selected as the Ukip candidate refused to make way for the defector. The secrecy of the defection meant that the party failed to alert Roger Lord who was told by the Ukip leadership that he would have to stand aside. Farage contacted Lord to smooth relations as the Tories moved to woo him.

Lord said: "There are some basic courtesies that one expects. Douglas Carswell could at least have called me an hour before his announcement. It's pretty arrogant of Douglas Carswell to assume that the voters and the electorate are like sheep and they will just go along with this." On Thursday night Ukip's party secretary said: "Roger Lord is not now, nor has he ever been the byelection candidate for Clacton. The National Executive Committee of the Party have voted to adopt Douglas Carswell as the candidate for the upcoming byelection. Roger Lord is mistaken in his belief that he is the candidate and he can best serve the party's and the country's interests by standing behind the decision of the NEC."

While Carswell's defection appears to be isolated for the moment, the announcement will destabilise the Tories on the highly fraught issue of Europe. There are fears that some MPs may now seek to insulate themselves from the Ukip threat by pledging to campaign for a no vote in Cameron's planned EU referendum regardless of the outcome of his negotiations.

A senior Tory MP who shared a suite of offices with Carswell at Westminster expessed surprise at his defection. Keith Simpson, who was ministerial aide to William Hague until last month, told the Guardian: "I was amazed as anybody else. In retrospect you could perhaps see a journey. But nearly all the evidence pointed to the fact that Douglas was saying the only way we will get a referendum is to have enough Conservative MPs and indeed a Conservative majority.

"I am sure that one part of him is absolutely genuine. But I do wonder if there is an element of calculation at two levels. One is that if he does have these doubts he sees this byelection as detonator that might set off wider explosions."

• This article was amended on 1 September 2014. An earlier version wrongly suggested that Douglas Carswell studied at the University of Oxford.