The Conservative Party conference began on Sunday with MPs and activists still reeling from the defection of Mark Reckless. And rumours about who might be the next to jump ship have dominated chatter in and around the Birmingham convention centre.

On Monday afternoon, former leadership contender David Davis told a meeting designed to address how to tackle the threat from Nigel Farage that the Tories "should not be insulting people who vote Ukip". Ken Clarke was obviously not listening.

Speaking to Conservative activists at a drinks reception on Monday evening, the former cabinet minister, 74, said Ukip attracted "elderly male people who have had disappointing lives".

And he said he was not "the slightest bit surprised" that Reckless and Douglas Carswell had joined Ukip. "I could never understand why they were not members of Ukip before," he said.

Clarke's comments came after Boris Johnson told a victory rally across the other side of the conference centre that Tories who defected to Ukip were the kinds of people who liked to have sex with vacuum cleaners.

Earlier in the day, Thatcherite Tory MEP Jacqueline Foster had described the Ukip MEPs she had encountered in Strasbourg as "collectively boorish".

Clarke, whose cabinet career came to and end in Cameron's last resuffle, likened the current situation to that immediately after the great fire of London. "A mob went through the streets beating up and hanging Frenchman despite all the rumours about a baker shop somewhere in London. They were concerned it was French arsonists who caused the problem. Ukip captures the spirit of that," he said.

He said the insurgent party was "largely appealing to the disappointed elderly, the grumpy old men, people who've had a bit of a hard time in life and are a bit disappointed and think the country is going to the dogs and are susceptible to the argument it's all the fault of foreigners".

Clarke added that while Farage had "absorbed the BNP vote" and "taken on board BNP followers. He said there was "a nasty undertone" in Farage's movement. "He does attract elderly male people who have had disappointing lives," Clarke said.

Clarke also said Cameron had a "mountain to climb" in order to win a overall majority in 2015. But he said the prospects of the Conservatives gaining more seats than Labour in a hung parliament was achievable. The veteran former cabinet minister observed in his opening remarks: "I belong to the Conservative Party that used to be able to win general elections."