Nigel Farage's comments about Romanians have been branded racist by three prominent Labour figures – Yvette Cooper, Diane Abbott and David Lammy – after the Ukip leader said people might not want to live next door to immigrants from the eastern European country.

Farage declined the opportunity to apologise for his comments several times in an interview with the BBC. After a day of criticism from senior politicians of all parties, Farage was asked if Ukip were the Millwall FC of British politics. "Perhaps we are the Millwall... I think in Westminster we're loathed and feared," he replied.

"What Nigel Farage said over the weekend was racist. So I'm clear, he's a racist," he said. "I am from a background where my parents arrived here as immigrants. I remember a context in which some people said: 'You don't want these people living next door to you.' That was racist."

Cooper, the shadow home secretary, told ITV News "It's not racist to be worried about immigration or to want stronger controls, but it is racist to somehow stir up fears about Romanians living next door. So Ukip should say they were wrong on that," she said.

Abbott, a former shadow health minister, said "his remarks were racist" and she was "glad everybody has stopped pussyfooting around".

It comes after David Cameron described the comments as "pretty unpleasant" and accused Ukip figures of saying a "succession of pretty appalling things", but said he would leave others to judge.

"I was asked if a group of Romanian men moved in next to you, would you be concerned? And if you lived in London, I think you would be," he told the interviewer, James O'Brien.

However, the Ukip leader backtracked from this stance on Sunday night, saying: "Do you know what, in life sometimes people get things wrong. I regret the fact that I was completely tired out and I didn't use the form of words in response that I would have liked to have used."

Ukip took out a full-page advertisement in the Daily Telegraph, insisting Ukip was not racist but warning about risks posed by organised criminal gangs from Romania. "The vast majority of Romanians who have come to the UK wish to better their lives and would make good neighbours.But there is a real problem, an unpalatable truth that our political class would rather not discuss. Since the welcome fall of communism and the awful dictator Ceausescu, Romania has struggled to complete a full transition into a western democracy," it said.

By leaving the EU and "taking back control of our borders" the UK could do the "necessary checks" on would-be immigrants, the open letter from Farage said.

"When this happens, my answer to the question: 'Should people be concerned if a group of Romanian men moved in next door?' will be no."

The Ukip leader later told the BBC that he had not said people "should" be concerned but that they "would" be concerned and that he would not want to give the impression that he was discriminating against Romanians.

"I think the mask is starting to slip and I think what's being revealed behind the beer-swilling bonhomie is a rather nasty view of the world," he told BBC1's The Andrew Marr Show.

"I think anyone who singles out one community, one nationality, and says: 'I don't want to live next door to them,' I really think that's the politics of division and I think it really should have no place in modern Britain."

Vince Cable, the Liberal Democrat business secretary, said Farage should not be getting involved in "national stereotyping".

Ukip is still a frontrunner to win the European elections this week, although Labour and the Conservatives have also topped some polls.

The party said on Monday morning that its membership was rising rapidly and was on course to reach 40,000 by July.