Immigrants face life as members of an unemployable "sub-class" living in ghettoes unless they are taught to speak English "like a native", the communities secretary, Eric Pickles, has said.

Labour criticised Pickles for his choice of words as he explained the government "integration strategy", launched last month, which calls for immigrants to speak English.

In an interview with the House Magazine, Pickles said: "In terms of wanting people, encouraging people, to be part of British society, they can't do that unless they have more than an understanding of English. If we don't get our resident population with an understanding of English, then they become a sub-class … virtually unemployable or stuck in a ghetto. We should not be turning people out of our schools who aren't able to speak English like a native."

Hilary Benn, the shadow communities secretary, said: "Eric Pickles should be thinking of practical ways to ensure that anyone settling in this country is able to speak English rather than talking about sub-classes and ghettoes."

Pickles spoke out after launching the government's integration strategy last month. It said the government "wants" immigrants to speak English.

In the interview, Pickles said there would be a £10m grant to promote the teaching of English. He said he wanted to achieve "real integration, an opportunity for people to meet, to mix, to be engaged in activities beyond their ethnic group". "I think that is massively important," he said.

Pickles said the government wanted to find common ground to allow communities to unite. "We have always been of the view that if the Muslim community of Britain, British Muslims, are seen as the enemy within, then the forces of extremism win. It is certainly my view that we should be able to unite on the things we have in common, rather than constantly harp on the things that we have that are different."

The communities secretary said he had no time for people who want to ban Muslim women from wearing the burqa. "We are a tolerant nation and, frankly, I have absolutely no patience in adopting a kind of French system that is going to remove people from wearing headscarves," he said. "I think we are a little bit more confident and relaxed in our culture for that."

Pickles explains that his background growing up in Keighley and Bradford in the 1950s and 1960s helped shape his views on integration. "I can remember asking my mum why there was a sign up in a shop that said 'no blacks in here' or something to that effect in Bradford, when I was not very old at all," he says. "It is quite right that we have introduced laws to outlaw that kind of thing. Nobody should feel frightened in the United Kingdom. People should feel safe in their beds, they should feel comfortable in their neighbourhoods. These dreadful extremists want to create fear in the minds of the community."