Housing benefit recipients under 25 currently receive an average of £90 a week. Photograph: Matt Cardy/Getty Images

Families with children are likely to be hit hardest by Conservative proposals to severely restrict housing benefit to under-25s, according to the housing charity Shelter.

Youth employment campaigners and the gay rights organisation Stonewall have also raised concerns about the proposal, which featured in George Osborne's speech to the Tory party conference, saying they would reduce labour mobility during a downturn and leave thousands of young gay adults at risk.

Shelter said that according to the latest Department for Work and Pensions (DWP) figures from June, cutting under-25s' housing benefit altogether would affect 385,000 households, 53% of whom are couples or single parents with children.

The benefit given to under-25s currently costs £1.8bn a year but represents only 8% of total housing benefit claims. The National Housing Federation chief executive, David Orr, said the proposals, which were also put forward by David Cameron in June, risked increasing homelessness. Stonewalls' Ben Summerskill said it would impact heavily on thousands of young gay adults who no longer had the option to move back in with their parents.

"We have real concerns that this policy might impact very unfairly on the significant number of young people who are still thrown out of their homes by their parents," Summerskill said.

"We know there are vulnerable young people who end up particularly in metropolitan areas, London and Manchester, on the streets … who have been thrown out of home by their parents," he said.

"The government does need to distinguish between young people who may be in real need and young people who could quite easily move back into the third or fourth bedroom of the rectory in Hertfordshire."

Osborne received loud applause from the Birmingham conference hall when he asked: "How can we justify giving flats to young people who've never worked?" Orr said, however, that housing benefit was a "crucial safety net that should be decided on the needs of the individual, not simply their age".

"Cutting the housing benefit of under-25s will put many young people at risk of homelessness, particularly those who can't go back to their family home. Some may not even have a home to return to," he said, adding that the government should focus on increasing the supply of housing if they wanted to keep benefit bills down.

Suzanne Beishon from the campaign group Young Londoners Forced Out said: "This is an absolute farce. At the same time as they telling us to get on our bikes and look for work, they are forcing many young adults back into living with their parents. Its measures like this that run the risk of infantilising a whole generation of young people."

Citing its report Housing Options and Solutions for 2020, the Joseph Rowntree Foundation's policy and research director, Emma Stone, said current projections for young people's housing were already "bleak for now and the years ahead".

The report, which was released in the summer, predicts that around 1.5 million more young people aged between 18 and 30 would be pushed towards living in the private rented sector by 2020, with just over a million fewer in home ownership.

"This feel likes a blunt and a potentially very harmful way of addressing issues that are very real and very live for a lot of young people, some of whom already have very limited options around housing, independence and access to education and jobs," Stone said.

Shelter said there was no evidence that young people were living outside the family home because they could take up housing benefit, adding that under-25's already received the lower shared room rate of benefit introduced in 1996.

The charity added that the DWP's own family resources survey showed that 56% of under-25s without children already lived with their parents, 19% were in shared student accommodation, 8% lived in flat shares and only 10% lived independently.

The Shelter chief executive, Campbell Robb, said: "Most under-25s who can live with their parents are already do so. Being under 25 is being used as shorthand for people who don't have responsibilities or adult commitments, whereas in reality more than half of young people who claim housing benefit already have a family of their own.

"For the small number of single young people renting privately and claiming housing benefit, it is mainly used as a short term safety net.

"Young people are already facing an incredibly tough time … Removing such a vital source of support will be a huge blow for young people struggling to set themselves up in life."