Charities are demanding an urgent rethink of government housing policy after a Guardian investigation found that almost half a million homes are lying empty in the UK – enough to put a roof over the heads of a quarter of the families on council house waiting lists.

The startling picture of neglect – we estimate that more than 450,000 properties have been empty for at least six months – at a time when there is an acute housing shortage was pieced together using information gathered from local councils under the Freedom of Information Act.

Our findings suggest the number of "long term vacant" properties is 25% higher than previously thought. David Ireland, chief executive of independent charity the Empty Homes Agency said the empty stock would go some way towards tackling the housing crisis – 1.8 million households are waiting for a council house – as opposed to the government's focus on building new homes to tackle the problem. "Refurbishing empty homes cannot deal with the entire housing crisis but it can make an important contribution," he said.

A lack of reliable information has hampered the efforts of campaigners trying to draw attention to the problem and research, gleaned from 284 councils, is the first time data from across Britain has been pulled together. Highlighting the scale of the problem, the figure provided by one London borough was nearly 30 times higher than that used in official figures.

The Guardian asked councils to identify the number of properties claiming a discount or exemption from council tax on the basis they had been empty for six months or longer. The sample took in 75% of councils in Britain and based on the data we extrapolated there are probably 450,000 long term empty homes around the country.

Ireland said the reservoir of property could be vital given the dent in the government's pledge to build 3m new homes made by the credit crunch and attendant property crash. "It is yet further evidence of the need for government to revisit a housing policy which is based almost entirely on building new houses," he said of the Guardian's research.

The Royal Institution of Chartered Surveyors (Rics), however, argues that the financial treatment of empty homes would have to change to make councils grasp the nettle. "For every empty home that is brought back into use the council sees a reduction in its annual central government grant," James Rowlands, policy project manager at Rics explained. "Many councils prefer the security of a guaranteed government grant rather than the less reliable income from the council tax on a refurbished property."

Ireland said the number of empty homes may be inflated if second homes have not been properly accounted or underestimated if regeneration projects which have been put on hold during the recession are excluded.

A number of big inner city regeneration projects resulted in large numbers of homes becoming empty, said Ireland, because residents have been moved out of housing estates. However, those properties are often excluded from empty homes data. So too are some private sector apartment developments which have been purposely left uncompleted to avoid incurring council tax.

Rowlands agreed that data collection on empty homes is imperfect – many councils, he said, rely on complaints from neighbours to identify empty homes. "There is no doubt that empty homes are a blight on local communities. They attract social disorder and have a very negative impact on the neighbourhood."

It has been estimated that the value of properties close to abandonded houses can be up to 18% lower than equivalent properties further afield.

Rics wants see VAT on home improvements reduced to 5% to make refurbishment of derelict properties more affordable. But Rowlands believes there also has to be a renewed effort to improve the regime to encourage owners to renovate their empty properties.

"The government introduced new rules which were intended to make it easier for local authorities to bring empty homes back into ownership but there are now doubts about the effectiveness of empty dwellings management orders and I think we need a new approach which improves the dialogue between councils and owners and encourages them to work together to address this issue," Rowlands said.

The TUC wants to see a tough approach in a bid to help those waiting for homes – it has argued for severe financial penalties to be imposed on owners of empty properties, such as council tax bills at five times the standard rate.

As part of its research the Guardian attempted to identify how many empty properties are owned by people or companies based overseas – but, only 179 councils could provide figures, or even estimates, for overseas owners of empty properties. The Guardian estimates that more than 11,000 empty properties are owned by people living abroad. However, that estimate is less reliable as fewer than half of councils were able to provide data.

Some councils are already grappling with the problem of empty houses. In Birmingham – the council with the biggest number of empty homes, at 9,000 – the council launched an empty property strategy in 2007, and vacancy rates are now down by 5,000 on their 2003 levels.

Councillor John Lines, Birmingham's cabinet member for housing said: "It's crucial in today's economic climate to bring much needed homes back into use for occupation by families who really need them and this is a priority for us."

The Guardian's research shows, as might be expected, that empty homes are at their highest in urban areas, where population density is greatest. Birmingham, for instance, has 1 million citizens and is the biggest council by population in Britain.

David Ireland at the Empty Homes Agency, said inner city brownfield sites, as well as vacant houses, offer potential for more homes: "Research conducted in two London boroughs, which examined the housing potential on underutilised brownfield areas concluded that there was sufficient land to build 26,000 new homes in each borough."

He is keen to see a new government housing policy which considers a wider range of initiatives including brownfield sites and empty homes as well as new houses. "My concern is that because it is difficult to quantify the number of empty homes and it is difficult for councils to encourage owners to refurbish them that the issue will become swamped by the imperative to build new homes which is actually a more attractive financial proposition for local authorities," he said.

Housing charity Shelter Scotland has secured Scottish government support for the appointment of an empty homes champion who will promote the use empty homes as a way of increasing housing supply and improving neighbourhoods. The position recognises that the financial climate is conducive to maximising the use of existing housing stock.

But Roger Harding, head of policy for Shelter England, insists that empty homes are only a part of the solution to the country's housing shortage: "You have to remember that we have about 5 million people who are waiting for council house accommodation. That is the issue we need to address."

• This article was amended on 7 April 2010. The original referred to the policy project manager at Rics as James Reynolds. This has been corrected.