When food runs low, yet some squirrel it away hoping to profit as others starve, we call that hoarding, and demand action. Hoarders rank high among history's villains. They're the cursed souls relegated by Dante to the fourth circle of Hell, rolling a heavy weight for all eternity. The merchants who deepened the Bengal famine by amassing grain in their godowns. Even now, food hoarders are first to be frogmarched off to jail by panicky governments in poor, hunger-struck countries.

So why do we tolerate the hoarding of houses? This weekend, the Guardian detailed how one of the priciest streets in Britain, The Bishops Avenue in north London, has become a stockpile of unused mansions worth around £350m. My colleague Rob Booth described how many of the homes are not only empty, but derelict: ferns sprout out of staircases; pigeon carcasses decompose on rotting carpets.

Doesn't such epic neglect cost money? Not a bit of it. The owners can still look forward to hefty profits from selling each of these ruins. London addresses are so sought after that people will pay over the odds even for a wreck.

Britain has a housing crisis. Nearly everyone accepts as much – from whichever political party, lobby or special interest group. Yet one of the flabbergasting aspects of the Guardian's story about 'Billionaires Row' was the calm acceptance of such spectacular property hoarding. It was the prerogative of the "very wealthy" owners, one local Conservative councillor shrugged. "Sometimes they live in them, sometimes they don't." Even the reactive statements put out by Labour and the Tories yesterday had all the urgency of a door being seen to be slammed – long after the horse had bolted.

Hoarding isn't the privilege of a few Saudi royals; it is a feature at almost every rung of the property ladder. For the oligarchs: over half the flats in One Hyde Park are registered either as empty or second homes. For the working rich: the sales director of one estate agency in Knightsbridge describes her clients as "international buyers looking for good two-to-three beds; a majority of them will use it just a few months a year".

Then there are the Londoners' second homes in pretty villages, occupied only at weekends even while the locals are priced out. In his forthcoming book on the property market, All That is Solid, Danny Dorling describes how all those extensions and loft conversions means that at least a third of bedrooms in England and Wales – 22m – are "empty on any given night". And right at the bottom, David Ireland, head of the Empty Homes charity, recounts how during last decade's boom, out-of-towners were snapping up cheap terraces in cities such as Hull. They stuck them "on their credit cards and left them empty", he says: as a punt on the housing market.

You could probably add to the list, or turn to a newspaper property page for other examples. It is a function of trying to use bricks and mortar both to house people and as speculative assets. And while the real-estate market was always a bit of both, the real-life Monopoly game has got ever bigger, and attracted ever more players from abroad, so that around 85% of new builds in central London are sold to foreigners.

It has also attracted a phalanx of middlemen from the City, ready to help those keen to have a London postcode, but not so keen on a British tax bill. On the website of solicitors Sykes Anderson is a very helpful Q&A on "how 'non-doms' … can avoid inheritance tax". It begins: "Q: I recently purchased a London property that is now worth around £6m. Is there any tax risk relating to the property?" The next question is: "I have heard that it is possible to use an offshore company to avoid this charge. Could I put my property into an offshore company?"

The cost of all this speculation isn't felt by others who get gazumped on that £6m house. It's borne by those lower down the chain, who have to deal with the ripples coming down towards them. The couples now spending half of their income on renting. Or the more than one in four young people who now live with their parents.

The typical politician or pundit response to such stats is to parrot the old line about how we have to build more. What property hoarding indicates is that it's not just about more concrete; it's also about distribution. The madness in the housing market is a function of a deeply unequal economy.

How to tackle it? Ireland suggests that Boris Johnson should stop making soundbites about "blocks of bullion in the sky" and underwrite an action by Barnet council to repossess one of the homes on The Bishops Avenue. That would send a signal, as would giving councils greater power and funding to take back and do up empty homes. We should put all owners of British property on to a register – not the directors of some nominee company – and use the tax system to penalise under-occupation. These aren't all the answers: the biggest one of all would be to take more property out of the market by building more public housing.