A solution to the housing crisis is to build houses on the green belt. Photograph: Richard Saker for the Observer

During idle moments between bellowing at Question Time, I like to play a game. I call it "Guess how many houses this panellist owns?" The appearances by MPs spike the figures, of course, because they are never knowingly underhoused: the fallen Maria Miller needed us to pay for a home in Wimbledon so that she wouldn't face the horror of a daily commute from Basingstoke (52 minutes from Paddington).

Overall, a third of MPs are buy-to-let landlords and some of them have mini-empires: Nadhim Zahawi has three flats in London and 31 acres in Warwickshire, for example, while Shaun Woodward owns rental properties in New York state, France and the West Indies. James Clappison, the Tory MP for Hertsmere, takes home top honours with 26 rented houses, 27 acres of land – and a cricket pitch.

And before you start, yes, the journalists on the panel will spike the figures too: most are in their 40s or 50s and benefiting from a London house bought before the last great boom. And God knows, whatever token "business leader" they have on that week won't be living in a rented bedsit above a kebab shop in Merthyr Tydfil, either.

The Question Time problem is a microcosm of the wider mess of our housing crisis. Everyone involved in shaping the debate, and making the policy decisions that matter, has a stake in prices going up. The houseless are also largely voiceless.

This isn't a new phenomenon. If you ever need to escape a difficult conversation with a Tory, change the subject by whispering three magic words: Right to Buy. Margaret Thatcher's most popular policy created a tight unit of winners, who saw their good fortune and attributed it to the government of the day. The negative effects, on the other hand, were indirect and dissipated among the population at large.

Every politician since has learned the wisdom of this: if you've got money to give away, make a big show of it. If you need to collect revenue, drown the proposals in subclauses and tortuous acronyms so no one notices the money disappearing from their pockets. Gordon Brown did it with his pension changes, while George Osborne shaved £11bn off the annual welfare bill by changing the uprating from RPI to CPI in 2011. (Told you it was complicated; essentially, pensions, benefits and tax credits now use a lower measure of inflation to calculate their annual increases.)

Osborne, incidentally, has learned both sides of the lesson: with Help to Buy, he will create a small number of homeowners who have him to thank for their purchase. (Privately, housing developers admit the scheme is mostly a ginormous subsidy, helping them to sell homes they would have built anyway, but for more money.) But it's a zero-sum game: without increasing supply, Help to Buy only helps inflate the market to the detriment of every other buyer.

Why does this chicanery matter? Because owning a house is like having an extra vote – a vote for house prices to keep rising, mortgage rates to stay low and for supply to be constricted to prop up the whole shebang. By owning your own property, you get a say in planning law too – your objection to that new tower block counts for far more than the indirect, dissipated unease of all the people hunting desperately for somewhere affordable in the vague vicinity of their work.

Housing prices have become a prime driver of inequality. First, between generations: a thirtysomething professional worker will now find it harder to buy a house than their exact peer 10 years ago. Second, within generations: the majority of people my age who own a house have drawn heavily on the Bank of Mum and Dad. They often end up letting out a spare room to a friend and the ultra-low interest rates of the last five years can mean the rent their mates is paying them is more than their mortgage. So the already well-off get an extra boost. Having parents who live in London also confers a huge advantage if you want to work in the arts, media, finance or law, because you can bunk up in the family home while you slog through unpaid internships or further study.

This state of affairs should worry all the political parties. For Labour, there's an obvious story about social mobility going into reverse and about the rich becoming richer. For the Lib Dems, there's a chance to win back some of the young voters who defected over tuition fees.

But perhaps the Tories have most at stake. The assumption has long been that homeowners become more conservative. More importantly, the Tories' core message – that with hard work comes rewards – is dented by a quick look at the housing market. On the country's median salary of £26,000, even a dual-earning couple will struggle to buy the average house at £250,000. (In London, the situation is flat-out ludicrous: the average salary is £34,000 and the average house now costs £458,000. And it will have risen another £200 between me typing this and you reading it.)

The worst thing about Britain's housing crisis is that the solutions are very simple in theory, but impossible to implement in practice. In an ideal world, more Londoners would move to the rest of the country, where house prices are lower and housing stock more plentiful. Except that's not going to happen while so many industries are based here and so many sandwich shops and dry-cleaners and wine bars are reliant on their custom. London is, as Vince Cable put it, a "dark star" sucking in skilled workers from the rest of the country and we just have to live with that.

No, the real solution is equally simple and equally impossible: build more houses. Politicians have cottoned on to this in the abstract, but they are more reluctant to be drawn on where these houses should be. That's because homes need to be in places where people want to live and that means the green belt. Oxford and Cambridge have particularly tight belts and high prices as a result. The south-east is spangled with golf courses that should be suburban homes. As a further insult, London bristles with two-storey Victorian terraces that are a wildly inefficient use of such coveted space.

But building on the green belt breaks that politicians' mantra: the losers shouldn't realise their fate. People like me – the houseless – might tentatively voice the opinion that Britain is not some olde worlde theme park, but a living, breathing society that needs to house its young and poor more securely – but who cares? We're not making the decisions, and our losses are indirect and diffuse, whereas that view of a rolling field can be appreciated every day.