Stacey Thomas with her family in their one-room-home at England’s Lane Hostel in Camden, north London.
Photograph: David Levene

To estate agents, England’s Lane, tucked away in a corner of Hampstead, is one of London’s most sought-after villages. Along its tree-lined length stretches a charming row of small shops, with a tearoom serving gentlemen’s relish on toast, a toy shop full of hand-crafted wooden toys, and a butcher that sells pheasant and grouse. In the window of the DesRes estate agency, flats are offered for rent at £1,500 a week, and just opposite, down a private side road, a seven-bedroom arts-and-crafts style mansion, set in a garden the size of a small park, is on sale for £6.7m.

Walking along England’s Lane, passersby might not notice a mildly forbidding building behind railings, an old nurses’ home that in 2004 was turned into a hostel for Camden’s homeless families, despite protests from local homeowners that it was “not suitable” for the location. The 160 families squeezed into this large red brick block do not visit the shops and cafes here, nor do they get facials at the naturopathic beauty salon, or gaze at the estate agent’s window. Each family has a very small room, originally designed for a single student nurse.

The England’s Lane hostel was intended to provide temporary accommodation for homeless families; however, once they move in here, “temporary” can mean years. The hostel is a modern day version of Dickens’s Marshalsea prison from Little Dorrit, a reluctant community with its own hierarchy of suffering, where years are ticked off by unlucky people who have run aground for one reason or another.

Janice has been here for five-and-a-half years, with her husband and two children now aged nearly two and five. In one corner of her room, on top of a small fridge, stand a couple of electric rings to cook on. That’s the kitchen. There’s a shower room in a cupboard and all the family’s possessions are stuffed into a stack of suitcases squashed by the door. There is only just enough room for the double bed which they all share, the four of them sleeping together restlessly.

Janice is a thoughtful, lively 30-year-old, a worried mother eager to get back to work some day. When I first met her 18 months ago, she was still bewildered to find herself in these circumstances. “I never, ever expected something like this to happen to me,” she said. At that time, however, she still had some glimmer of hope that she would be rehoused.

The hostel is a modern version of Dickens’s Marshalsea prison a reluctant community with its own hierarchy of suffering

Janice was born in north Kensington, the shabby end of Notting Hill. Her father came from Egypt long before she was born, working all hours at three low paid jobs. She said he’s a passionate Anglophile, “worships the royal family, more English than the English”, and proud of Janice’s English education. But he was appalled that anyone in his family might be reduced to living in a hostel or need their income topped up with benefits of any kind. He was adamant, she said, that I should not use her real name in this story, or photograph her, though she wouldn’t have minded. Her mother is ill with a heart condition, so Janice takes the baby on the 45-minute bus ride over to visit them nearly every day.

It took only a frighteningly small step for Janice and her husband to slide from what seemed like security, to ending up here. They both worked, she had a good job as an administrator at Hendon Police College, but she was only on contract and hadn’t completed a year – so when she got pregnant, she didn’t qualify for maternity leave and lost her job when the baby was born. They were living in a bedsit where the damp seeped through the ceiling, but when they complained to the landlord that it was unhealthy for the baby, he evicted them. The practice is so widespread it has a name, “revenge eviction”, and it happens to some 200,000 private tenants a year, according to the housing charity Shelter. (A bill in the House of Commons is currently trying to ban this practice.) Landlords know they can instantly re-let to someone willing to accept almost any squalor: windowless, damp basements, kitchens divided from toilets by flimsy partitions.

The England’s Lane hostel for the homeless in Camden, north London. Photograph: David Levene for the Guardian

On eviction day, Janice called the council in a panic but was told there was no redress, nothing could stop them being turned out on to the street, so they should come to the town hall with their baby and their luggage, as they would qualify for temporary housing.

Later that night they were brought to England’s Lane. “I was so scared at the very word hostel. I was horrified,” Janice said. “I expected the very worst. I didn’t know what sort of people would be here. But by then, we were grateful for anywhere. At first, you have to learn how to survive, how to avoid arguments over small things – like noisy neighbours. You need to make friends.” The hostel wasn’t as bad as Janice’s worst fears, but neither did it turn out to be temporary.

In June, a year on from our first meeting, Janice was still living in England’s Lane, almost all her hope gone, looking more defeated as she saw their chance of escape receding with every year. She looked drawn and wan, her two children subdued, lying on the bed watching television.

“I had a degree, a good education and a good job so why would I expect to end up here?” she said, smoothing back her dark hair, as she dandled her youngest child on her knee. She has a degree from Middlesex University and had worked as an administrator in a law office before the job at Hendon Police College earning £26,000 a year. Her husband runs the postroom in an office block in Kensington: not a well paid job, but it offers security, he’s had it for a few years and likes his employers.

England’s Lane is better than many hostels for the homeless. The building is solid and imposing, the entrance hall deceptively grand with its 1940s wood panelling, a place well cared for by its managers. It doesn’t smell, though there are mice and insect-trap boxes in every corridor. Mostly, say its residents, it’s not scary but it is nonetheless a dark place. “When people leave, they want to forget they were ever here,” says June, one of the two warm but firm women managers on site all day, who keep trouble and rows to a minimum. At one point there had been drug dealers, rowdy all-night parties, and one woman who took to banging on her neighbours’ doors at 3am, looking for her imaginary missing man.

June has a tough-love approach to the residents here: she counsels the parents and children, advising, soothing, keeping things calm. But she confesses she is puzzled that the residents put up with it when they could move out to private rented flats. Years ago, when she was a mother of two sets of twins with nowhere to live, she pestered every housing association in the land until she got herself a place. She cycles to work, an hour and a half each way from Newham. But the longer the residents of England’s Lane stay in this place of quiet desperation, the more their confidence falters, as their chance of being rehoused recedes.

* * *

For those trying to navigate its corridors, the housing system is an arcane maze. Janice and her husband’s options seem to shrink every year. They could move out of the hostel and the council would find them a private rented flat miles away in Newham, Redbridge or Enfield. Nationally over a third of the homeless in 2013 were rehoused far away, and many London families were sent to Birmingham. The shortage of new housing is causing London boroughs to outbid each other, driving up costs in their frantic search for private rentals for their homeless families. When Camden bought this nurses’ hostel from the NHS, it paid well over the odds because Kensington and Chelsea barged in and sent the bidding price shooting up: there is no central authority to stop councils outbidding each other and wasting money.

The residents are among a rising number of households in temporary accommodation, 60,000 currently in England

Janice has opted not to go into private rental. If they moved to a private flat found by the council in, say, Redbridge, an hour away, her husband would lose his prized job in Kensington, their eldest daughter would have to move school and they would be too far from her parents who rely on her, and she on them. “But we would do all that gladly,” Janice says, “we’d go far away, lose the school and the job – but only if it was permanent. But it wouldn’t be. The risk is we’d have to do it all over again after a year and maybe over and over.” Private tenancies only last six months and few landlords now take people on housing benefit. “Each time you start again they’d want a high deposit that we haven’t got.”

The residents of the England’s Lane hostel are among a rising number of households in temporary accommodation, 60,000 currently in England, just one step up from living on the streets. Some, like Janice’s family, were evicted by landlords, some lost their council flat when they couldn’t pay the new bedroom tax for having a spare room and some fled domestic violence. A few are refugees who, in a bizarre welcome to citizenship, find the day they are granted official refugee status they are instantly evicted from accommodation for asylum seekers.

Many are turned away if councils deem they have made themselves homeless “intentionally”, thrown out by parents, fallen into rent arrears, deemed to be responsible for their own predicament. The national waiting list for social housing stands at 1.7m households – up by 65% since 1997. The law says families with children must be given shelter, but their chance of moving into a coveted Camden council flat or housing association home is vanishingly small. Camden has some of north London’s most desirable residential streets and parks, yet it’s one of the country’s poorest boroughs too, a prime example of the extremes of Britain’s housing problems. To buy an average house in Camden now, you need to earn at least £200,000 a year. No one can afford to live here apart from the extremely wealthy and those poor enough to qualify for council housing. Meanwhile, the pressure grows on Camden council to find somewhere to put their swelling ranks of the homeless.

In England’s Lane, the residents suffer the effect of one particular political calamity – the right to buy. The signature policy that swept Margaret Thatcher to power in 1979 has caused two million council homes to be sold at a discount. At the time it was hailed as a policy that endorsed working-class aspiration and promised upward mobility. But its long-term effect has been to deprive others of anywhere affordable to live.

In Camden, 40% of council homes have been sold. Of those bought by their tenants, says the council, some 40% were sold on to buy-to-let landlords, many of them companies who rent flats back to the council at inflated rates: a council rent might be £150, but the next door ex-council flat rented out by a landlord will go for £450 a week. Prime minister David Cameron promised before the last election that any further council property sold off would be replaced one for one. Since 2012, however, when Labour’s programme for pre-financed social housing finally ran out, more council houses have been sold than built. Cameron has also increased the discount for tenants to 70%, making council flats even cheaper to buy.

The result is that it is harder and harder for people such as Janice to find somewhere viable to live. In the meantime, she is stuck. She does her best, she’s enterprising and energetic, she seeks out anywhere free to take the children, anywhere with space to play – the park, the swimming pool – anywhere to escape this tiny room built for a student nurse, not for a family of four. The library was one haven, but after cuts it’s only open three days a week. “I feel lost, and sometimes I can’t breathe,” she said. “I can’t cook here, my children can’t play, I can’t be myself, all of us locked in. In winter, when it’s dark early, it’s unbearable.”

* * *

The longer families stay in England’s Lane hostel, the harder it gets to leave. If they made the leap into some distant, insecure private flat, they would lose the precious council housing points they have accumulated.

Points – they are the key to everyone’s life here. Residents of England’s Lane talk about points all day long, compulsively comparing them, how to acquire them, who has most, who struck lucky and how. Points are more precious than money, painfully earned by sticking it out here year after year: points give you bidding power when a vacant flat comes up. Yet this is a strangely elusive and dwindling currency.

Here’s how it works: you get points for children, points when they go over the age of 10, points for sickness, points for having suffered domestic violence and harassment, points for years lived in Camden and points for every year spent in England’s Lane. A strongly worded GP’s letter about health issues might perhaps get you 40 points, and a hospital consultant’s statement that living conditions are having a severe health impact might just get you 80 points. Every year your points rise by 10%, so in theory you can count the number of years it will take. But here’s the catch: every year the number of points required to be rehoused rises – the result of growing numbers of homeless people (partly as a result of the introduction of the bedroom tax), and the acute shortage of affordable housing.

Everyone talks about points, compulsively comparing them, how to acquire them, who has most, who struck lucky and how

When I first met Janice in 2013, a family needed 600 points, on average, to be allocated a Camden council flat. Janice had just 220 but she was told by her housing officer she’d acquire enough points in six years’ time. Eighteen months later, the number of points needed has risen to 700, but Janice has only 242. Now she’s told it will be another 10 years before she qualifies. “I won’t cheat, like some people,” she says. “I wasn’t brought up to cheat, to go to the doctor and beg for letters and try to get extra points that way. That’s not how I am.” As Janice has mild diabetes, she might clock up some more points, but her reluctance means she may be leapfrogged by others who are less scrupulous.

Every Wednesday night at midnight – actually at one minute past on Thursday morning – Camden council puts online the few flats available. For the residents of England’s Lane, a frantic bidding process begins. Any of a possible 25,000 people on Camden’s waiting list can log on at the same time, desperately seeking one of the 10 to 20 flats that come up in a typical week. The more bedrooms a flat has, the more points it costs. Most weeks about 250 families get online to declare their points and are told what number they are on the list for that flat. “As I have so few points, I am usually well over number 100,” Janice says. “One wonderful Thursday I was number 30!” (That might have been because of luck or an undesirable flat, she didn’t know why.) “But that still meant 29 other people had to look at the flat and turn it down before I had a chance. Of course I didn’t get it.”

Janice wants to go back to work, but the vagaries of the benefit system mean she would be worse off. She didn’t plan to have a second child, these things happen. If her husband and she had waited for a stable home, they’d never have had children at all. She looks despairing at the prospect that she can never return to work, but it’s difficult. Her husband is paid £1,300 a month – £15,600 a year. The hostel rent, paid by housing benefit, is £1,167 a month, energy bills included. If she worked, they would lose housing benefit and she would have to pay for childcare and travel to work, making them even worse off. “But I will still do it, because I have to, I need to, even if it’s at a loss for us and even if we have less. I am just so depressed.” She looks tired and defeated. Going to work would make her feel better but it wouldn’t solve their housing problem.

Some residents of England’s Lane hostel are even more trapped than Janice. The longest-term resident, Stacey Thomas, lives in a room with her husband, her son of seven and daughter of four. “We came here when Callum was just three days old,” she said, and that’s seven and a half years ago.

They are not even allowed to bid for flats when they come up: “We’ve been suspended from bidding as we owe over £2,000 in rent arrears, and I don’t know how we’ll ever pay that back,” Stacey told me. How did it happen? They both had jobs for a while, but they failed to get their housing benefit adjusted immediately, so they were overpaid for a couple of months, and now owe the money back. They will never be rehoused unless they get those back-payments down to under £500 – and she doesn’t see how she’s going to manage that as she only has £60-a-week shopping money left for the four of them. Stacey waves a bundle of letters she is collecting from advisers, doctors and her children’s school warning of the harm being done to them. So far they have produced no effect.

* * *

Some of the 160 families in the hostel hang on in limbo, hope kept alive by the chance, the remote chance, that they might end up as winners in the great property gamble. If only they can tolerate living here long enough, they might win the Thursday night lottery. Sami and his family recently won that jackpot, arousing hope and envy in those they left behind.

When I first met Sami (not his real name) in early 2013, he had already been in England’s Lane for five years with his wife Faria and six-year-old daughter Aida. They had made their minuscule room remarkably elegant, with a small fish tank quietly burbling oxygen, a potted fern, a neat sofabed and the kitchen shelf screened off, giving the momentary impression as you stepped in from the corridor that this was just their sitting room, a polite front parlour. Surely bedrooms, kitchen, dining room, storage space must be through some other door beyond? But everything was tucked away here. The illusion of space and gentility mattered to them.

Prime minister Margaret Thatcher at the Conservative party conference in 1979. Photograph: PA

Sami was born and brought up in Birmingham, but he had been the black sheep of his Pakistani parents’ family and had lost touch with them. “I was naughty, a problem teenager, but I learned better now.” He has worked on the machines at the McVitie’s biscuit factory in Leicestershire. After working as a repair engineer, Sami ended up in Camden as a minicab driver until he broke his arm and shoulder, and had to claim housing benefit for a while. But his landlord didn’t want any benefit cases, not even in their miserable two-roomed, lino-floored flat above a chicken shop in Camden. The landlord kicked down their door and threw all their possessions on to the street. They tried to find another private flat, but they didn’t have the £400 deposit that landlords were demanding. So, with one-year-old Aida, they landed with their suitcases outside Camden town hall. “The shock of that eviction was something we’d never want to go through again, never,” said Sami.

When I met him last year, Sami was working nights as a concierge and security guard in an office block. Though he claimed he only worked two nights a week, he looked tired, and the hostel managers reckoned he worked all the hours there were, saving, saving. Dapper and well organised, Sami spoke fast and expressively, exuding determination. Calculating exactly how many hours to work officially to claim housing benefit and tax credits is a skill anyone living with a high rent needs. Their hostel rent was £291 a week to be paid in full if he worked full time, along with £76 a month council tax. By dropping down (officially) to part time, he only need pay £87 a week, and housing benefit picked up the rest.

Here they stayed – where time is measured out in points – for five long years, while Sami worked out how to keep them all afloat, determined to escape some day. “I suffer bad depression,” he said. “I sleep with a breathing device that’s noisy for my family. I’m on medication, I often can’t sleep.” His daughter, he said, was sick too. “She’s depressed, she’s under the psychiatrist. Sometimes she bangs her head against the wall for no reason.” After six years, he had clocked up 515 points for the family, adding in the sicknesses, his wife’s pregnancy and 10% extra for every year they clung on there.

Sami said he was green when he arrived: “But you soon learn the ropes from other people.” He learned, for example, the various ways some people cheat the system. “Some rooms are empty, just used for storing stuff, because some people don’t really live here at all, they just pretend. They live outside with friends or family and just clock up the points each year. They’re not really homeless, they do their waiting-list time somewhere else.” June, the manager, says they stamp down on these kinds of scams, monitoring who comes and goes.

Sami himself is clever but above all, he’s been exceptionally lucky. The number of points he needed last year was 650 – but every year the necessary points were rising faster than his own, forever just out of reach. In May 2013, he told me, “I don’t mind where we go, where they send us, so long as it’s secure and permanent. I want somewhere a private landlord won’t ever evict us again, and somewhere near enough so I still commute to work.” That’s what he was holding out for.

And that’s what he got. When I contacted Sami and Faria in September, he had hit the rollover prize. “I bid every Thursday evening for anything, but always we were 17th or 20th on the list, never enough points, always other people with more points took the flats ahead of us.” Then suddenly, one Thursday they were only seventh in line for a flat on the 16th floor of a high-rise block. “When we came for the viewing I couldn’t believe our luck. All the seven families were there together inspecting it, same time as us, and every one in front of us turned it down. Why? I don’t know. They were saying it made them scared and dizzy to be this high up. Idiots!”

With the new baby just a few weeks old, they had moved into their new flat. When I visited, he showed me with pride their two bedrooms, nice floors, well-painted rooms, bright light from airy windows, clean lifts that work, a smart entrance hall, a park outside and a tube station round the corner. Camden is well known for high council housing standards and this gleaming block is so well kept you wouldn’t know it wasn’t private. For them, their six years of purgatory in England’s Lane has paid off handsomely, bringing them a council flat that guarantees a lifetime’s secure roof over their head at a rent they can afford. But it brings them much more than that. This stroke of luck has vaulted them in one leap from the league of losers to housing winners in the great British property game.

The day Sami and Faria got the key to their council flat, they were gifted a prime central London property. What’s it worth? An identical flat was for sale in this block at £399,995. After three years, Sami and his wife can buy it with no deposit, at a tenant’s discount that knocks off £102,700 (£77,000 outside London), a discount the government keeps increasing. Sami may have savings and a good enough job to get a mortgage in a few years, easy when the price is marked down so low beneath market value. But if he doesn’t have the money, council tenants find plenty of eager agents knocking on their door who will put up the cash to buy the flat for them.

London housing estates in desirable districts are leafleted regularly by companies tempting residents with offers of a bundle of money in exchange for giving up their lifelong secure tenancy. Photograph: The Guardian

London housing estates in desirable districts are leafleted regularly by companies tempting residents with offers of a bundle of money in exchange for giving up their lifelong secure tenancy. When I lived briefly on an estate in Lambeth, where I was researching a book in 2003, a company called Berkley Alliance kept soliciting me to sell my flat. These property investors are often fly-by-night outfits, with shifting company titles in this business, but it’s perfectly legal to urge council tenants to sacrifice their security for an instant pile of cash. One called London Property Agents has recently been offering “up to £100,000” to council tenants in prime London districts. That’s a strong temptation to people in debt – but once they exercise their right to buy, they lose any right ever to be housed again.

Last week, Sarah Hayward, the leader of Camden council, showed me round one of her borough’s finest council estates, a Victorian block overlooking Hampstead Heath where 48% of the flats have now gone to right to buy, mostly to private landlords. “In 2010 we only had to let 14 go,” Hayward said, but this year 110 flats have been sold. The council isn’t even allowed to keep the money to build more homes, but forced to send most of it to the Treasury. “The madness of housing finance is that almost all of government subsidy goes on housing benefit, mostly straight into the pockets of private landlords, but only 5% is spent on building new homes,” said Hayward. We stood gazing up at the elegant iron work and ornamental tiling. A plaque showed social justice theorist RH Tawney once lived here. What would he make of it being sold for buy-to-let? “We just valued a flat here for a tenant exercising their right to buy – it’s worth £590,000,” Hayward said ruefully, powerless to defend her council’s property.

As more council houses are sold, the stranded England’s Lane families have even less chance of moving out of their bedsits. In spite of the discounts, June, the manager, reckons only a small number of the hostel residents would be interested in buying (and then selling) their council flat. “Most would just be so grateful for a council home to live in for ever, without having to keep moving and keep taking their children out of schools.”

Meanwhile, Janice and her family sit and wait in their single room for a chance that may never come. When we spoke in October, she was hoping for a place on a Camden-run back-to-work scheme, but she had no hope of moving out of England’s Lane, with winter approaching and nowhere to take the children on dark after-school days. “There’s nothing I can do, except try to keep myself sane,” she said, counting her precious points year by year, knowing they lead nowhere.

Yan Yates is one of a growing number of young people who are turning their backs on London's unaffordable housing in favour of alternatives. He lives as a 'continuous cruiser' on London's canals moving from one place to the next every two weeks. But as the moorings fill up, some fear that the overcrowding happening on land is beginning to blight the capital's canals