A row of great brick arches marches along a back street in Stratford, east London, framing the front doors to a row of new homes, as if a Victorian railway viaduct had mated with a Georgian terrace. This sturdy brick mass is an arresting sight to find among a jumble of 1960s council housing in Newham borough, rising four storeys to a crenellated top, giving it the fortified air of a casbah. Walking through one of the gaping vaulted archways into a central courtyard, you almost expect to find a lively souk of carpet sellers hawking their wares.

This is council housing as imagined by Peter Barber, one of the most original architects working today. Over the past decade he has built a reputation for his ingenious reinventions of traditional house types and his ability to craft characterful chunks of city out of unpromising sites. Where the rest of us might see a sliver of useless verge or a bit of leftover car park, Barber can imagine a row of back-to-backs or a cluster of cottages. He is a master of humane high-density, designing that rare thing: new housing that feels in tune with the grain of London, in the form of neither alienating slabs nor tacky towers, without resorting to pastiche.

“The thing most people forget,” he tells me, when we meet in his King’s Cross studio, a tiny Dickensian shopfront crammed full of cardboard and plaster models, “is that housing accounts for 70% of all the buildings in London. When we design housing, we are designing cities.”

Judging by the bulk of new residential projects in the capital, it appears to be something most developers have forgotten. Conceived in splendid isolation, their steroidal slabs greet the street with bin stores and service hatches, shuffling their residents in through segregated lobbies, with few apparent concerns beyond the sanctity of their own property boundaries, save for the occasional garnish of pseudo-public space.

But when you visit one of Barber’s projects, you get a sense that it is a fragment of a much bigger urban idea. There are indications that he has thought long and hard about how people might occupy their porches, how neighbours might interact between balconies, how people might dwell in the space around the edges. There is an indication that housing, as the physical stuff that makes streets, might offer more than just private shelter behind the front door.

The site in Newham, before Barber got involved, had been slated for a pair of apartment towers. He argued that the same number of homes could be achieved in a courtyard formation of what he calls “tower houses”: four-storey homes that have a kitchen-diner on the ground floor, two floors of bedrooms, and a top-floor living room that opens on to a roof terrace, arranged back-to-back around a tree-lined square. They are a combination, he says, of the “bandbox” house, a compact 19th-century American house type with a room per floor, found in Philadelphia, and the good old northern English back-to-back – which became associated with overcrowding and squalor, and was outlawed by the 1909 Housing Act.

Barber thinks they are due a revival. “Back-to-backs deliver high density, while providing people with a house rather than a flat,” he says, noting that he has rotated the top-floor living rooms by 90 degrees, so they enjoy a dual aspect. “You’re sharing three walls, so they’re cheaper to heat, and there’s more potential for neighbourliness.” He hopes the deep recesses of his arcaded facades will encourage people to hang out, “like a kind of level-access Mississippi stoop”, while the interior courtyard, with its balconies, windows and archways, feels like a stage set ready for a great domestic opera to play out.

The sense of theatre is no accident. Barber talks of his housing projects as multi-levelled stages to be animated by the activities of daily life (he can’t wait to see washing lines strung across the terraces), and thinks architecture can play a crucial social role. “The focus of all of these projects is the design of a public or shared space that brings people into close proximity,” he wrote in a recent essay about his work, “making them highly visible to one another and therefore more likely to actually meet.”

Such thinking was at the core of his recent project for a mews of homes for people over 60 in Barking, partly designed to tackle loneliness. On a former strip of lock-up garages, he has built two rows of little bungalows with brightly coloured front doors that face on to a pedestrian twitten – a word for an alleyway in Brighton, where Barber lives. (Being from Yorkshire, I’d call it a ginnel.) The fun, wavy roofline of the facades indicates the presence of generous rooms with airy barrel-vaulted ceilings, while big windows tilt open on to the herringbone brick-paved passage.

“It’s much more sociable than where I was living before,” says resident Pauline Branch, who moved here from a block of flats. “We’ve been sitting out here in the evenings, singing together. It reminds me of an old East End street – I’m always seeing my neighbours.”

Windows on the world … Donnybrook Quarter in Hackney. Photograph: Morley von Sternberg

Down the road, near Barking station, another row of 16 cottages is nearing completion, built on what was a thin strip of verge. Barber calls them “side-to-side” houses, rather than back-to-backs, each just four metres wide, arranged in pairs along their long edges. Gates lead into little walled patios, from where the homes are entered, while a dogleg stair leads up to a vaulted first floor, with the feeling of being in an upturned ship’s hull. Occasional porthole windows add to the nautical air.

At the end of the row, a larger house for a wheelchair user enjoys a corner window seat popping out of a landing, the kind of detail you might expect from a private house commission, but rare to find in council housing. Barber included a similar projecting corner window halfway up the stairs in his 2006 Donnybrook Quarter scheme in Hackney; a resident recently told him her daughter sat there to do her homework for years, while watching the world go by. In this thoughtful approach to design, a staircase can be more than just a way of getting upstairs.

Back in his studio, where his team of just nine is busy working on plans for about 700 homes across eight London boroughs, Barber rifles through some other projects currently on the boards or recently completed. There’s a mansion-block-meets-ziggurat in Peckham, back-to-back-cum-courtyard housing in Greenwich, an almshousey scheme of little studios for homeless people in Camden, designed around an allotment garden. Each draws on well-established, much-loved housing types, mostly from the premodern era, brought up to date and served with a Barber twist. There are sketches for imaginary cities, too, scribbled on his commute from Brighton, which he posts almost daily on Instagram at @one_year_365_cities. One of them has taken on a life of its own, with a model exhibited at the Royal Academy this summer and a short film to be presented at an exhibition of his work at the Design Museum next month.

Utopian visions … Peter Barber. Photograph: Morley von Sternberg

“If you developed a 200-metre wide strip of land in the last bit of suburbia all around London,” he says, “you could get a million homes.” His utopian proposal for a Hundred Mile City would see the capital ringed with a “dense, intense edge”, a continuous belt of houses, schools, shops and little factories laid out in terraces, with narrow pedestrian streets and squares, the whole thing connected by a monorail (“Bexley to Brentford in 40 minutes”). He describes it as “a linear Barceloneta, a circular Rome, a stretched Porto”. It protects the green belt by intensifying suburbia: the city would grow inwards, rather than sprawl outwards, seeing “Metroland consolidated, back filled, integrated and urbanised”. It is a mad but brilliant proposition, a window on to another world that is just the kind of radical creative thinking that our housing problem so dearly needs.

Having long worked with homelessness organisations, Barber is well aware of the urgency of the situation, and that a solution is feasible, given the political will. “We were building 150,000 homes a year and funding the NHS in the aftermath of the second world war, when we were broke,” he says. “We need to end the right to buy, introduce private-sector rent controls, and have a massive social housing programme, funded by direct taxation.” With a few more people like Barber around, a world of well-designed housing for all would seem eminently possible.