The way Eric Klinenberg perceives the interviewees for his book on living alone, you really wouldn't want to be one of them. Nicky's voice is "soft and a bit squeaky". Kimberley from New York has "a sweet but somewhat sinister smile", while Ella "is brazen and brilliant, with muscular arms". Frankly, it is amazing Dr Klinenberg got out alive.

Kimberly, Ella and Nicky are just three of the 300 people whom the NYU sociologist talked to for a seven-year project designed to tease out the full picture behind the extraordinary fact that more than 50% of adult Americans are single, with one out of every seven living alone. And what Klinenberg found surprised him. Instead of stumbling on a subculture of anxious, angry or just plain weird single householders, he encountered a serene if slightly smug set of men and women who were quite convinced that they had got the best deal. Far from seeing singleness as a transitory blip on the path to the sunny uplands of couplehood, Klinenberg's guinea pigs felt that they had arrived exactly where they wanted to be.

This rush to singleness is hardly confined to America: Britain, Scandinavia and Japan all post even higher figures. But it is symptomatic of Klinenberg's rigour that he refuses to deploy lazy arguments about galloping narcissism or diminished public life. Instead he looks at the flinty data and concludes that an upsurge of settled singles is a symptom of a society's growing wealth, like owning cars or eating meat. Essentially, if we can afford to live alone, then we do, seeing it as "a mark of distinction, not a social failure". Anecdotally, too, the current recession seems to have sharpened the longing for a room – or, better still, a nice one-bedroom flat with low service charges – of one's own. Sharing with housemates, even bunking up with your boyfriend, is increasingly what losers do.

The same thinking prevails at the other end of the life cycle. Elderly single people – most often widows – are not, as you might imagine, fighting over the handful of old men still just about standing. These real-life Golden Girls know only too well that if they remarry, they will soon end up as full-time carers all over again. For precisely that reason they are not keen to go and live with their married sons and daughters either, telling Klinenberg that they are wary of being put to work as unpaid domestic skivvies. Some go even further, revealing that, actually, they're not even sure they like their children. Ava complains at the way her daughter bores on about her failing marriage, while Joan, a retired psychologist, snaps that her 11-year-old granddaughter is "incredibly manipulative. Not to be trusted an inch."

All this evidence of the unfettered American self following its highest calling makes perfect sense in a nation steeped in the visionary spirit of Emerson and Thoreau. Yet Klinenberg is equally careful to attend to the shadow side of such upbeat accounts. SROs – single room occupancy facilities – started in the 1930s as "plain hotels for plain people", but are now essentially hostels for capitalism's casualties. Here live a gang of "unmarriageable men": poorly educated, unskilled and often struggling with chronic ill health caused by multiple addictions. They live alone because, more often than not, they are too ashamed to go home to Wisconsin or Mexico – they worry about the welcome that awaits them. So instead they hunker down, developing what Klinenberg calls "defensive individualism", a spiny armour designed to repel neighbours who may be planning to pop round to rob them or suggest getting high. As Tim, a veteran of Manhattan's SROs, puts it: "I'm not anti-social but I have a hard enough time with my own problems without other people's problems."

Gathering up these threads – of those who aspire to live alone, those who happily already do, and those who accept that it is probably the best they can do – Klinenberg argues that we need to stop worrying about what it all means and concentrate instead on making it work. The first, and most profound, thing to do is acknowledge that solo living is actually a fantasy underwritten by the very real presence of the family, communities and the state. The 35-year-old woman who loves her aloneness does so because she has a dry-cleaner on the corner and a regular Sunday yoga class. Klinenberg's elderly widows are fine in their own homes as long as there are meals on wheels and visits from friends who play a mean game of mahjong. The unmarriageable men are able to live stable lives because housing charities have continued to invest in the SROs and attendant services. Even Thoreau, it turns out, used to get deliveries of home-cooked meals from his mum.

What we need to do, Klinenberg concludes, is craft new ways of living alone together, ones that acknowledge and nurture the links between the solitary and the communal. To see this in action he goes to Stockholm, "the global capital of living alone", where 60% of households are headed by one person. His first stop is Färdknäppen, a community-owned facility where 43 single people live in companionable autonomy. It sounds wonderful. You get your own little flat, but you can pop into the restaurant when you can't be bothered to cook (dinner is about £3). There's a library, a gym and – it must be a Swedish thing – a "weaving room". There's even a dedicated space for parties. I was just about to go online and put my name down, when I stumbled over this sentence: "Every six weeks a resident must help with the cooking and cleaning." At a stroke, what sounded like a lovely future turns out to be exactly like the dismal past. Solo living, as endorsed by the happily married Klinenberg, resembles nothing so much as youth hostelling in the 1970s, complete with the whiff of other peoples' socks.

• Kathryn Hughes's biography of Mrs Beeton is published by Harper Perennial.

The number of people living alone has skyrocketed. What is driving the phenomenon, asks Eric Klinenberg. And solo dwellers Colm Tóibín, Alex Zane, Carmen Calli and others reflect on life as a singleton