When I was young, there was nothing so bad as being asked to work. Now I find it hard to conjure up that feeling, but I see it in my five-year-old daughter. “Can I please have some water, daddy?”

“You can get it yourself, you’re a big girl.”

“WHY DOES EVERYONE ALWAYS TREAT ME LIKE A MAID?”

That was me when I was young, rolling on the ground in agony on being asked to clean my room. As a child, I wonderingly observed the hours my father worked. The stoical way he went off to the job, chin held high, seemed a beautiful, heroic embrace of personal suffering. The poor man! How few hours he left himself to rest on the couch, read or watch American football.

My father had his own accounting firm in Raleigh, North Carolina. His speciality was helping people manage their tax and financial affairs as they started, expanded, or in some cases shut down their businesses. He has taken his time retiring, and I now realise how much he liked his work. I can remember the glowing terms in which his clients would tell me about the help he’d given them, as if he’d performed life-saving surgery on them. I also remember the way his voice changed when he received a call from a client when at home. Suddenly he spoke with a command and facility that I never heard at any other time, like a captive penguin released into open water, swimming in his element with natural ease.

At 37, I see my father’s routine with different eyes. I live in a terraced house in Wandsworth, a moderately smart and wildly expensive part of south-west London, and a short train ride from the headquarters of The Economist, where I write about economics. I get up at 5.30am and spend an hour or two at my desk at home. Once the children are up I join them for breakfast, then go to work as they head off to school. I can usually leave the office in time to join the family for dinner and put the children to bed. Then I can get a bit more done at home: writing, if there is a deadline looming, or reading, which is also part of the job. I work hard, doggedly, almost relentlessly. The joke, which I only now get, is that work is fun.

Not all work, of course. When my father was a boy on the family farm, the tasks he and his father did in the fields – the jobs many people still do – were gruelling and thankless. I once visited the textile mill where my grandmother worked for a time. The noise of the place was so overpowering that it was impossible to think. But my work – the work we lucky few well-paid professionals do every day, as we co-operate with talented people while solving complex, interesting problems – is fun. And I find that I can devote surprising quantities of time to it.

What is less clear to me, and to so many of my peers, is whether we should do so much of it. One of the facts of modern life is that a relatively small class of people works very long hours and earns good money for its efforts. Nearly a third of college-educated American men, for example, work more than 50 hours a week. Some professionals do twice that amount, and elite lawyers can easily work 70 hours a week almost every week of the year.

Work, in this context, means active, billable labour. But in reality, it rarely stops. It follows us home on our smartphones, tugging at us during an evening out or in the middle of our children’s bedtime routines. It makes permanent use of valuable cognitive space, and chooses odd hours to pace through our thoughts, shoving aside whatever might have been there before. It colonises our personal relationships and uses them for its own ends. It becomes our lives if we are not careful. It becomes us.

When John Maynard Keynes mused in 1930 that, a century hence, society might be so rich that the hours worked by each person could be cut to ten or 15 a week, he was not hallucinating, just extrapolating. The working week was shrinking fast. Average hours worked dropped from 60 at the turn of the century to 40 by the 1950s. The combination of extra time and money gave rise to an age of mass leisure, to family holidays and meals together in front of the television. There was a vision of the good life in this era. It was one in which work was largely a means to an end – the working class had become a leisured class. Households saved money to buy a house and a car, to take holidays, to finance a retirement at ease. This was the era of the three-Martini lunch: a leisurely, expense-padded midday bout of hard drinking. This was when bankers lived by the 3-6-3 rule: borrow at 3%, lend at 6%, and head off to the golf course by 3pm.

The vision of a leisure-filled future occurred against the backdrop of the competition against communism, but it is a capitalist dream: one in which the productive application of technology rises steadily, until material needs can be met with just a few hours of work. It is a story of the triumph of innovation and markets, and one in which the details of a post-work world are left somewhat hazy. Keynes, in his essay on the future, reckoned that when the end of work arrived:

For the first time since his creation man will be faced with his real, his permanent problem – how to use his freedom from pressing economic cares, how to occupy the leisure, which science and compound interest will have won for him, to live wisely and agreeably and well.

Karl Marx had a different view: that being occupied by good work was living well. Engagement in productive, purposeful work was the means by which people could realise their full potential. He’s not credited with having got much right about the modern world, but maybe he wasn’t so wrong about our relationship with work.

Marx is not credited with having got much right about the modern world, but maybe he wasn’t so wrong about our relationship with work

In those decades after the second world war, Keynes seemed to have the better of the argument. As productivity rose across the rich world, hourly wages for typical workers kept rising and hours worked per week kept falling – to the mid-30s, by the 1970s. But then something went wrong. Less-skilled workers found themselves forced to accept ever-smaller pay rises to stay in work. The bargaining power of the typical blue-collar worker eroded as technology and globalisation handed bosses a whole toolkit of ways to squeeze labour costs. At the same time, the welfare state ceased its expansion and began to retreat, swept back by governments keen to boost growth by cutting taxes and removing labour-market restrictions. The income gains that might have gone to workers, that might have kept living standards rising even as hours fell, that might have kept society on the road to the Keynesian dream, flowed instead to those at the top of the income ladder. Willingly or unwillingly, those lower down the ladder worked fewer and fewer hours. Those at the top, meanwhile, worked longer and longer.

It was not obvious that things would turn out this way. You might have thought that whereas, before, a male professional worked 50 hours a week while his wife stayed at home with the children, a couple of married professionals might instead each opt to work 35 hours a week, sharing more of the housework, and ending up with both more money and more leisure. That didn’t happen. Rather, both are now more likely to work 60 hours a week and pay several people to care for the house and children.

Why? One possibility is that we have all got stuck on a treadmill. Technology and globalisation mean that an increasing number of good jobs are winner-take-most competitions. Banks and law firms amass extraordinary financial returns, directors and partners within those firms make colossal salaries, and the route to those coveted positions lies through years of round-the-clock work. The number of firms with global reach, and of tech start-ups that dominate a market niche, is limited. Securing a place near the top of the income spectrum in such a firm, and remaining in it, is a matter of constant struggle and competition. Meanwhile the technological forces that enable a few elite firms to become dominant also allow work, in the form of those constantly pinging emails, to follow us everywhere.

This relentless competition increases the need to earn high salaries, for as well-paid people cluster together they bid up the price of the resources for which they compete. In the brainpower-heavy cities where most of them live, getting on the property ladder requires the sort of sum that can be built up only through long hours in an important job. Then there is conspicuous consumption: the need to have a great-looking car and a home out of Interiors magazine, the competition to place children in good (that is, private) schools, the need to maintain a coterie of domestic workers – you mean you don’t have a personal shopper? And so on, and on.

The dollars and hours pile up as we aim for a good life that always stays just out of reach. In moments of exhaustion we imagine simpler lives in smaller towns with more hours free for family and hobbies and ourselves. Perhaps we just live in a nightmarish arms race: if we were all to disarm, collectively, then we could all live a calmer, happier, more equal life.

But that is not quite how it is. The problem is not that overworked professionals are all miserable. The problem is that they are not.

Drinking coffee one morning with a friend from my home town, we discuss our fathers’ working habits. Both are just past retirement age. Both worked in an era in which a good job was not all-consuming. When my father began his professional career, the post-war concept of the good life was still going strong. He was a dedicated, even passionate worker. Yet he never supposed that work should be the centre of his life.

Work was a means to an end; it was something you did to earn the money to pay for the important things in life. This was the advice I was given as a university student, struggling to figure out what career to pursue in order to have the best chance at an important, meaningful job. I think my parents were rather baffled by my determination to find satisfaction in my professional life. Life was what happened outside work. Life, in our house, was a week’s holiday at the beach or Pop standing on the sidelines at our baseball games. It was my parents at church, in the pew or volunteering in some way or another. It was having kids who gave you grandkids. Work merely provided more people to whom to show pictures of the grandkids.

This generation of workers, on the early side of the baby boom, is marching off to retirement now. There are things to do in those sunset years. But the hours will surely stretch out and become hard to fill. As I sit with my friend it dawns on us that retirement sounds awful. Why would we stop working?

Here is the alternative to the treadmill thesis. As professional life has evolved over the past generation, it has become much more pleasant. Software and information technology have eliminated much of the drudgery of the workplace. The duller sorts of labour have gone, performed by people in offshore service-centres or by machines. Offices in the rich world’s capitals are packed not with drones filing paperwork or adding up numbers but with clever people working collaboratively.

The pleasure lies partly in flow, in the process of losing oneself in a puzzle with a solution on which other people depend. The sense of purposeful immersion and exertion is the more appealing given the hands-on nature of the work: top professionals are the master craftsmen of the age, shaping high-quality, bespoke products from beginning to end. We design, fashion, smooth and improve, filing the rough edges and polishing the words, the numbers, the code or whatever is our chosen material. At the end of the day we can sit back and admire our work – the completed article, the sealed deal, the functioning app – in the way that artisans once did, and those earning a middling wage in the sprawling service-sector no longer do.

The fact that our jobs now follow us around is not necessarily a bad thing, either. Workers in cognitively demanding fields, thinking their way through tricky challenges, have always done so at odd hours. Academics in the midst of important research, or admen cooking up a new creative campaign, have always turned over the big questions in their heads while showering in the morning or gardening on a weekend afternoon. If more people find their brains constantly and profitably engaged, so much the better.

Smartphones do not just enable work to follow us around; they also make life easier. Tasks that might otherwise require you to stay late in the office can be taken home. Parents can enjoy dinner and bedtime with the children before turning back to the job at hand. Technology is also lowering the cost of the support staff that make long hours possible. No need to employ a full-time personal assistant to run the errands these days: there are apps to take care of the shopping, the laundry and the dinner, walk the dog, fix the car and mend the hole in the roof. All of these allow us to focus ever more of our time and energy on doing what our jobs require of us.

There are downsides to this life. It does not allow us much time with newborn children or family members who are ill; or to develop hobbies, side-interests or the pleasures of particular, leisurely rituals – or anything, indeed, that is not intimately connected with professional success. But the inadmissible truth is that the eclipsing of life’s other complications is part of the reward.

It is a cognitive and emotional relief to immerse oneself in something all-consuming while other difficulties float by. The complexities of intellectual puzzles are nothing to those of emotional ones. Work is a wonderful refuge.

This life is a package deal. Cities are expensive. Less prestigious work that demands less commitment from those who do it pays less – often much less. For those without independent wealth, dialling back professional ambition and effort means moving away, to smaller and cheaper places.

But stepping off the treadmill does not just mean accepting a different vision of one’s prospects with a different salary trajectory. It means upending one’s life entirely: changing locations, tumbling out of the community, losing one’s identity. That is a difficult thing to survive. One must have an extremely strong, secure sense of self to negotiate it.

I’ve watched people try. In 2009 good friends of ours packed their things and moved away from Washington, DC, where we lived at the time, to the small college town of Charlottesville, Virginia. It was an idyllic little place, nestled in the Appalachian foothills, surrounded by horse farms and vineyards, with cheap, charming homes. He persuaded his employer to let him telework; she left her high-pressure job as vice-president at a big web firm near Washington to take a position at a local company.

My wife and I were intrigued by the thought of doing the same. She could teach there, we reckoned, and I could write. It was a reasonable train ride from Washington, if I needed to meet editors. We would be able to enjoy the fresh air, and the peace and quiet. Perhaps at some point we would open our own shop on the main street or try our hand at winemaking, if we could save a little money.

It wasn’t the stress of being on the fast track that caused my chest to tighten and my heart rate to rise, but the thought of being left behind by those still on it

Yet the more seriously we thought about it, the less I liked the idea. I want hours of quiet to write in, not days and weeks. I would miss, desperately, being in an office and arguing about ideas. More than that, I could anticipate with perfect clarity how the rhythm of life would slow as we left the city, how the external pressure to keep moving would diminish. I didn’t want more time to myself; I wanted to feel pushed to be better and achieve more. It wasn’t the stress of being on the fast track that caused my chest to tighten and my heart rate to rise, but the thought of being left behind by those still on it.

Less than a year after moving away, our friends moved back. They had found themselves bored and lonely. We were glad, and relieved as well: their return justified our decision to stay in the city.

One reason the treadmill is so hard to walk away from is that life off it is not what it once was. When I was a child, our neighbourhood was rich with social interaction. My father played on the church softball team until his back got too bad. My mother helped with charity food-and-toy drives. They both taught classes and chaperoned youth choir trips. They socialised with neighbours who did these things too.

Those elements of life persist, of course, but they are somewhat diminished, as Robert Putnam, a social scientist, observed in 1995 in “Bowling Alone: America’s Declining Social Capital”. He described the shrivelling of civic institutions, which he blamed on many of the forces that coincided with, and contributed to, our changing relationship to work: the entry of women into the workforce; the rise of professional ghettoes; longer working hours.

One of the civic groups that Putnam cites as an important contributor to social capital in ages past was the labour union. In the post-war era, unions thrived because of healthy demand for blue-collar workers who shared a strong sense of class identity. That allowed the unions’ members to capture an outsize share of the gains from economic growth, while also providing workers and their families with a strong sense of community – indeed, of solidarity.

The labour movement has unravelled in recent decades, and with it the network that supported its members; but these days a similar virtuous circle supports the professional classes instead. Our social networks are made up not just of neighbours and friends, but also of clients and colleagues. This interlaced world of work and social life enriches us, exposing us to people who do fascinating things, keeping us informed of professional gossip and providing those who have good ideas with the connections to help turn them into reality. It also traps us. The suspicion that one might be missing out on a useful opportunity or idea helps prod us off the sofa when an evening with “True Detective” beckons seductively.

This mixing of the social and professional is not new. It is not unlike Hollywood, where friends have always become collaborators, actors marry directors, and an evening out on the town has always been a public act that shapes the brand value of the star. Or like Washington, DC, in which public officials, journalists and policy experts swap jobs every few years and go to the same parties at night: befriending and sleeping with each other, exchanging ideas, living a life in which all behaviour is professional to some extent. But as hours have lengthened and work has become more engaging, this social pattern has swallowed other worlds.

There is a psychic value to the intertwining of life and work as well as an economic one. The society of people like us reinforces our belief in what we do. Working effectively at a good job builds up our identity and esteem in the eyes of others. We cheer each other on, we share in (and quietly regret) the successes of our friends, we lose touch with people beyond our network. Spending our leisure time with other professional strivers buttresses the notion that hard work is part of the good life and that the sacrifices it entails are those that a decent person makes. This is what a class with a strong sense of identity does: it effortlessly recasts the group’s distinguishing vices as virtues.

Life within this professional community has its impositions. It makes failure or error a more difficult, humiliating experience. Social life ceases to be a refuge from the indignities of work. The sincerity of relationships becomes questionable when people are friends of convenience. A friend – a real one – muses to me that those who become immersed in lives like this suffer from Stockholm Syndrome: they befriend their clients because they spend too much time with them to know there are other, better options available. The fact that I find it hard to pass judgment on this statement suggests that I, too, may be a victim.

My parents have not quite managed to retire, but they are getting there. Even with one foot in and one foot out of retirement, their post-career itinerary is becoming clear. They mean to see parts of the world they couldn’t when they were young and had no money, or when they were older and had no time. Their travels occasionally bring them to London to see me and my family. On a recent visit the talk shifted, as it often does, to when I might be planning to return to the east coast of America, much closer to the Carolinas, which is where they and most of the rest of my extended family still live. As my father walks around the house, my three-year-old son trotting adoringly behind him, they ask whether I couldn’t do my job as easily closer to home.

I get hung up on as easily. The writing I could do as easily, just about. Building my career, away from our London headquarters, would not be so easy. As I explain this, a circularity threatens to overtake my point: to build my career is to make myself indispensable, demonstrating indispensability means burying myself in the work, and the upshot of successfully demonstrating my indispensability is the need to continue working tirelessly. Not only can I not do all that elsewhere; outside London, the obvious brilliance of a commitment to this course of action is underappreciated. It looks pointless – daft, even.

And I begin to understand the nature of the trouble I’m having communicating to my parents precisely why what I’m doing appeals to me. They are asking about a job. I am thinking about identity, community, purpose – the things that provide meaning and motivation. I am talking about my life.