Tucked away behind York Minster – the grand cathedral adorned with medieval stained-glass windows that dominates the North Yorkshire city’s skyline – is a cobbled street that has become an informal labour exchange. Each day, just before lunch, couriers dressed in the distinctive mint green and black uniform of Deliveroo, the online food delivery company, arrive at the end of this street, park their bikes and scooters next to a bench, and talk among themselves. Clutching their smartphones, they wait for someone, somewhere in the city, to place an order with one of the nearby restaurants and cafes. When an order comes through, one of the couriers will pick it up and deliver it in exchange for a small fee. They will then return to the bench to wait.

Plenty of people in early 21st-century Britain can identify with the experience of working for a company like Deliveroo. Drivers for the taxi firm Uber, for example, know only too well what it’s like for work to arrive in fits and starts via an app. But even more people are employed on zero-hour contracts in a wide variety of jobs, from stacking shelves to waiting tables to caring for the elderly. According to the Office for National Statistics, around 900,000 workers rely on a job with a zero-hour contract. These people start every week not knowing how much work they will get or how much money they will earn.

Informal or casual employment of this kind helps explain why Britain’s unemployment rate has not sky-rocketed since the financial crash of 2008. By contrast, almost a century ago, during the struggles of the 1920s and the Great Depression of the 30s, unemployment regularly climbed above 10%; at the most difficult moments, it went above 20%, with the true level – including those who were out of work but not officially registered as unemployed – even higher. Unemployment was also a serious problem – and one that suffered from the same difficulties of measurement – during the 1980s, when it climbed steadily to more than 12% during the early Thatcher years and, despite a steady decline, ended the decade at almost 7%. Despite the past decade seeing one of the slowest economic recoveries in history, unemployment has not got out of hand for long periods. After peaking at 8.5% in 2011, the rate has recently dropped below 4.5%.

The Conservative-led governments of the past seven years argued that declining unemployment rates are a sign that austerity is working. In the wake of the financial crash, in which banks collapsed and ATMs were hours away from refusing to dispense cash, David Cameron, George Osborne and their colleagues argued that there were too many skivers, sleeping off a life on benefits, while everyone else – the strivers, as they were labelled – trudged to work to support them. Cutting benefits would solve all manner of problems: it would get the skivers back to work, bring public spending down, and be good for the general health of the economy.

Like the unemployment statistics, these claims are deceptive. Millions of people are “just about managing”, to use a phrase the prime minister, Theresa May, was once fond of, and many are faring much worse. In the 12 months before March 2017, the Trussell Trust, Britain’s largest food bank charity, gave out more than 1m three-day emergency food parcels to people in desperate need. At the same time, as the Guardian has reported this week, debt has ballooned in the UK, returning to pre-financial crash levels, with household debt at 150% of income in 2015. This debt has been fuelled by low-to-no wage growth, inflated house prices and, thanks to historically low interest rates, credit made available for items such as cars. But the main issue for the estimated 8.3 million people living with unmanageable debt is needing to borrow money to survive.

According to some commentators, much of this economic insecurity – a major contributor to the discontent that made Vote Leave’s slogan “take back control” so powerful in the EU referendum last year – is rooted in a profound set of changes taking place across western economies. Traditional ways of working and archaic vested interests are being challenged by new and powerful forces. The gig economy epitomised by the likes of Deliveroo and Uber, for example, is often talked about as “disruption”, with digital technology a new and irresistible means of transforming business practices and satisfying their customers. Tremendous entrepreneurial individualism and flexibility is being unleashed, the world just needs to catch up.

The difficulty with these arguments is that we’ve been here before. The sight of workers standing in large groups waiting for work would have been familiar to the residents of British cities such as York more than a century ago. Those workers were painfully aware that irregular and low-paid employment offered few guarantees. While they might be able to obtain enough work each day, week or month, they could be stopped in their tracks at any moment by injury or illness. For all their willingness to work, those casual labourers, like their successors now, might not be able to make ends meet – and while not troubling the unemployment figures, were at constant risk of falling into debt and destitution. Those earlier generations’ answer to their problems, however, was the welfare state – the very thing that successive governments have blamed for the country’s current situation.

Many of the issues at the heart of the current malaise in British politics can be traced back to the late 19th and early 20th centuries, when what we now call the welfare state was slowly being assembled. From a legislative perspective, the welfare state was initially focused on a specific problem that had grown since the early 1800s: that many workers struggled to earn regular and reliable wages throughout the entire year. But these labour market problems were believed to be bound up with other issues: squalor, ignorance, want, idleness and disease – the “five giant evils” William Beveridge identified in his famous report, published in 1942. Schools, hospitals, council houses and benefits for those out of work were just some of the threads woven together to create the tapestry of the modern welfare state. The unravelling of that settlement has seen a resurgence of the original problem governments tried to tackle more than a century ago.

The country had grown wealthy during the industrial revolution, via the financial might of the City of London, the manufacturing power of the north of England, and an enthusiastic embrace of free trade. The poor, however, had not disappeared. The poor law, established in 1601, at the end of Elizabeth I’s reign, made Britain’s guarantee of help for the destitute unique among European nations. In the 1830s, an influential group of reformers, who later would be known as “modernisers”, changed the terms on which that help was offered. Assistance should amount to less than what the lowest-paid labourers could obtain with their wages, reformers insisted. Furthermore, help should only be available to people who were prepared to live in a workhouse – a dark, dank and miserable place where they were given an ill-fitting uniform and forced to carry out menial tasks in exchange for shelter and meagre rations of the most basic food.

The theory was deterrence: make the poor law frightening and only the most desperate – those truly in need – would trouble the authorities and public purse. Yet theory had a difficult relationship with reality. By the closing decades of the 19th century, hundreds of thousands of people – a total close to the population of Liverpool – were still using the poor law every year. The number was so large that many local authorities could not accommodate them in workhouses and had to continue offering cash handouts or food, as had been the case before the 1830s. Who were these people and why were they still asking for help?

Kensington High Street in London circa 1895. Photograph: Heritage Images/Getty

Shipping magnate Charles Booth organised a survey of London in 1886, which collected information about what went on behind closed doors in the capital’s slums. Booth then divided London’s population into categories based on their economic means and – somewhat questionably to modern eyes – their habits and behaviour. The result was a shock for his middle-class readers: 30% of London’s population seemed unable or only just about able to meet the basic costs of living.

Booth’s research threw light on dark corners of Britain and implied that the poor were a much bigger group than even the government’s statistics on the number of poor law claimants suggested. Many people wondered if he was right. One was Seebohm Rowntree – a member of a York-based Quaker family that manufactured confectionery and prided themselves on being responsible employers. They tried to know all their workers’ names and introduced welfare schemes, including an eight-hour day. But this approach became more difficult as business boomed and their company grew during the late 19th century. When Booth’s report came out, they worried they were not close enough to their employees to know if Booth’s conclusions applied to them too.

Rowntree and his assistants went out on to the streets of York in 1897 to investigate. Armed with notebooks, they criss-crossed the city, frequently passing the place where Deliveroo couriers would congregate more than a century later. They visited more than 45,000 people in the following two years, asking how much they earned, what they paid in rent, what food they bought, and all manner of other questions about their lives. Rowntree made sure to compile information on wages from local employers and to consult the latest medical research on the number of calories men, women and children needed to consume every day. He used this information to draw a “poverty line” – a calculation of the goods and services an individual needed to survive in modern society and how much money they needed to acquire them – and figured out how many people fell below it.

To Rowntree’s surprise, Booth’s findings applied to York as well as London. But Rowntree did not agree with his static description of the poor. Booth’s classification included numerous sub-divisions and distinctions: those he considered criminal, morally weak and semi-savage were separated from the poor who had not displayed an obvious – and unacceptable – flaw, such as a weakness for drink. Rowntree, however, believed there was much movement between these categories. The poor seemed always to be with us, he explained, but the poor were not always the same people.

Rowntree identified what he called the “poverty cycle”. Many people earned enough money to support themselves, he argued. From time to time, though, their circumstances changed – they got married, had a child or a relative died. These quite ordinary events stretched resources, sometimes for just a few weeks, but often much longer. But when they were over, the pressure on household finances was lifted, meaning people rose above the poverty line. Nevertheless, there was always something around the corner, waiting to drag people back down again; most obviously old age, when all those years of being stretched to the limit and unable to save would take their final toll.

Social reformers and charity workers across the country observed similar patterns of interruptions to, and pressures on, people’s earnings throughout the late 19th and early 20th centuries. One of the most important vantage points was Toynbee Hall, a university settlement located between Whitechapel and Spitalfields in the East End of London, where a small group of Oxford graduates lived among the poor, doing voluntary work and social research, before taking up employment – often of a much more lucrative and prestigious kind – elsewhere.

Among Toynbee Hall’s residents between 1903 and 1905 was William Beveridge. Beveridge spent time in the East End working with the unemployed, observing their daily routines, assisting with schemes that aimed to get them back into work, and following caseworkers from charities. In the process, Beveridge had come to a number of important conclusions. One was that unemployment was “at the root of most other social problems” because society “lays upon its members responsibilities which in the vast majority of cases can be met only from the reward of labour”. The other was that conventional wisdom about the causes of unemployment was wrong. For some commentators, unemployment was a question of character and motivation. An increasingly large part of mainstream opinion certainly accepted that a reality of modern industrial capitalism was periods when there would be no work available for some people – because trades were seasonal, or markets fluctuated. But Beveridge believed even this was a superficial understanding of the issue.

The biggest contribution to unemployment outside the downward slopes of the trade cycle, Beveridge argued, was the inefficiency of industry when it came to hiring workers. He asked readers of his book Unemployment: A Problem of Industry (1909) to imagine a scene he had encountered on many occasions: 10 wharves that each employed between 50 and 100 men per day, half of whom were regular staff and half of whom were reserves. While each wharf would experience similar high and low points in trade throughout the year, they were also likely to have their own individual fluctuations within those patterns. Anyone looking at the 10 wharves as a whole would not see these smaller deviations. The problem was that those smaller deviations were all that mattered to the reserve labourers walking from wharf to wharf asking for work each morning, because they meant the difference between them and their families eating, or going hungry.

If there was better communication and planning, Beveridge argued, almost all of those men would be able to find work each day. The problem was that business and industries were quite happy with the situation: they often had many more workers than vacancies, and did not need to pick up the costs of supporting those who couldn’t find work. Beveridge believed the state was the only institution with both the power to solve this problem and the interest in doing so. The political will to act on this conviction would have far-reaching implications for the millions of people who have found themselves out of work since. But we have slid backwards into a situation where precarious work paid by the hour is considered a sign of progress.

The Liberal party administrations that governed Britain before the first world war changed Britain for ever. They modernised the tax system, differentiating between earned and unearned incomes, and introduced graduated rates for the roughly 3% of the population who qualified to pay income tax. David Lloyd George, the Liberals’ charismatic chancellor of the exchequer, announced a “People’s Budget” in 1909 – one for “raising money to wage implacable warfare against poverty and squalidness”.

The principal aim of the budget was to tackle interruptions to earnings among the working-age population. Following the example of similar schemes in Germany, national insurance involved weekly contributions from three groups: workers, their employers and the state. All wage earners aged 16 to 70 and earning less than £160 a year who paid weekly contributions of four pence a week (three in the case of women) could claim sick pay for up to 26 weeks a year, and treatment from a government-approved doctor. But another aspect of the Liberals’ plans had not been tried anywhere else before: 2.25 million men in a number of trades and industries, such as construction and shipbuilding, where work could be brought to a halt by something as unpredictable as the weather, were to be enrolled in a scheme of compulsory unemployment insurance, which offered benefits of seven shillings a week for up to 15 weeks a year in exchange for contributions of two and a half pence a week.

These schemes had obvious limitations. Pensions were meagre; unemployment insurance was mainly for skilled men; health insurance excluded hospital care, and spouses and children. Almost everyone found something to be unhappy about. The British Medical Association complained about the prospect of doctors being forced to become government employees, while friendly societies, trade unions and private insurance companies thought the state was trying to force them out of business. Middle-class households resented being made to pay out to insure their domestic help. Moreover, Labour MPs complained about the contributory system. What about those who couldn’t pay in or who found themselves out of work for longer than 15 weeks? Why not follow the example of pensions and pay benefits to all out of general taxation?

The Liberal government recognised that national insurance on its own would not tackle interruptions to earnings; interventions into the economy would be required, too. Beveridge was drafted in by Winston Churchill, president of the board of trade, to help roll out a system of labour exchanges – another idea borrowed from Germany. A forerunner of the modern jobcentre, labour exchanges were an important part of the government’s plans for administering national insurance, with employers offered incentives to advertise vacancies in the exchanges and the unemployed asked to visit them to demonstrate they had looked for a job. For Beveridge, this had the potential to create an “organised fluidity of labour” that eliminated the kinds of problems he had observed in east London.

Migrant workers at a jobs noticeboard in west London. Photograph: David Levene for the Guardian

Labour exchanges certainly helped some people find jobs, but they were never the dynamic sites of free-flowing information and recruitment that Beveridge imagined. Pensions and national insurance proved much more successful and durable, though. After the first world war, a succession of governments extended and reformed the schemes in significant ways. The result, building on the centuries-old guarantee of help for the destitute, was an imperfect yet impressive system that offered assistance to many, though far from all, people in times of need.

During the Great Depression of the 1930s, however, when the Labour prime minister Ramsay MacDonald was expelled from his own party after he pushed through a 10% cut to unemployment benefits, there was widespread concern that these schemes were unsustainable. Interruptions to earnings looked like a minor problem when many people feared the complete collapse of the global economic system. What Britain seemed to have, Beveridge later suggested, was a series of “patches” – things that could be sewn on to country’s tearing fabric, rather than solutions to its underlying problems. Perhaps capitalism – a system that treated unskilled workers without the fallback of insurance as dispensable – was the real problem. There had to be a way to manage the economy that would transform life for people in Britain and enable national insurance to offer genuine security to all.

The economist John Maynard Keyneswas always clear about whose side he was on. “I can be influenced by what seems to me to be justice and good sense,” he explained, “but the class war will find me on the side of the educated bourgeoisie.” Although many of his contemporaries threw their lot in with Labour during the interwar years, believing they were the only realistic hope for progressive reform in an era of universal suffrage, Keynes stood firm. He was a Liberal and he intended to do everything he could to help the party – going so far as to help formulate its economic policy under Lloyd George’s leadership during the late 1920s and early 30s. Keynes was not alone. Beveridge might have kept his allegiance quiet in a bid to appear neutral, but in 1944 he won a byelection for the Liberals, and ran the party’s 1945 general election campaign.

Keynes cemented his status as the most important economist of the 20th century during the mid-1930s, when he published The General Theory of Employment, Interest and Money – the book that would serve as the set text for one side of the argument about how governments should respond to downturns and recessions. The world has revisited this argument regularly since the interwar years, including after the financial crash of 2008, when Nobel prize-winning economist Paul Krugman urged governments not to forget the lessons that Keynes had once taught them.

Thrashed out over five years of debate and discussion with his research students and colleagues at Cambridge, The General Theory is now widely known for a relatively small number of ideas and a simple message. Governments should resist the temptation to cut back during recessions, Keynes argued, because their root cause was a contraction in aggregate demand – the total amount of goods and services that are purchased in an economy – which collapsed when people and organisations, uncertain about the future, simultaneously chose to hold on to their money. Keynes explained how spending money had effects that rippled outwards in the economy, including the creation of employment, as the demand for goods and services increased. Governments should stop worrying about deficits and sound finance when times were bad – they could take care of them once everything was moving again.

Unlike Keynes, Beveridge was not invited to help run the economy during the second world war, and was disappointed not to be fully involved. By the summer of 1941, the government had tired of his sniping from the sidelines, and gave him a job they thought would keep him out of sight and mind for some time to come: a review of national insurance. In November the following year, he delivered the result: his ground-breaking report, Social Insurance and Allied Services.

The Beveridge Report on Social Services & Allied Services from 1942. Photograph: The National Archives

The Beveridge report was not quite what people were expecting, or what people now think it is. The British people did not want a “Santa Claus state” that handed out gifts to everyone, he argued; they wanted a form of economic and social security that reflected a history of paying into the system. Benefits should cover a much wider range of risks, Beveridge explained, but they should be simple to understand: everyone – workers, employers and the state – should pay flat-rate contributions and get flat-rate benefits paid out in return.

On the face of it, there was no reason for the government to be worried. Beveridge’s benefits scheme involved significant extensions and new responsibilities, but it also required people to pay hefty weekly contributions. However, Beveridge had made a number of recommendations, which he modestly called “assumptions”. The first two were an allowance for each child born (after the first) and a National Health Service, free for all people, both of which would be paid for out of general taxation. The other was that the government commit to a new way of running the economy: one in which they made sure that unemployment never went above 8.5%.

Beveridge had called these recommendations assumptions because he believed a reformed system of national insurance could not work without them. Unemployment had to be kept below 8.5% so that people could build up a history of contributions and the system did not collapse under the weight of demand when they needed help. Indeed, the easiest system to administer was one in which authorities could safely assume there were jobs for the vast majority of people.

Initially, the government was not as enthusiastic about committing to his recommendations as the public, who bought an astonishing 100,000 copies of Social Insurance and Allied Services during the month after it was published. The prime minister, Winston Churchill, refused to comment on the report for three months, and offered a vague endorsement when he eventually did. By the end of 1944, however, when victory over Germany looked certain, a series of white papers committed the British state not only to Beveridge’s plan, but also to a number of other new policies, such as a new secondary school system with a leaving age of 15. The roadmap for social reconstruction had been drawn.

The welfare state that came into beingduring the late 1940s underpinned a whole way of life that politicians only started to pull apart from the early 1980s onwards. The intention during the third quarter of the 20th century was to bring capitalism under control, specifically its tendency to interrupt and put downwards pressure on people’s earnings, rather than dispense with the system entirely. The Labour party, which won a historic landslide in the election of July 1945, put its mark on the whole project, in particular by nationalising whole swaths of industry. Yet, after half a century of debate and legislation, each political party had left fingerprints on the final product.

These points matter for a number of reasons. One is that we often assume the welfare state was a collectivist venture. But even strident individualists found reasons to support it. Indeed, the era of social democracy helped create successive generations of individualists, including the working-class people who suddenly found themselves socially mobile during the 1950s, 60s and 70s. Looking back, and quite understandably, those generations can often give in to the temptation to imagine this progress was all down to their own hard work. Yet, as sociologists such as John Goldthorpe have shown, these generations rode the economic and social wave created by the policies adopted after 1945. Economic growth expanded the middle class by creating new management-level jobs into which working-class people could move, in both the public and private sectors, meaning there was “more room at the top”. Moreover, in an era of full employment, home ownership started to rise, not only because new houses were built, but also because it was perfectly reasonable for banks to assume that people would hold down a job for 25 continuous years and therefore pay back any money they borrowed.

Could that strategy be repeated today? The answer that has been given repeatedly for the past decade – and in some cases longer – is no. We have come to see the welfare state simply as a cost to be kept down rather than part of an economic and social strategy that aims to deliver security for all and opportunities to obtain more for those who want to. The idea that these goals are no longer obtainable is clearly false. A good start would be to reconnect with the liberal idea, now more than a century old, that everyone sees returns when they pool risks, whether it’s the individuals who can stop worrying about what is around the corner, governments that might otherwise cut their headline costs but succeed only in shifting it somewhere else, or the companies that benefit from healthy and educated workers operating in a safe environment. A successful economy requires all these actors to understand that they need to give, not just take, in order to build an environment in which they and those that follow them are able to succeed.

Migrant workers waiting to be collected for farm and factory shifts in East Anglia. Photograph: David Levene for the Guardian

Are more radical measures required? In the long term, yes. The world has changed since the early 20th century: businesses and individuals behave differently and the “assumptions”, as Beveridge would have called them, that go with national insurance have evolved. The trend has been to pay for things by pushing the costs on to individuals, as has been done with university tuition fees. But there seems only so much mileage in this approach when debt is reaching dangerous levels, wages are stagnant and, as the economist Thomas Piketty has shown, income generated by wealth has increased rapidly for those lucky enough to have it.

One appropriate response would be to breathe new life into the radical strand of liberalism that differentiated between earned and unearned incomes back at the start of the 20th century. Piketty has argued for a global tax on wealth. But there are domestic policies that would go some way to achieving similar ends. We could consider applying capital gains tax to property – recouping some of the considerable profits that those generations who benefited from the welfare state have acquired from the houses they were able to buy, in part because of it.

A world without retirement

Read more

Some commentators suggest what seem like even more radical ideas, such as universal basic income (UBI): a guaranteed regular payment for every citizen that would keep them above the poverty line, even if they chose not to work. UBI would deliver security, but faces numerous technical challenges, not least the significant differences in living costs across the country, which make a “universal” sum impossible to settle on, even before tackling the political problems of accusations that it would simply make everyone a benefit claimant. Yet versions of the idea have found support across the political spectrum, from neoliberals such as Milton Friedman to the leftwing economist and one-time Greek finance minister Yanis Varoufakis. For the left, a basic income would give people security and dignity. For the right, however, that basic security would be valuable because it would mean people would be free to take the kind of irregular work offered by the gig economy or zero-hours contracts. The lesson of these differences and convergences of opinion is that tackling economic insecurity need not be done at the expense of efficiency, competitiveness or innovation.

Main image: unemployed labourers waiting for work at a dockyard in March 1931. Photograph by Fox Photos/Getty

Bread for All by Chris Renwick is published by Allen Lane at £20. To buy it for £14 go to bookshop.theguardian.com or call 0330 333 6846. Free UK p&p over £10, online orders only. Phone orders min p&p of £1.99.

• Follow the Long Read on Twitter at @gdnlongread, or sign up to the long read weekly email here.