If London’s economically inactive population were a city, it would be the second largest in the UK. Beyond the glittering towers, London is a city that blends economic dynamism with social lethargy. Despite notable successes in domains such as transport and culture, the strategic institutions established in 2000 with a Mayor at their pinnacle have done little to redress this. London’s economic surplus has a social deficit as its shadow.

Recent data from the Centre for Cities highlights several dimensions to this deficit. London has far and away the highest number of startups and business stock per head in the country. Its productivity is the highest in the UK. However, London is the UK’s third most unequal city and has the second most unaffordable housing. This raises questions about whether London’s form of devolution affords the social interventions that might create a more cohesive city.

Contrast this with Scotland’s devolved institutions, only a year older – yet Scotland is having a very different debate about its social and economic future. The point here is not to assess the success or otherwise of successive Scottish governments. There are serious policy failures, such as those indicated by poor literacy, numeracy and science outcomes in comparative international data through PISA scores. What is clear, however, is that there are the beginnings of a different kind of democratic engagement with a range of challenges. Scotland’s democratic discourse is a cacophony of concerns and ideas about how the nation can develop along more inclusive lines, even if its policymaking structures still lag somewhat behind.

Lately, this debate has connected with the idea of Universal Basic Income (UBI). A UBI is a regular payment made to every citizen without condition. Proponents emphasise its putative benefits in providing a basic level of security in an environment of rapid economic change, especially in the jobs market. UBI is seen as one means of cushioning the impacts of technological change while enabling adaptation in an often-frenetic jobs market. The idea was tested in Canada in the 1970s with encouraging results, which included young people staying longer in education and improved health.

Concerns about poverty, about the intrusive and ineffective nature of the modern welfare state, and about the future of working lives have begun to drive a local and national debate on the possible benefits of UBI. Two places in the UK – Fife and Glasgow – are exploring the feasibility of a UBI experiment. The Scottish government and parliament are looking at whether a UBI could better meet the needs of Scots and considering how this might be tested. The debate is becoming an exploration of what is possible and desirable, and thoughtful contributions have been made to it across all the political parties in Scotland. The point here is that greater democratic engagement with issues of social concern and greater devolution are starting to open up avenues of innovation.

London, with all its wealth, power, poverty and inequality, is silent in such discussions. This in itself suggests a weakness of public debate. Some might say that to compare London – a city – with a nation such as Scotland is unfair. Scotland has greater sway over its resources, services, and the social support it can offer. London has a strategic authority rather than a national government and is the nation’s capital as well as a powerful city. This latter role creates a dual identity: London is a dominant site of British nationhood as well as a global city and powerhouse in its own right. Powers are tightly curtailed and dispersed in London – even fragmented – whereas Scotland’s institutions have far more authority and resources at their disposal.

All of this is true. But Scotland has taken the 1999 settlement as a starting point from which to endeavour to repatriate as much power and authority as possible. London has largely accepted its settlement, concentrating on transport, economic and physical development, and policing – with a push into areas such as adult skills, but little attempt to change the basic outlines of the original settlement. Peculiarly, this has led to Greater Manchester’s overtaking the London Mayor and GLA in its ability to influence social and economic policy in its area. Other cities, too, are superseding London in terms of the tools at their disposal.

These are not just constitutional niceties. The gap between London’s economic and social performance is a function of both the centralisation and fragmentation of power. To appreciate this, think of the complexity that would have to be managed and marshalled to innovate a new approach to supporting those out of the work and the low-paid – all the governance, the funding, and organisational incentives – and the costs involved in coordinating it. London’s political leaders have yet to find a convincing way of expressing, let along confronting the gap between London’s prosperity and its social challenges. This gap could be addressed through democratic and policy innovation, but it would require London to draw together power and resources in new ways.

The starting point has to be a clear expression of the fact that national policies are clearly not meeting London’s social needs. This is precisely the clarity of message we are getting from other major cities in the UK, as well as from its constituent nations. Second, London needs to consider distinctive approaches to meet the needs of its citizens. National welfare, skills, economic and housing policies do not meet the capital’s needs, yet London appears stuck with them (and there is no greater illustration of London’s particular needs than the likely negative impact of Brexit on a city that didn’t wish to see it happen).

Freedom to innovate is essential to modern cities, but London lacks the means to innovate in social policy in the way that it does (with excellent results) in tech, science, and business. And while the rest of the country may think that London gets enough of its own way, the reality is that social problems are soon exported out of the city, as the most needy look elsewhere for their needs to be met. London’s social challenges don’t remain within the boundary of the M25.

In the Netherlands, Scotland, Canada and Spain, cities are increasingly coming to the conclusion that national welfare, skills, jobs and tax policies are not working for them. The same deficiencies are evident in London, but its voice is muffled. Cities such as Utrecht, Barcelona and Glasgow, alongside the Canadian province of Ontario, are experimenting with new forms of individual and family support that are better adapted to the local economies with which those on low earnings or out of work have to contend. Some of these experiments and explorations seek to understand elements of the contribution that a UBI could make.

London’s labour market is at the high end of flexibility in the UK. It is dynamic, but low pay and skills are endemic. Faced with a flexible labour market, national welfare and income support systems such as Universal Credit can sometimes generate even greater insecurity by emphasising short-term paid work at all costs, making the situation for many even more insecure. Little wonder, then, that Londoners are more likely than the national average to be mentally ill; to sleep rough; to be out of work; to live in poverty despite being in work; to live in poor housing; and to have low life expectancy. Radical ideas for provision of a better level of security for people in an atmosphere of relentless insecurity – such as UBI – could place London on a different social trajectory without harming its economic dynamism. They are worth exploring, given that the current tax, welfare and work support systems do not adequately meet the city’s social needs.

It would be a travesty to accept London’s relentless social deficit as part of the capital’s condition, an externality of a dynamic, global city. Instead, a series of fresh approaches are needed. Rather than posing the question “What can we do with the powers we’ve got?”, there is a need for a re-framing: “What do we need to do, and then how can we get the powers we need?” The Mayor’s voice is important in this regard, as are those of the boroughs and London’s socially aware businesses and communities. London’s democracy should confront these bigger challenges. The shackles of low ambition and small-scale response can be released, but only with concerted political effort.

There is nothing about seeking to experiment with new ways of supporting London’s citizens that precludes engagement with equally important approaches to better, more secure and affordable housing, support of living wages, or the apprenticeship programme. In fact, creating more space in London’s democracy for bigger questions of social change would open up a spectrum of public discourse whose elements would not be mutually exclusive. Stronger housing policies, living wages, and a dynamic apprenticeship and skills programme all sit quite neatly alongside innovations such as UBI.

More time and energy should be devoted to creating a stronger London social agenda: until this is developed, the promise of London’s devolution will not be realised. London’s democratic voice has been muffled for too long – which is one of the reasons why the city has become so misunderstood by the rest of the country in a way that has in itself been harmful to London’s interests. Unless the capital is able to consider and experiment with ideas such as UBI, then the sizeable social deficit will remain. And that will make London a city much harder to live in than it needs to be.

I’m very grateful to Anthony Painter and to Centre for London, a friend of this website, for giving me their blessing to re-publish this piece. More on UBI from Anthony here and here.

On London is run by Dave Hill, formerly the Guardian's award-winning London commentator, and written by him and an array of fellow Londoncentrics. It aims to improve the quality of coverage of London politics, development and culture.