(I recently took the difficult decision to resign from IBM after nearly 20 years to become IT Director for Smart Data and Technology for Amey, one of the largest infrastructure and services companies in the UK, and a subsidiary of the Ferrovial Group. It’s a really exciting opportunity for me to build a team to create new Smart City services and infrastructures. If you’d like to work in the Smart Cities field, please have a look at the roles I’m hiring for. I’ll be continuing to write the Urban Technologist, and this seemed a good point to share my view of the current state of the Smart Cities movement.)

The last year has shown a huge acceleration of interest and action in the Smart Cities market – in the UK, and around the world. What has long been a topic of interest to technology companies, academics, urban designers and local authorities was covered extensively by mainstream media organisation such as the BBC, the Independent newspaper, New Statesman magazine and marketing magazine The Drum.

But what progress has been made implementing Smart Cities ideas?

In the UK, many local authorities have implemented Open Data portals, usually using Open Source platforms such as CKAN and investing a few £10,000s of resources. These are important first steps for building the ecosystems to share and build new service models using data. Some cities, notably Glasgow and Milton Keynes, have been successful deploying more sophisticated schemes supported by research and innovation grants – though as I pointed out last year, exciting as these initiatives are, research and innovation funds will not scale to support every city in the country.

Further afield, local authorities in Europe, the United States and Asia have constructed more substantial, multi-million Euro / Dollar business cases to invest their own funds in platforms that combine static open data with realtime data from sensors and infrastructure, and which use social media and smartphones to improve engagement between citizens, communities, businesses and both public- and private-sector service providers. The Center for Data Innovation recently wrote a nice summary of two reports explaining the financing vehicles that these cities are using.

This has not happened in the UK yet to the same extent. The highly centralised nature of public sector spending means that cities here have not yet been able to construct such ambitious business cases – Centre for Cities’ report “Outlook for Cities 2014” highlighted this as a general barrier to the UK’s cities carrying out initiatives to improve themselves, and reported that UK cities have autonomy over only about 17% of their funding as compared to an average of 55% across countries represented by the OECD.

(The London Underground is just one example of a transport operator using technology to help it operate more efficiently, safely and effectively)

Where similar technology platforms and channels of engagement are nevertheless starting to appear in the UK is through business cases based on efficiencies and increased customer satisfaction for private sector organisations that offer services such as transportation and asset management to cities, citizens and local authorities.

This approach means there’s even more of a need for collaboration between stakeholders in local ecosystems in order to establish and express common objectives – such as resilience, economic growth and social mobility – which can then guide the outcomes of those smart services through policy tools such as procurement practises and planning frameworks. Recent recommendations from the British Standards Institute on the adaptation of city planning policy to enable the Smart City agenda have highlighted the need for such collaboration.

As a consequence of this increased activity, more and more people and organisations of every type are becoming interested in Smart Cities – from oil companies to car manufacturers to politicians. This broadening of interest led to some extraordinary personal experiences for me last year, which included discussing Smart Cities with ex-US Vice President Al Gore (whose investment company Generation IM explores opportunities to invest in assets, technologies and developments that promote sustainability) and very briefly with the UK’s Princess Anne, a supporter of a leadership training scheme that will focus on Smart Cities this year.

But to be honest, I still don’t think we have really understood what a “Smart City” is; why it’s one of the most important concepts of our time; or how we can turn the concept into reality broadly and at scale.

I’ll explore six “inconvenient truths” in this article to describe why I think that’s the case; and what we can do about it:

1. The “Smart City” isn’t a technology concept; it’s the political challenge of adapting one of the most powerful economic and social forces of our time to the needs of the places where most of us live and work

(Photograph of Macau in the evening by Michael Jenkin illustrating some the great complexity of cities: economic growth, social inequality and pollution)

One topic that’s endlessly revisited as more and more people encounter and consider the idea of a Smart City is just how we define that idea. The best definition I thought I had developed is this, updated slightly from the article “7 Steps to a Smarter City“:

A Smart City systematically creates and encourages innovations in city systems that are enabled by technology; that change the relationships between the creation of economic and social value and the consumption of resources; and that contribute to achieving a vision and clear objectives that are supported by a broad and active collaboration amongst city stakeholders.

But such definitions are contentious. Most obviously there’s the basic issue of whether “smart” implies a central role for digital technology – every technology company takes this approach, of course – or whether it’s simply about being more creative in the way that we manipulate the resources around us to achieve the outcomes we desire, whether that involves digital technology or not.

More broadly, a “city” is such a terrifically broad, complex and multi-disciplinary entity – and one whose behaviour is the aggregate of the millions of individual behaviours of its inhabitants, both enabled and constrained by the environment they experience – that it’s pretty much impossible to create any concise definition without missing out something important.

And of course those who live or work in towns and rural areas raise the challenge that limiting the discussion to “cities” omits important stakeholders from discussions about our future – as do those concerned with the national infrastructures that are not located wholly in cities, but without which neither cities nor any other habitations could survive as they do today.

That technology may be an important tool for them to use in doing so, having reached the point where it has exploded out of the back office functions of organisations into the consumer world, the information economy and the “internet of things“, and created the opportunity to innovate in every aspect of our lives.

In this context, it’s less useful to attempt to precisely define the concept of a smart city, and more important to encourage and enable each of us – every community, city, government and organisation – to develop our own understanding of the changes needed to overcome the challenges and take the opportunities before us, and of the rapidly evolving role of technology in doing so.

Why is it so important that we do that?

In their report “Cities Outlook 1901“, Centre for Cities explored the previous century of urban development in the UK, examining why at various times some cities thrived and some did not. They concluded that the single most important influence on the success of cities was their ability to provide their citizens with the right skills and opportunities to find employment, as the skills required in the economy changed as technology evolved.

The challenges faced by cities and their residents in this century will be unlike any we have faced before; and technology is changing more quickly, and becoming more powerful, than it ever has before. Creating “Smart Cities” involves taking the right political, economic, social and engineering approaches to meeting those challenges.

Cities that do so will be successful. Cities that don’t, won’t be. That is the digital divide of the 21st Century, and for everyone’s sake, I hope we are all on the right side of it.

Let me tell a short tale of two cities and their Smart transformations.

For a long time I’ve written occasional articles on this blog about Sunderland, a city whose leaders, people and social entrepreneurs have inspired me. Sunderland is one of the very few cities in the UK who have spent significant sums of their own money on Smart City projects and supporting technologies, justified by well-constructed business cases. They have publicised investments of well over £10 million, most recently including their visionary “City Intelligence Hub” initiative.

It is no coincidence at all that one of the cities that has been boldest in investing in technology to support its economic, social and environmental objectives has a Chief Executive who shows belief, leadership and engagement in the ideas of Smart Cities.

It is similarly no coincidence that – like Bristol, London and Glasgow, to name just three more – Milton Keynes Council have senior leadership figures – initially the then Chief Executive, Dave Hill, followed by Director of Strategy, Geoff Snelson – who regularly attend Smart Cities conferences and government bodies, and who actively convene Smart Cities collaborations. Their very visible presence demonstrates their belief in the importance of Smart City approaches to those organisations seeking to invest in developing them.

A strategy to transform the operations of a local authority (or any other organisation) using technology, and to re-invest the savings achieved by doing so into new services and initiatives that create economic growth, social mobility and resilience is not going to succeed without direct Executive leadership. Similarly, technology vendors, service providers and research funding bodies are most attracted to invest in developing new ideas and capabilities in cities whose most senior leaders are directly seeking them – they all need the outcomes of their investment to achieve real change, and it’s only through the leaders that such change will happen.

For the most part, where this level of leadership is not engaged I have not seen cities create business cases and issue procurements for Smart City solutions, and I have not seen them be successful winning research and innovation investments.

Finally, let’s be really clear about what most of those city leaders need to do: they need to follow Sunderland’s lead, not Milton Keynes’s.

The research and innovation funding from the EU and the UK that Milton Keynes has attracted will only fund projects that explore for the first time the capabilities of new, technology-enabled approaches to urban challenges. Those funding sources will not support the widespread deployment of successful approaches in cities around the UK and around the world.

The vast majority of cities will only benefit from Smart Cities initiatives by financing them through robust business cases based on a combination of financial efficiency and social, environmental or economic value – as Sunderland and some cities outside the UK are already doing.

Cities won’t get smart if their leaders aren’t involved in actively driving their institutions to adopt new business cases and operating models. Those that don’t risk leaving the fate of their cities not to chance; but to “the market”.

3. We can’t leave Smart Cities to the market, we need the courage to shape the market

(Photograph by Martin Deutsche of plans to redevelop Queen Elizabeth Park, site of the 2012 London Olympics. The London Legacy Development’s intention, in support of the Smart London Plan, is “for the Park to become one of the world’s leading digital environments, providing a unique opportunity to showcase how digital technology enhances urban living. The aim is to use the Park as a testing ground for the use of new digital technology in transport systems and energy services.”)

As I wrote in my last article on this blog, as the price of digital technologies such as smartphones, sensors, analytics, open source software and cloud platforms reduces rapidly, market dynamics will drive their aggressive adoption to make construction, infrastructure and city services more efficient, and hence make their providers more competitive.

But those market dynamics do not guarantee that we will get everything we want for the future of our cities: efficiency and resilience are not the same as health, happiness and opportunity for every citizen.

In far too many discussions of Smart Cities I hear the argument that we can’t invest in these ideas because we lack the “normalised evidence base” that proves their benefits. I think that’s the wrong view. There are more than enough qualitative examples and stories that demonstrate that these ideas have real value and can make lives better. If we insist on moving no further until there’s a deeper, broader corpus of quantified evidence, then there’ll be no projects to deliver the evidence – a chicken and egg problem.

Writing in “The Plundered Planet”, the economist Paul Collier asserts that any proposed infrastructure of reasonable novelty and significant scale is effectively so unique – especially when considered in its geographic, political, social and economic context – that an accurate cost/benefit case simply cannot be constructed in advance based on comparable prior examples, because those examples don’t – and never will – exist.

“Private investment shapes cities, but social ideas (and laws) shape private investment. First comes the image of what we want, then the machinery is adapted to turn out that image. The financial machinery has been adjusted to create anti-city images because, and only because, we as a society thought this would be good for us. If and when we think that lively, diversified city, capable of continual, close- grained improvement and change, is desirable, then we will adjust the financial machinery to get that.”

The “anti-city images” Jacobs was referring to were the vast urban highways built over the last half century to enable the levels of road traffic thought to be vital to economic growth. Since Jacobs’ time, a growing chorus of urbanists from Bogota’s ex-Mayor Enrique Penalosa to town planner Jeff Speck, architect Jan Gehl and London’s current Mayor Boris Johnson has criticised those infrastructures for the great harm they cause to human life – they create noise, pollution, a physical barrier to walking through our cities, and too often they injure or kill us.

Fifty years after she wrote, we should follow Dan’s example and take Jane Jacobs’ advice.

4. Smart cities aren’t top down or bottom up. They’re both.

(The SMS for Life project uses the cheap and widely used SMS infrastructure – very much the product of “top-down” investment – to create a dynamic, collaborative supply chain for medicines between pharmacies in Africa – a “bottom-up” innovation. Photo by Novartis AG)

It’s perfectly obvious that we need both: the “bottom up” creativity through which everyone seeks to create a better life for themselves, their family, their business and their community from the resources available to them; and the top-down policies and planning that – when they work best – seek to distribute resources fairly so that everyone has the opportunity to innovate successfully.

It’s only by creating harmony between these two approaches that we will shape the market to create the cities we want and need.

Over the last few years I’ve been inspired by extraordinary thinkers from many disciplines who have tackled the need for this balance. Some of them are creating new ideas now; others created amazing ideas years or decades ago that are nevertheless imperative today. All of them are worth reading and learning from:

The economist E F Schumacher, who identified that investment in the distribution and accessibility of “appropriate technologies” was the best way to stimulate and support development in a way that gave rise to the broadest possible opportunities for people to be successful.

Andrew Zolli, head of the philanthropic PopTech foundation, who describes the inspiring innovators who synthesise top-down and bottom-up approaches to achieve phenomenal societal changes as “translational leaders” – people with the ability to engage with both small-scale, informal innovation in communities and large-scale, formal institutions with resources.

Jan Gehl who inspired the “human scale cities” movement by relating the scale of city structures – from pavements to housing blocks to skyscrapers – to the human senses, and the nature of our lives and movement.

And, of course, Jane Jacobs, whose book “The Death and Life of Great American Cities” was the first written in the context of modern society and cities to point out that cities, however vast their physical size and population, can only ever be understood by considering the banal minutiae of the daily lives of ordinary people like you and I – why we walk along this street or that; how well we know our neighbours; how far it is to walk to the nearest school, shop or park; and whether we and our families feel happy and safe.

5. We need to tell honest stories

(Photograph by Meshed Media of Birmingham’s Social Media Cafe, where individuals from every part of the city who have connected online meet face-to-face to discuss their shared interest in social media.)

Any “smart city” initiative that successfully uses digital technology to create a financially sustainable social, economic or environmental improvement, in a particular physical place and on behalf of a particular community, must draw together skills from a wide variety of disciplines such as architecture, economics, social science, psychology and technology. Experts from these disciplines use a vast and confusing array of language and terminology; and all of us are frequently guilty of focussing on the concerns of our discipline, rather than communicating the benefits of our work in plain language.

The leaders of city institutions and businesses, who we are asking to take the courageous and forward-looking decisions to invest in our ideas, are understandably not familiar with this torrent of technical terminology, which can easily appear to be (and too often is) jargon; and new ideas that appear to be presented in jargon are unlikely to be trusted.

Another Birmingham initiative, the Northfield Ecocentre, crowdfunded £10,000 to support their “Urban Harvest” project. The funds helped the Ecocentre pick unwanted fruit from trees in domestic gardens in Birmingham and distribute it between volunteers, children’s centres, food bank customers and organisations promoting healthy eating; and to make some of it into jams, pickles and chutneys to raise money so that in future years the initiative can become self-sustaining.

In the village of Chale on the Isle of Wight, a community not served by the national gas power network and with significant levels of fuel poverty, my IBM colleague Andy Stanford-Clark has helped an initiative not only to deploy solar panels and smart meters to generate energy and measure its use by each household; but to co-design with residents how they will use that technology, so that the whole community feels a sense of ownership and inclusion in the initiative. The project has resulted in a significant drop in rent arrears as residents use the technology to reduce their utility bills, in some cases by up to 50 percent. Less obviously, the sense of shared purpose has extended to the creation of a communal allotment area in the village and a successful campaign to halve bus fares to nearby towns.

None of these initiatives are purely to do with technology. But they all use technologies that simply were not available and accessible as recently as a few years ago to achieve outcomes that are important to cities and communities. By understanding how the potential of technology was apparent to the stakeholders in such initiatives, why it was affordable and accessible to them, and how they acquired the skills to exploit it, we can learn how to design Smart Cities in a way that encourages widespread grass-roots, localised innovation.

No single person or organisation can shape the Smart Cities market so that it delivers the cities that we need. Local governments have the ethics of civic duty and care but lack the expertise in financing and business model innovation to convert existing spending schemes into the outcomes they desire. Private sector corporations as institutions are literally amoral and strongly incentivised by the financial markets to maximise profits. Many social enterprises are enormously admirable attempts to fuse these two models, but often lack the resources and ability to scale.

Most of us won’t ever be in a position to determine the reporting model or approach to recycling of corporations as large as Amey or IBM. But all of us make choices every day about the products we buy, the organisations we work for, the politicians we vote for, the blog articles we read, share and write and the activities we prioritise our resources on.

Those choices have real effects, and digital technology gives us all the opportunity for our choices to have more impact than ever before. This blog, which costs me nothing to operate other than the time it takes me to write articles, now reaches thousands of readers in over 150 counties. Air BnB took 2 years to accumulate the same number of rentable rooms that it took the Hilton Hotel chain 50 years to build.

It has never been easier to express an opinion widely or create a new way of doing things. That’s exactly what Shaleen Meelu did when she started the Harborne Food School to promote healthier, more sustainable approaches to food, with the support of Birmingham’s Smart City community. It’s an opportunity all of us should seize; and it’s absolutely the best opportunity we have to create better cities and a better world for ourselves.