Extending Open Data Beyond Governments

I help business understand everything from big data to cloud computing

In Governments and related agencies around the world, the idea that non-sensitive data should be freely available for use and re-use has clearly caught on. Many of the benefits apply just as much to data collected by and for private corporations, but boardrooms are proceeding far more cautiously than parliaments.

According to the 2nd edition of the Open Data Barometer, published earlier this year, national governments are hard at work opening up large swathes of the data they hold. The UK, United States and Sweden hold the top three spots, as they did in the first survey back in 2013. Among the other 83 countries surveyed, big climbers include Indonesia (up 16 places to 36), China (up 15 places to 46), Peru (up 14 places to 33), and Ecuador (up 12 places to 38). Amongst the fallers, Kenya dropped 27 places to 49, Thailand dropped 26 places to 57, Botswana dropped 23 places to 78, Morocco dropped 15 places to 55, and Iceland dropped 14 places to 27.

The UK Parliament (Source: Flickr/ Michael D Beckwith)

In the UK, and elsewhere, attitudes have shifted. Instead of having to make a case for data (locations of bus stops, river levels in flood season, the nature of high-value government contracts, and more) to be released, the logic has flipped. Public sector data is now largely presumed to be open, unless a case can be made to keep it closed. That's quite a change, and it's driving increased transparency in Government and new innovations in the commercial sector. All those apps, for example, that tell you when your next train or bus is due? Powered by open data, released by a Government agency under a license that explicitly makes it available for reuse.

And yet, only a few short years ago, campaigners like Tom Steinberg (then) at MySociety, Rufus Pollock at the Open Knowledge Foundation and Charles Arthur (then) at The Guardian seemed engaged in a philosophical battle they wouldn't — couldn't — win. Transparency, the prevailing wisdom seemed to suggest, was all very well in theory, but checking, maintaining and supporting data would simply cost too much. The liability issues alone were too scary to contemplate. Government was acting in the best interests of its citizens, and so long as the right people (civil servants) had access, the rest of us should simply trust them to get on with it. In some areas, government departments began to be actively encouraged to sell specific data. The value of the data was — slowly — being recognized, but the ridiculously high price demanded for much of it acted to squeeze rather than nurture new markets.

And then things changed. Partly a response to public disgust at scandals like politicians' abuse of the expenses system, partly a recognition that corporate giants like Google were beginning to invest in alternative ways of gathering some of this data, partly a victory by all those open data campaigners, and partly a realization that the economic benefits could dramatically outstrip any revenue raised by limited data sales, the United States, the UK, and a small number of others began to change rules and policies to get data released.

Portals like data.gov and data.gov.uk were set up, making raw data available for download by anyone who wanted it. Licenses were developed, which made it clear that the data was intended for widespread use and reuse. Unsurprisingly, perhaps, millions of citizens did not flock to their government portals to pore tirelessly over the minutiae of government. So long as someone can, most of us don't need to. But apps, websites and whole businesses emerged that are only feasible because good, comprehensive, current data is being made available by governments. Potholes in the road are reported, transport timetables are consulted, and trends in crime, journey times or the fluctuating housing market are studied, interpreted, and acted upon. All powered by government data that anyone can now see.

In a recent survey, Open Data means business, the Open Data Institute in London discovered that 70% of the companies they asked use government open data. Rather more surprisingly, almost half (49%) use open data from a non-government source. OpenStreetMap accounts for a lot of this non-government use, as does data from government-ish bodies such as privatized rail and bus operators.

But real re-use of business data by other businesses lags far behind, and there are clear opportunities to explore and exploit the potential here. As the ODI report notes,

Developing your company’s capacity to take advantage of data, both at the analytic and at the strategic level, is vital. To make the most of the potential that data has to offer, your employees need to be able to work with data, understand which data to trust, and incorporate data publication and use into your business model.

Government already has a complex set of issues to consider, ensuring that personally identifiable or sensitive information is protected. For commercial organizations, where it's currently too easy to cite commercial sensitivity (or customer privacy) as reasons to do nothing, it may be time to take a closer look at the lessons from government. Consume government data, for sure. But maybe there's a business opportunity in contributing a little more, too.

I am an analyst and consultant at the Cloud of Data, based in the UK and working with clients around the world. I work with both vendors and buyers of technology, exploring the potential for new solutions and highlighting their impact inside existing businesses and existing ...