Government Open Data: How To Get Involved

A workshop on open government data pathways by API Evangelist and current White House Innovation Fellow, Kin Lane, shows there is still plenty of potential for developers, startups and interested citizens to influence the government open data agenda and make use of government data assets. Lane ran the workshop as part of the pre-opening day's workshop series at API Strategy and Practice, being held in San Francisco for the rest of the week.

Lane points to several roadblocks that are creating barriers to the uptake of APIs in government agencies as a way to implement White House policy aimed at making all public data machine readable from the get go.

"In government, it's really hard to stand things up," Lane said, giving the example that if a government agency wants to start using an open source platform like CKAN to manage open data, it can cost in the vicinity of $300,000 and take 9 months to go live. "Open data and API management can get a bad name because it's the same old backend procurement process that is being used, it's not using the transparent models that we are used to with the cloud."

Lane's role within the White House has been to assist government agencies in releasing their data via API, i.e. in machine readable (JSON) format, and to coax agencies towards implementing an open data policy framework. The role has taken him "to both ends of the spectrum at once". On the one hand, he is explaining the basic tenets of an open data policy framework: answering questions about what is open data, data privacy, and security issues. On the other hand, he is needing to show what JSON file formats look like and what solutions an API can provide external users, when available.

One example Lane gave is turning website data on Veterans' Affairs facilities from an online search form format into a CSV file and then into JSON, and showing how the end result could then be used by community-based or private services to improve access and awareness of locally available resources supporting veterans with post-traumatic stress disorder, for example.

Along the way, Lane has had to make use of a range of data scraping and conversion tools to take existing public data from government agency websites and turn it into machine readable format, as well as create new open source tools. From there, it needs to be published to a central repository so that the JSON datasets are available.

The hope is that Lane's work will give rise to a profession of government data stewards that are responsible within their agencies to maintain the lifecylce of data: knowing where it is stored, knowing when it is updated, managing metadata details, converting into machine readable formats, managing uploads, and knowing when it is out-of-date.

Another stumbling block is the lack of analytics built-in to the release of open data and even open data APIs. As a result, while work is being done on the supply side to open up data sources, agencies are unclear of the demand and end use of the open data, including if, how, and how much the data is being used beyond the government's walls. "We need more good case studies available on how businesses and community are using open data and government APIs," said Lane.

Lane urges both developers and non-programmers to "adopt an agency" and contribute to the open data agenda. Individuals can use tools to convert agency data sources into machine readable format, work with local organizations to create solutions that would draw on government data supplies, or augment government data sources with local knowledge or independent sources to create even more useful datasets. Another need, in light of the recent US Federal Government shutdown, is to ensure continuous access to government data by setting up external caches of datasets so that political point scoring of-the-day doesn't grind access to (what should be) publicly owned data to a halt.

So far, Lane's work has been highly productive for the government open data agenda. "It has resulted in building a toolbox and now I'm starting to build up a curriculum around this," Lane told the workshop audience.

Anyone interested in contributing to the approach can access Lane's Github resource list and review the examples provided. "There's a lot of work that can be done outside of government: community members can convert datasets, and create simple APIs," said Lane.

About the author:Mark Boyd
is a ProgrammableWeb writer covering breaking news, API business strategies and models, open data, and smart cities.
I can be contacted via email, on Twitter, or on Google+.