Many cities have started to create open data websites to post their collected geospacial data. These sites usually allow bulk download of their datasets in standard machine-processable file formats, which is key. Some have visualizations of the data too, though this is a secondary feature to open data. These sites are either created by the city, or they use certain vendors to provide easy, out-of-the-box solutions for formatting, opening, and mapping their raw data.

UPDATE – Oct 28, 2013

Since creating this post a number of other organizations have started keeping a list of cities, counties, and states with open data portals. As such I’m not going to update this page any more. Please visit the following 3 pages to see a list and either edit their pages or connact them to have them add new ones.

New York City has followed in the footsteps of DC and San Francisco by opening some of their data for developers to create apps, websites, and services. But they’ve taken it a step further by not only showcasing the submissions, but also have public and private voting for cash prizes and dinner with the Mayor Michael Bloomberg. The result is the NYC BigApps contest.

The cash prizes are a terrific way to get some really cheap, high-quality development and ideas that will remain free to the public for a year after the contest ends. The 85 submissions are very diverse, ranging from iPhone and Android apps, full websites, GIS and Google API maps, and parts of larger data mapping platforms. They make great use of the data available from the NYC DataMine, and have generated tremendous press for the city and the mayor.

Why It’s Great

Open Data – Opening up some of the stores of public data that a city has locked up in private databases is always a good thing.

Publicity – Great press for the city, and it encourages other cities to do the same.

Cheap Developers – The city gets some great applications of their data out to the public quickly and cheaply, and reduces their FOIA requests.

Developer Press – The programmers that do something with the data get recognition and fame, putting them on the map.

Why It’s Not So Great

Limited Data – Only a fraction of the data has been made public. For example, there are no crime reports, the most commonly requested dataset.

Ballot Stuffing – For the public voting portion, the fact that many iPhone apps that haven’t even been made public in the App Store have 50+ votes is disturbing. Also, you can vote once per email address, so I’m sure there is a bit of voting fraud going on. Luckily the panel of judges is more important.

Incomplete Data – Some datasets have only been updated through August, or only contain a certain number of records.

Horrible Formats – Each dataset has its own distinct formatting, structure, and organizational logic, requiring developers to wade through arcane field names and meta data just to understand the data.

Of course, the final issue is the one that we know how to solve. If all the raw data could be presented in a format like the OMG Standard, then it would be easy to understand, human-readable, and able to be automatically machine processed into any standard MIME-type: rss, kml, csv, xml, html, etc. One of the biggest barriers of entry for this contest was making sense of the raw data for each dataset. Eliminate that barrier by provided data in one human and machine readable format, and you’ve just exponentially increased your number of submissions.

We congratulate the city of New York, the Challenge Post organizers, and the talented developers who are making this open data contest a success!

The New York Times has a great article today about how local governments are following data.gov’s lead and opening their data. DC, San Francisco, and now New York City have done this, then have ‘App Contests’ to garner great uses of the data from world-wide developers.

A good summary of the goal:

Many local governments are figuring out how to use the Internet to make government data more accessible. The goal is to spawn useful Web sites and mobile applications — and perhaps even have people think differently about their city and its government.

A discussion of the format. Of course we think the OMG Standard format is the best way to release the data.

By releasing data in easy-to-use formats, cities and states hope that people will create sites or applications that use it in ways City Hall never would have considered.

And there are some interesting tidbits about the issues and challenges with getting access to public data. Bold emphasis added.

Still, asking for the data is often not enough. Software developers in New York have been unsuccessful in getting data feeds of pedestrian and bicyclist injuries and fatalities from the Police Department, said Noel Hidalgo, who is director of technology innovation for the New York State Senate and has been working with developers on building city-data applications. He envisions applications that overlay accident information on city bike maps.

Paul J. Browne, a deputy commissioner of the New York City Police Department, said it releases information about individual accidents to journalists and others who request it, but would not provide software developers with a regularly updated feed. “We provide public information, not data flow for entrepreneurs,” he said.

There have been other scuffles over who has the right to data. Routesy, an iPhone application, uses data from San Francisco’s Municipal Transportation Agency to show train and bus schedules and locate stations on a map. It stopped working for a while because a private contractor working with the agency wanted to charge a licensing fee for the information. The agency now requires its contractor, NextBus, to make the data freely available.

There is evidence that governments’ attitude toward publicizing data is changing. Two years ago, when a Web design and research firm called Stamen Design started a Web site, Crimespotting, that mapped crime data for Oakland, Calif., the city cut off access to the data a week after the site went up.

Bob Glaze, the city’s chief technology officer, said the frequent data requests from the site were disrupting the city’s own crime site. The city eventually changed its mind. And in August, Stamen’s designers unveiled a San Francisco version of Crimespotting with Mayor Newsom at their side.

These open data contest are a good thing for everyone: the city releasing the data, the developers using the data, and the citizens who get access to new information. Let’s keep it up.