U.S. government's open data proves a treasure trove for savvy businesses

Cindy Waxer |
March 25, 2014

Hoping to capitalise on free government information, IT leaders are discovering the value -- and vexation -- of converting terabytes of data into new revenue streams.

"There's a big difference between free open data and actionable intelligence," says Alfred. "Companies don't want to purchase data that requires data analysts and infrastructure and integration. So there's a ton of work on both the manual side and the engineering side to get the data into a format that can be reliably used by our clients." That's a complex process that, he says, could be simplified with a bit of help from the government.

The good news is that the government is taking steps to improve data quality. Calcbench has benefited from that effort. The New York-based startup has created a sophisticated engine that turns complex financial data such as earnings statements, cash flow statements and companies' balance sheets into a more readable format. To ensure accuracy, Calcbench uses proprietary artificial intelligence tools that sift through the data to detect errors such as misdated year-end reports. Finance professionals such as investors, auditors and industry researchers use the resulting repackaged and cleansed data to compare financial ratios, examine entire industries and review competitors' disclosure data.

"A lot of what we're charging for isn't the data," says Alex Rapp, Calcbench's CTO and co-founder. "It's how we structure it, how we store it for you and how we solve a tremendous number of business problems by making data comparable across different companies and industries. That's something the data doesn't do itself."

A lot of what we're charging for isn't the data. It's how we structure it, how we store it for you and how we solve a tremendous number of business problems by making data comparable across different companies and industries. That's something the data doesn't do itself. Alex Rapp, CTO and co-founder, Calcbench

Nor is it a service that would have been possible without the U.S. government's insistence, starting in 2009, that the financial information in corporate SEC filings must be in XBRL (Extensible Business Reporting Language) — a freely available and global standard format for exchanging business information. Whether you're downloading quarterly filings directly into spreadsheets or analyzing year-end statements using off-the-shelf software, XBRL converts fragmented financial data into a machine-readable format. And that makes life easier for Calcbench's techies.

But as government agencies inch toward cleaner, more structured data sets, a handful of startups are offering powerful new tools for dealing with older data. "These intermediaries are going to make open data more accessible to other types of companies because the reality is, open data is still pretty messy at the local, state and federal level," says Gurin.

One of those startups is New York-based Enigma Technologies, which has created a platform that allows users to access open data via an API or a Web dashboard. The company uses Web crawlers that gather public data from sources such as dot-gov websites and FCC documents regardless of whether the information is stored in a Zip file, an Excel spreadsheet or a multigigabyte database. Next, the platform creates a relational infrastructure that centralizes all of the open data and creates links between matching data sets. A single interface offers users access to more than 100,000 sources of finely curated data. And because Enigma's solution is offered as an API, developers can borrow the technology to build dashboards that serve their industry-specific data retrieval needs.