Unstructured data

Unstructured data covers much of the data you will come across – from data buried in PDFs and web sites, to mining data from social networks, but it all requires analysis to extract meaning. There are many tools for getting at the data – see the previous section on scraping data for a range of tools – but the Sunlight Foundation’s Text Analysis in Transparency talk is a great introduction to that world of text analysis and natural language processing.

Extracting meaning from text

Once you have your data in a nicer format you may well need to tackle the problem of pulling something meaningful out of it. Fortunately, there are a lot of good analysis and natural language processing libraries around these days that will allow you to automatically find the meaningful keywords in a body of text.

Natural language processing may be a bit of a heavy topic to dive into during a hackathon, but if you’re feeling brave there are a few goodtutorials on the subject to get you started (if you’d like some more academic articles check this StackOverflow question).

As always, there are web-based tools – such as TextRazor and Yahoo Content Analysis – that may be able to save you the trouble of diving into code and learning too much about the theory and practice of NLP whilst time is tight.

There are a surprising number of good NLP libraries around for all of the major languages though:

Beyond the world of NLP you might consider going straight to a search engine that provides similar text interrogation capabilities along with a database to store your data and APIs to query it. Solr and Elastic (formerly ElasticSearch) are pretty well know in this space – but Sphinx and Constellio are worthy entries.

Visualising unstructured text

Being able to visualise unstructured information is key to making sense of it – be it a word tree of a text blob, a whole web page, or a social media feed – tools like Word Tree, Overview, and even Google Charts will help you turn out some quick visualisations. On the academic end of the spectrum the National Science Foundation have made their Jigsaw toolbox available.