ScraperWiki implementation to extract data from PPP monthly pdf docs. For use by Morph.io.

PPP_Scraper

This is an implementation of the scraperwiki package. The app connects to the UN Department of Peacekeeping (DPKO) report on troop contributions for the previous month hosted on their website. The url pattern uses the pattern http://www.un.org/en/peacekeeping/contributors/<year>/<3 or 4 letter month abbr><last 2 numbers of year>_3.pdf. March 2014's file would be at http://www.un.org/en/peacekeeping/contributors/2014/mar14_3.pdf for instance.

The app then converts the pdf to xml and uses page position to parse into a sqllite db. Country names are positioned between 130 and 140 px left, mission designations are between 276 and 280 px left, and data is anything more than 350 px left. This app is built to run on the Morph.io platform which contains an API that is used by the PPP_Loader app. It is also possible to run scraperwiki locally, though I've had trouble getting it to work.

Morph.io runs this script directly from GitHub once daily and data is only inserted if it doesnt exist in db.