Secretary of State for Communities & Local Government Eric Pickles followed this up with a letter to councils saying, “I don’t expect everyone to do it right first time, but I do expect everyone to do it.” Great. Raw Data Now, in the words of Tim-Berners Lee.

Now, however, with barely the ink dry, the reality is looking not just a bit messy, a bit of a first attempt (which would be fine and understandable given the timescale), but Not Open At All.

As a member of the Local Public Data Panel, I’ve worked with other members and councils to draw up some clear and pragmatic draft guidelines for publishing the local spending data. We’ve had a great response in the comments and in conversations, and together with some lessons I did on importing the existing data, I think these will allow us to do a second draft soon.

One thing we weren’t explicit in that first draft – because we took it for granted – was that the data had to be open, and free for reuse by all. Equality of access by all is essential.

So I’ve been watching the activities of Spikes Cavell’s SpotlightOnSpend with some wariness and now those fears seem to have been borne out, as the company seems to set out not to consume the open data that councils are publishing, but to control this data.

The idea seems to be that councils should give Spikes Cavell privileged access to their detailed invoice information, which the company then adds to their proprietry and definitely non-open database, and then publishes an extract of this information on the SpotlightOnSpend website. Exactly what information they get, and under what terms isn’t disclosed anywhere.

The website’s got most of the buzzwords: transparency, accessible, efficiency. It’s even got a friendly .org.uk domain. If that’s not enough to convince councils, liberally sprinkled around the site is an apparent endorsement from the Secretary of State himself:

I’m really excited about the opportunities of transparency and it’s something this government is utterly committed to. spotlightonspend demonstrates that, when innovative businesses work with far-sighted public bodies, we can inform the public, reduce costs and improve democracy both locally and nationally.

Eric Pickles

Secretary of State

Communities and Local Government

However, when you go to the data and click on the download link this is what you get:

Note the “This data is for your personal use only” (not to mention the fact that the use of a captcha’ to screen out machines downloading the data means, er, you can’t use machines to automatically download the data, which is sort of the point of publishing the data in a machine-readable way).

Never mind, surely you can just head over to the council’s website and download the data from there? No chance. This is what you get on the Guildford website:

You can search and view this financial data using a new Spotlight on Spend national website. Just follow the link found in the offsite links section of this page.

What about Mole Valley Council:

This data is now available on the spotlight on spend website. You can look at categories and individual suppliers to see how much has been spent in each area or you can download all the data to see individual transactions.

But what about Windsor & Maidenhead, who are closely affiliated with the project, and who are publishing data on their website? Well, download the data from SpotlightOnSpend and it’s rather different from the published data. Different in that it is missing core data that is in W&M published data (e.g. categories), and that includes data that isn’t in the published data (e.g. data from 2008).

So the upshot seems to be this, councils hand over all their valuable financial data to a company which aggregates for its own purposes, and, er, doesn’t open up the data, shooting down all those goals of mashing up the data, using the community to analyse and undermining much of the good work that’s been done.

It’s worth linking here to the Open Knowledge Foundation’s draft guidelines on reporting of Government Finances (disclosure: I helped draw them up), of which the first point is ‘Make data openly available using an explicit license’. And let me be absolutely clear here: this is not open data, not a desirable approach, will not achieve the results of transparency or of equality of access, and is not good for the public sector.

I’m hoping this is a matter of councils and the Secretary of State not understanding the process and implications of giving this data to Spike Cavell on a privileged basis. If not, perhaps it could be the first test case for the newly setup of Public Sector Transparency Board to rule on.

6 thoughts on “The open spending data that isn’t”

Although some Councils have already started to publish this information, there is obviously a reticense to develop costly reporting solutions. Councils therefore appear to publish the information in downloadable format either as Excel spreadsheets or PDF documents. Although this may ‘tick’ the required disclosure requirements, we do not believe that this provides real value to the public.

Information such as this only has real value when viewed in context (i.e. is a payment normal or abnormal). For a user to be able to make this interpretation, they may need to be able to look at spend in a month for a particular expense category against similar spend in previous periods or against other categories.

BIOLAP has developed an application, driven by arcplan technology, that we will provide to Councils free of charge to allow members of the public to analyse expenditure, slice and dice information and drill through to the underlying transactions.

Councils whose expenses are already included are:

Blaby District Council
Corby Borough Council
Greater London Authority
Guildford Borough Council
King’s Lynn & West Norfolk
Richmond upon Thames
Mole Valey District Council
Reigate & Banstead
Windsor & Maidenhead
South Oxfordshire District Council
Spelthorne Borough Council
Vale of White Horse District Council
Waverley Borough Council
Woking Borough Council

Couldn’t actually get it to produce reports on my Mac, so wasn’t able to see the functionality, but from the screenshots there appears to be some filtering by category — is this just using the text provided with the data (which isn’t comparable between councils), or are you doing extra analysis on it?

Thank you for your feedback. We now have more data on our app (15 councils and double the amount of transactions) compared to OpenlyLocal, but our aim is the same – to open up the data and to provide users with more choice to interact with the data.

Sadly, MacOS does not support Java anymore (even though we designed our dashboard on a Makbook Pro (using Win7)but we have the option to publish the application in .Net as well as DHTML formats. Before we do this though I wanted to see whether users like the basic design in the first instance. I hope therefore that you can view the dashboard on another platform and let me know what you think.

As for filtering – yes we have provided the ability to slice and dice, sort and filter as well as drill through to the underlying data.

At the moment we have simply loaded data already publicly available. Some Councils provided less analysis detail than others in which case we had to annotate ‘Unspecified’ data tags. The dashboard is context sensitive though and the filter options are dependent on the options available for a selected council to make the viewing experience as easy as possible for users.

Good to know. Are you sure about Mac OS & Java? I’ve got Java on my install (Snow Leopard), and the applet loads, just won’t produce reports. OpenlyLocal should prob have caught up by the weekend — been on holiday for a couple of weeks, which is why the most recent councils aren’t on there.