Free Our Data: the blog

Archive for April, 2008

The Green Party has been doing some sterling work trying to get the government to answer questions arising from the Cambridge economics study into the potential benefits of making data sets available for free.

We draw your attention to the exchange in the Lords (where the Greens have a representative, Lord Beaumont of Whitley) over this. Though you may find the answers uninspiring, at first:

Lord Beaumont of Whitley asked Her Majesty’s Government: Whether they intend to make the Ordnance Survey’s MasterMap available free of financial or legal restrictions. [HL2714]

The Parliamentary Under-Secretary of State, Department for Communities and Local Government (Baroness Andrews): As announced in the Budget, the Government will look closely at public sector information held by trading funds including Ordnance Survey, to distinguish more clearly what is required by government for public tasks and ensure that this information is made available as widely as possible for use in downstream markets. In the lead up to the next spending review, the Government will ensure that information collected for public purposes is priced so that the need for access is balanced with ensuring that customers pay a fair contribution to the cost of collecting this information in the long term. In the mean time Ordnance Survey will continue to generate the revenue it requires to cover its costs, to fund investments and to provide a return to government, from sales of paper mapping and from licensing use of the Crown copyright and Crown database rights in its data, including OS MasterMap.

This is the same “customers pay a fair contribution” line we’ve been hearing since the report came out (first used by DBERR, as we recall).

Of course the key to really good questions in Parliament is to ask something that the government can’t disagree with, but which isn’t part of its policy at present – because that puts it into the logical bind in which it should take up that policy. The case of economic advice given to the government is the classic one. (But you hear it all the time at Prime Minister’s Questions – the ne plus ultra of this game – in which opposing MPs try to expose gaps like this.)

Whether they intend to make unrefined information held by trading funds available free of legal and financial restrictions, as recommended in the recently commissioned study, Models of Public Sector Information Provision via Trading Funds; and [HL2713]

Why certain financial information in the Ordnance Survey and United Kingdom Hydrographic Office sections were redacted as confidential in the recent Models of Public Sector Information Provision via Trading Funds report; and [HL2715]

Whether they intend to make the Met Office’s unrefined information available free of legal and financial restrictions, following the finding of the recently commissioned study Models of Public Sector Information Provision via Trading Funds (p76) that this would provide a net benefit to society of £1.03 million; and [HL2733]

Whether in light of the finding of the recently commissioned study Models of Public Sector Information Provision via Trading Funds that there is a net benefit to society if trading funds release unrefined information at marginal cost, they will review the financial and legal restrictions on all unrefined information held by public bodies. [HL2734]

Whether they will define the “public tasks” for trading funds in order to identify information that should be released free of financial and legal restrictions, as highlighted in the recently commissioned study Models of Public Sector Information Provision via Trading Funds (chapter 3, paragraph 3.49). [HL2735]

Lord Davies of Oldham:

As announced in the Budget, the Government will look closely at public sector information held by trading funds to distinguish more clearly what is required by Government for public tasks and ensure that this information is made available as widely as possible for use in downstream markets. In the lead-up to the next spending review the Government will ensure that information collected for public purposes is priced so that the need for access is balanced with ensuring that customers pay a fair contribution to the cost of collecting this information in the long term.

For central government bodies other than trading funds, the clear policy is that raw information should, subject to any statutory provision, be freely available or provided at the marginal cost of dissemination.

In drafting the report Models of Public Sector Information Provision via Trading Funds, Cambridge University relied on the co-operation of and provision of data from trading funds. Some of the financial data provided was commercially confidential and was therefore not published in the report. This did not alter the overall conclusions of the report.

Tom Chance, a coordinator for the Green Party (who helped draft the questions, notes on his blog that the commitment to make information freely available “subject to any statutory provision” is encouraging:

That’s good to know, and backs up the Green Party’s case for making it accessible as well, e.g. Parliamentary procedure in an open, machine-readable format rather than plain HTML, or key data on domestic energy use in one place as a canonical source rather than being scattered across different sections of government departments (Defra, BERR, CLG, etc.)

One other thing he remarks on (about the Guardian’s mapping):

It would be nice if we could supply those guys with a decent set of OpenStreetMap graphics for use in articles rather than using non-free sources too!

It’s actually an avenue we’re exploring – I’ve swapped emails with Steve Coast, who says that a change in the licensing for OSM may make this possible in the near future. It would certainly cut some of our mapping costs.

The question has some urgency because if you can think of what you’d like to do with data from the Land Registry, Companies House or the Met Office, then you could be in line to be the first to benefit from it – and show the benefits of making more data free.

The Cambridge report noted three sets of data that could be made free with minimal revenue impact: Land Registry, Companies House, and Met Office.

Let’s revisit them, so you can think what to do with them.

For Land Registry, the analysis only looks at “Property Data Services” – which are the ‘Property Price Data’ and ‘Polygons’, whose respective revenues were £893k, the majority of which was from a bulk form of the product, and £405k. (That compares to Land Registry’s total revenues of XXX, 86% of which comes from compulsory registrations.)

Those, it should be noted, are tiny compared to its revenues. Land Registry’s fee income in 2006/7 was £474million (in 2005/6, £395m). Its costs are very high, but it still had an operating surplus of £96m – nearly as large as Ordnance Survey’s entire revenues.

The reason: you’re obliged to tell LR when you buy or sell land or put a charge (such as a mortgage) on registered land.

The property data could surely be used for some imaginative analysis – though note that Land Registry bans the use of its data for unsolicited mailshots. (An interesting question is how, if one moved to a free data model, one would spot uses which broke rules like that. Would you drop the rule, or include intentionally fake data which would tip you off if you received a mailshot addresses to it?)

Companies House is next: £72m revenue, again almost all from obligatory registrations. (Although search is the most profitable area – as you’d expect: it’s easier to search data than to accept and check it.) Unfortunately most of the data there is marked confidential in the analysis – but again, one can imagine that it might be useful to find people who are persistent directors of companies that aren’t acting lawfully…

Finally there’s the Met Office. Can anyone think what you’d do with a lot of weather data?

More analysis and suggestions of how to use these three organisations’ output data – if it were free – are welcome.

Today’s Guardian notes the arrival of the NLPG (national land and property gazetteer) in a commercial form, created by local authorities and Intelligent Addressing.

Yours, for

between £15,000 and £20,000 a year. Profits will be shared among local authorities to help them keep data up to date.

So let’s delve briefly:

The gazetteer is not the only address database on the market. The state-owned Ordnance Survey also offers addresses as part of its MasterMap digital geographical database of Britain.

Depending where you stand, this is either healthy competition or a wasteful duplication of effort. In recent years, a tortuous dispute over the licensing of intellectual property in state-generated address databases has exposed some of the damaging consequences of public agencies trying to compete with each other in the information business. Last year, the dispute exposed a hole at the heart of the government’s information strategy when the Advisory Panel for Public Sector Information said it was unable to rule on the matter.

So what do we advise?

Technology Guardian’s Free Our Data campaign urges a simple solution – that a taxpayer (or otherwise centrally funded) basic database of addresses be made available to all comers, for free. Despite some advances in the campaign, including the support of the Cabinet Office minister responsible for government IT, we have a long way to go.

the Minister for Ordnance Survey responsible for the shareholder relationship between the Department and the agency, dealing with strategic and day-to-day issues arising in connection with its activities, particularly in terms of financial and Government matters. My ministerial colleague the noble Baroness Andrews leads for the Department on issues relating to the purchase of Ordnance Survey products and services.

there is continuing confusion between [OS’s] public duty and the private competition that it has to have as a trading fund. The pan-government agreement, which regulates how different Government Departments and agencies use Ordnance Survey, came to an end yesterday. We have no news of what is going to be put in its place, so will he tell us? When will the regulatory framework be updated and amended to bring an end to all this confusion, which is getting in the way of Ordnance Survey’s excellent work?

The reply:

In respect of his important point about the pan-government agreement, that was established, as he is aware, to ensure that the Government have access to mapping data in order to develop and implement policy at a reasonable price. We are looking into that, and I will update the House accordingly.

There is an argument that [OS] information should be made more freely available, free of charge. Has he read the book which was published alongside the Budget, “Models of Public Sector Information via Trading Funds”—quite a racy read—and which rebuts the claim that a move to free data would damage the work of Ordnance Survey? It should be made freely available to citizens of this country, and that can be done in a way that produces funds rather than absorbs them.

As a fellow accountant, I can imagine that I would find it racy as well. [Ah, Parliamentary humour – CA] My hon. Friend raises an important point about the provision of data. He said that Ordnance Survey breaks even as a trading fund. In fact, it provides about £6.2 million in surplus that is then passed back to the public purse via dividends. That is to be encouraged. The business model, with changing market conditions and technology, is being considered and, as Minister with responsibility for Ordnance Survey, I will continue to do so. [Emphasis added – CA]

Interesting: that the Cambridge report is now getting debate time, that the free data model is being considered, and that the minister responsible is considering the business model “with changing market conditions”.

Since we’re playing with hypothecated taxes – which we should point out is not a central tenet of the Free Our Data campaign; we think governmnent should fund this centrally, but recognise that in tight economic times it’s hard to persuade government to spend more rather than doing revenue-neutral “experiments” – let’s examine the suggestion (made elsewhere) that OS’s free data should be funded by a surcharge on planning applications.

After all, the logic goes, planning applications are voluntary; and they require OS detail. (One has to submit a number of copies of OS maps of the area the application applies to.)

[A disclosure up front: I have a planning application in process. However, the following simply emerges directly from the available statistics.]

Let’s recall the amount that needs to be raised, according to the Cambridge study, to compensate OS for the loss of revenues from selling its “raw” or “public task” data: £12m – £30m.

Now, how many planning applications were there for the UK? For England and Wales, the total applications received looks like this:

This is only for England and Wales, of course, but shows that planning applications, even at their busiest, are an order of magnitude less than Land Registry transactions. True, this excludes Scotland, but even if that has as many planning applications as England and Wales (which seems unlikely), that would only be a total of about 1.3m applications in the busiest time.

That in turn would imply a surcharge per application received of roughly £10 to £30. Whether that’s small or large compared to the cost of the overall transaction (which might be putting up an extension on a house or a change in use, requiring no actual building) isn’t clear.

(Locally, making an application costs between £135 – £265. It may vary in other locations. The surcharge would also have to be collected from multiple councils, which would impose an administrative overhead; by comparison, there’s only one Land Registry.)

But as an actual amount per transaction, it’s comparatively large, meaning it would be very sensitive to variations in the number of applications. The figures there vary from 501,000 to 689,000 – a 28% or 37% variation, depending which you take as your numerator.

Why might this be? Perhaps Land Registry transactions include commercial purchases, which occur more frequently. And also that people buy and sell houses more frequently than they do things to them. Either way, it’s doesn’t look like the optimum way forward. It’s certainly a lot more expensive – per transaction – than a Land Reggistry surcharge.