Law change put UK internet archive beyond reach of plebs

As experts pointed out after prime minister David Cameron's Internet Summit at Downing Street last Monday, you've got to be a fairly feeble child abuser to be typing taboo words into Google - especially after the NSA scandal.

Search engine giants Google and Microsoft had offered to filter paedo words from their search results. Some experts didn't credit them because horrid criminals do not usually advertise themselves to search engines in the first place. Stuff you can find easily on the internet is controlled by people who want you to see it.

One thing that got lost amidst the hullabaloo about Cameron's paedo Summit (and other moral matters stoked by his office last week) was his response to the revelation he had deleted (literally deleted) his old claim that the internet would make politicians like him more transparent and accountable; and that his party had taken steps to ensure Conservative election pledges became one of those things you couldn't find on the net so easily.

One reason Cameron's response got lost amidst the hullabaloo was because he neglected to make one. His office said it was a party political matter. It was not Number 10's business, said a spokeswoman.

Cameron had made the promise of transparent politics the primary characteristic of his opposition leadership in 2006. It defined his election campaign. He even repeated the pledge when fresh in office. If his government was not going to concern itself with carrying out that pledge, then who would?

Number 10 said the coalition government had done lots of things to fulfil Cameron's transparency pledge. It had published government spending data. It had published other government data. It had (like the last government) published speeches by sitting government ministers. It had done lots of things. These things were however all about government transparency.

Cameron always pitched that as only half the story. He promised to deliver the other half as well: political transparency.

Patter

Take for example the PM's "Podcast on a train", made on 29 May 2010, shortly after he became prime minister.

"It's not just about efficiency and saving money," said Cameron then.

"Transparency can help us to re-build trust in our politics.

"One of the reasons people don't trust politicians is because they think we've always got something to hide.

"Well by the time we've finished there will be far fewer hiding places," said the eager young PM.

He said he was planning a "transparency revolution".

It was the same patter he had before the election: political transparency and government transparency. That was what David Cameron stood for.

It wasn't just a passing fad. It wasn't just sales patter to steal power. He really meant it. (Didn't he?). There on that train, fresh from his election win, Cameron could have made his people's podcast about anything. He used it to utter those words that have been repeated often since he deleted all his transparency speeches last month - words that are almost never referenced because the reference is hard to find.

"I want our government to be one of the most open and transparent in the world," he said.

He went one step further even than that.

"Its going to make a big difference," he said.

"People will be the masters. Politicians the servants. And that's the way it should be."

There was some alarm when he first deleted his transparency speeches. It subsided too soon. The Conservative Party had not just deleted them from its website. It imposed a block that would have removed almost any trace of them from the net - by issuing a computer instruction to have them removed from the major internet repositories. It responded to the outrage by removing the block. The outrage subsided.

But even now, the speeches are not readily available elsewhere.

Not in the way Cameron, champion of the plebs, meant. Not in the way he meant when he talked about making politicians accountable to the people.

When Cameron used to talk about political transparency it was actually true. You could stick something like "Cameron transparency speech" into Google and the whole lot of them would come up in one proudly convenient batch of links to the Conservative Party website.

The Conservative website itself was astonishing. You could select a Conservative MP from a drop-down list and see any speech they had given in any year you chose. There, it was true. You could use the internet to hold Cameron to account like he said you could. There were no smarmy obstacles or small-print letdowns. Cameron stood for decency. Like English cricket. Like cucumber sandwiches, elderflower cordial and ice-cubes tinkling in late afternoon sunshine and jolly jolly tittering good humour.

But no, the speeches are not readily available anymore. Not in the way Cameron meant.

Tradition

The Internet Archive - the first stop for things people have deleted off the internet - cannot be searched in a way that really satisfies Cameron's pledge. You have to know what you are looking for. If you don't know the exact URL - and who does? - you have to know the website where the information you want used to be when it was available, and then you have to navigate laboriously through a calendar of copies the archive took before the speeches were deleted to get just the right spot where you can find a single speech. A lot of the navigation pages are simply inaccessible. Some of the content cannot be found. It's like driving to the commons across a mud flat in a Reliant Robin. This was not what Cameron meant when he said he would make politicians accountable to the people.

There are other places where Cameron's information revolution doesn't quite deliver either.

There's the Oxford University Bodleian Library, which published a misleading note about the deleting affair, in apparent defence of the prime minister, against the people.

"As disappointing as it may seem to many conspiracy theorists," it said, "these speeches are all safely in the care of the Conservative Party Archive (CPA) at the Bodleian Library in Oxford, and still freely available to anyone who wishes to consult them."

It then published one of the missing speeches in a series of 16 .jpg images, which stretched down the page in mocking parody of the accountability its words described. (.jpgs, really). If you saw this article and felt placated you were a fool.

If you scrolled - and scrolled, and scrolled - down beyond the .jpgs you would have read that Cameron's deleted speeches had indeed been donated to the Bodleian Library.

But you could only read them if you actually visited the library in Oxford. What are the chances you would get a day pass if you turned up in your mud-spattered Robin Reliant and said, "Excuse me, I'm a pleb. Please can I read the prime minister's speeches?".

The Bodleian assured readers of its blog that it had been working for two years on a system that would "soon" present the deleted speeches to people on the internet.

"Ultimately, when funds allow, the intention is to enable free-text searching of all the Party's vast number of speeches, online, and free-of-charge," it said.

This must be what the PM called, on launching his bid for government in 2006, the "democratisation of information". What are the chances the Bodleian will have it finished before the next election? And what are the chances it will be searchable by plebs using Google?

Take the British Library, for example - an institution your correspondent looks up to with a sort of plebeian awe (like Jude the Obscure). The British Library had taken copies of the Conservative speeches and put them on a curated collection of websites called the Web Archive. But it blocked access to that archive by search engines using a robots.txt exclusion, in the same way the Conservative Party obscured the prime minister's speeches from the internet after it deleted them in October.

The Library's exclusion would prevent anyone finding the prime minister's speeches unless they went specifically to its domain to look for them.

They would have to know where to look. The same is even true of the Internet Archive, which may have a permissive robots policy ("Please crawl our files," it says), and even has a boggy collection of the prime minister's speeches, but they still did not come up in Google and other search engines polled during ongoing research conducted for this story since September.

This was not the vision David Cameron conjured when he flattered Google on the campaign trail for its "truly amazing" democratization of information.

He didn't say: "You won't be able to find my speeches pledging transparent politics when you look them up in Google in a few years time. But you might find a few of them if you scratch around long enough. With any luck you won't have any time left to hold me to them because it was just a load of sales patter anyway."

The British Library said in a statement it had just finished its first crawl of the entire UK web domain after it got legal powers in April (under the Legal Deposit Libraries (Non-Print Works) Regulations 2013 Act). But plebs were again given seriously limited access to this information.

The 2013 regulations said the UK Web Archive would be available only to people who visited a "Legal Deposit" like the British Library in person and sat at one of its terminals. Any pleb with ideas above his station knows not just anyone can get a reader's pass to the British Library.

Likewise, the list of other Legal Deposit Libraries reads like a who's who of gated cloisters: the Bodleian Libraries of the University of Oxford, Cambridge University Library, The National Library of Scotland, The Library of Trinity College, Dublin, and The National Library of Wales.

Can I av wonerem reader pass fings?

Get this though.

Cameron's 2013 internet legislation said that not only would you have to visit some posh library to read the UK web archive, but it could only be viewed by one person at a time! If you were sat at terminal A in the Bodleian Library or whatever and you wanted to look up that other speech Cameron gave at a Google conference when he was crawling to be prime minister, and someone else was already reading it on terminal B, you would have to wait till they'd finished reading it first.

That other speech, by the way, was the one where Cameron said:

"Technological advance - supported by a liberal regulatory regime - has transformed the amount of information that's available, the number of people who can get hold of it, and the ease with which they can do so."

He did not say, "When I come to power I will make sure the internet's historic record is locked up in some inaccessible library vault where it will be as inaccessible to plebs as information ever was before."

He said his transparency ticket was "noble" and he sort of likened himself to John F. Kennedy.

Cameron reckoned the dead president's "Ask not what your country can do for you - ask what you can do for your county" skit really meant something, now that plebs had information, had knowledge, had power: "So a new generation of politicians can help make that noble dream a reality."