Lobbyists beware: judge rules metadata is public record

The Arizona Supreme Court has ruled that the metadata attached to public …

The Arizona state Supreme Court has ruled that the metadata attached to public records is itself public, and cannot be withheld in response to a public records request. Such a ruling on file metadata may not seem like a huge win for open government advocates, but it definitely is, given that metadata has unmasked more than one lobbyist's effort to influence Congress.

In the Arizona case, a police officer had been demoted in 2006 after reporting "serious police misconduct" to his superiors. He suspected that the demotion was done in retaliation for his blowing the whistle on his fellow officers, so he requested and obtained copies of his performance reports from the department. Thinking that perhaps the negative performance reports had been created after the fact and then backdated, he then demanded access to the file metadata for those reports, in order to find out who had written them and when.

The department refused to grant him access to the metadata, and the matter went to court. After working its way through the court system in a series of rulings and appeals, this past January an Arizona appeals ruled that even though the reports themselves were public records, the metadata was not. It turned out that Arizona state law doesn't actually define "public record" anywhere, so the appeals court relied on various common law definitions to determine that the metadata, as a mere byproduct of the act of producing a public record on a computer, was not a public record itself.

The case was then appealed to the Arizona state Supreme Court, which has now ruled that the metadata is, in fact, a public record just like the document that it's attached to.

Metadata follies, and the case of Google

If you want to know how important metadata can be in public policy deliberations, Google's history with it can be instructive, since the search giant has been both hurt and helped by metadata snooping.

Last year, the The Australian Competition Commission and Consumer Commission (ACCC) received hundreds of electronically submitted feedback letters opposing eBay Australia's decision to go PayPal-only for accepting auction payments. One of the most impressive letters was a 38-page missive that had obviously been written by someone with extensive and intimate knowledge of payment systems. A look at the letter's PDF metadata revealed that the author of the letter was none other than Google, which was upset that Google Checkout was being excluded in favor of PayPal. The metadata also revealed, embarrassingly enough, that the PDF had been written not in Google Docs, but in Microsoft Word.

The very next month, the tables were turned when the American Corn Grower's Association somewhat surprisingly threw its weight behind the idea that Congress should launch a hearing to look into the possible anti-trust implications of the Google-Yahoo advertising deal. CNET's Declan McCullagh took a look at the PDF letter that the group submitted to Congress, and found that it had been authored by a staffer at the LawMedia Group, a DC lobbying shop whose client list includes the anti-Google, anti-net neutrality National Cable and Telecommunications Association.

To leave Google's metadata mixups and go back even further in time, one of the most famous metadata lobbying goof-ups occurred in 2004, when Wired busted California Attorney General Bill Lockyer circulating an anti-P2P letter that, after a look at its Word metadata, appeared to have been either drafted or edited by the MPAA.

As open government projects that solicit feedback from the public gain traction at the federal and local level, these types of metadata-related discoveries will become more and more common. Guaranteeing that file metadata is available to the public will make help to ensure that we know who is trying to influence public discussion.

Based on the article (I haven't gone looking for more in-depth information), it sounds like we could use laws that go further than this. It's not enough to simply allow viewing of incriminating metadata that happens to be inadvertently left intact, but to require that the original authors and chain of editing for government submissions be logged and available as public record.

Yes, laws need to be created to make sure that the metadata isn't just automatically wiped when the document is entered into the company's document management system (or failing that, when the document is turned over under FOIA/discovery).

Doormat, not sure I followed your idea but I'm confused. The courts just said that the metadata is part of the FOIA, so that part seems already done. I'm not really sure why you'd think metadata has to be submitted with any document though. A company creates a letter and sends it to an official and you'd say they have to submit the metadata with it and aren't allowed to clean it off? I don't get that. If I send a paper letter I don't have to submit my drivers license and proof of voter registration with it to prove who I am, why should I have to with electronic documents?

I understand the urge to shrink government, I think the 2000 page healthcare bill is an abomination, but to make things better but it doesn't make sense to continuously make it harder to interact with the government with more regulations.

Originally posted by Pont:Legislators should be forced to use Subversion/CVS or equivalent with a public web front-end. Laws, being structured text, lend themselves to change tracking software very well.

Originally posted by Pont:Legislators should be forced to use Subversion/CVS or equivalent with a public web front-end. Laws, being structured text, lend themselves to change tracking software very well.

I can see it now: "Mr. President, after months of struggle we have managed to get a version of this law through Congress and the Senate. If you would just sign it, we can pass it along to the last and most powerful link in the chain: the guy with SVN commit rights."

In all seriousness, it's a good idea, but I don't see it happening any time soon.

Sounds to me that people in favor of this want to make it easier to launch ad-hom attacks. There's a big difference between a cop wanting to see if documents were altered or filed after his case (which is attempt to uncover wrong-doing) and wanting to see who drafts whatever letters and documents become public record.

The content of such public records should stand alone. It should take judicial oversight to reveal anything beyond that. I'd argue that anyone who believes otherwise, yet thinks that anonymous bloggers should be able to stay that way want to have their cake and eat it too.

Imagine if you're some poor clerk who's job it is to edit letters for spelling and grammar, but your name is required by law to remain on documents that you had nothing to do with the content...

Originally posted by jdietz:I want to know what the police officer found when he examined the metadata...

^-- This. Why isn't the most interesting point covered!?

//Edit: Well, that's interesting. After digging the case for a while, it appears the Ars article is incorrect in a few points:

1) The officer suspected his superiors of backdating negative reports.2) The officer could have received all the data in printouts. Their opinion was "the law only says we have to give it to you, it doesn't say how we have to give it to you."

Gee, I wonder why being bungholes (yes, I watched South Park yesterday, how did you know?) made a case turn against them.

Originally posted by Miwa:Sounds to me that people in favor of this want to make it easier to launch ad-hom attacks. There's a big difference between a cop wanting to see if documents were altered or filed after his case (which is attempt to uncover wrong-doing) and wanting to see who drafts whatever letters and documents become public record.

The content of such public records should stand alone. It should take judicial oversight to reveal anything beyond that. I'd argue that anyone who believes otherwise, yet thinks that anonymous bloggers should be able to stay that way want to have their cake and eat it too.

Imagine if you're some poor clerk who's job it is to edit letters for spelling and grammar, but your name is required by law to remain on documents that you had nothing to do with the content...

I would disagree. I believe it lends much more clarity to a discussion/argument when it's clear who's actually behind some of the arguments. I seriously doubt anyone's going to care that Joe Schmoe edited the document ... they'll be looking for the bigger names, not the smaller ones. Unless, of course, Mr. Schmoe purposely changed the nature of the document.

In my opinion (and that's really all it is), the positives greatly outweigh the negatives with this.

Originally posted by jdietz:I want to know what the police officer found when he examined the metadata...

^-- This. Why isn't the most interesting point covered!?

//Edit: Well, that's interesting. After digging the case for a while, it appears the Ars article is incorrect in a few points:

1) The officer suspected his superiors of backdating negative reports.2) The officer could have received all the data in printouts. Their opinion was "the law only says we have to give it to you, it doesn't say how we have to give it to you."

Gee, I wonder why being bungholes (yes, I watched South Park yesterday, how did you know?) made a case turn against them.

This comment was edited by Renx on October 30, 2009 10:15

Thanks for doing the legwork. But we still don't know. What did he find?

Originally posted by Miwa:Sounds to me that people in favor of this want to make it easier to launch ad-hom attacks. There's a big difference between a cop wanting to see if documents were altered or filed after his case (which is attempt to uncover wrong-doing) and wanting to see who drafts whatever letters and documents become public record.

Even if this is the case, it's a small price to pay. What we're talking about here are documents that either detail the workings of, or are trying to influence, government. There is an inherent public interest in keeping everything about such documents within the public sphere (as the article noted) -- not just their contents.

quote:

The content of such public records should stand alone. It should take judicial oversight to reveal anything beyond that.

Why?

quote:

I'd argue that anyone who believes otherwise, yet thinks that anonymous bloggers should be able to stay that way want to have their cake and eat it too.

This is where your argument falls apart. This case doesn't involve individuals writing for their own purposes. This covers documents involved with the workings of government. As our current form of government is (should be) beholden to its constituents, there is an interest in keeping those workings transparent to said constituents.

quote:

Imagine if you're some poor clerk who's job it is to edit letters for spelling and grammar, but your name is required by law to remain on documents that you had nothing to do with the content...

Do the lobbyists even care, really? If anybody would find this to be their worst nightmare, it would be politicians. Lobbyists are doing the job they're paid to do (whether you agree with what they do or not is irrelevant to the matter at hand), but the connections that might be uncovered through such metadata may be quite embarrassing to any politicians they are connected with.

The content of such public records should stand alone. It should take judicial oversight to reveal anything beyond that.

Why?

I'd venture a guess that the push-back would be to erect more barriers and obfuscations - more closed-door committee meetings where nothing is committed to paper until the last minute, effectively adding more bloat and more red tape to the process and ensuring the little that actually gets accomplished in government goes away.

Originally posted by madmikefisk:Do the lobbyists even care, really? If anybody would find this to be their worst nightmare, it would be politicians. Lobbyists are doing the job they're paid to do (whether you agree with what they do or not is irrelevant to the matter at hand), but the connections that might be uncovered through such metadata may be quite embarrassing to any politicians they are connected with.

Of course they care. If things like this end up embarrassing politicians, they could backtrack on support for the legislation that the lobbyists are pushing for. That would be bad for the lobbyists.

Originally posted by jdietz:I want to know what the police officer found when he examined the metadata...

^-- This. Why isn't the most interesting point covered!?

//Edit: Well, that's interesting. After digging the case for a while, it appears the Ars article is incorrect in a few points:

1) The officer suspected his superiors of backdating negative reports.2) The officer could have received all the data in printouts. Their opinion was "the law only says we have to give it to you, it doesn't say how we have to give it to you."

Gee, I wonder why being bungholes (yes, I watched South Park yesterday, how did you know?) made a case turn against them.

This comment was edited by Renx on October 30, 2009 10:15

Thanks for doing the legwork. But we still don't know. What did he find?

It sound like the Arizona Supreme Court made their ruling - but that nothing yet has come of the case. It could end up potentially heading for Federal Court next. So until everything is exhausted there will be no final conclusion yet.

Originally posted by ricbach229:Doormat, not sure I followed your idea but I'm confused. The courts just said that the metadata is part of the FOIA, so that part seems already done. I'm not really sure why you'd think metadata has to be submitted with any document though. A company creates a letter and sends it to an official and you'd say they have to submit the metadata with it and aren't allowed to clean it off? I don't get that. If I send a paper letter I don't have to submit my drivers license and proof of voter registration with it to prove who I am, why should I have to with electronic documents?

I understand the urge to shrink government, I think the 2000 page healthcare bill is an abomination, but to make things better but it doesn't make sense to continuously make it harder to interact with the government with more regulations.

I don't know may be it is like having created a document on letterhead, and then when asked to produce it for the plaintiff, covering up the letterhead.

The content of such public records should stand alone. It should take judicial oversight to reveal anything beyond that.

Why?

I'd venture a guess that the push-back would be to erect more barriers and obfuscations - more closed-door committee meetings where nothing is committed to paper until the last minute, effectively adding more bloat and more red tape to the process and ensuring the little that actually gets accomplished in government goes away.

So it is better to let these guys cover up things and fabricate evidence because we don't want to inconvenience them by red tape?

Originally posted by odcchaz:Well, now that the metadata issue has been outed, it'll take everyone who's got something to hide and extra 5 minutes to retype or copy/paste their documents before making them public.

Originally posted by odcchaz:Well, now that the metadata issue has been outed, it'll take everyone who's got something to hide and extra 5 minutes to retype or copy/paste their documents before making them public.

If records are either requested via a public records request or through a subpoena then it is going to look extremely suspicious that the meta data just so happens to have a "created" date that shows the file was created right after the record was requested. Especially if the particular record was supposed to have been created quite a bit of time before what the created date suggests.

I realize there will be some occasions where people with something to hide will be able to do something sneaky, but this solution is not pretending to be some kind of silver bullet. It is claiming that it will make the process of hiding information more difficult and that's a fact.

Originally posted by odcchaz:Well, now that the metadata issue has been outed, it'll take everyone who's got something to hide and extra 5 minutes to retype or copy/paste their documents before making them public.

If records are either requested via a public records request or through a subpoena then it is going to look extremely suspicious that the meta data just so happens to have a "created" date that shows the file was created right after the record was requested. Especially if the particular record was supposed to have been created quite a bit of time before what the created date suggests.

I realize there will be some occasions where people with something to hide will be able to do something sneaky, but this solution is not pretending to be some kind of silver bullet. It is claiming that it will make the process of hiding information more difficult and that's a fact.

Well in the cases where letters were found to be generated by suspicious or self serving parties, that evidence could be dealt with before the letter even makes it out into the open, not just before someone questions its origins.

Originally posted by odcchaz:Well in the cases where letters were found to be generated by suspicious or self serving parties, that evidence could be dealt with before the letter even makes it out into the open, not just before someone questions its origins.

I don't quite follow you. Are you strictly talking about metadata? That is all I am talking about.

Again, no one is claiming that this change is a silver bullet (no such bullet exists), but the fact is that it will help.

Originally posted by odcchaz:Well, now that the metadata issue has been outed, it'll take everyone who's got something to hide and extra 5 minutes to retype or copy/paste their documents before making them public.

Exactly. That's why (as I said) we need new laws that require such metadata, so that people can't simply strip it out, and that the metadata must be accurate under penalty of perjury.

Originally posted by odcchaz:Well, now that the metadata issue has been outed, it'll take everyone who's got something to hide and extra 5 minutes to retype or copy/paste their documents before making them public.

How is that going to alter the date the file was created?

Set your system clock to whatever date you please and then create the file. This isn't rocket science.

Originally posted by odcchaz:Well, now that the metadata issue has been outed, it'll take everyone who's got something to hide and extra 5 minutes to retype or copy/paste their documents before making them public.

How is that going to alter the date the file was created?

Set your system clock to whatever date you please and then create the file. This isn't rocket science.

Perhaps... But most of this stuff will show up on automated back-ups as well which can be supeoned... and now you're on the hook for fraud.

The content of such public records should stand alone. It should take judicial oversight to reveal anything beyond that.

Why?

I'd venture a guess that the push-back would be to erect more barriers and obfuscations - more closed-door committee meetings where nothing is committed to paper until the last minute, effectively adding more bloat and more red tape to the process and ensuring the little that actually gets accomplished in government goes away.

So it is better to let these guys cover up things and fabricate evidence because we don't want to inconvenience them by red tape?

I was going on the vein of "force politicians to do it". It's not about inconveniencing them... there are plenty of ways to get around it that would only assure that even less comes from any given legislative term. By increasing bureaucracy's bloat you're not inconveniencing bureaucrats - they're pretty used to it - just expect even less out of them in terms of results than before. They have lots of meetings now, y'know.