"Whenever the people are well-informed, they can be trusted with
their own government. Whenever things get so far wrong as to
attract their notice, they may be relied on to set them to
rights." - Thomas Jefferson to Richard Price, 1789

Access to the public is exactly appropriate for any arbitration hearing that affects the public's money. We should go further, however, and demand that all negotiations that affect public funds, such as those highlighted by Councilman Oliverio, should be recorded and made public after a suitable embargo period. This embargo would allow negotiators enough latitude to propose compromises, while remaining eventually accountable to their constituents–whether union members, shareholders, or voters. Even India, with some of the worst corruption in the world, has implemented their Right to Information act, putting similar information into the public domain after 30 days. Democracy only works when our right to know is respected. It is ridiculous that our access to information is worse than India's. Kohl S. Gill Sunnyvale

10 December 2009

Today's scientists are effectively public servants, and should start behaving accordingly. The public has a right to know what scientists do and how they do it. To prevent scandals like Climategate, scientific correspondence, critiques, and even data manipulation must be done in public. The University of East Anglia and associated climate scientists didn't understand that. I'm not sure they do, even now that their emails were hacked and used to sow doubt about climate science. Jon Stewart sums up the problem better than I can:

The hackers who extracted emails from UEA committed a crime. In so doing, they exposed that several scientists called people dismissive names, manipulated access to blogs, adjusted data in non-obvious ways, and derided their obligations under the law such as the U.S. and U.K. Freedom of Information Acts (FOIAs).

What bothered me most was the disregard for the law. The most damning example from the emails I've found includes this passage, from Phil Jones to Michael Mann (emphasis mine):

Mike,

I presume congratulations are in order - so congrats etc !

Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don't leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I'll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? - our does ! The UK works on precedents, so the first request will test it.

We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it - thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who'll say we must adhere to it !

The passage goes on to other topics, but you get the picture.

The reaction to this crime should not be bigger and thicker firewalls, or at least not solely that. Scientists should realize by now that privacy is not absolute, and in most cases, it's not even a good idea. They should embrace transparency, by moving correspondence to more open formats like blogs, wikis, and Google Wave. This move is critical to maintaining the long-term respect and public support for science. Obviously these platforms need to be adjusted to accommodate the complex manipulations of large data sets; the scientific community is in the best position to demand for and contribute to the development of appropriate applications. They will find great allies in the library science and open government communities, folks tackling the same essential problem. One great example is the Alliance for Taxpayer Access. In any case, scientists need to get out in front regarding transparency, or risk letting deniers define the route, themselves.

Some scientists may argue at this point, claiming that such presumptions of transparency will stifle free and open discussion and debate, that scientists working in the public sphere will censor themselves, and that the best ideas will not come forward. They might worry that science would become even more politicized than it already is. These might be fair points; after all, much of scientific inquiry is based on proposing and discarding ideas, ideas that seem crazy at first, but may just be right.

The answer is to embrace the embargo. Embargoes are already used throughout the scientific enterprise. Usually, they're used by scientific journals to delay access to non-paying readers, but they can be used in other ways, as well. Scientist A collects data and wants to analyze it and publish. While working on it, she shares her data with Scientist B so as not to delay further analysis. Oftentimes, as a courtesy or by agreement, Scientist B holds his own results under an embargo until Scientist A has had a chance to publish her results. This avoids confusion over which person should get credit for the initial data and analysis.

Wary scientists should know that everything they do, every email they write, every correction of data, every keystroke, could eventually wind up in the public domain. The platforms that will serve those scientists best will incorporate a time-bound embargo, with definite and obvious, rolling expiration dates.

[G]iven the growing policy relevance of climate data, increasingly higher standards must be applied to the transparency and availability of climate data and metadata. These standards should be clarified, applied and enforced by the relevant national funding agencies and professional societies that publish scientific journals... The need for public credibility and transparency has dramatically increased in recent years as the policy relevance of climate research has increased. The climate research enterprise has not yet adapted to this need, and our institutions need to strategize to respond to this need.

This actually isn't a loss of privacy–which doesn't really exist, anyway–but rather a move to make science even more legitimate and accessible to the public. We should all recognize the great value the world has derived from access to earlier scientific correspondence. Occasionally, the public needs to be reminded that scientific inquiry is a human enterprise. Transparent scientists are ones they can believe in.

08 December 2009

Ever wonder why you get the particular advertisements you see when using tools like Yahoo? Most of the time, the ads relate to the content you're accessing at the time, like ads for pet services when you spend all day looking at cute pictures of kittens instead of applying for jobs like you promised you would.

But even if you're not logged in at sites like Yahoo!, Google, or Facebook, those companies tend to collect quite a bit of information on you, creating their own consumer profile of you, which they use to match you up with targeted ads. Wouldn't it be great to take a look at that profile? Wouldn't you like the option to edit such a profile, or turn it off altogether?

Thanks to increasing pressure from transparency advocates and the U.S. Federal Trade Commission, Yahoo! is taking a baby step in this direction. Yahoo! is launching a test version of it's Ad Interest Manager (AIM), which shows you a summary of the info that Yahoo! has on you. I haven't logged in at Yahoo! in a while, so without logging in, I took this screenshot of the AIM site. Now, it definitely gets some things right, like my age range, gender, OS and browser. It still thinks I'm in DC, which is odd because I'm on the other coast, now. When I log in, I see exactly the same profile, except that "Entertainment" has been added as an interest.

Note that Yahoo! appears to know who I am, even when I'm logged out! Yahoo's policy is to de-identify data after 6 months at the most, but they never delete it.

Statistical corruption has been found in China for years, largely for two reasons. First, economic growth is a key factor determining the promotion of government officials. Statistical data and numbers are regarded as a reflection of economic growth, which is used to evaluate the performance of the officials. This is the so-called "numbers make leaders" phenomenon ("shu zi chu guan" in Chinese). Second, the statistical organizations are not independent entities in China. They are a part of the government and hence are vulnerable to government interference. Without specific laws and regulations to punish statistical corruption, government leaders can intervene in statistical reporting with low political risks. They may tailor statistics for different purposes, such as inflating statistical numbers that indicate economic achievements and decreasing statistical numbers for environmental pollution and damage. This is the so-called "leaders make numbers" phenomenon ("guan chu shu zi" in Chinese).

My guess is that the move by the NBS to curb official corruption is at least partly due to the recent scandal surrounding Qiu Xiaohua, former director of the National Bureau of Statistics, who was sacked and expelled from the Chinese Communist Party in 2007.

We hold these truths to be self-evident that all men are created equal, that they are endowed by their Creator with certain inalienable rights, among these are life, liberty, and the pursuit of happiness, that to secure these rights governments are instituted among men. We...solemnly publish and declare, that these colonies are and of right ought to be free and independent states...And for the support of this declaration, with a firm reliance on the protection of divine providence, we mutually pledge our lives, our fortunes, and our sacred honour.

Despite the fact that the passage in yellow was never written by Jefferson, this looks fair enough, until you take a look at the actual text of the Declaration:

We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness. That to secure these rights, Governments are instituted among Men, deriving their just powers from the consent of the governed, That whenever any Form of Government becomes destructive of these ends, it is the Right of the People to alter or to abolish it, and to institute new Government, laying its foundation on such principles and organizing its powers in such form, as to them shall seem most likely to effect their Safety and Happiness.

The passage in green, which Jefferson thought was the whole point of the Declaration, was dropped. Some editing to conserve space is appropriate, but this gutting of his message is ridiculous. Of course, I'm not the first to notice this censorship. Let's hope I won't be the last.

In light of Mr. Jefferson's prominent role as the supplier of our new lead quotation here, let's raise a glass to this Founding Father, censored in his own memory.

Update: Since folks have asked, my point is that this type of editing is dangerously Orwellian. Even our national monuments may be lying to us. (The memorial was constructed just a few years before Orwell completed 1984.)

15 September 2009

Over the past few years, I've become something of a transparency nut. One of the problems with transparency as movement is that very few people even recognize the term, much less know what it is. Once they see an example of transparency, they're often quite supportive.

Privacy as a social phenomenon is not terribly old. Indeed, the first taste that most Americans experienced was the privacy of isolation, and then of anonymity, which they got by moving to the American wilderness, and then to cities and suburbs.

Privacy is inconvenient. As we see in the social networking phenomenon and online advertising, folks are trading more information about themselves and their habits in exchange for better-tailored goods and services. Though most social networks, e.g., Facebook, have detailed privacy settings, most users leave these settings off for convenience.

Privacy is illusory. Private detectives agree that the most effective way to get information about someone is not through cameras, microphones, wiretaps, or keystroke monitoring. Rather, it's much easier to find all you need to know by using Google, sifting through publicly accessible civil and divorce court records, criminal and driving records, credit histories, physical mail, digging into garbage, and chatting up that gossipy neighbor or brother-in-law.

Privacy is distracting. Privacy is used as a solution to more fundamental social concerns. You shouldn't have to keep your information private to prevent harassment, identity theft, or stalking. Those are already crimes, in and of themselves, and privacy is a flimsy barrier to them, at best.

Privacy is expensive. Privacy is now a huge industry. Privacy is not insurance, which has the value of increasing your resilience in a measurable way. Rather, privacy purports to increase your protection, with few guarantees. You trade your information, often along with your money, to Company A for goods and services. Then you pay Company B to make sure that Company A doesn't release that same information. And only rarely do you get the opportunity to see the information that has been collected about you – to see how it is being used, for example – even to simply correct the information itself.

Privacy is insidious. You want to be paid well, and to be paid fairly, for what you do, but you're often told to keep your wages private. This privacy allows you to be paid unfairly, by allowing arbitrary decisions, such as those based on gender, race, age, or hairstyle, to affect your wage. Also, since everyone is quiet about wages, you never know the true value of your work, and are never able to bargain based on that value. This has the net effect of suppressing the wages of truly valuable workers. This stems from the idea that evaluations, especially of individuals, are more accurate if they are private. Private evaluations are a sign of ineffective management. And making them public can even be fun.

Privacy is not necessary. There is an alternative. You can choose differently.

You can choose to create a more transparent society. Imagine a society in which you can voluntarily release information, with the understanding that you can always see what is done with that information, how it's used, how it's changed, and what the benefits and detriments are. Imagine a society where your neighbors don't snoop on you, not because they can't, but just because it's rude and they have better things to do. Imagine a society where you can do good by working hard, paying your taxes, and serving your community with the assurance that you can see just how other people are doing those things, too. Imagine a society where your governments, businesses, and other organizations don't waste precious resources just because they can get away with it. It could be a society where we all dispense with financial privacy to strike a resounding blow against organized crime, world-wide.

At the very least, such a society could be quite interesting.

Is this a society worth striving for? If so, how do we get there from here?

14 September 2009

I'm excited to resume writing about transparency, and I have lots and lots in the hopper to share. I'm also looking for paying work in the Bay Area as I move to Sunnyvale, CA, in October. The job search is tough, but it does allow me some time to devote to writing.

As you watch this space, you'll see several interweaving series of posts, on my last couple of jobs in the federal government, on my progress in finding the next great job, on why transparency matters, on great sources of information, and a whole lot more.