NSA, the FISA Court, and Risks of Tech Summaries

Yesterday the U.S. government released a previously-secret 2011 opinion of the Foreign Intelligence Surveillance Court (FISC), finding certain NSA surveillance and analysis activities to be illegal. The opinion, despite some redactions, gives us a window into the interactions between the NSA and the court that oversees its activities—including why oversight and compliance of surveillance are challenging.
The opinion has enough (unredacted) technical detail to get a basic picture of what the NSA was doing. The court was considering the NSA’s capture of traffic from Internet backbone links (“upstream capture”), which accounted for about 9% of the Internet traffic captured by the NSA. (The other 91% came directly from service providers, for example when email providers turned over the contents of targeted accounts.) The eavesdropping apparatus would do some kind of pattern matching on the traffic it saw, and would record traffic that “hit” on an approved pattern. Captured data would go into a database where human analysts could search and examine it.

The NSA was supposed to be capturing one discrete “communication” (e.g. an email message) at a time, but due to limitations in the capture technology they could only capture data in a unit called a “transaction” (e.g. an interaction between a user’s computer and an email server while the user was reading email). About 90% of the transactions that were captured contained only a single communication. The remaining 10% of transactions were Multi-Communication Transactions (“MCTs”), meaning that they contained at least one communication that hit on a pattern, along with one or more other communications that might not be a hit.

Based on the opinion plus remarks by officials in a press call yesterday, it appears that the NSA had a list of “targeted” email addresses that it was allowed to monitor, and the capture technology would grab a chunk of data such as an Internet packet, whenever one of the targeted email addresses was contained in that chunk. The NSA said it had no reasonable way to capture more precisely—although the detailed limitations of the NSA’s capture technology were redacted. The Court took these technical limitations at face value but said that these limitations did not make the resulting over-capture legal.

The Court scolded the NSA:

The Court is troubled that the government’s revelations regarding NSA’s acquisition of Internet transactions mark the third instance in less than three years in which the government has disclosed a substantial misrepresentation regarding the scope of a major collection program.

In March, 2009, the Court concluded that its authorization of NSA’s bulk acquisition of telephone call detail records from [redacted] … “ha[d] been premised on a flawed depiction of how the NSA uses [the acquired] metadata,” and that “[t]his misperception by the FISC existed from the inception of its authorized collection in May 2006, buttressed by repeated inaccurate statements made in the government’s submissions, and despite a government-devised and Court-mandated oversight regime.”

(footnote 14 on p. 16)

This footnote has been much-quoted as evidence of the Court’s disappointment. What struck me in this passage, though, is that the Court was unhappy because the NSA withheld important technical details.

Whenever technologists have to interact with less-technical courts or policymakers, there are communication challenges. It’s not helpful to tell the non-expert everything about how the system works—supplying too much detail is can be just as detrimental to decision-making as supplying too little. You have to supply a summary.

The trick is to supply a summary containing just the right details, that is, the details that matter to the non-expert in light of what they are trying to accomplish. If a non-expert judge needs to decide whether an activity is lawful, they need to know the details that relate to the legal questions they are considering.

Choosing which details to reveal can be very challenging—you have to understand something of the law, along with the technology—even if you’re doing your level best to help the decisionmaker. And if your motivations are less pure, it’s all too feasible to cook up a misleading summary of the facts. The Court seemed to suspect that the NSA had been doing exactly that.

Our legal system normally addresses this unreliable-summary problem by relying on the adversarial process. The opposing party gets access to the underlying facts and can cross-examine the expert who provides the summary. This process isn’t perfect but it does tend to deter and correct unreliable summaries.

This safeguard is not in place when the FISC is considering NSA activities. The FISC has no choice but to rely on the NSA’s own summary of the technical details. Even if NSA employees are trying in good faith to tell the Court what it needs to know, it’s not too hard to see how human error plus institutional pressures could lead the agency to not reveal “unhelpful” facts.

Congress could address this problem by changing the law to create an opposing party in FISC procedures—a change that the President recently endorsed. But a really effective opposing party will need to have access to technical evidence and expertise as well.

Comments

“Congress could address this problem by changing the law to create an opposing party in FISC procedures”

They could. But I would remain highly suspicious of such a change as long as there still is no transparency in the process.

The SEC is theoretically an “adversarial party” to Wall Street and the financial sector, but it’s pretty clear that even with some transparency (though probably not enough), the SEC acts at least as much as a farm league for aspiring brokers as it does a watchdog.

An opposing party in FISC procedures would not have precisely that type of relationship of course (except inasmuch as the NSA relies on private contractors to do some of their dirty work). But even so, as long as the government gets to decide who gets to act as the opposing party, there remains a conflict of interest and little assurance that the opposing party will indeed have the expertise required to effectively oppose FISC procedures, never mind a strong motivation to do so.

It seems to me that the secrecy involved in the NSA and FISC is a far deeper problem than the simple lack of an opposing party. Yes, an adversarial process is an important step in the right direction. But historically, secretly run government agencies have done a very poor job of self-restraint and adhering to lawful behavior, and it seems that the NSA has been no exception to this rule.

“But even so, as long as the government gets to decide who gets to act as the opposing party, there remains a conflict of interest”

Yep, conflict of interest is the biggest problem I see. Of course the government is no stranger to that; they just love to delve into anything that would be a conflict of interest with no qualms. They don’t even pretend to be worried about it let alone to actually put a stop to it.

The ONLY way there would not be a conflict of interest would be if the opposing party was the targets of the collection of data. E.G. DUE PROCESS in every other court of law.

And in the case of the NSA’s relentless collection of every kind of e-mail (of which they are only claiming they were “hitting” on certain patterns, the proof is actually contrary, and they have been cataloging all e-mails, all telephone calls etc.) the opposing party would indeed have to be the American Public.

Oh, but wait, what good would a secret court be, if the secret court couldn’t conduct anything in secret and the public was allowed to have full access to all of the technical information.

Well we do call that “peer review” in scientific circles; “due process” in legal circles; and “transparency” in political circles.

The problem being here is that the U.S. Government abhors all three things (peer review, due process, and transparency); that is why they created the secret court in the first place. I don’t see them backing away from their conflict of interest anytime soon.

High praise for Ed Felten’s new declaration, supporting the ACLU in its federal lawsuit against James Clapper and the NSA, et al. The ACLU is challenging the indiscriminate collection of metadata from the calls of most American citizens. His declaration makes clear how metadata, even standing alone, enables the government to construct detailed profiles on any or all of us. It is easy to take for granted what it means for an expert academician to step forward and take strong positions on such critical matters of public policy. I hope all of us will let Ed know that we don’t take his contribution for granted.

Freedom to Tinker is hosted by Princeton's Center for Information Technology Policy, a research center that studies digital technologies in public life. Here you'll find comment and analysis from the digital frontier, written by the Center's faculty, students, and friends.