The Obama administration overruled recommendations from within the US Department of Homeland Security and implemented new guidelines earlier this year that allow the government to gather and analyze intelligence on every single US citizen. Since the spring, a little-know intelligence agency outside of Washington, DC has been able to circumvent the Fourth Amendment to the US Constitution and conduct dragnet surveillance of the entire country, combing massive datasets using advanced algorithms to search and seize personal info on anyone this wish, reports the Wall Street Journal this week. There’s no safeguard that says only Americans with criminal records are the ones included, and it’s not just suspected terrorists that are considered in the searches either. The National Counterterrorism Center (NCTC) has been provided with entire government databases and given nearly endless access to intelligence on everyone in the country, regardless of whether or not they’ve done anything that would have made them a person of interest. As long as data is “reasonably believed” to contain “terrorism information,” the agency can do as they wish. What’s more is the NCTC can retain that information for years, reviewing it whenever they’d like to take a look. The update to the agency’s policies, reported by RT at the time and reexamined this week in the Journal, expose any person in the country to invasive and nearly endless government surveillance.

According to documents obtained by the Journal through Freedom of Information Act requests and conversations between the paper and persons familiar with that Situation Room sound-off, Ms. Callahan unsuccessfully argued against updating a 2008 Justice Department memo about what intel the NCTC can have and how they use it. Just weeks after that meeting, new guidelines were authorized and, within months, Ms. Callahan was working elsewhere. Despite her efforts, a 32-page document, “Guidelines for Access, Retention, Use and Dissemination by the National Counterterrorism Center and other Agencies of Information in Datasets Containing Non-Terrorism Information,” went into effect, and with that the NCTC was no longer restricted to only terrorism-related intelligence. Indeed, the changes aren’t just within the name of the document. The 2012 update to the NCTC’s data-mining policies expand the intelligence the agency can comb while at the same time removing safeguards that were in place for privacy’s sake. Under the new rules, data on innocent Americans can be retained for five years, and intel on anyone “reasonably believed to constitute terrorism information” can be kept until the end of time.

According to the paper, “flight records, casino-employee lists, the names of Americans hosting foreign-exchange students and many others” can be collected indefinitely and searched at will within the NCTC, an agency only nine years old and not nearly as well-known as her sister spy groups: the CIA and FBI. Once the NCTC has the info, though, they can decide who else can be made privy to it. If the US government is so inclined, intelligence on specific citizens can be sent to any foreign nation in the world. On the blog PrivacySOS, civil liberties advocate Kade Crockford condemns the spy program by saying any safeguard that could be implemented wouldn’t end what appears to be a serious constitutional violation. “And even if it was an effective anti-terrorism technique, widespread, warrantless surveillance of every single living human being – suspicious or not – damn sure isn’t democratic practice. We are supposed to be innocent until proven guilty in this country, not the other way around,” Crockford writes. Meanwhile, in-between Calabrese’s original post and the Journal’s article from this week, search giant Google confirmed that the federal government has sent more requests for personal user data in 2012 than ever before. “This is the sixth time we’ve released this data, and one trend has become clear: Government surveillance is on the rise,” Google explained last month.

The latest revelation from the Journal of course is but the most recent installation in what has become a remarkable year in terms of finding out the truth about Uncle Sam’s shocking full-fledged surveillance. Throughout 2012, several former employees of the National Security Agency (NSA) have stepped up and given interviews about the grievances with the office, particularly their disregard for the privacy of Americans. “When you open up the Pandora’s Box of just getting access to incredible amounts of data, for people that have no reason to be put under suspicion, no reason to have done anything wrong, and just collect all that for potential future use or even current use, it opens up a real danger — and to what else what they could use that data for, particularly when it’s all being hidden behind the mantle of national security,” NSA whistleblower Thomas Drake told Current TV host Eliot Spitzer earlier this year.

Earlier this month, former NSA analyst William Binney spoke with RT and said that the FBI — who maintains databases that can be requested by the NCTC under their latest policies — has been storing the emails of every person in America for at least a decade. “So, yes, this can happen to anyone. If they become a target for whatever reason – they are targeted by the government, the government can go in, or the FBI, or other agencies of the government, they can go into their database, pull all that data collected on them over the years, and we analyze it all. So, we have to actively analyze everything they’ve done for the last 10 years at least,” he said. Upon winning a Callaway award for civic courage in DC last month, Mr. Binney explained that he and other former NSA agents “could not be accessories to violations of the US Constitution.” Ms. Callahan has since left her post within the NCTC and is now practicing law in the nation’s capital focusing specifically on privacy.

The Wall Street Journal today published (alternate link) an in-depth review of a new, relatively unknown program run by the National Counterterrorism Center (NCTC). Although we have been warning about the dangers of the program for months, and Itestified before Congress about the issue in July, the Journal’s story conveys how controversial the program was even inside the government. It also describes the broad scope of new authority the government is granting itself.

The program is striking in so many ways. Innocent people can be investigated and their data kept for years. It can be shared with foreign governments. All of this in service of not just terrorism investigations but also investigations of future crimes. In effect, the U.S. government is using information it gathers for its ordinary business to turn its own citizens into the subjects of terrorism investigations. Meanwhile, all of this is supposed to be against the law. The Privacy Act of 1974 says that information collected by the federal government for one purpose is not supposed to be used for another. However, agencies are attempting to circumvent these rules by publishing boilerplate notices in the Federal Register. Sadly, that practice has become far too common. Worse, all of this happened in secret, approved by National Security Advisor John Brennan and signed off on by Attorney General Eric Holder. No public debate or comment and suddenly, every citizen can be put under the terrorism microscope.

Ironically, all of these changes to the rules came in response to an attempted attack that had nothing to do with information collection or a U.S. citizen. The government cites the attempted 2009 Christmas bombing by Umar Farouk Abdulmutallab as the impetus for the changes. However, as the Journal story makes clear, Abdulmutallab wasn’t a U.S. citizen, and collecting information on him wasn’t a problem. Instead, his own father had identified him to the U.S. government as a potential terrorist. In short, an attack by a known foreign terror suspect was used to justify changes to rules about collecting information on U.S. citizens.

Finally, credit must be given to those who fought the program. It’s clear that DHS, especially the Privacy Officer, Mary Ellen Callahan, and the Office of Civil Rights and Civil Liberties pushed back hard against this. Nancy Libin, the chief privacy officer at the Department of Justice, also expressed serious reservations and fought an internal battle against the changes. It’s probably not a surprise that none of them are still in government. If you want to learn more here is a simple guide to the main changes created by the 2012 NCTC guidelines. And here are the Freedom of Information Act documents that we have gathered on NCTC—we will post more as we receive additional records.

Mary Ellen Callahan, then-chief privacy officer of the Department of Homeland Security: ‘This is a sea change in the way that the government interacts with the general public.’

Top U.S. intelligence officials gathered in the White House Situation Room in March to debate a controversial proposal. Counterterrorism officials wanted to create a government dragnet, sweeping up millions of records about U.S. citizens—even people suspected of no crime. Not everyone was on board. “This is a sea change in the way that the government interacts with the general public,” Mary Ellen Callahan, chief privacy officer of the Department of Homeland Security, argued in the meeting, according to people familiar with the discussions. A week later, the attorney general signed the changes into effect. Through Freedom of Information Act requests and interviews with officials at numerous agencies, The Wall Street Journal has reconstructed the clash over the counterterrorism program within the administration of President Barack Obama. The debate was a confrontation between some who viewed it as a matter of efficiency—how long to keep data, for instance, or where it should be stored—and others who saw it as granting authority for unprecedented government surveillance of U.S. citizens.

The rules now allow the little-known National Counterterrorism Center to examine the government files of U.S. citizens for possible criminal behavior, even if there is no reason to suspect them. That is a departure from past practice, which barred the agency from storing information about ordinary Americans unless a person was a terror suspect or related to an investigation. Now, NCTC can copy entire government databases—flight records, casino-employee lists, the names of Americans hosting foreign-exchange students and many others. The agency has new authority to keep data about innocent U.S. citizens for up to five years, and to analyze it for suspicious patterns of behavior. Previously, both were prohibited. Data about Americans “reasonably believed to constitute terrorism information” may be permanently retained.

The changes also allow databases of U.S. civilian information to be given to foreign governments for analysis of their own. In effect, U.S. and foreign governments would be using the information to look for clues that people might commit future crimes. “It’s breathtaking” in its scope, said a former senior administration official familiar with the White House debate. Counterterrorism officials say they will be circumspect with the data. “The guidelines provide rigorous oversight to protect the information that we have, for authorized and narrow purposes,” said Alexander Joel, Civil Liberties Protection Officer for the Office of the Director of National Intelligence, the parent agency for the National Counterterrorism Center.

The Fourth Amendment of the Constitution says that searches of “persons, houses, papers and effects” shouldn’t be conducted without “probable cause” that a crime has been committed. But that doesn’t cover records the government creates in the normal course of business with citizens. Congress specifically sought to prevent government agents from rifling through government files indiscriminately when it passed the Federal Privacy Act in 1974. The act prohibits government agencies from sharing data with each other for purposes that aren’t “compatible” with the reason the data were originally collected. But the Federal Privacy Act allows agencies to exempt themselves from many requirements by placing notices in the Federal Register, the government’s daily publication of proposed rules. In practice, these privacy-act notices are rarely contested by government watchdogs or members of the public. “All you have to do is publish a notice in the Federal Register and you can do whatever you want,” says Robert Gellman, a privacy consultant who advises agencies on how to comply with the Privacy Act. As a result, the National Counterterrorism Center program’s opponents within the administration—led by Ms. Callahan of Homeland Security—couldn’t argue that the program would violate the law. Instead, they were left to question whether the rules were good policy.

Under the new rules issued in March, the National Counterterrorism Center, known as NCTC, can obtain almost any database the government collects that it says is “reasonably believed” to contain “terrorism information.” The list could potentially include almost any government database, from financial forms submitted by people seeking federally backed mortgages to the health records of people who sought treatment at Veterans Administration hospitals. Previous government proposals to scrutinize massive amounts of data about innocent people have caused an uproar. In 2002, the Pentagon’s research arm proposed a program called Total Information Awareness that sought to analyze both public and private databases for terror clues. It would have been far broader than the NCTC’s current program, examining many nongovernmental pools of data as well. “If terrorist organizations are going to plan and execute attacks against the United States, their people must engage in transactions and they will leave signatures,” the program’s promoter, Admiral John Poindexter, said at the time. “We must be able to pick this signal out of the noise.”

Adm. Poindexter’s plans drew fire from across the political spectrum over the privacy implications of sorting through every single document available about U.S. citizens. Conservative columnist William Safire called the plan a “supersnoop’s dream.” Liberal columnist Molly Ivins suggested it could be akin to fascism. Congress eventually defunded the program. The National Counterterrorism Center’s ideas faced no similar public resistance. For one thing, the debate happened behind closed doors. In addition, unlike the Pentagon, the NCTC was created in 2004 specifically to use data to connect the dots in the fight against terrorism. Even after eight years in existence, the agency isn’t well known. “We’re still a bit of a startup and still having to prove ourselves,” said director Matthew Olsen in a rare public appearance this summer at the Aspen Institute, a leadership think tank. The agency’s offices are tucked away in an unmarked building set back from the road in the woodsy suburban neighborhood of McLean, Va. Many employees are on loan from other agencies, and they don’t conduct surveillance or gather clues directly. Instead, they analyze data provided by others.

The agency’s best-known product is a database called TIDE, which stands for the Terrorist Identities Datamart Environment. TIDE contains more than 500,000 identities suspected of terror links. Some names are known or suspected terrorists; others are terrorists’ friends and families; still more are people with some loose affiliation to a terrorist. TIDE files are important because they are used by the Federal Bureau of Investigation to compile terrorist “watchlists.” These are lists that can block a person from boarding an airplane or obtaining a visa. The watchlist system failed spectacularly on Christmas Day 2009 when Umar Farouk Abdulmutallab, a 23-year-old Nigerian man, boarded a flight to Detroit from Amsterdam wearing explosives sewn into his undergarments. He wasn’t on the watchlist. He eventually pleaded guilty to terror-related charges and is imprisoned. His bomb didn’t properly detonate. However, Mr. Abdulmutallab and his underwear did alter U.S. intelligence-gathering. A Senate investigation revealed that NCTC had received information about him but had failed to query other government databases about him. In a scathing finding, the Senate report said, “the NCTC was not organized adequately to fulfill its missions.”

“This was not a failure to collect or share intelligence,” said John Brennan, the president’s chief counterterrorism adviser, at a White House press conference in January 2010. “It was a failure to connect and integrate and understand the intelligence we had.” As result, Mr. Obama demanded a watchlist overhaul. Agencies were ordered to send all their leads to NCTC, and NCTC was ordered to “pursue thoroughly and exhaustively terrorism threat threads.” Quickly, NCTC was flooded with terror tips—each of which it was obligated to “exhaustively” pursue. By May 2010 there was a huge backlog, according a report by the Government Accountability Office. Legal obstacles emerged. NCTC analysts were permitted to query federal-agency databases only for “terrorism datapoints,” say, one specific person’s name, or the passengers on one particular flight. They couldn’t look through the databases trolling for general “patterns.” And, if they wanted to copy entire data sets, they were required to remove information about innocent U.S. people “upon discovery.” But they didn’t always know who was innocent. A person might seem innocent today, until new details emerge tomorrow.

“What we learned from Christmas Day”—from the failed underwear bomb—was that some information “might seem more relevant later,” says Mr. Joel, the national intelligence agency’s civil liberties officer. “We realized we needed it to be retained longer.” Late last year, for instance, NCTC obtained an entire database from Homeland Security for analysis, according to a person familiar with the transaction. Homeland Security provided the disks on the condition that NCTC would remove all innocent U.S. person data after 30 days. After 30 days, a Homeland Security team visited and found that the data hadn’t yet been removed. In fact, NCTC hadn’t even finished uploading the files to its own computers, that person said. It can take weeks simply to upload and organize the mammoth data sets. Homeland Security granted a 30-day extension. That deadline was missed, too. So Homeland Security revoked NCTC’s access to the data. To fix problems like these that had cropped up since the Abdulmutallab incident, NCTC proposed the major expansion of its powers that would ultimately get debated at the March meeting in the White House. It moved to ditch the requirement that it discard the innocent-person data. And it asked for broader authority to troll for patterns in the data.

As early as February 2011, NCTC’s proposal was raising concerns at the privacy offices of both Homeland Security and the Department of Justice, according to emails reviewed by the Journal. Privacy offices are a relatively new phenomenon in the intelligence community. Most were created at the recommendation of the 9/11 Commission. Privacy officers are often in the uncomfortable position of identifying obstacles to plans proposed by their superiors. At the Department of Justice, Chief Privacy Officer Nancy Libin raised concerns about whether the guidelines could unfairly target innocent people, these people said. Some research suggests that, statistically speaking, there are too few terror attacks for predictive patterns to emerge. The risk, then, is that innocent behavior gets misunderstood—say, a man buying chemicals (for a child’s science fair) and a timer (for the sprinkler) sets off false alarms. An August government report indicates that, as of last year, NCTC wasn’t doing predictive pattern-matching. The internal debate was more heated at Homeland Security. Ms. Callahan and colleague Margo Schlanger, who headed the 100-person Homeland Security office for civil rights and civil liberties, were concerned about the implications of turning over vast troves of data to the counterterrorism center, these people said. They and Ms. Libin at the Justice Department argued that the failure to catch Mr. Abdulmutallab wasn’t caused by the lack of a suspect—he had already been flagged—but by a failure to investigate him fully. So amassing more data about innocent people wasn’t necessarily the right solution.

The most sensitive Homeland Security data trove at stake was the Advanced Passenger Information System. It contains the name, gender, birth date and travel information for every airline passenger entering the U.S. Previously, Homeland Security had pledged to keep passenger data only for 12 months. But NCTC was proposing to copy and keep it for up to five years. Ms. Callahan argued this would break promises the agency had made to the public about its use of personal data, these people said. Discussions sometimes got testy, according to emails reviewed by the Journal. In one case, Ms. Callahan sent an email complaining that “examples” provided to her by an unnamed intelligence official were “complete non-sequiturs” and “non-responsive.”

In May 2011, Ms. Callahan and Ms. Schlanger raised their concerns with the chief of their agency, Janet Napolitano. They fired off a memo under the longwinded title, “How Best to Express the Department’s Privacy and Civil Liberties Concerns over Draft Guidelines Proposed by the Office of the Director of National Intelligence and the National Counterterrorism Center,” according to an email obtained through the Freedom of Information Act. The contents of the memo, which appears to run several pages, were redacted. The two also kept pushing the NCTC officials to justify why they couldn’t search for terrorism clues less invasively, these people said. “I’m not sure I’m totally prepared with the firestorm we’re about to create,” Ms. Schlanger emailed Ms. Callahan in November, referring to the fact that the two wanted more privacy protections. Ms. Schlanger returned to her faculty position at the University of Michigan Law School soon after but remains an adviser to Homeland Security. To resolve the issue, Homeland Security’s deputy secretary, Jane Holl Lute, requested the March meeting at the White House. The second in command from Homeland Security, the Justice Department, the FBI, NCTC and the office of the director of national intelligence sat at the small conference table. Normal protocol for such meeting is for staffers such as Ms. Callahan to sit against the walls of the room and keep silent. By this point, Ms. Libin’s concern that innocent people could be inadvertently targeted had been largely overruled at the Department of Justice, these people said. Colleagues there were more concerned about missing the next terrorist threat. That left Ms. Callahan as the most prominent opponent of the proposed changes. In an unusual move, Ms. Lute asked Ms. Callahan to speak about Homeland Security’s privacy concerns. Ms. Callahan argued that the rules would constitute a “sea change” because, whenever citizens interact with the government, the first question asked will be, are they a terrorist? Mr. Brennan considered the arguments. And within a few days, the attorney general, Eric Holder, had signed the new guidelines. The Justice Department declined to comment about the debate over the guidelines. Under the new rules, every federal agency must negotiate terms under which it would hand over databases to NCTC. This year, Ms. Callahan left Homeland Security for private practice, and Ms. Libin left the Justice Department to join a private firm.

Homeland Security is currently working out the details to give the NCTC three data sets—the airline-passenger database known as APIS; another airline-passenger database containing information about non-U.S. citizen visitors to the U.S.; and a database about people seeking refugee asylum. It previously agreed to share databases containing information about foreign-exchange students and visa applications. Once the terms are set, Homeland Security is likely to post a notice in the Federal Register. The public can submit comments to the Federal Register about proposed changes, although Homeland Security isn’t required to make changes based on the comments.