For 17 years, Bob Gellman sat in the U.S. House of Representatives keeping tabs on what the federal government did with the information it collected on Americans. This was back in the days when data were stored on magnetic tapes and punch cards, or literally warehoused as paper records, and processed—if at all—by hulking mainframe computers. But even then, there was enough of it piling up in government’s hands to trigger worries about citizens' privacy.

It was 1977, and Gellman, just four years out of Yale Law School, joined the committee with oversight of the federal government.A relatively new privacy law had set rules about how Washington used personal data, from who could look at Medicare beneficiaries’ files to what the proper uses were for passport records; Gellman’s job was to review the privacy plans agencies were required to submit. For all the rules, says Gellman now, “privacy” often amounted to a leap of faith in the federal employees who had access to the information. “They have to have some degree of discretion—otherwise the world doesn’t work,” he says. “You hope everyone does the right thing, whatever that happens to mean to them. Welcome to the world. Welcome to Washington.”

Today, the government has a great deal more data on Americans than it did during Gellman’s time on the Hill, housed in huge numbers of computers networked to one another, accessible with a few keystrokes and vulnerable to hacking threats unimaginable 40 years ago. “Every agency, depending on its mission, has these great stores of data,” says Gregory Wilshusen, director of information security issues at the Government Accountability Office. “And some of it – maybe even much of it – is very sensitive.”

The federal portrait of Americans’ lives varies in detail depending on how much you interact with the government, but can be quite intimate. The IRS has details of your income from year to year; the Department of Education knows how well you’ve kept up with your federal student loan payments. If you’ve served in the military, your fingerprints are probably on record; and if you’ve ever been in prison, there’s a chance your tattoo is filed away in a federal database. If you've ever applied for a security clearance, then the portrait likely extends beyond you to your friends and family, plus their friends and family. And a “death master file” under lock and key with the Social Security Administration keeps tabs on the moment you stop generating data.

When Gellman was overseeing it, that data tended to live in many silos, and structurally it still does -- with two dozen major agencies and countless subprograms collecting and storing information. The trend in recent decades, though, has been for the federal government to collaborate internally when it comes to bridging those silos, leading computer scientists to sound the alarm about what they call “the mosaic effect” – that is, the risk that all this data, when combined, could paint a far more complete picture than any of us knew we were allowing.

Ed Felten, a decorated Princeton computer scientist who served as deputy chief technology officer of the United States in the Obama White House, has explained it this way: "One file might contain detailed information about behavior and another might contain precise identity information. Merging those files links behavior and identity together." And even if the blended data doesn’t contain a name or Social Security number, the image that comes into focus can quickly be so specific to plausibly belong to only one person, or a handful of people. Currently, the Justice Department and the Department of Housing and Urban Development have a high-level agreement to allow that sort of matching. So do the IRS and the Social Security Administration, Health and Human Services and the Defense Department, and the list continues.

If the potential for knitting all this together is a fresh concern, the underlying worry about its enormous power goes back decades. In 1973, in a country rattled by Watergate and the accompanying disclosures about the willingness of Uncle Sam to dig into the backgrounds of President Richard Nixon’s so-called enemies, the then-Department of Health, Education and Welfare issued a report warning of the emergence of a new class of “technicians as record keepers” in whose hands our information was kept. Often these bureaucrats, HEW argued, were remote from both the Americans whose records they collected and those who would eventually make use of the data once it was passed along a federal daisy chain. That setup, HEW officials worried, encouraged a “’dragnet’ behavior” in which bureaucrats would feel free to dig into the private lives of Americans who’d done nothing wrong.

Senator Samuel J. Ervin J, seen here presiding over a Senate Subcommittee meeting, was a leading proponent of the Privacy Act of 1974, one of the first laws to limit the data the government collects on individuals. | Stan Wayman/The LIFE Picture Collection/Getty Images

With the HEW report as the backdrop, 77-year-old Senator Sam Ervin – a North Carolina Democrat who, however improbably, had made a name for himself as both a segregationist and civil libertarian – thundered in a speech in the Senate in June 1974 about the risk to Americans. “When [the] quite natural tendency of government to acquire and keep and share information about citizens is enhanced by computer technology and when it is subjected to the unrestrained motives of countless political administrators,” railed Ervin, “the resulting threat to individual privacy makes it necessary for Congress to reaffirm the principle of limited, responsive government on behalf of freedom.”

At Ervin’s urging, Congress scrambled to pass the Privacy Act, meant to both limit the government’s ability to collect information on citizens and boost the rights of those Americans to understand what was being collected. Five months after Nixon resigned, President Gerald Ford signed the bill into law, calling it “a major first step in safeguarding individual privacy.” But from Day One, say experts, the law has been at worst a mess, and at best a beast to interpret and enforce. Federal employees have discretion to share whatever data they might have with colleagues in their home agency, if necessary for work. How an agency might share information beyond its walls, including with other agencies, has to be detailed in the Federal Register—but those notices are little read, often out of date and routinely full of loopholes. Lawsuits have been brought again and again to test the edges of the Privacy Act. In one 1980 case, a senior manager at the IRS was found to have improperly handed in-house bond salesmen a list of employees who’d failed to purchase U.S. savings bonds, despite Nixon’s call for federal workers to be on the front lines of the bond-buying movement. But, in practice — what data get shared and how — is decided at the desks of Ervin’s “countless political administrators,” with widely varying oversight. This can limit the use of data, as well as promote it, says Gellman: "Somebody can say to somebody in another office who wants a record, 'I'm not going to give it to you.’”

The attacks of Sept. 11, 2001, marked a major shift in the relationship between the federal government and so-called Big Data. Scores of federal workers worried that an information deficit had left the country vulnerable to that day’s horrific attacks. Partly in response, the Department of Homeland Security was conjured into existence in 2003, pulling together some 22 federal agencies and offices, in whole or part, some of them with wildly different missions. Mary Ellen Callahan was DHS’s chief privacy officer from 2009 to 2011, and said that a first step to preventing the sprawling new agency from stampeding all over privacy was limiting its appetite for data. “One of the things I always talked with my federal clients about was, ‘Just because it exists, doesn’t mean you should collect it,” says Callahan, “but the answer is always yes.”

Government agencies have more and more data at their fingertips, from biometric records to property deeds. TOP LEFT: A Customs and Border Protection Officer directs an international air traveler in a wheel chair how to use the biometric scanner. TOP RIGHT: An officer learns how to use an identification system based on iris recognition biometric technology. BOTTOM LEFT: An analyst with the FBI examines fingerprint comparisons in the National Instant Criminal Background Check System. BOTTOM RIGHT: These backup tapes for the National Instant Criminal Background Check System are part of the system that can perform eight million background checks per day. | Getty Images

In the 16 years since 9/11, Washington' data portrait of Americans has taken a new turn into biometrics, or the measurement of individuals’ unique biological characteristics, from irises to palm prints to facial recognition. In 2010, the FBI’s criminal justice information-services division began working on a “next generation identification” system to replace its outmoded fingerprint system. In May 2016, a GAO study found that some 30 million photographs of nearly 17 million people had been fed into NGI; about 70 percent were mugshots, with the rest pulled from sources like security-clearance applications and immigration records. The FBI updated its privacy policy only last year to account for this next generation information, even as it works in partnership on the project with states – including Michigan, New Mexico and Texas – and other agencies. One experiment has the FBI using State Department passport photos to hunt for wanted persons. (If you have a passport photo, your “face has been turned into an algorithm,” points out Clare Garvie of Georgetown law school’s Center on Privacy & Technology.) The worry among privacy advocates is that once bureaucrats have a face database, the temptation is to use it.

So what’s to prevent the government from linking your passport photo to your IRS records, and then stopping you at the airport on your way to spring break if you’ve dawdled in filing your taxes? Well, there’s one powerful counterbias that has arguably grown only more powerful as data become more crucial to government: The propensity of people who have data to hoard it.

In 2011, Alon Peled, a political scientist at the Hebrew University of Jerusalem, concluded in a paper that a top-down order by President Barack Obama to open up federal information caches to the public had generally failed to gain traction at least partly because “datasets are valuable assets which agencies labor hard to create, and use as bargaining chips in interagency trade, and are therefore reluctant to surrender these prized information assets for free.” When the FBI wanted to gather biometric data as part of its anti-terrorism efforts, Peled pointed out, the agency spent a full year trying to find someone inside the Defense Department willing to agree to share it with them.

So who’s watching out for us? In theory, one layer of protection is the privacy plans agencies are required to publish, but Gellman isn't especially optimistic about those: Today, he's a data-privacy consultant to federal agencies, and says agencies routinely detail in their stated privacy plans uses of data that are overly broad, unnecessary, or even illegal. “But nobody objects,” says Gellman, “because for a lot of these things, nobody cares."

On the other hand, agencies jealously guarding their own data might be our best hope. So perhaps there is maybe some reassurance to take from the fact that our data are protected by a force perhaps as powerful in Washington as it was back when Bob Gellman came to Congress: the bureaucratic turf war.