The National Security Agency has collected a vast number of digital photos from Internet traffic and the internal networks of foreign governments in order to identify and track persons of interest, according to a report by TheNew York Times. The images, reportedly extracted from Internet traffic such as e-mail messages and from video conferencing streams, have been used as part of the NSA’s “Identity Intelligence” (I2) program to “track, exploit, and identify targets of interest,” according to a 2011 NSA presentation slide.

According to the documents cited by the Times, the agency began performing facial recognition searches using captured images in 2010, matching photos in Pinwale (the NSA’s longterm store of captured content from external sources) and a terrorist watch list database called Tide. By 2011, the NSA was capturing millions of images daily—and of those, about 55,000 images are “facial recognition quality.” And the NSA expanded its collection and cross-referencing of facial images, pulling from CIA and State Department data from the border crossing stations of a number of countries, as well as airline passenger data and foreign national identity card databases. According to the NSA documents, in 2011 the agency was trying to gain access to the national identity card databases of Saudi Arabia, Pakistan, and Iran.

All of these sources can be used to help identify images mined with Wellspring, the NSA’s program that extracts images from Internet communications and calls out those that appear to be passport images or other ID photos. Indexes for images have been built using a combination of internally developed facial recognition software and technology from Pittsburgh Pattern Recognition (PittPatt)—a firm acquired by Google in 2011. Other software allows the NSA to match the details of outdoor photos with satellite and aerial imagery to pinpoint where the photos were taken.

The fishing’s getting (a little) harder

The NSA’s collection efforts here are focused on individuals and organizations that have been specifically “tasked” or on images collected from databases that have a high value as facial recognition targets. Collecting those images requires a lot more than just skimming Internet traffic.

When the NSA began its facial recognition efforts, the technology of facial recognition was still experiencing growing pains, but capturing images from Internet traffic was fairly easy, thanks to the lack of protection from encryption on many Internet services—and the NSA’s active efforts to gather data from within the networks of those services that did offer encryption for Web clients.

Of the major Web mail providers, only Google was providing SSL encryption at the beginning of 2010. Microsoft added SSL encryption to Hotmail in November of 2010. But SSL wasn’t even an option for Yahoo mail until early in 2013—and Yahoo didn’t turn it on by default until October of 2013.

As part of a collaborative project with National Public Radio and penetration testing provider Pwnie Express—the results of which we’ll publish later this week—we took a look at what sort of content could be fished out of Internet traffic from major Web and mobile services today. Yahoo, Google, Microsoft, Apple, and Facebook now all encrypt images and other content from servers to Web browsers—though there are some exceptions in the mobile realm.

Facebook, for example, encrypts all the images transmitted from its content delivery network to users’ browsers, though the images can still be reached through an unencrypted interface. During our testing, Pwnie Express founder and CTO Dave Porcello found that on an Android 4.1.1 “Jelly Bean” device—admittedly an older phone, but still in wide use—Facebook profile pictures and images were transmitted unencrypted to the Facebook app. Our tests on newer platforms found that the images were encrypted.

However, the NSA doesn't necessarily have to pull images from raw Internet traffic to build its database—particularly for individuals outside the US who are the subject of a Foreign Intelligence Surveillance Act (FISA) Court warrant. The NSA, through the FBI, could simply order service providers to hand over images associated with specific accounts. And a few overseas Web mail providers don't use encryption by default yet, leaving their services exposed to the passive capture of contents.

On the other side of the problem is facial recognition technology itself. The Times’ report indicates that, while the NSA had some successes with facial recognition early, the technology suffered from a high “false positive” rate. A 2011 presentation viewed by the Times showed that for a query using an image of Osama Bin Laden, the NSA’s system returned photos of four men who shared only one obvious facial characteristic with the Al Qaeda leader—a beard.

The problems with facial recognition technology based on ID card databases were demonstrated by the manhunt for the Boston Marathon bombers. While the technology has advanced recently, much of the recognition capability is dependent on the angle and quality of the two images. On many current systems, even matching photos taken from the same angle can be made more difficult by variations in lighting and resolution.

But faces aren’t the only piece of data that the NSA works with in correlating images with individuals. There’s a great deal of contextual data that can be collected along with images—especially those pulled from the databases of foreign governments and from e-mail—that can be used to identify individuals in them and help to build a library of related images that could generate more accurate results.

"NSA doesn't necessarily have to pull images from raw Internet traffic to build its database—particularly for individuals outside the US who are the subject of a Foreign Intelligence Surveillance Act (FISA) Court warrant."

Not only do these individuals in the pictures both have beards, but they're purported to have similar lives as well. It's said that Osama rarely ever left his compound for fear of being spotted by someone. And, well...

I'm unsure of what sort of intelligence the NSA hopes to gain from this sort of thing. What kind of terrorist would include pictures of himself in his communication?

Orwellian is becoming inadequate as a word to describe the NSA's activities:

- We're collecting data on who you call and who calls you.- We're recording a vast amount of your calls.- We're in your e-mail and your social media accounts.- We're intercepting your instant messenger chats and video streaming.- We're logged into your favorite online game and monitoring your chats there as well.- We're following you at work through infiltration of your corporate networks.- We're following you at home through all your personal accounts.- We're scanning everyone's internet traffic and can single your traffic out any time we want to.- We're developing malware, exploits and viruses to target you with if you ever become of interest to us.- We're weakening international crypto standards so have no defense against us.- We're spying on your democratically elected officials.

for a query using an image of Osama Bin Laden, the NSA’s system returned photos of four men who shared only one obvious facial characteristic with the Al Qaeda leader—a beard.

My limited understanding of facial recognition software is that it looks for specific features, and then compares the results at a higher level. I would expect the underlying algorithms to accept some kind of tuning or weighting parameters that allow the users to de-emphasize certain features (like beards).

It would be darkly humorous (and yet not surprising based on my experience with government software contractors) if their problems all boil down to designers and developers not providing a "settings" interface for this crucial ability, exacerbated by a typical lack of communication with the users to understand their actual needs.

I'm unsure of what sort of intelligence the NSA hopes to gain from this sort of thing. What kind of terrorist would include pictures of himself in his communication?

Orwellian is becoming inadequate as a word to describe the NSA's activities:

- We're collecting data on who you call and who calls you.- We're recording a vast amount of your calls.- We're in your e-mail and your social media accounts.- We're intercepting your instant messenger chats and video streaming.- We're logged into your favorite online game and monitoring your chats there as well.- We're following you at work through infiltration of your corporate networks.- We're following you at home through all your personal accounts.- We're scanning everyone's internet traffic and can single your traffic out any time we want to.- We're developing malware, exploits and viruses to target you with if you ever become of interest to us.- We're weakening international crypto standards so have no defense against us.- We're spying on your democratically elected officials.

Don't worry though. We're the good guys.

— and I have become disgusted and embarrassed by my country's actions. "Land of the free" my ass.

I'm unsure of what sort of intelligence the NSA hopes to gain from this sort of thing. What kind of terrorist would include pictures of himself in his communication?

Orwellian is becoming inadequate as a word to describe the NSA's activities:

- We're collecting data on who you call and who calls you.- We're recording a vast amount of your calls.- We're in your e-mail and your social media accounts.- We're intercepting your instant messenger chats and video streaming.- We're logged into your favorite online game and monitoring your chats there as well.- We're following you at work through infiltration of your corporate networks.- We're following you at home through all your personal accounts.- We're scanning everyone's internet traffic and can single your traffic out any time we want to.- We're developing malware, exploits and viruses to target you with if you ever become of interest to us.- We're weakening international crypto standards so have no defense against us.- We're spying on your democratically elected officials.

Don't worry though. We're the good guys.

— and I have become disgusted and embarrassed by my country's actions. "Land of the free" my ass.

I'm unsure of what sort of intelligence the NSA hopes to gain from this sort of thing. What kind of terrorist would include pictures of himself in his communication?

Orwellian is becoming inadequate as a word to describe the NSA's activities:

- We're collecting data on who you call and who calls you.- We're recording a vast amount of your calls.- We're in your e-mail and your social media accounts.- We're intercepting your instant messenger chats and video streaming.- We're logged into your favorite online game and monitoring your chats there as well.- We're following you at work through infiltration of your corporate networks.- We're following you at home through all your personal accounts.- We're scanning everyone's internet traffic and can single your traffic out any time we want to.- We're developing malware, exploits and viruses to target you with if you ever become of interest to us.- We're weakening international crypto standards so have no defense against us.- We're spying on your democratically elected officials.

Don't worry though. We're the good guys.

— and I have become disgusted and embarrassed by my country's actions. "Land of the free" my ass.

What? Nixon wasn't embarrassing enough?

As a veteran who has served his country, I am offended by the perversion of all that I had taken an oath to uphold, defend, and protect. There are many, cumulative embarrassments in every country's history; however, a pervasive system that is obliged to attribute guilt to its citizens (and every world citizen for that matter) without real, reasonable suspicion, jurisdiction, and without due process, is, perhaps, a far greater embarrassment than, yes, even Nixon.

I'm unsure of what sort of intelligence the NSA hopes to gain from this sort of thing. What kind of terrorist would include pictures of himself in his communication?

The kind that applies for a passport to travel to his target destination (and such an application could include all kinds of info, phone, email etc.). The ammount of personal information people choose to include in digital communications is kind of scary.

The good news: facial recognition still kind of sucksThe bad news: when it is good enough to be genuinely scary, "reforms" will already have passed and most folks won't pay as much attention to this news anymore.

"NSA doesn't necessarily have to pull images from raw Internet traffic to build its database—particularly for individuals outside the US who are the subject of a Foreign Intelligence Surveillance Act (FISA) Court warrant."

Why would NSA need a warrant to spy on foreign nationals?

Everyone knows those poor bastards do not have any rights

/s

This really hits home as the general moral weight of the constitution leans toward human rights or rights that are uninfringable, but our govenment and some fascist citizens don't see it that way. They think that rights should only be had by themselves or their own group, when really human rights and freedom should not be infringed upon by anyone, anywhere. Our own govenment has been continually degrading our rights for about 100 years now. People just accept being forced to pay retarded income taxes when major corporations pay zero. People just accept propety tax like the govenment has some magical right to our land People just accept being forced to pay for insurance that is not needed. It's stupid and nobody cares. Hell. I bet half of you have an investment with fidelity, which invests your money into our private prison complex, which lobbies for stricter punishments for non violent "crimes". EG: Spitting your gum on the ground can get you jail time in places with strict anti littering laws... sound like some other olace you know? (coughkoreacough). Pretty nazi faciest commie if yoy ask me. but hey. what the hell do i know?

the NSA’s “Identity Intelligence” (I2) program to “track, exploit, and identify targets of interest,”

I got a chuckle out of the order of their 3 objectives. First we'll track you, then we'll exploit you, and then when it's all over we'll identify if you were a target of interest that we should have been tracking and exploiting in the first place.

"NSA doesn't necessarily have to pull images from raw Internet traffic to build its database—particularly for individuals outside the US who are the subject of a Foreign Intelligence Surveillance Act (FISA) Court warrant."

Why would NSA need a warrant to spy on foreign nationals?

Everyone knows those poor bastards do not have any rights

/s

This really hits home as the general moral weight of the constitution leans toward human rights or rights that are uninfringable, but our govenment and some fascist citizens don't see it that way. They think that rights should only be had by themselves or their own group, when really human rights and freedom should not be infringed upon by anyone, anywhere. Our own govenment has been continually degrading our rights for about 100 years now. People just accept being forced to pay retarded income taxes when major corporations pay zero. People just accept propety tax like the govenment has some magical right to our land People just accept being forced to pay for insurance that is not needed. It's stupid and nobody cares. Hell. I bet half of you have an investment with fidelity, which invests your money into our private prison complex, which lobbies for stricter punishments for non violent "crimes". EG: Spitting your gum on the ground can get you jail time in places with strict anti littering laws... sound like some other olace you know? (coughkoreacough). Pretty nazi faciest commie if yoy ask me. but hey. what the hell do i know?

This message makes me embarrased to share your position on the NSA, because your entire worldview is skewed.

Quote:

People just accept being forced to pay retarded income taxes when major corporations pay zero.

Yes, the tax code needs to be stricter on corporate taxes. Money you store overseas shouldn't be allowed to be used in any form without taxes being paid. The only issue is rules need to be written so that they don't disproportionately screw the smaller businessed that keep their cash overseas because they're located overseas. You probably won't believe it, but the reason the tax laws aren't rewritten is more because of the difficulty of coming up with an equitable solution. Monetary contributions might make politicians think it's not worth the effort, but they're not the sole reason why nothing's being done. Contrary to popular belief, politicians generally don't make decisions while looking at their donation sheet. It's all through personal relationships. Does one follow the other? Yes, of course. But that doesn't necessarily mean "we have the finest leaders money can buy."

As to "ridiculous income taxes," the U.S. has one of the lowest effective income tax rates in the world. Why? Because the U.S. government provides less services than many other governments. Yeah, there's a significant amount of waste, but that's going to be there whether corporations pay their fair share or not.

Quote:

People just accept propety tax like the govenment has some magical right to our land

Do you live in a city or town? Do they provide services? Where do you think that money comes from? Property taxes are collected entirely at the local level. If you live in the town, you're entitled to that town's schools, that town's snowplows, that town's water and sewer infrastructure, and so on. You should be entitled to that town's phone and internet infrastructure, but unfortunately that's a different debate. It's not the government's right to your land, it's your responsibility as a citizen to provide for your city's services.

Quote:

People just accept being forced to pay for insurance that is not needed. It's stupid and nobody cares.

Yeah, everyone just rolled over and accepted that one.

Oh wait, no, people fought tooth and nail to preserve our ridiculously low income tax rates by keeping the services provided by government to a minimum.

Or maybe you're referring to things like car insurance, which everyone's forced to buy so that you don't go bankrupt paying for someone else's bodywork out of pocket. I've yet to meet anyone who hasn't made use of their insurance sometime in the past decade, but if it's really useless, you can buy a barebones plan for not all that much. The insurance market is reasonably competitive by me.

Quote:

Hell. I bet half of you have an investment with fidelity, which invests your money into our private prison complex, which lobbies for stricter punishments for non violent "crimes". EG: Spitting your gum on the ground can get you jail time in places with strict anti littering laws

You're right, we do need to reform incarceration, even though FIdelity doesn't just dump your money into private prisons. Nothing like a gross oversimplification leading into some fearmongering that we're turning into Singapore.

Quote:

sound like some other olace you know? (coughkoreacough).

Yeah...Singapore. Believe it or not, when you do a U.S./North Korea analysis, there are bigger differences that come up.

Quote:

Pretty nazi faciest commie if yoy ask me. but hey. what the hell do i know?

Sean Gallagher / Sean is Ars Technica's IT Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland.