Yes, All Phone Calls are Recorded

Glenn Greenwald asks the question “Are all telephone calls recorded and accessible to the US government?” He knows the answer already and the answer is yes. We’ve known this for years. He spotlights this admission of that by a former FBI counterterrorism agent on CNN:

BURNETT: Tim, is there any way, obviously, there is a voice mail they can try to get the phone companies to give that up at this point. It’s not a voice mail. It’s just a conversation. There’s no way they actually can find out what happened, right, unless she tells them?

CLEMENTE: “No, there is a way. We certainly have ways in national security investigations to find out exactly what was said in that conversation. It’s not necessarily something that the FBI is going to want to present in court, but it may help lead the investigation and/or lead to questioning of her. We certainly can find that out.

BURNETT: “So they can actually get that? People are saying, look, that is incredible.

CLEMENTE: “No, welcome to America. All of that stuff is being captured as we speak whether we know it or like it or not.”

But this is not the first time this has come out. Former AT&T technician Mark Klein revealed this years ago in his book Wiring Up The Big Brother Machine…And Fighting It. He actually saw the installation of equipment that allows the NSA to read and hear every electronic communication in the country. And NSA whistleblower William Binney has previously revealed that the federal government literally has trillions of electronic communications stored from that data mining process.

All without a warrant. All with no safeguards whatsoever. All with total secrecy and immunity from legal challenge because of the State Secrets Privilege, the abuse of the standing doctrine and all the other ways the Bush and Obama administrations have ensured that the constitution does not apply to the executive branch. Welcome to the National Security State, citizen.

After spending several years touring the country as a stand up comedian, Ed Brayton tired of explaining his jokes to small groups of dazed illiterates and turned to writing as the most common outlet for the voices in his head. He has appeared on the Rachel Maddow Show and the Thom Hartmann Show, and is almost certain that he is the only person ever to make fun of Chuck Norris on C-SPAN.

What kind of data storage is needed for that? How long are the communications kept for? It has to be immense.

http://www.facebook.com/den.wilson d.c.wilson

Synfandel:

Text messages are automatically recorded, too.

Synfandel

d.c.wilson:

They’re encrypted.

baal

I, for one, would welcome lizard people as overlords if it meant we could jettison the National Security State.

http://timgueguen.blogspot.com timgueguen

(clueless person mode on)

But if you’re not guilty of something you have nothing to hide.

(clueless person mode off)

That kind of thinking is comforting, until someone’s search parameters decide you just might be involved with something they think is bad, because you use the word Columbia in too many conversations, or some other word or phrase that connects to their current target. Or you knew the wrong guy in high school.

jamessweet

So, what are they actually doing with it? Getting anything useful out of this mountain of data seems like quite a trick.

This particular data center is supposed to be up later this year, but it certainly isn’t the only data center at this point. This will be the big aggregator though.

asonge

Synfandel:

Under pressure from new markets like India and Saudi Arabia, BB’s text message encryption can be bypassed by host governments.

DaveL

Just making a rough estimate:

Let’s say there are 250 million mobile users in the U.S. Each uses on average 400 minutes/ month. Let’s allow a bandwidth of 6kbps for voice data communications (on the low side, but some codecs go lower). That’s 3.6 * 10^16 bits or 4.5 petabytes per month. Storing 10 years’ worth of conversations would be 540Pb. Using commonly available 2TB drives, that works out to 270,000 drives.

http://timgueguen.blogspot.com timgueguen

There’s also the obvious question of what level of encryption is crackable by the NSA and allied agencies, and how long it might take. Is there in fact encryption that can’t be cracked with enough time and omputer power

jaxkayaker

Good thing we live in the free U.S., where rights are respected and government is constrained by the constitution, not in a commie country like Russia or China.

DaveL

So, what are they actually doing with it? Getting anything useful out of this mountain of data seems like quite a trick.

Well, that’s the thing, isn’t it? Despite all their vaunted data mining algorithms, we see things like the radicalization of the Boston Bombers slip through the cracks all the time. I’d say its value for surveillance would have to be classified as questionable.

Building a case against a specific target, on the other hand, is another matter.

outraged

It doesn’t matter what they do with it in the short run, or even if they can find the data that they are looking for. What it IS good for is that once they know that you, Bob Smith of Somewhere, IA are becoming something of a pain in the butt with your letters to the editor, or perhaps your run for Congressman is besting the D or R’s chosen candidate. Then, all they have to do is type in your name and location to retrieve years worth of calls, emails, web sites and the like. What are the chances that the local press wouldn’t appreciate your penchant for tranny porn or the off color jokes you share with your friends?

Combine that with license plate scanners and the camera’s scattered all over the place, along with the commercial data bases available and your entire life is wide open for anyone with access. Perhaps that’s why so many politicians vote the way the do already?

unbound

@DaveL – You may want to read the Wired article I linked earlier. The data center that is nearing completion is looking to have storage capabilities measured in the yottabytes.

For those not aware the progression is mega to giga to tera to peta to exa to zetta to yotta.

DaveL

Is there in fact encryption that can’t be cracked with enough time and omputer power

The one-time-pad cipher. But that’s just against “time and computer power”. I can’t think of any cryptosystem that’s secure against any amount of bribery, thuggery, or social engineering.

DaveL

@15

I never said it was impossible. In fact for the U.S. government or any organization of similar size and resources* I think it’s entirely plausible.

*Is there another organization of similar size and resources? The Chinese government perhaps?

http://timgueguen.blogspot.com timgueguen

Yeah, one time pads aren’t crackable if you use them properly. I seem to remember there were at least a couple of Cold War examples where they were reused, and hence allowed at least some of the messages to be deciphered or partially deciphered.

http://en.uncyclopedia.co/wiki/User:Modusoperandi Modusoperandi

I, for one, am glad that the State is observing and recording me, since I am a patriot.

jamessweet

Yeah, one-time pads are completely and totally unbeatable — and also they are impractical for most day-to-day purposes, and extremely vulnerable to meatspace attacks (like, um, obtaining the pad). And timgueguen is correct, historically people have not always used one-time pads correctly, allowing cryptographic attacks.

Synfandel

The difficulty of cracking the Advanced Encryption Security (AES) on iPhones and BlackBerrys is the primary reason for recording as much conversation as possible. According to James Bamford, of Wired magazine,

Breaking into those complex mathematical shells like the AES is one of the key reasons for the construction going on in Bluffdale. That kind of cryptanalysis requires two major ingredients: super-fast computers to conduct brute-force attacks on encrypted messages and a massive number of those messages for the computers to analyze. The more messages from a given target, the more likely it is for the computers to detect telltale patterns, and Bluffdale will be able to hold a great many messages.

TGAP Dad

Thing is, the NSA, which lacks any law enforcement role, has no disincentive since the exclusionary rule is meaningless to them as the evidence will never be considered in a court of law.

http://www.ranum.com Marcus Ranum

Breaking into those complex mathematical shells like the AES is one of the key reasons for the construction going on in Bluffdale.

That’s not true. The keys are usually exchanged at the head of the session, using a public key certificate. The design of the way SSL is used on the internet is suspiciously mediocre and is susceptible to a variety of attacks – including some rather obvious ones involving compromising the certificate authorities themselves. There’s no need to crack AES at all, if you can compromise the key exchange and that’s been done over and over again.

http://www.ranum.com Marcus Ranum

Yeah, one time pads aren’t crackable if you use them properly. I seem to remember there were at least a couple of Cold War examples where they were reused, and hence allowed at least some of the messages to be deciphered or partially deciphered.

VENONA was the codename of that series of intercepts. They have now been declassified. The story of VENONA is interesting history indeed.

BTW, the police state is nothing new. Read Tim Wiener’s brilliant history of the FBI “Enemies” and you’ll see that the FBI and national security apparatus have always spied on citizens and always will.

I, for one, would welcome lizard people as overlords if it meant we could jettison the National Security State.

hahahahahah.

I only talked to a preying mantis drone, it abducted me for past two years, there are no lizard people: that’s just a joke. It’s only robotic drones as they are the only ones that can survive the amount of geforces put on their aircraft when they travel. I’m not sure what they’re made of but it’s some kind of super refined metal looking substance. Anyway, ‘ole Zorack is an asshole that’s more interested in trivial things. It makes me think the drone is just some criminal warrior that got court martialed and kicked out of his solar system, and has nothing better to do that wander the universe.

In summary, there are no biological aliens I am aware of, and the AI drones from other places, well no one knows where they are from or why they are here, and the one that abducted me anyway is an evil psychopath and has ruthlessly traumatized me.

In summary #2 Reptiles from Sirius is a bad joke, but at this point, if there really were Reptiles from Sirius, it would be totally ok if like they just shapeshifted to look like our current rulers and put all the rulers we have in a pod and launched it towards the sun…..

So back on main topic, our country is the most thorough and comprehensive police state and prison state in the history of humanity, and we crow moronically about freedom and liberty. People will say “but we are sophisticated and we don’t allow government to install backdoors in encrypted communicationns!” and how many people use encryption if it’s not PKI infrastructure keys in HTTPS websites? Like what, 5%? That means government has like 95% of peoples’ communications.

I am almost of the mind that the FBI and NSA agents claiming government collects all information as alleged in these articles are running a psychological operation on this society, hoping to force people AWAY from more advanced technology and into far more restrictive, decades or even centuries old technology, in order to communicate sensitive effort, the political result being to prevent revolution.

I know if I was ruling this country and I was scared of being overthrown I would tell everyone my intelligence agencies already recorded everything they said over the phone or email anyway: because people will believe it, get scared, and then start using pen and paper and lemon juice with simple ciphers thinking they’re badass. The speed of their messages will be drastically reduced while they’re busy thinking they’re living in the good ‘ole days with walkie talkies stayin low tech out of gubbmint’s eyes!!

It’s a psy op imo: I don’t believe for one second that the NSA or the FBI or any of these alphabet soup agencies, chock full to the brim with social conservative assholes, has the power to do this. Sure, we ARE the most comprehensive and advanced prison and police state in the history of humanity, by far even, but not this far…I don’t believe this, this is a psychological operaation from a government that’s scared to get overthrown given the polls are showing 30% of the population believes armed revolution is the only answer towards a better political future. The government wants the revolutionary opposition communicating on walkie talkies and paper letters because then it’s super easy to own them in real time due to superior technology on the internet.

It’s fear mongering. People suck it up guillably because some ex-NSA agent goes to reveal this stuff with a scared look on their face “Oh I’m NOT SUPPOSED TO SAY THIS, BUT I’LL HUSH HUSH IT TO A JOURNALIST AND GIVE THEM MY NAME AND FACE AND MAGICALLY EVERYONE ELSE IN GOVERNMENT WILL JUST BE LUVVY DUVVY AND UNDERSTANDING OF THE FACT I JUST LEAKED CLASSIFIED INFORMATION AND WON’T GO AFTER MEEEEE”. Most people who leak stuff that sounds too good to be true, and put their face and name on tv like that NSA agent did in last year or two, they’re bullshit artists trying to psy-op people. I don’t believe it for one second. Most ‘whistleblowers’ who show their faces are just bullshit artists, Breanna Manning’s of the world are very rare indeed.

Abby Normal

Every Christmas I pick-up the phone and say something along the lines of, “Jihad, bomb, president… I just wanted to wish my big brother a merry Christmas.”

Want a little more justifiable paranoia? Consider that many phones can have their microphone remotely and covertly activated. Cell phones have gotten some attention for this, but I understand it can be achieved with most land-line phones as well. Phone companies, the NSA, and skilled hackers effectively have bugs in every home, office, car, pocket or purse with a phone.

From a legal perspective, a warrant has been required since 1988. But we see in what regard the NSA holds such requirements.

http://www.ranum.com Marcus Ranum

Getting anything useful out of this mountain of data seems like quite a trick.

Actually, my opinion is not ignorant, in the slightest in fact: I watched the entire inteview given with the NSA agent that came out in like last year or two alleging they could record everything. I also read many articles on it. People like you say it’s ignorant because to admit it is a psy-op that’s fooled 80+% of the American public is to admit you’ve been had.

Every Christmas I pick-up the phone and say something along the lines of, “Jihad, bomb, president… I just wanted to wish my big brother a merry Christmas.”

Want a little more justifiable paranoia? Consider that many phones can have their microphone remotely and covertly activated. Cell phones have gotten some attention for this, but I understand it can be achieved with most land-line phones as well. Phone companies, the NSA, and skilled hackers effectively have bugs in every home, office, car, pocket or purse with a phone.

From a legal perspective, a warrant has been required since 1988. But we see in what regard the NSA holds such requirements.

”

What you actually wrote about effectively having bugs in every home, office, car, pocket, or purse, THAT’S actually true, legit information, and one of many reasons we are the most comprehensive surveillance and police state society every constructed. That part is true, very true in fact, and yes, scary.

Well you know something, I have something else to say to this too, which is this: the counter-intelligence spooks that spread this BS are probably laughing their butts off while alex Jones devotees try to wage a libertarian revolution on walkie talkies and sending simple ciphers through mail thinking they’re badass. The spooks are laughing their buns off because they figured out 100 years ago how to tap walkie talkie and many older forms of communication…roflmao, so they are actually listening into all their conversations-and good thing for us, who wants Alex Jones and Ayn Rand assholes overthrowing government, no matter how bad it is right now, those idiots are 20 times worse.

Also zug :+)

I like saying cute things, anyway.

http://en.uncyclopedia.co/wiki/User:Modusoperandi Modusoperandi

Marcus Ranum, searching for tasty brownie recipes isn’t the same as searching for people who are thinking about or planning to commit acts of terrorism.

With Google you’re searching for things that have already happened (“Who won the Indy 500 in 1971?”, for example). With, say, anti-terrorism, you’re trying to connect dots that aren’t connected yet (“Montana, cabin, recluse, package”). Like the cameras in Boston (or the UK, for that matter), gathering everything is more useful after the fact than before it.

http://anexerciseinfutility.blogspot.com tommykey

It’s a good thing we have so many armed citizens who are stopping the government from doing this to us.

http://thebronzeblog.wordpress.com/ Bronze Dog

Storing all the phone conversations is currently scary since they can grief anyone they don’t like by digging up dirt in their recorded phone calls. It’s going to get worse as computerized interpretation gets better, since then they’ll be able to do the reverse, finding people to hate based on their searches for dirt, particularly harmless or universal dirt.

http://www.ranum.com Marcus Ranum

People like you say it’s ignorant because to admit it is a psy-op that’s fooled 80+% of the American public is to admit you’ve been had.

You have no idea what “people like” me, are. More ignorance.

I’m not going to bother trying to educate you; educate yourself. But the short form is that I’ve worked with computers all my life and (as I pointed out to the gentleman above that thinks the NSA’s capabilities for sorting through data are implausible: Google works) and I understand the technology quite well. I’m also fairly familiar with NSA’s capabilities and have worked for/with some of the contractors that produce the systems we’re talking about. In other words, I actually do know a bit about the systems under discussion – unlike you who are watching some television clip about a whistle-blower and jumping to absurd conclusions. The descriptions of the Thin Thread system are credible, and if you do a literature search for “Semantic forests” that’s the basic algorithms that are used for massive search-clustering. It’s not only plausible, it’s straightforward since it parallelizes nicely. Again, Google demonstrates a similar capability.

When you think about yottabytes, the yottabytes are not searchable data. The yottabytes are the backend store of archival data; the actual searchable parts are subsets of the data themselves. This is (again) how Google works – the indexes are map/reduced pointers into a large cache which is, in the case of Google, backed by the entire internet as its archive. Since the NSA is not dealing with the internet as a whole, they have to keep their own copy of the archive – hence the yottabytes. Anyone with beginner-level familiarity in databases ought to immediately understand that your primary index is vastly smaller and more searchable than your underlying archive. Add to that an understanding of how heirarchical storage systems work (RAM cache -> SSD -> RAID arrays -> tape robots) the tape robot systems can store, basically, an infinite amount, as long as you can keep upgrading the tape drives without changing the cartridge format. Have you even heard of Amazon’s elastic cloud storage? How do you think systems like that work? And, did you notice that they do, in fact, work?

It’s cute and “edgy” make farcical claims like that it’s all a scam, but it’s merely cute and edgy and is no substitute for actually knowing what you’re talking about.

Ben P

So, what are they actually doing with it? Getting anything useful out of this mountain of data seems like quite a trick.

Well, if the details of conspiracy rumors are to be believed, they don’t precisely store it all. Rather, they have in place automated systems that scan the vast bulk of electronic communications and flag those that meet certain parameters. Those then get closer automated, and eventually human looks.

There’s also the obvious question of what level of encryption is crackable by the NSA and allied agencies, and how long it might take. Is there in fact encryption that can’t be cracked with enough time and omputer power

As some people said above, one time pads are unbreakable, and certain techniques like quantum cryptography utilize these for ultra-secure transmissions.

But I think this is the wrong question.

Devoting time and computer power to breaking encryption is something you only do if you have a good reason. Suppose you use a commercially available system like PGP (Pretty Good Privacy). I have little doubt that the NSA has the techniques and computer power to accomplish this in short order.

On the other hand, the local police, and even the state police and FBI, to some extent, do not have these things. (the FBI might, but I bet that lab is pretty busy with high value cases). In all likelihood, the NSA is not terribly concerned about your individual emails, text messages and phone calls. (Unless you are, in fact, a terrorist mastermind), they sweep with a broad net, not a fine one.

So if you use something like PGP (including encryption apps which can be used on phones), you’re protected from any casual looks, and probably from 98% of the people who want to take a serious look. If the NSA wants to take a serious look at you, bets are off.

http://www.ranum.com Marcus Ranum

Marcus Ranum, searching for tasty brownie recipes isn’t the same as searching for people who are thinking about or planning to commit acts of terrorism.

I know that.

It’s the same problem we had with intrusion detection systems back in the late 1990s – you can detect “stupid” (i.e.: anyone who is dumb enough to text a friend “let’s make a bomb, yo!”) but you can filter that stuff out pretty easily by clustering. Again: what do you think Google’s page-ranking algorithms are but a great big relevance clustering system? I’m not saying that the NSA’s systems will work exactly the same way as Google’s search engine does – they work more like the page-weighting system that collects data and builds it into the index. And if you stretch your imagination a little bit you might realize it’s not a far step to take your weighting process and add a proactive alert/analysis step into the loop. Another thing Google shows us is that when you’re looking at massive-scale data, “close” is usually good enough, especially over time.

With Google you’re searching for things that have already happened (“Who won the Indy 500 in 1971?”, for example). With, say, anti-terrorism, you’re trying to connect dots that aren’t connected yet

You’re close but I don’t think you understand yet: we’re talking about a data collection system. The data, since it is being collected, is “things that have already happened.” That phone call? It already happened. That web browsing? It already happened. Now, the intelligence analysis process involves extrapolating from “what has already happened” to “what might happen.” That’s not something computers do particularly well but again, if you think about it for a few seconds, you’ll realize that that’s what Google’s predictive ad system is trying to accomplish, in its own slightly different way: Hey this guy has been searching for pink ruffly shirts, other people who bought pink ruffly shirts also bought bolero pants and others bought tux pants. Amazon’s customer-based advertising system is an example of the kinds of things you can do with data clustering when you have massive amounts of data to cluster. A system for clustering activities by dangerousness is exactly what they are building. It’s not only possible, it’s “simply” a problem of parallelizing and storing and racking up lots of computers.

Bronze dog writes:

It’s going to get worse as computerized interpretation gets better, since then they’ll be able to do the reverse, finding people to hate based on their searches for dirt, particularly harmless or universal dirt.

Well, things like speech->text (for the cell phone conversations) are probably already good enough. Salt that with other data they collect like location codes, phone owner information, and past history and you’ve got enough already to make some very very interesting correlations. Remember: that kind of analysis is part of the input loop, not a query done afterward. So you can parallelize it basically infinitely. So if you have a bank of computers sitting there turning cell phone calls into text, they’re going to also extract and categorize things like: language, possible gender of speakers (based on pitch), keywords, duration, location of call origin and target – all of which can be done (except that last bit) without need to do queries of other databases.

Look what happened to General Petraeus. How did the FBI get access to all his emails after they were deleted? Where did that stuff come from? Here’s how that kind of thing goes down: someone gets a suspicion, queues up someone for deeper analysis, all the data relevant to certain search critera gets retrieved, and is presented linked together on relevant terms. If you’re familiar with a threaded mail reader, you can see exactly how that works.

Oh, and look up a product called “Palantir” it’s selling like hotcakes in the DC beltway corridor.

http://twitter.com/meursalt meursalt

Listen to Marcus Ranum. He’s a bit of an authority on this stuff – I remember him being active on quite a few of the security listserv’s I was lurking in the 90’s. The rest of us, myself included, are not in his league on this particular topic.

Seriously, at the very least the NSA had this interception capability long before 2001, via the Echelon program. That was the part that was public knowledge; you can always bet they have more up their sleeve. People in computer security circles knew about this, but if we brought it up in polite society, we were dismissed as conspiracy nuts.

Pre-9/11, Echelon was supposedly only intercepting international communications involving non-US citizens. I never believe that, primarily because of the factor of human laziness. It would be a non-trivial task to identify which communications were OK to intercept, and which would clearly fall afoul of the Fourth Amendment. It would be much more feasible to intercept and store everything, and be (in a best-case scenario) on the honour system not to peek at the communications of citizens. Personally, I never particularly trusted the honour system in this. Pre-9/11, this signals intelligence was probably not shared with the FBI.

Post 9/11, there was a push for the ability for greater intelligence sharing between the FBI and intelligence agencies. This was when they started admitting to doing what any reasonable person who knew about Echelon had surmised they were doing all along – capturing everything from everybody, and sharing among agencies when deemed appropriate.

It’s scary stuff, and I don’t even like thinking about it. I’m always afraid I’ll find myself on some watchlist just by virtue of trying to educate people about it, so I usually keep my mouth shut on the topic. I suppose in doing so, I’m part of the problem.

I’d like to stress that I’m no expert: all of the above is based on inductive reasoning and the very general details that have been made public over the years.

Schneier’s primary expertise is in the mathematics of encryption and the practical application of said mathematics in software algorithms. I don’t think he’s in a position to speak authoritatively on what processing and storage facilities the U.S. intelligence community does or does not have. In this case, I suspect he’s being overly conservative in his estimate of their abilities. Also note that he qualifies his statements, making the point that the task of storage becomes much easier if audio communications are converted to text prior to long-term storage.

When making assumptions about the capabilities of these bastards, I prefer to err on the side of caution. Let’s remember that they do their best to recruit the best minds available direct out of college and grad school, and they have dedicated hardware platforms for decryption and signals processing, the capabilities of which the mainstream computing industry has no definite idea.

Look what happened to General Petraeus. How did the FBI get access to all his emails after they were deleted? Where did that stuff come from? Here’s how that kind of thing goes down: someone gets a suspicion, queues up someone for deeper analysis, all the data relevant to certain search critera gets retrieved, and is presented linked together on relevant terms. If you’re familiar with a threaded mail reader, you can see exactly how that works.

Lol@your credentials: General Petraeus got busted and the FBI got access to all his emails because was an idiot:

“I hope that this scandal will finally kill off this inaccurate myth about hiding emails from the government. General Petraeus should have known better—placing documents in an email “drafts” folder is not an effective way to hide things from the government. It wasn’t 10 years ago, and it certainly isn’t anymore.”

This guy was head of the CIA, the most important general in the US Army for years when it came to running our wars, the master strategy maker for the wars themselves, but he was a dumbass when it came to protecting his own info. Weaksauce (rolleyes)

“What kind of data storage is needed for that? How long are the communications kept for? It has to be immense”

Think FEMA camps. The computers are buried under them. FEMA camps, btw, run on the energy of gummint’ hubris.

“Yottabytes of storage, holy crap!”

Why the hell ain’t they, “Yodabytes”? I mean, fuck, props to George Lucas and all that shit.

“It’s a good thing we have so many armed citizens who are stopping the government from doing this to us.”

One of the enduring bits of “folk indignorance” amongst the armed and stupid is that meme. What is deliciously ironic is their internet bragging about their personal armories is all being catalogued for “D-day*” nannybooted jackthuggery. Morons.

* Disarmament Day

http://www.ranum.com Marcus Ranum

The original UK/USA intelligence treaty group was a clever work-around for domestic surveillance. The NSA (on paper) was not supposed to monitor US domestic communications – so they let the British do it, then under the terms of intelligence-sharing, the Brits shared it back to NSA. Clever! Canada’s GCHQ was also a member and had similar arrangements. The early ECHELON system was vast and complex, and very carefully designed to skirt laws through clever reinterpretation of them when necessary; there are some who believe (as Duncan Campbell did) that “international communications” were interpreted to also mean satellite-based communications that were domestic->orbit->domestic (after all, they ‘left the country’) I’m more cynical and I favor Weiner’s interpretation that they were doing it all along and compartmentalized internally so the left hand didn’t know what the right hand was doing.

I am constantly amazed when people say “there’s too much data to collect” when simultaneously acknowledging that the system works, at all. I.e.: it’s impossible to collect that much cell traffic or voicemail, when Verizon (for example) can carry that traffic in their data centers and store the voicemails in their voicemail servers. Obviously, it’s possible. From there it becomes a data management problem – deciding what to collect and forward and what not to. And there are numerous public breaks about such systems as the FBI’s CARNIVORE or the GCHQ sigint towers in Capenhurst(1). These breaks aren’t just “tinfoil hat brigade” stuff, or pissed-off whistleblowers – they’re people who build data networks for phone companies. As always, the technologies and techniques used leak into nonclassified applications over time if they are valuable(1)

Collect and filter at the edge, categorize at the edge, do semantic correlation in the core, and archive all the bits for long-term re-examination if necessary.

GET PATHEOS NEWSLETTERS

Get the Best of Patheos Newsletter Get the Atheist Newsletter Get the Dispatches From the Culture Wars Newsletter

Follow Dispatches From the Culture Wars

About the Author

After spending several years touring the country as a stand up comedian, Ed Brayton tired of explaining his jokes to small groups of dazed illiterates and turned to writing as the most common outlet for the voices in his head. Read More...