There Is Simply No Scientific Backing For TSA's Behavioral Detection Program

from the hearsay-isn't-evidence dept

The Government Accountability Office has taken a run at the TSA's Behavioral Detection program in the past. Its findings were far from complimentary. Specially-trained "Behavior Detection Officers" (BDOs) were basically human coin flips. Deciding whether or not someone was a threat came down to a lot of subjective readings of human behavior, rather than proven principles.

In response to this report, the TSA started trimming back the number of BDOs it deployed, converting about 500 of them back into regular TSA officers. But the TSA still believed there was something to its pseudoscience patchwork, so it's still sending out 2,600 BDOs to covertly stare at travelers' throats and eyes (no, really) until terrorism reveals itself.

The GAO's second report focuses on the TSA's stubborn insistence that the Behavior Detection program is worth what we're paying for it. It has made claims to oversight that this program is scientifically-backed and scientifically-based. The GAO's investigation [PDF] finds almost nothing that backs these assertions.

We reviewed and categorized all 178 sources that, as of April 2017, TSA cited as providing support for specific indicators in its revised list of behavioral indicators to identify the extent to which they present valid evidence. We defined valid evidence as original research that meets generally accepted research standards and presents evidence that is applicable in supporting TSA’s specific behavioral indicators.

[...]

Of the 178 total sources TSA cited, we determined that 137 are news or opinion sources, and we took no further action because these do not meet our definition of valid evidence. We determined that 21 of the 178 sources are reviews of studies— sources that rely on the source author’s assertion of support for the indicator rather than original analysis, methods, or data that can be independently used as valid evidence.

Of the 178 sources cited by the TSA (with great confidence, one imagines), only 20 met the GAO's definition of valid evidence.

The TSA claims its BDOs have the ability to sniff out terrorism by looking for behavioral indicators. There are long lists of indicators to look for but, again, the TSA has nothing in the way of evidence to back these up.

In our review of all 178 sources TSA cited in support of its revised list, we found that 98 percent (175 of 178) of the sources do not provide valid evidence applicable to the specific indicators that TSA identified them as supporting. In total, we found that TSA does not have valid evidence to support 28 of its 36 revised behavioral indicators, has one source of valid evidence to support each of 7 indicators, and has 2 sources of valid evidence to support 1 indicator.

In fact, it appears the TSA may just be Googling for indicators, as its supporting "evidence" is extremely scattershot.

For example, among other sources, TSA cited four separate news articles that provide eyewitness accounts of a single 2005 suicide bombing incident at an Israeli bus station as support for 2 behavioral indicators. In another example, TSA cited a 2004 newsletter published by the New Hampshire Police Standards and Training Council as support for 11 behavioral indicators.

A partial snapshot of the chart created by the GAO shows the utter failure of the TSA's behavioral "science."

The individual breakdown of the scientific "support" for behavioral indicators is even more damning.

As a result of our analysis, we determined that TSA does not have valid evidence supporting 28 of its 36 revised behavioral indicators. Specifically, every source TSA cited in support of these 28 indicators is a news or opinion source, a review of studies source, or an original research source that either does not meet generally accepted research standards or is not applicable to the behavioral indicator it was provided to support. For example, TSA cited 105 sources to support the use of indicator number 11, which involves BDOs identifying individuals who seem to be attempting to conceal their normal appearance. However, we found that none of the 105 sources present original research that meets generally accepted research standards. In another example, TSA cited 63 sources to support the use of indicator number 5, which involves BDOs identifying individuals who seem to be sweating heavily. While we found that one of the 63 sources cited is original research that meets generally accepted research standards, we also found that this source does not present evidence that is applicable as support for this indicator.

To which is appended this footnote:

Some reviews of studies sources TSA cited as support for behavioral indicators present conflicting information on whether the use of behavioral indicators and behavior detection more generally is useful in identifying individuals who may pose a threat to security. For example, one article states that indicators of suicide bombers “are drawn from multiple sources and have not been formally or empirically validated.” Another states that indicators used by law enforcement to identify potential suicide bombers are “vague, contradictory, and so broad as to be useless.”

The DHS responded to the GAO's draft by claiming it still had plenty of research and science justifying the deployment of BDOs. More "evidence" was submitted and found to be just as useless as everything the GAO had already examined.

[I]n its letter, DHS cited passages from a 2013 RAND report as providing support for TSA’s use of behavior detection. However, the report DHS cited does not clearly support TSA’s use of behavior detection. As DHS noted, the RAND report states that there is value and unrealized potential for using behavioral indicators as part of a system to detect attacks. However, the indicators reviewed in the RAND study could not be used in real time in an airport environment, as we reported in 2013. Further, the RAND report refers to behavioral indicators that are defined and used significantly more broadly than those used by TSA. For example, the RAND report includes indicators such as mobile device tracking, monitoring online activity, and changes in lifestyle patterns.

The DHS also said the program was justified because trained BDOs also sometimes saw things even untrained non-professionals might be able to spot.

DHS also stated that there are certain common-sense indicators that TSA cannot reasonably ignore, with or without valid evidence. Specifically, DHS highlighted the indicator “unusual exposed wires or electrical switches on a person.”

Of course, these could (MAYBE) be detected by normal screeners, not specially-trained officers costing the taxpayers millions each year. The GAO agrees BDOs should ignore common sense. But much of what the DHS considers to be "common sense indicators" are highly-subjective takes on normal human behavior.

We recognize DHS’s position that certain common-sense indicators of mal-intent should not be ignored. However, TSA’s revised list of 36 behavioral indicators also includes indicators that are subjective, such as assessing the way an individual swallows or evaluating the degree to which an individual’s eyes are open.

Finally, the DHS protests it should be able to do it because other agencies (CBP, FBI) and other countries do it. The GAO patiently responds that behavioral detection may work for others, but the point is the TSA has yet to prove it works when it does it and can offer no solid evidence to back its claims.

The TSA's Behavioral Detection program isn't dead yet, but a few more nails have been driven into the coffin. If the regular TSA screening procedures are security theater, the BDOs are the mimes. A whole lot of moving around, but in the end, there's really nothing there.

Re: Re: Re: Sure, there is. It's called &quot;training the slaves to obey authority without question&quot;. Been written down and practiced since at least ancient Rome.

You both stubbed your toes.

I agree with your question "What's your point?""science is not the same as application" has no meaningful context.

"SHEESH. No matter how neutral and uncontroversial of comments,..."

Everything is uncontroversial until someone creates a controversy out of it.

"... on this site full of contrarians you can be sure that some weenie will try to gainsay."

This is the nature of social interaction. Someone usually has something to say about something. It is not bad to offer new ideas, even if they are stupid. If the ignorant appear, they become teachable moments. If knowledge is presented, then everyone has the opportunity for enrichment.

Re: Re: Re: Sure, there is. It's called &amp;quot;training the slaves to obey authority without question&amp;quot;. Been written down and practiced since at least ancient Rome.

Re: Re: Re: Sure, there is. It's called &quot;training the slaves to obey authority without question&quot;. Been written down and practiced since at least ancient Rome.

Yeah, and when it is pointed out how there is nothing scientific about any of it, one still has to make a comment about how 'scientific doesn't mean good'.

Scientific does, however, mean scientific. Calling something scientific when it is, in fact, not, is either a lie or extremely sloppy thinking. Or both.

It also does not help to confuse the scientific method (which is what "scientific" means here), and any products of a scientific endeavour, which may be used for good or ill, and in which the particular endeavour itself may have been a bad idea or outright nefarious to begin with.

In that sense, scientific is not necessarily a good, but is also irrelevant when the subject at hand is not in the least scientific to begin with.

Further, there is little which is scientific in the example given. If one wanted to specifically address Nazi sciences, some of them are entirely unscientific also, merely being labelled as science. Others are entirely common to any system, such as base R&D which might be applied to weapons systems, but most of the development is actually engineering issues. (But it's funny how i have never heard similar commentary for engineering as i have heard regarding science, or what someone wishes to label as science. Which is the sign of something being super trope-y and not particularly a reflection of reality.)

Controversial? I don't care to speculate. But it is highly inaccurate. And there is at least one weenie somewhere who will always support some poorly reasoned statement or line of outright bullshit, no?

Re: Re: Re: Sure, there is. It's called &quot;training the slaves to obey authority without question&quot;. Been written down and practiced since at least ancient Rome.

Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

Don't know where you get the notion that "scientific" must mean good

That'd come from a proper high-school science class. Where you're taught how to properly put a claim or theory to the test, with control groups and whatnot to avoid false conclusions and bias. Taught how to rule out - or acknowledge - other possible explanations. Taught to make your test repeatable by other independent testers to further rule out mistakes, deception and bias.

That includes the TSA methods described above. There's no scientific backing for it. Nazis, "legalizing state actions" and corporatocracy have nothing to do with science. Technology is only vaguely related.

You complain of "training the slaves to obey authority without question." Science class is all about teaching kids to question everything. Add a method for uncovering the truth and debunking garbage claims, and yes, it's a good thing.

Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

Science class is all about teaching kids to question everything.

Well, that's what it SHOULD be about. In most cases, it's about regurgitating currently accepted "facts" no matter how ridiculous they seem when considered logically. Critical thinking is indeed NOT taught as then you have a generation of people capable of thinking for themselves, which makes them unsuitable to be cogs in the wheel.

Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

Re: Re: Re: Re: Sure, there is. It's called "training the slaves to obey authority without question". Been written down and practiced since at least ancient Rome.

You kinda read my post backwards. It's common in schools to NOT teach evolution, or to teach creationism along side evolution. This is what I mean when referring to "currently accepted 'facts'". That's also just a small example - history classes are almost devoid of real factual information and are almost completely composed of anecdotes written by popular fiction authors of the late 1800's and early 1900's.

Re: There is!

A different approach

Given:*A ride on public transit is a low-cost thing to steal, (most systems incur near zero marginal cost per passenger until they have to increase capacity somewhere) and* public transit is generally subsidized, either by the fare-paying public or by taxes,

How about not being too worried about a few free tickets to the enterprising or the poor who just might need them?

Re: A different approach

Except this is part of quality of life. NYC, in the eyes of quite a few people who live there, is going downhill, backwards. Now they just passed a law saying that people who urinate in the street won't be arrested. Fare jumpers won't be arrested.

Re: A different approach

Israel uses this. People getting on planes know they are being looked at, evaluated, if things seem out of sorts, they will talk to people, search them more thoroughly.

I can see how this would be effective. People that are up to no good and know they are being watched tend to exhibit signs. Of course, some people may just be nervous and end up being delayed a bit, but whatever. When is the last time you heard of an El Al flight having a problem?

Same with good cops, good police work involves watching people and seeing things that are outside the norm and then questioning. That is just good police work.

Re:

The scale of security needed in Israel is also much more limited. For example, the U.S. has more than 100 times the number of air passengers as Israel does. “It would be impossible to scale what they’re doing,” Jacobson said.

Also, Israel has a workforce full of military veterans, thanks to the country’s mandatory military service, including some with training in interrogation or security. Sela said it would take a huge effort for the U.S. to institute the kind of training that would be required to fulfill Trump’s vision of putting in place Israel-style security. “The problem is, he has no clue how expensive this is,” Sela said.

It wouldn't be necessary to copy the whole Ben Gurion program.Just train the existing profiling staff in the same way.They already had plenty of those and could easily staff up.They also have thousands times as many veterans available.

Re:

When is the last time you heard of an El Al flight having a problem?

It's extremely rare for any flight to have a security problem. We keep finding out it's really easy to get guns, bombs, etc. onto US flights, but when does it ever happen? The US and Israel spend a lot on security. Is there evidence they're more secure than countries that spend less?

In lots of countries you can get on a train without any security check. When was the last time you heard of a train having a problem? All the ones I can think of were unintentional derailments.