'Mystery shopper' technique could be effective counterterrorism tool

The business of connecting the dots of information received from diverse sources and then developing conclusions about terrorist threats is more complicated than often suggested in media accounts. Nonetheless, we have a right to demand that counterterrorism organizations measure their performance and learn how to refine their analysis methods so they do the best possible job of protecting us.

One approach that might help is for the National Counterterrorism Center, an organization established in 2004 specifically to work on connecting dots, to adopt a version of the “mystery shopper” technique that many businesses use to measure and improve their performance.

In traditional settings, retail companies hire a mystery shopping service to send “undercover” shoppers into their stores to collect information on the customer experience, identifying both strengths and weaknesses. Management then uses that data to devise training programs to improve customer service.

Here’s how it would work in the National Counterterrorism Center: A dedicated staff would be responsible for sending information about fictional individuals into different parts of the intelligence system, whether National Security Agency telephone intercepts or reports from agents on the ground. The unit would be responsible for tracking whether and how analysts evaluated the information — that is, did they successfully connect the dots?

The center’s performance would be measured by the percentage of cases the analysts correctly identified as exhibiting a high risk of terrorist behavior. The metric could be tracked by organization, by team and eventually — when enough cases have been assembled — by analyst.

Before starting, the mystery shopper unit would need to develop a consensus about which information patterns suggest a high risk of terrorist activities. And it would need to avoid planting too many high-risk rumors, which could overwhelm the center’s resources. So the unit would also need to send noise or low-risk information as well and track the percentage of good hits versus low-risk hits. A mystery shopper unit would also want to vary the types of information it sends to see if certain sources or types are less likely to be successfully identified.

Like performance measures in general, if such metrics are used as a management tool, they should improve the performance of the counterterrorism system. Establishing challenging performance targets will motivate employees to work harder and be more vigilant. Varying the kinds of information fed into the system will allow officials to diagnose its weaknesses so agencies can better target training or changes in techniques.

The center would need to create a learning environment in which honest errors are discussed but not punished — what professor Amy Edmondson, who has done research on how hospitals can best reduce medical errors, calls psychological safety.

The country needs the best possible performance from its National Counterterrorism Center. There are management tools that have proven effective for improving performance in many contexts outside counterterrorism. The government can no longer afford to ignore them.

About the Author

Kelman is professor of public management at Harvard University’s Kennedy School of Government and former administrator of the Office of Federal Procurement Policy. Connect with him on Twitter: @kelmansteve

The Department of Homeland Security banned the Russian cybersecurity company Kaspersky Lab’s products from federal agencies in a new binding operational directive.

Reader comments

Tue, Feb 9, 2010
Kenny Burroughs

The analysis group needs more time to identify potential terrorists as the amount of data "explodes" that they are reviewing. Spending time finding fake terrorist will surely detract from spending time looking for the real ones. There has to be a better way than wasting security experts time playing what essentially is a game rather than looking for the next "real" bomber.

Tue, Feb 9, 2010
Mike Moxcey
Fort Collins, CO

I think the idea is standard and foolish. Identifying high risk terrorist behavior has already been done and procedures set up. Adding in the mystery shopper is a _compliance_ measure now, to ensure that everybody in the analyst chain is following the identified procedures.
It doesn't improve security; it merely measures the way we are already tracking data that we already believe is high risk.
Typical security mentality thinks compliance with the rules means the rules work. We can use computers to force password complexity and length and prevent repeat use, but that essentially forces people to write their password down.
Those security analysts subject to mystery shopper techniques will get very good at identifying what they are told to measure by those above them on the security chain. What they will miss is anomalous data and there will not be any incentive (or time) to investigate what will probably be the next style of terrorism to occur.
As long as the initial identification of high risk behavior is correct, then measuring the compliance of the worker to meet that goal will help achieve the goal of better security.
But if the terrorists change tactics due to security, then we are essentially tying the hands of analysts, forcing them to look for targets that have become irrelevant; fighting the last war so to speak instead of preparing for the next one.

Mon, Feb 8, 2010
Henry

This is great until the name used by the mystery shopper is close to yours and you end up on the no fly list and Local Law Enforcement is kicking your door in? How do you "un-connect" the dots, once you have entered false/msileading information into the system? There has got to be a better way.

Please post your comments here. Comments are moderated, so they may not appear immediately
after submitting. We will not post comments that we consider abusive or off-topic.