Get the latest federal technology news delivered to your inbox.

The government’s security clearance process currently “is stuck in the industrial age."

Days after Navy contractor Aaron Alexis murdered12 people during a shooting rampage at the Washington Navy Yard on Sept. 16, 2013, Pentagon officials acknowledged they had neglected to follow up on a Rhode Island police report the previous month showing that Alexis, who died in a shootout with police, had complained of hearing voices. That turned out to be just one of many red flags in Alexis’ background that Navy officials and security clearance investigators were not aware of prior to the tragedy.

The Navy Yard shooting sparked an outcry about how the government handles the process for granting and reauthorizing security clearances. At the time, “Alexis was one of roughly 4.9 million Americans—over 1.5 percent of our country’s population—that hold security clearances,” a House Oversight and Government Reform report on the shooting noted.

Since then, officials have worked to significantly strengthen the way clearances are granted and managed. With the creation last year of the National Background Investigations Bureau housed at the Office of Personnel Management, security clearance professionals across government have been wrestling down an investigations backlog, which stood at 343,557 unprocessed clearances at the secret level and 72,566 at the top secret level by the end of the third quarter of fiscal 2016. The backlog of periodic re-investigations stood at 156,172.

The background checks are also taking much longer. A February House Oversight and Government Reform report this February found that in 2015, it took on average 95 days to process a secret clearance and 179 to process a top secret clearance; by fiscal 2016 the average for secret clearances had risen to 166 days and top secret clearances to 246 days.

One result of heightened concerns is that the Defense Department and the Office of the Director of National Intelligence are relying more on continuous monitoring to detect insider threats and for the periodic re-investigations of current employees and contractors.

Could technology accelerate the process while improving thoroughness?

The software industry has stepped up with an array of subscription cloud-based products being used by private employers to monitor individuals. They continuously scour open source data to flag events that might indicate that an employee is experiencing a personal crisis that could make them an insider threat.

Thomson Reuters’ Clear,TLOxp and Endera (formerly IDentrix) are among the current offerings, along with others in the dozens of firms that promise human resources monitoring on the General Services Administration’s schedule 738X. “Disgruntled employees, malicious insiders, outside contractors and compromised coworkers will cost you upwards of $7 million this year,” says Endera’s ad for products already in use in the airline industry and other realms of corporate America.

Congress has been pushing the NBIB to better exploit social media—it’s been done only on a pilot basis—to monitor employee states of mind. But agencies and contractors must work within the Fair Credit Reporting Act, which requires the consent of employees and employers before they incorporate social media in their software.

Debate continues over whether such electronic tools are cost-effective. And there is doubt by some that they are fair to employees who may fear that the human resources department will abuse information from their private lives.

“There is little to no evidence that social media monitoring has any actual effectiveness to balance out the significant harm that inaccuracies could cause,” said Jay Stanley, senior policy analyst at the national office of the American Civil Liberties Union. Though the law permits the tapping of public open sources, he said, “not only is there a risk of inaccuracies, there is good reason to believe the inaccuracies would not be distributed evenly in terms of ethnic and racial groups and income groups.”

Even so, the new bureau is warming to adoption of monitoring software as it works to improve a process that has endured years of criticism. “We work closely with industry and we welcome ideas from industry,” Director Charles Phalen said in a statement to Government Executive. “Products such as these are more useful to companies and agencies that have or are developing mandated insider threat programs.”

Past Government Efforts

The government’s security clearance process “is stuck in the industrial age,” said Raj Ananthanpillai, the chairman, CEO and president of Endera, which for more than a decade has been bidding on contracts with the FBI, Transportation and Homeland Security departments. Today’s clearances and continuous monitoring requires that agencies be “quick on their feet and come at it from multiple angles. It’s got to be done by automation to provide accurate and relevant information,” he said.

Endera’s platform taps mostly public sources in 13 areas ranging from arrests to driving violations to financial stress, Ananthanpillai said. “There’s a treasure trove out there. If the data are relevant to the risk, they can produce a timely alert, and then you can take action.” Agency employees and contractors with security clearances are “supposed to self-report on bankruptcies, divorce and foreign travel, but the majority don’t do that anymore,” he said. “So we have to figure out how to do continuous vetting.”

“We have the tools, technology and services to help with insider threat risk mitigation. But these scare some people,” said James Henderson, CEO of the Insider Threat Defense consulting firm, who has contracted as an instructor for agencies, defense contractors and businesses. The data is out there, and from an insider threat perspective, you only know what’s inside the door,” he said. “Email alerts of an employee’s arrest or other indicators of concern from continuous monitoring software such as Endera are invaluable today, he added. “Every little nugget of information helps.”

“When a company or agency uses continuous monitoring and finds an arrest or something that is not right, it could help the company prevent another workplace shooting or other incident”, Henderson said. “But some companies are a little on edge on using continuous monitoring and collecting all this information on an employee.”

Some organizations, Henderson added, don’t do a good job of sharing information about employee concerns with other departments, such as human resources, security and IT offices. “If disgruntled or behavioral indicators are not shared, this information lives in silos, and does not help with insider threat risk mitigation and give a complete picture of an employee’s threat level,” Henderson said. How many times after an incident, he asks, “do you hear the signs were there, but no one spoke up?”

It’s unclear whether such risks to privacy are justified by results. One case study of Endera by the Security Executive Council research firm documented that the software identified more than 800 identity changes of Homeland Security Department employees, “of which 24 actionable alerts were deemed to disqualify the noted persons from continued participation.”

“There is little in the way of empirical evidence that shows continuous screening results in any advantage to employers … There are no studies to suggest, on a cost-benefit basis, such checks produce results,” he said. “If such checks are done, the next issue is how. If databases are used, then there is the possibility of both false positives and false negatives since databases available to private employers are not always complete, accurate, or up to date.”

Contractor Support

Alan Chvotkin, executive vice president and counsel at the 400-company Professional Services Council, said his contractors group strongly supports the NBIB’s efforts to anticipate insider threats both in pre-employment screening and continuous monitoring of staff and contractors already hired. But the NBIB“does need to do a better job of relying on technology” in background checks, he said. “Knocking on neighbors’ doors is silly—your neighbors have no idea what your behaviors are.”

Companies and agencies have to consider the privacy issues, Chvotkin added, but individuals who apply for a security clearance consent to allowing investigators to solicit their personal information. And employees already hired are “told through their employment agreement that the company owns the resources,” so agencies don’t need employee consent to monitor.

The real challenge, he added, is not privacy but the fact that “the databases are not very good. Not every state has a single database, and not every law enforcement agency participates in state or local databases,” he said. “As we saw in the Navy Yard shooting, “a lot of activity we expected to have been reported was never reported.” That means companies and agencies “can’t rely on any single source—they rely on multiple technologies.”

Trey Hodgkins, senior vice president for public sector at the Information Technology Alliance, is also an advocate of monitoring software. “We have data either in government or the private sector, all of which can or should be available to the oversight entities, that can identify when an event necessitates a deeper investigation,” he said. “The end state we’ve advocated is a single digital record for each person, which starts when they fill out a form online, and [eventually] determines whether that individual should get a clearance.”

That record would be shared among government organizations. The goal would be to create “reciprocity processes, instead of the individual having to get multiple badges, which is the current setup,” Hodgkins said.

His group is keeping an eye on an NBIB report due soon on reducing the background check backlog. “Industry has argued you have no right to privacy in this process,” Hodgkins added. “You sign on the dotted line asking for the privilege of the government granting you a clearance.”

But the ACLU’s Stanley questions whether monitoring key events in an employee’s life really helps predict dangerous behavior. “Is divorce correlated with insider threat behavior?” he asked. “Is there actual evidence, or is it a theory?” He said he’s seen polices based on theories without evidence. An example is “security guards all over America are harassing photographers on the theory that taking photos is an indicator of suspicious activity or terrorism. If the monitoring “is done electronically,” Stanley added, “it’s not clear how much worse the privacy invasion will be if it’s repeated.”

Ananthanpillai bases his hopes for Endera on the idea that the government is now centralizing background checks. “We’ve come full circle” since 2004, when the Defense Department got out of the background check business and outsourced it to multiple companies hired by OPM, he said. “Every agency has its own security department, but why not provide reciprocity across agencies?” he asked. “We automate the rap sheets from the 50 states,” he said. “OPM can’t possibly be monitoring people every five years. The threats are now so asymmetric. You have to stay on top of them. Tougher vetting is possible,” he added. “It’s not rocket science.”