On August 7, as Def Con was kicking off far below in the bowels of the Rio Hotel’s convention center in Las Vegas, I was ushered into a suite on the 19th floor to see a man who has one of the most high-profile security gigs in the industry: Joe Sullivan, Facebook’s chief security officer. An acquisition of a security startup company announced that same day and a huge internal investment in security technology development have created a software security giant that has but one paying customer—Facebook itself. Sullivan explained the PrivateCore deal as an investment in Facebook’s future—especially when viewed within the context of the company’s Internet.org effort to bring affordable Internet access (and Facebook) to the still-unwired parts of the planet. “PrivateCore is a perfect fit for the future of Facebook,” Sullivan told Ars.

A VM in a vCage

The technology PrivateCore is developing, vCage, is a virtual “cage” in the telecom industry’s usage of the word. It is software that is intended to continuously assure that the servers it protects have not had their software tampered with or been exploited by malware. It also prevents physical access to the data running on the server, just as a locked cage in a colocation facility would.

The software integrates with OpenStack private cloud infrastructure to continuously monitor virtual machines, encrypt what’s stored in memory, and provide additional layers of security to reduce the probability of an outside attacker gaining access to virtual servers through malware or exploits of their Web servers and operating systems. If the “attestation” system detects a change that would indicate that a server has been exploited, it shuts it down and re-provisions another server elsewhere. Sullivan explained that the technology is seen as key to Facebook’s strategy for Internet.org because it will allow the company to put servers in places outside the highly secure (and expensive) data centers it operates in developed countries.

“We’re trying to get a billion more people on the Internet,” he said. “So we have to have servers closer to where they are.”

Enlarge/ The architecture of PrivateCore—now a Facebook internal product.

By purchasing PrivateCore, Facebook is essentially taking vCage off the market. The software “is not going to be sold,” Sullivan said. “They had a couple of public customers and a couple of private ones. But they took the opportunity to get to work with us because it will develop their technology faster.”

Sullivan said the software would not be for sale for the foreseeable future. “The short-term goal is to get it working in one or two test-beds,“ he said. “We have to understand how to tune the technology to where you don’t get a lot of false positives.” When asked if Facebook would eventually open-source PrivateCore’s technology, as it has done with much of Facebook’s other core technology, Sullivan said, “We’re definitely excited about open source in a security context. We’re not ready to comment on anything. But we are going to do everything that we can to secure open source.”

Facebook has a kill chain

It’s been 18 months since Facebook was hit by a Java zero-day that compromised a developer’s laptop. Since then, Facebook has done a lot to reduce the potential for attacks and is using the same anomaly detection technology the company developed to watch for fraudulent Facebook user logins to spot problems within its own network and facilities.

“We’ve been doing anomaly detection on Facebook logins forever,” Sullivan said. “So I think we have a world-class technology there, and we need to apply it to the enterprise.”

The system uses the same graph database technology that powers Facebook’s social graph, but it has been integrated into a number of third-party software products to adapt it to the security role. The security approach at Facebook has become one focused on isolating breaches rather than hoping they never happen—which, given the number of potential targets for an attacker within Facebook, would be fantasy. “It’s one thing for an employee’s laptop to be compromised,” Sullivan said. “It’s another to let someone move laterally within the company—to get access to something further. Most of my attacks are targeted at assuming the identity of an employee.”

To prevent that sort of attack—a piece of malware using the victim’s credentials to gain access to other systems on Facebook’s network—Sullivan has pushed Facebook to adopt two-factor authentication for all employees. “So every employee has a Duo or YubiKey device—we’ve worked with both,” he said. Additionally, Facebook has invested in building on its anomaly detection technology to build a system that can watch for anomalous network and physical security events—“like if an employee badge is used in two different places at the same time,” Sullivan explained.

Once an anomaly is detected, it’s fed into Facebook’s “kill chain” process—an approach named for a long-standing military model for a much more kinetic form of warfare, adopted by the security industry over the past five years to address “advanced persistent threat” (APT) attacks and other threats. The "kill chain" refers to the attack methods used by advanced persistent threats, rather than some militarized approach to hunting down and killing malware—the malware is usually just part of a long tail of the attackers' process

As threats are identified within Facebook’s alert system, they are taken on by security engineers—compromised machines are taken offline, sandboxed from the rest of the network, and analyzed. “I’ve got 50 to 100 engineers at any time who are just working on that,” Sullivan said.

It’s nice to share (intelligence)

The Java zero-day, he said, “drove home that it’s impossible to secure an employee’s computer 100 percent.” To minimize what an attacker can get to, Facebook has moved virtually everything that employees work with into its own cloud—reducing the amount of sensitive data that resides on individual employees’ computers as much as possible.

Sullivan said that he doesn’t have a long list of things that he worries about at this point as far as Facebook’s security goes. But he knows that “there are no silver bullets for a lot of security threats. I worry about not knowing about the next attack until it’s worked.” Another lesson from the Java zero-day plays into that concern—the need to share information."

"One thing we've learned is it’s OK to talk about attacks,” Sullivan noted. As a result of the information Facebook shared about the Java attack, a number of other companies discovered they had been affected.

Last Wednesday, In-Q-Tel's Dan Geer suggested in his Black Hat keynote that there shouid be some sort of mandatory breach reporting. Sullivan said he wouldn't go that far, but he does want the feds to share intelligence information more freely. “I hate bureaucracy in security but love the idea of sharing,” he said. "I was really excited about the government talking about sharing IOCs [indicators of compromise].”

Sean Gallagher
Sean is Ars Technica's IT and National Security Editor. A former Navy officer, systems administrator, and network systems integrator with 20 years of IT journalism experience, he lives and works in Baltimore, Maryland. Emailsean.gallagher@arstechnica.com//Twitter@thepacketrat