Facebook computers compromised by zero-day Java exploit

Facebook officials said they recently discovered that computers belonging to several of its engineers had been hacked using a zero-day Java attack that installed a collection of previously unseen malware. In an exclusive interview with Ars Technica, company officials said that the attack did not expose customer data, and it was contained to the laptops of a small number of Facebook engineers. But other companies who were affected by the same hacking campaign may not have been so lucky.

Facebook's internal security team worked with a third party to "sinkhole" the attackers' command server, taking over the network traffic coming into it from systems infected by its malware. They discovered traffic coming from several other companies, according to Facebook Chief Security Officer Joe Sullivan. Facebook notified those companies of the attack, and it has turned the case over to federal law enforcement. An investigation is still ongoing. While some of the affected companies were aware of an ongoing attack, others were unaware of the problem before being notified by Facebook.

The attack was discovered when a suspicious domain was detected in Facebook's Domain Name Service request logs. According to Sullivan, the requests were tracked back to the laptop of an engineer working on mobile application development projects. Forensic analysis of the files on the laptop led to the discovery of a number of other compromised systems.

The patterns of the attack, which appear eerily similar to the Facebook war-game drills Ars recently chronicled, don't appear to be related to any previous attacks on Facebook or other organizations. "This looked like a new campaign that wasn't linked to previous Advanced Persistent Threat activities," Sullivan told Ars.

Lurking at the watering hole

Rather than using typical targeted approaches like "spear phishing" with e-mails to individuals, the attackers used a "watering hole" attack—compromising the server of a popular mobile developer Web forum and using it to spring the zero-day Java exploit on site visitors.

"The attack was injected into the site's HTML, so any engineer who visited the site and had Java enabled in their browser would have been affected," Sullivan told Ars, "regardless of how patched their machine was."

Through forensic analysis, Facebook was able to identify the exploit and report it to Oracle. (Oracle had previously documented the flaw, but the company expedited the release of a patch when it learned an exploit of the problem was "in the wild.")

The exploit was used to download a collection of malware to victims' computers—a mix of tools that ran on both Windows and Apple computers. Facebook's security team has a dedicated malware researcher, Sullivan said, who was able to identify the malware. After analyzing it, the Facebook security team shared signature and forensic data from the malware with law enforcement and other companies.

Antivirus software was unable to detect the malware because "it was novel," said Sullivan. "The fact that the machines were patched didn't slow down the attackers."

An analysis of the activity of the malware showed that "they were trying to move laterally into our production environment," Sullivan said. The attackers gained "some limited visibility" into production systems, but a forensic review found no evidence that data was exfiltrated from that. However, some of the information on the laptops themselves—"what you typically find on an engineer's laptop," Sullivan said—was harvested by the hackers, including corporate data, e-mail, and some software code.

This is not a drill

The exploit that was used to attack Facebook is just the latest in a long string of well-publicized security issues related to Java browser plugins. Facebook had begun work to reduce its exposure to Java exploits even before this attack was discovered. "We had already started an initiative to reduce our dependence on products that require Java plugins," Sullivan said. "But it's hard to do, because there are so many enterprise applications that require it."

But while disabling Java would block already existing attacks, Sullivan added, it wouldn't eliminate the risk of future threats. "If it wasn't a Java plugin vulnerability, it could have been another," he said.

Sullivan pointed to the ongoing security drills that Facebook conducts (as recently reported by Ars' Dan Goodin) as being key to the company's ability to quickly detect and respond to the attack. "The fact hat we do those drills and have people trained to deal with these situations meant we were able to work really quickly to get the problem resolved," he said. "People stayed cool under fire. To me, that felt like a good kind of response to a bad situation."

Promoted Comments

Java doesn't have to die, although it does need a serious security overhaul which will not be a patch-as-it-happens reactionary strategy it uses currently. And nothing is ever secure, internet or in real life.

96 Reader Comments

But while disabling Java would block already existing attacks, Sullivan added, it wouldn't eliminate the risk of future threats. "If it wasn't a Java plugin vulnerability, it could have been another," he said.

Could have been, yes. But it just seems inevitable with Java, doesn't it?

Well, good to see the Facebook team on top of things. I know Facebook gets a lot of (deserved) hate over their privacy issues when it comes to customer data, but at least those in the trenches are active and knowledgeable enough to stamp out these little events before they do real damage. Kudos to them.

But while disabling Java would block already existing attacks, Sullivan added, it wouldn't eliminate the risk of future threats. "If it wasn't a Java plugin vulnerability, it could have been another," he said.

Could have been, yes. But it just seems inevitable with Java, doesn't it?

Facebook is, at least, a high enough value target to burn 0day in more robust components, if they are being targeted specifically. However, Java is and continues to be The Biggest Problem.

I like to think there were red alarms going off and an action team was dispatched screaming go-go-go down the corridors as they bum rushed the mobile developer. Maybe one guy leaping over the file cabinets screaming "noooo...".

I like to think there were red alarms going off and an action team was dispatched screaming go-go-go down the corridors as they bum rushed the mobile developer. Maybe one guy leaping over the file cabinets screaming "noooo...".

Don't forget the one guy sitting at his CRT with the green text filling in one letter at a time (accompanied by a beep for each character) saying, "I think they've hacked the mainframe but I can block them with encryption!"

Goes to reinforce two opinions, Java must die, and also, when it comes to the internet, there is no such thing as secure.

Well, let's instead say, "security is a process, not a product." We're obviously hearing about Facebook's security problems because they've decided to let us know - but the story they're telling is one of process maturity. And that is a good thing, especially for a relatively young company.

It would be great if we could start to get more transparency across the entire IT industry - and not just from Internet service companies. Presumably attacks like this happen every day in banking, healthcare, construction, travel, insurance, retail... but outside of occasional FBI reports, we really don't have a way to track them.

(If I understand correctly, the new executive order doesn't address private-sector threats at all?)

I don't understand all the hate towards Java. Now, I'm not a Java developer, but what makes Java so much worse than, say, .NET (C#/VB), C++, Objective C, etc? I can certainly write programs in these other languages and cause harm, so why is Java being blamed and not the other languages?

I can see how a Java plugin for a browser can be a problem, but why are people saying a language is a problem, when the language doesn't do anything more or less than the other languages?

Java doesn't have to die, although it does need a serious security overhaul which will not be a patch-as-it-happens reactionary strategy it uses currently. And nothing is ever secure, internet or in real life.

It would be great if we could start to get more transparency across the entire IT industry - and not just from Internet service companies. Presumably attacks like this happen every day in banking, healthcare, construction, travel, insurance, retail... but outside of occasional FBI reports, we really don't have a way to track them.

I'm curious, does hearing this type of story increase or decrease your respect for the security of your data at Facebook?

I'm sure that every large tech company has similar events on a regular (monthly, yearly, whatever) basis. If they were more open about this, would it do them harm or help them?

IMHO, if they are forth right about the attacks and the risk to their customers/users in a timely manner my respect goes up. If they timely inform people then they react appropriately and hopefully quickly. Websites will be attacked all the time; so are targeted more frequently. Important software has holes and bugs unfortunately and often they can be exploited a 0-day attacks.

I don't understand all the hate towards Java. Now, I'm not a Java developer, but what makes Java so much worse than, say, .NET (C#/VB), C++, Objective C, etc? I can certainly write programs in these other languages and cause harm, so why is Java being blamed and not the other languages?

I can see how a Java plugin for a browser can be a problem, but why are people saying a language is a problem, when the language doesn't do anything more or less than the other languages?

The problem is not Java the language.

The problem is exploits in the Java runtime environment (the interpreter) which runs and is *supposed* to sandbox the Java code from untrusted resources (the internet at large) and prevent it from doing something persistent like dropping malware on a machine.

The problem is that Java the runtime environment (which is used to run Java the language, compiled into bytecode) is roughly as watertight as a screen door based on the steady stream of exploits against it. Further, it's installed in damned near everything connected to the internet.

This is why I don't give Facebook any real information. I fake all the other crap they ask for, because eventually somebody will hack them and then start attacking other services I use if they have my real info

I don't understand all the hate towards Java. Now, I'm not a Java developer, but what makes Java so much worse than, say, .NET (C#/VB), C++, Objective C, etc? I can certainly write programs in these other languages and cause harm, so why is Java being blamed and not the other languages?

Quote:

Quote:

I still find incredible that Java, an interpreted language, supposed to be in a sandbox, has so many exploits...Like flash...

The issue is to do with how Java/(and old Flash) manages its own heap and stacks. They basically don't use the safety features that modern operating systems provide. In a way they subvert the built in OS features like DEP and ASLR because they need to manage their own heaps in a cross platform way. This means they need to store and address code on that heap in a predictable way. If there are any holes found in the Java client itself, this makes exploit code is much easier to write.

.NET/Silverlight in windows uses more of a dictionary to JIT'd code rather than a straight stack / heap. It can then use OS protection rather than subverting it. This helps it mitigate potential security problems.

OBJ-C would be prone to buffer overrun attack. Apple take a different approach to security that makes use of a lot of cyptography and code signing to establish levels of trust for executing code. This is actually extremely effective for all non-jail broken machines, for example if you tried to insert shell code in an executable it simply wouldn't run because it's code signature would no longer match the executable.

I don't understand all the hate towards Java. Now, I'm not a Java developer, but what makes Java so much worse than, say, .NET (C#/VB), C++, Objective C, etc? I can certainly write programs in these other languages and cause harm, so why is Java being blamed and not the other languages?

I can see how a Java plugin for a browser can be a problem, but why are people saying a language is a problem, when the language doesn't do anything more or less than the other languages?

The problem is not Java the language.

The problem is exploits in the Java runtime environment (the interpreter) which runs and is *supposed* to sandbox the Java code from untrusted resources (the internet at large) and prevent it from doing something persistent like dropping malware on a machine.

The problem isn't the runtime environment. The problem is that the Java *browser plugin's sandbox* has flaws.

Just because a bunch of bloggers dump on Java (as opposed to the plugin/sandbox)...yeesh.

The .NET runtime enviornment isn't secure at all. I could easily write a program that would collect anyone's info and send it off to a random server. Writing a program to delete files on a partition would also be trivial..

Obviously if you write and execute the program yourself you can do that.

I'm talking about exploiting the framework to run shellcode. So for example you paste the shellcode to turn off the firewall, add a new user, and give it remote desktop privileges to a textbox in an asp.net web form and execute it.

I still find incredible that Java, an interpreted language, supposed to be in a sandbox, has so many exploits...Like flash...

This article is talking about Java, not JavaScript. They are completely different.

Both Java and JavaScript are interpreted languages running in a sandbox. Both now support JIT'ing to native code making both a vector for certain kinds of attacks that will bypass the sandbox.

Sure, JVM bytecode is interpreted. Java (the language) is compiled. Either way, I probably should have been more forgiving in my original comment. There's just so much public confusion around Java and it's frustrating.

How long is it going to be before someone launches a civil suit over Java's crappy security and tries to take a big chunk out of Oracle?

Bet you such a suit would either be settled out of court for an undisclosed sum or be held up in litigation until the plaintiff runs out of money.

It'll happen right after someone launches a civil suit aimed at Adobe over *their* crappy security for Flash and Reader. And after the one aimed at Google, because if that "Pinkie Pie" dude can escape Chrome's sandbox, it's been done elsewhere, and they surely must be liable for not getting it perfect the first time around. And Apple -- surely every security fix they've released for the OS represents a lawsuit that *should* have happened, right?

If such a suit were to succeed, it would pretty much be the end of the software industry as we know it. If a vendor *warrants* its software as fully secure, or fails to take reasonable measures, that's different.

If you took a stack of software written in Java and rewrote it into C, in most cases you'd end up with a vastly larger security problem than the one you started with. Ever wonder why almost all security disclosures start with "buffer overrun"?

The .NET runtime enviornment isn't secure at all. I could easily write a program that would collect anyone's info and send it off to a random server. Writing a program to delete files on a partition would also be trivial..

Obviously if you write and execute the program yourself you can do that.

I'm talking about exploiting the framework to run shellcode. So for example you paste the shellcode to turn off the firewall, add a new user, and give it remote desktop privileges to a textbox in an asp.net web form and execute it.

You're talking about an exploit in the plugin. There is nothing wrong with the Java language. Just because a plugin has a problem that doesn't mean that there is anything wrong with the language (as someone of the comments above would lead you to believe).

I'm curious, does hearing this type of story increase or decrease your respect for the security of your data at Facebook?

I'm sure that every large tech company has similar events on a regular (monthly, yearly, whatever) basis. If they were more open about this, would it do them harm or help them?

IMHO, if they are forth right about the attacks and the risk to their customers/users in a timely manner my respect goes up. If they timely inform people then they react appropriately and hopefully quickly. Websites will be attacked all the time; so are targeted more frequently. Important software has holes and bugs unfortunately and often they can be exploited a 0-day attacks.

Facebook's concern with security is high because the data they protect is what generates their revenue, enabled by the sense of security among users who share it.

In other words, they have to be able to charge for it getting out; they can't have people getting it for free.

You're talking about an exploit in the plugin. There is nothing wrong with the Java language. Just because a plugin has a problem that doesn't mean that there is anything wrong with the language (as someone of the comments above would lead you to believe).

In a related story - I work at a mobile related company. Everybody there is a high earner, we're all engineers, not people I would expect to do stupid things.

Yesterday one of the engineers got malware via Skype. This spread to his contacts in the office. It's not my job to do desktop support (i'm a Linux server guy now) but the support guy was busy so I got asked to sort it out.

The malware was simple. It installed a Skype API handler, then installed a dropper that did <whatever>. It was only noticed because it didn't work on 64 bit systems and put up a Windows alert when it needed to run, which didn't help when most of our desktop kit is 32bit XP. I isolated one, worked out what it did and neutralised it.

What annoys me:

1. That a person I would assume to be internet savvy was utterly stupid, and executed a .scr (screensaver, exe) file just to see a funny video

2. That his peers did exactly the same

Stupid people on the internet annoy me, but you would be amazed by how many people don't care, including people you would trust. Some of the computers I looked at had mywebsearch, along with all sorts of other shite, how do these people breathe?

The only reason I personally have Java on my machine is because it's required by the Thales n-cipher kit I use as a HSM. Even then I block it from the browser, or anything else.

My point, loose as it may be, is that stupidity is

1. As loose as the weakest link

2. Not limited to those you would assume

How can we, as professionals, mock users when our peers are equally as stupid?

I still find incredible that Java, an interpreted language, supposed to be in a sandbox, has so many exploits...Like flash...

This article is talking about Java, not JavaScript. They are completely different.

EDIT: On second thought, I probably should have assumed you were talking about JVM bytecode. Sorry.

Java not javascript, I know the difference still I don't remember many exploit that get outside the sandbox on javascript. For Java it should be even simpler to make it secure, the bytecode should be simpler to parse, and the JVM at this point should be secure even when doing strange things with memory during JIT. I just don't get how an exploit against java work, except for error in implementing security at the library level. But even then after so much time is it that difficult to simple block access to the IO api if the applet is untrusted???