Scanning an enterprise organisation for the critical Java deserialization vulnerability

On November 6, security researchers of FoxGlove Security released five zero day exploits for WebSphere, WebLogic, JBoss, Jenkins and OpenNMS. These software products are used everywhere in enterprise organizations and with the published exploits remote malicious code can be unauthenticated executed.

The underlying vulnerability was known for years but it was vague and believed to be hard to exploit. That situation changed when Chris Frohoff (@frohoff) and Gabriel Lawrence (@gebl) published in January of an exploit generator for it in their talk on AppSecCali . Their generator didn’t get much attention on the internet. FoxGlove didn’t find the original (underlying) vulnerability, but they did find vulnerable products and wrote exploits for it. They call the vulnerability “the most underrated, underhyped vulnerability of “. When I read about FoxGlobe’s research on Twitter on November 8, , my guts told me immediately that this was something big.

Validating the security research

I needed to validate this research and so I tried out the exploits against a fully patched and hardened WebSphere server on November 9, . It worked! Woah! I could easily run arbitrary code on WebSphere servers. This is something really big! For organizations that have a large Java stack, it’ll be certainly a lot bigger than Heartbleed.

After validating the research I informed my hacker friends John de Kroon and Ger Schinkel to start investigating the vulnerability further. We exploited a WebLogic and Jenkins server and improved the published WebLogic exploit code. We also directly started developing vulnerability scan software to scan large computer networks for vulnerable servers and put our scan software on Github.

Escalating the vulnerability to enterprise organizations

We saw that a lot of enterprise organisations and critical infrastructure were vulnerable. These large organizations didn’t know about it yet and at the time that we found out on November 10, , not much attention was given to the zero day exploits on the internet. We informed the Dutch National Cyber Security Center and also shared our intelligence with security departments of quite of lot of enterprise organizations in The Netherlands and financial institutions world wide. That had effect and security departments were starting their investigations after our notification.

As the underlying vulnerability is a complex one, some security departments didn’t give the vulnerability the attention and priority that it deserved. Hopefully that will change with this post.

Via this article I want to share our knowledge about scanning large enterprise networks for vulnerable software components. I’ll update the article continuously with new information about the situation (last update was November 21, ). Security departments can use the information in this article to get up to speed and in control of the vulnerability. This article functions as a security test plan and we’re also releasing a vulnerability scanner so you can check your own networks for vulnerable WebSphere, WebLogic, JBoss and Jenkins servers.

The Java deserialization vulnerability

The zero day exploits that were released for five different products on November 6, , all exploit the same underlying vulnerability in the Apache Commons Collection library (VU#576313) that occurs when deserializing Java serialized objects. Custom Java applications using vulnerable versions of third-party libraries, such as Apache Commons Collection, Groovy, or Spring, may also be vulnerable.

This means that a lot more products are also vulnerable for the Java deserialization vulnerability. The vulnerability has the potential to affect almost all applications written in Java (!). When you search on Github for applications that use Apache Commons Collection then you’ll find more than 1,300 applications.

The security problems with Java deserialization was already known for quite some time:

Pierre Ernst wrote in 2013: “When Java serialization is used to exchange information between a client and a server, attackers can try to replace the legitimate serialized stream with malicious data. [..] Java serialization enables developers to save a Java object to a binary format so that it can be persisted to a file or transmitted over a network. Remote Method Invocation (RMI) uses serialization as a communication medium between a client and a server. Several security problems can arise when a service accepts binary data from a client and deserializes the input to construct a Java instance. [..] [an] attacker [can] cause a denial-of-service condition or €” in extreme cases €” to inject arbitrary code. You might believe that this kind of attack is impossible [..]”. The extreme case Pierre Ernst referred to was CVE-2012-4858.

Those “extreme impossible cases” are not that extreme and impossible anymore because in January of a generator called Ysoserial was released to easily exploit applications that made use of Java deserialization functionality that was available in the Apache Commons Collection, Groovy and the Spring Java library. This exploit generator didn’t get any attention in the security community.

This means that currently it’s possible to easily hack 12,175 servers. This number is probably is just the tip of the iceberg of vulnerable internet-facing servers.

Impact and security risk rating

According to our research it seems that the vulnerability mostly occurs on internal networks due to (management) interfaces that are most of the time not internet-facing. If a vulnerable interface can be reached, then it’s game over: an attacker can execute arbitrary malicious code on the vulnerable server.

We observed that the published WebSphere, WebLogic and Jenkins exploits are very reliable and haven’t resulted in denial of service of servers.

The injected malicious code will be executed under the privileges of the operating system user that the vulnerable service runs under. In hardened services this will not be a root or administrator account. Within WebSphere an attacker will be ‘root’ under all applications that WebSphere runs: all applications can get compromised. An attacker will also have access to database connections that WebSphere Java applications use. An attacker can thus also compromise linked databases. This all makes it a critical security vulnerability.

Via a privilege escalation vulnerability that the server might have, an attacker can gain root or administrator rights on the server, which results in full compromise of the server.

Attacks possible on internal network launched via internetEven through most vulnerable servers can only be reached via an internal network, we know ways to exploit these servers via the internet. For now, we won’t disclose that attack scenario, as most companies haven’t patched their systems yet.

Metasploit exploit for JBoss servers
On November 18, , security researcher Nick Fox released an exploit for the Metasploit framework that targets vulnerable JBoss servers:

This Metasploit exploit makes it much easier to exploit JBoss servers in comparison with the already released JBoss exploit. Via this new polished exploit JBoss servers now face also the threat of script kiddies.

Security mitigation phases

The security mitigation plan should be based around the following phases:

First things first. A threat analysis should be made for all known vulnerable products, after that your custom written software should be reviewed and finish by finding yet unknown deserialization vulnerabilities in other products.

Phase 1.1: Vulnerability scan plan for known vulnerable products

An organization should first find out if they have IBM WebSphere, Oracle WebLogic, JBoss, Jenkins, OpenNMS and Atlassian Bamboo servers running in their network.

If this is the case, then it should be validated if the released exploits work against these servers:

It’s important to test if existing security controls are blocking or minimizing the attack surface, such as firewalls and intrusion prevention systems.

It could be the case that the version of the software that is used is not vulnerable for the exploit. In other words: find out if your organization is running the latest version of the software (I hope so, otherwise you’ll probably have a lot of security updates that are missing).

But perhaps the most important one: have hard proof in hand that your organization is vulnerable for the exploit(s). This is very important information to convince higher management to form a security incident response team to start controlling the vulnerability and have the mandate to use human resources with high priority of your organization.

Now that you found out which of the six products are running in your network and which of those are exploitable (probably all of them), this issue should be escalated to management so you’ll have the time and team to work on it.

The next action is to make an inventory of all vulnerable servers located in your network:

Hopefully your organization has Configuration Management Databases (CMDB) were this information is stored in. I specifically name the existence of multiple CMDBs, because most IT departments have their owns lists of servers that are not (yet) consolidated into the main CMDB of the company. If you have found a CMDB, hopefully it’s up-to-date. It probably won’t be the case, because IT assets will get attached, detached and relocated on a network all the time. Administration is always outdated.

Contact should be made with the IT departments that deliver and maintain the vulnerable software packages and servers. The IT department that initially delivers a software package/server can be different from the department that maintains it. Ask these departments to log in on all their servers to get a list of vulnerable components from a white box perspective on their landscape. It’s very important to get to know on which TCP and/or UDP ports the vulnerable services are listening, so let them run Netstat on the server. We need this information to validate the vulnerability from a black box scan perspective and to find other vulnerable servers on the network that the IT department is not aware of.

Apart from the administration retrieved, you should fire up port scanner Nmap and start scanning the internal network to find servers that are not on the official lists (you’ll be surprised!). It’s important to limit the amount of ports that you’ll scan with Nmap in order to minimize the risk of denial of servicing production machines and network components. You’ll not be the first security analyst who brings down a network firewall and cause management to panic because production servers are not reachable anymore. Limit the port list to all default ports (see further in this article for a list) that the vulnerable software components use and complete it with the custom ports that your organization uses for these components.

Now that you have a list of potential vulnerable servers, you need to confirm the vulnerability on these server from a black box scan perspective. First scan the internet-facing side, and then the internal network.

Indicators of the vulnerability

Indicators of the Java deserializing vulnerability:

When the hexadecimal string AC ED 00 05 and in base64 encoded format rO0 is visible in HTTP requests or responses (parameters, ViewStates or cookies).

The vulnerable Java library is commons-collections*.jar. This is a file that exists on the file system in most cases (fat .jar files are an exception to this rule). The star functions as a wildcard, as it can contain various version numbers. This library is responsible for the code execution possibility when Java objects are deserialized. Existence of this library means that the software that uses it has the possibility of the deserialization vulnerability. The following steps should be followed to analyse and determine the security risk further:

Application Java code should be analyzed if untrusted Java objects are deserialized. If this is the case, then the application is vulnerable.

To determine the risk value associated with the vulnerability, the attack surface should be determined: how easy is it for an attacker to exploit the vulnerability?

8080: HTTP
9080: HTTP
After connecting to port 8080 or 9080, Jenkins will give the client a ‘random’ port back to connect to: binary communication.

JBoss

8080: HTTP

OpenNMS

1099: TCP

Atlassian Bamboo

Not researched so far.

Java serialized objects could be used anywhere on any port. TCP would be used in 99% of the cases, but also UDP could be used in rare cases. Java objects are frequently communicated via HTTP, HTTPS and SOAP. Most web servers use TCP port 80 and 443, but also 81, 444, 8080, 8443, 8888, 9000 and 9443 are frequently used as (alternative) web server ports.

Common used ports according to Shodan

It’s wise to also scan for vulnerable servers on common used ports. According to Shodan the most used ports are:

We had a huge list of servers that we wanted to check and unfortunately there was no vulnerability scan software available that had signatures to test for the Java deserialization vulnerability. So we created our own scanner, dubbed it SerializeKiller and put it on Github on November 11, . The scanner is written in Python can check for vulnerable WebSphere, WebLogic, JBoss and Jenkins servers.

Our scanner can scan 1,000 servers in less than two minutes (via multi threading) and is non intrusive. No exploits will be fired and only the default installation ports will be scanned. You can easily add extra ports by modifying the code.

To run our scanner, Python, CURL and Nmap should be installed. After that the scanner can be called via command line:

I haven’t tested this extension yet, but it looks like it can only detect vulnerable JBoss servers. This free unofficial extension can only be loaded in the commercial edition of vulnerability scanner and HTTP proxy Burp Suite.

If custom Java code is developed for your organisation, then all Java code should be reviewed if object serialization is used. As this can take up quite some time, you should prioritize on your internet-facing applications, and then first review your most important applications based on their Confidentiality Integrity Availability (CIA) rating.

Fortify Static Source Code Analyzer
The (commercial) static source code analyzer (SCA) of HP Fortify can detect the usage of Java serialization. This white box detection is however very simple (compare it to a simple grep search) and doesn’t take into account possible attack surface. All Fortify SCA findings should be manually verified, because the rudimentary signature is prone to quite a lot of false positives.

Apart from servers, work stations can also be vulnerable. Analyzing servers should have a higher priority than clients.

A security analyst should then process the results of the collected data to research new Java based products that are vulnerable for the deserialization vulnerability. Key is to check if serialized Java objects are visible in attack surface in the products that are to be researched.

Network based intrusion detection systems can help with the inventory

Signatures for Java deserialization can be created by yourself via the vulnerability indicators above. The most simple indicator is to search network traffic for the hexadecimal string AC ED 00 05 and in base64 encoded format rO0. We found out that searching for JRMI also gives interesting results (possible new zero days).

Detection of vulnerable services via an IDS is particularly helpful in a complex network to get the inventory of known and unknown vulnerable services as complete as technically and organizationally possible.

Emerging Threats has rules
On November 9, , Emerging Threats released in their daily ruleset update the following (commercial) rules for Snort (open source IDS) and Suricata intrusion detection systems:

HP TippingPoint has also rules
TippingPoint (commercial IDS) has released detection rules on November 18, . I’ve seen those in action and first impression is they work great!

Phase 2: Damage control (short term)

Java deserialization is broken in the design and in its core it’s not an implementation vulnerability. The vulnerability has not been patched in the Apache Commons Collection library and that probably won’t be the case any time soon.

Just for the record: Apache released a patch (in their nightly build) which added a kill switch for the deserialization functionality, but I don’t call that a real patch. If your application is using Java serialized objects, then you won’t get safe if you just install Apache’s patch.

Solution for custom written Java software

Using Java object deserialization functionality is deprecated and should not be used. If it’s nonetheless used, then the exchanged serialized objects should be protected via strong encryption so a malicious person won’t be able to view and modify the Java object and inject malicious exploit code. To further protect the exchange of Java objects, Java object white listing should be implemented.

Alternative for the official patch is to close the management interface of WebSphere on port 8880 and/or 5005 via iptables on the WebSphere server itself. This should have minimal impact. When iptables is used, the server doesn’t have to be rebooted in order to close the port; this means no down time for the server and thus this mitigation shouldn’t have customer impact.

Solution for Oracle WebLogic

WebLogic is working on a fix, but hasn’t released a patch yet. The patch is planned to be released in week 48. Please note that it’s expected that the patch will only be available for the latest version and will not be backported.

The Oracle WebLogic vulnerability is tracked under CVE–4852. Oracle has released mitigation advice under MOS Note 2076338.1. The advice was put however behind a paywalled login screen. The Oracle mitigation advice is just to block the WebLogic port (that uses the T3/T3S protocol) on the firewall.

Solution for Jenkins

Jenkins has released this week a patch for the vulnerability via version 1.638 and 1.625.2. We’ve confirmed that this patch results in the fact that the Java deserialization exploit won’t work anymore.

Solution for JBoss

We’ve not researched the solution for this product so far. You can always use iptables to block remote access to vulnerable TCP port 8080.

Solution for OpenNMS

We’ve not researched the solution for this product so far. You can always use iptables to block remote access to vulnerable TCP port 1099.

Solution for Atlassian Bamboo

Atlassian has released a patch and you should upgrade to the latest version.

Solution via WAF or IPS?

Currently no web application firewalls (WAF) and intrusion prevention systems (IPS) are known that can mitigate exploits against the Java deserialization vulnerability.

The long term solution would be to add an entry in the Java development security policy to deprecate the usage of serialized Java objects. If serialized Java objects are used, these should be protected via strong encryption and object white listing should be implemented.

Check developed Java software via security testing and code reviewing before deploying it to production.

Time table

A chronological overview on all the dates and events named in this article:

25 Responses to Scanning an enterprise organisation for the critical Java deserialization vulnerability

I have a fundamental dislike of many Web Server Platforms and, for the last year, have investigated the Node model for API needs The risks for Node vulnerabilities, the frame works and modules used and the programming practices used are “important”
Node still may have a long way yet, so I hope that the issues in the established Web Server platforms can produce a sharp learning curve that can result in best practices when exposing Data via public, or private, API’s in the Node Of Things (NOT)

The movements of The Big Players (3 letter abbreviation) into the NOT should be monitored to ensure that security is the prime directive

I wrote an article that talks about what to do about this vulnerability (and the limitations of each approach) from the point of view of a programmer (also talking about application security in general):

I have set up IDS rules as defined by ET. How do I test these rules? I tried submitting a form over HTTP with fields :
content1=rO0ABXNyA, content2=jb21tb25zLmNvbGxlY3Rpb25z
but that didn’t trigger the rule for
alert tcp any any -> $HOME_NET any (msg:”ET EXPLOIT Serialized Java Object Calling Common Collection Function”; flow:to_server,established; content:”rO0ABXNyA”; content:”jb21tb25zLmNvbGxlY3Rpb25z”; fast_pattern; distance:0; reference:url,github.com/foxglovesec/JavaUnserializeExploits; classtype:misc-activity; sid:2022114; rev:1;)