Our analysis of this malware, dubbed Linux/Cdorked.A, reveals that it is a sophisticated and stealthy backdoor meant to drive traffic to malicious websites. We urge system administrators to check their servers and verify that they are not affected by this threat. Detailed instructions to perform this check are provided below. (Update 5/1/2013: An improved tool coded in C replaced the Python script we originally published.)

In fact, Linux/Cdorked.A is one of the most sophisticated Apache backdoors we have seen so far. Although we are still processing the data, our Livegrid system reports hundreds of compromised servers. The backdoor leaves no traces of compromised hosts on the hard drive other than its modified httpd binary, thereby complicating forensics analysis. All of the information related to the backdoor is stored in shared memory. The configuration is pushed by the attacker through obfuscated HTTP requests that aren’t logged in normal Apache logs. This means that no command and control information is stored anywhere on the system.

Technical analysis of Linux/Cdorked

Here we provide the first technical analysis of Linux/Cdorked, which seems to be affecting hundreds of webservers right now. In the Linux/Cdorked binary all the important or suspicious strings are encrypted. As shown in the following image, a function is responsible for decrypting the strings on demand with a static XOR key.

The version of Linux/Cdorked that we have analyzed contains a total of 70 strings that are encoded this way. As shown in the following screenshot, the key used for encoding the data is 27A4E2DADAF183B51E3DA7F6C9E6239CDFC8A2E50A60E05F.
As mentioned before, Linux/Cdorked does not write any files on the disk. Instead, it allocates around six megabytes of shared memory to keep its state and configuration information. This memory block, a POSIX shared region of memory (shm), is used by all Apache subprocesses but can also be accessed by any other process since the malware authors didn’t limit its permission. The following screenshot shows the (read, write for everyone) permission rights assigned to the shared memory region.

There are two ways the attacker can control the behavior of the backdoored server: through a reverse connect shell or through special commands, all of them are triggered via HTTP requests.

The Linux/Cdorked.A backdoor

The HTTP server is equipped with a reverse connect backdoor that can be triggered via a special HTTP GET request. It is invoked when a request to a special path is performed with a query string in a particular format, containing the hostname and port to connect. The client IP of the HTTP dialog is used as a key to decrypt the query string as a 4 byte XOR key. Additionally, IP specified in X-Real-IP or X-Forwarded-For headers will override the client IP as the XOR key. This means we can craft a X-Real-IP header that will in effect be a “\x00\x00\x00\x00” key. Query string also needs to be hex-encoded before sending.

While the shell is used by the attacker, the HTTP connection creating it is hung (the backdoor code does not implement forking). This implies that malicious shells can be found if one has access to the server and checks for long-running HTTP connections. On the other hand, the HTTP request does not appear in Apache’s log file due to the way the malicious code is hooked into Apache.

Redirection in Linux/Cdorked.A

When redirecting a client, the malware adds base64 encoded string to the query containing information like the original visited URL and whether or not the request was originally to a javascript file so the server could provide the right payload.

The “surl” parameter shows the infected host and the “suri” indicates what the original requested resource was.

After the redirection, a web cookie is set on the client so it is not redirected again. This cookie is also set if a request is made to a page that looks like an administration page. The backdoor will check if the URL, the server name, or the referrer matches any of the following strings : ‘*adm*’, ‘*webmaster*’, ‘*submit*’, ‘*stat*’, ‘*mrtg*’, ‘*webmin*’, ‘*cpanel*’, ‘*memb*’, ‘*bucks*’, ‘*bill*’, ‘*host*’, ‘*secur*’, ‘*support*’. This is probably done to avoid sending malicious content to administrators of the website, making the infection harder to spot. The following screenshot shows part of the code responsible for handling the web cookie.

A few other conditions must be met before redirection happens; for example, a check is done for the presence of the Accept-Language, Accept-Encoding, and Referrer header.

Other Linux/Cdorked.A commands

We found 23 commands in Linux/Cdorked.A that can be sent to the server via a POST to a specially crafted URL. The request must also contain a cookie header starting with “SECID=”. The query string value must hold 2 hex encoded bytes that are encrypted with the client IP, using the same technique as the shell. The SECID cookie data will be used as arguments to some of the commands. We believe that the URLs to redirect clients are sent to the backdoor using this method. The redirection information will be stored encrypted in the allocated shared memory region. We also believe that the conditions for redirection are set this way, for example, a white list of user agents to redirect can be preconfigured and a black list of IPs to avoid redirection.

Finally, some information about the status of the backdoor is returned in the ETag HTTP header, as shown in the screenshot below. We are still investigating the purpose of each of the commands and will publish our results as soon as the analysis is completed. In short, they all either add content to, or remove it from, the configuration in the shared memory region.

Linux/Cdorked.A Remediation

As previously mentioned, the permissions on the shared memory allocation are loose. This allows other process to access to memory. We have made a free tool to allow systems administrators to verify the presence of the shared memory region and dump its content into a file (you can download dump_cdorked_config.c or copy and paste the code from the listing below).

We also recommend using debsums for Debian or Ubuntu systems and `rpm –verify` for RPM based systems, to verify the integrity of your Apache web server package installation. (However, remember to temper this advice with the reality that the package manifest could have been altered by an attacker.) Checking for the presence of the shared memory is the recommended way to make sure you are not infected. We would be interested in receiving any memory dumps for further analysis.

At the time of writing, the ESET Livegrid monitoring system is showing hundreds of webservers that seem to be affected by this backdoor with thousands of visitors being redirected to malicious content. We will publish more information on the scale and complexity of this operation in the days to come.

You need to install the ctypes library, it should be provided by the python-ctypes package on CentOS. Please note that we have released a new C program as the python script was not working will all variants of Linux/Cdorked, you can find it at the end of the article. You will need gcc to compile it, the compilation command is specified in the comment at the top of the file. Let us know if you have any issues.

Thanks,

Sébastien

Weng Fu

Why does not the CIA track down the criminals who write these codes and either jail or execute them?

Stephen Cobb

Mark – We have updated the code. There is now a C program that will work for you, and it detects more variants than the original python script.

Stephen Cobb

We are working on it. Hope to have something new on Monday, if not sooner.

Stephen Cobb

I take your point, and believe me, we are researching the method of compromise. Are they getting in via Plesk, cpanel, Drupal, WordPress? Are they paying off sysadmins?

I would say the fancy stuff is not just about sensationalism, it is important to know that this malware is much harder to detect than older malware, relative to the level of management normally accorded to LAMP systems.

kamicc olo

So, is there any information what do those thousands of compromised servers have in common? I hope it might narrow the search of way of compromisation easier…

LOL, that’s true. But really, one have to admire the tenacity in creating applications such as this. I think it admirable.

Alessandro Forghieri

I have written a perl version of the test script (which does not
require non-core modules). As I have no way to test it on an infected
server, I canot validate THAT case (but it should work as the
translation is very straightforward). It can be downloaded form
pastebin: http://pastebin.com/EittPYmK

Why are the X-Real-IP and the special path to get the reverse shell censored? It’s not clear what is the method to obtain them.

Valentino

Hello ESET.

I’m wondering if Hetzner was hit exactly by Linux/Cdorked.A:

“The malicious code used in the backdoor exclusively infects the RAM. First
analysis suggests that the malicious code directly infiltrates running Apache
and sshd processes. Here, the infection neither modifies the binaries of the
service which has been compromised, nor does it restart the service which has
been affected.”