Thursday, September 13, 2012

MoVP 1.4 Average Coder Rootkit, Bash History, and Elevated Processes

Month of Volatility Plugins

In this post I will begin showcasing some of Volatility’s
new Linux features by analyzing a popular Linux kernel rootkit named “Average
Coder”.These new features will include
recovering .bash_history from memory, finding userland processes elevated to
root by kernel mode components of the rootkit, and discovering overwritten file
operation structure pointers.

If you would like to follow along or recreate the
steps taken, please see the LinuxForensicsWiki
for instructions on how to do so.

Obtaining the Samples

To have samples to test against, I used two sources. The
first was the sample stored on the SecondLook “Linux Memory Images” page(big thanks to SecondLook for creating the
samples and hosting them!), and the second sample was from my own virtual
machine that I infected to showcase parts of the rootkit not used by
SecondLook.

Average Coder Rootkit

The Average Coder Rootkit is a Linux kernel rootkit that operates as a loadable kernel module
and provides the ability to hide processes, logged in users, and kernel modules.
It also provides the ability for a userland process (e.g. bash) to elevate
itself to root privileges by writing to a file inside of /proc.

Average Coder gains control over a computer by loading as a
kernel module and then overwriting a number of file_operationstructures within the kernel. file_operations
is a C structure used by the Linux kernel to provide generic handling of files
by a number of subsystems within the kernel and contains function pointers such
as read, write, readdir, open, close, etc.Each active file
in the operating system has its own file_operations
structure that is referenced every time activity is performed on
the file (e.g when a file is read, the readfunction pointer of the structure is eventually called to handle reading
the file contents from disk).

We will now go through each piece of functionality the
rootkit offers, how it accomplishes it, and how we can detect it with
Volatility.

The Loadable Module

The first thing we notice, by using the linux_lsmodplugin, is that the rootkit does not unlink itself from
the module list (it is named rootkit):

Instead, as
we will see, this rootkit hides from userland by hooking /proc/modules, and while this does effectively hide from userland,
tools analyzing kernel memory will have no issues finding the module.

By reading
the rootkit source,
we also see that the rootkit takes optional parameters for configuration files
and directories. We can recover these options with the –P/--params parameter to linux_lsmod:

In this
case we see that the SecondLook engineers did not pass any options when they loaded
the rootkit, so all the values are null.

Since
rootkits that are deployed in the wild will hide themselves from lsmod, to
perform the rest of the analysis, I modified Volatility’s module finding code
to skip the rootkit module in order to mimic a real compromise. We will see in a later
blog post how Volatility detects modules that unlink themselves from the module
list.

Hiding
Users

Effect on Forensics

To hide
users, Average Coder hooks the readmember
of /var/run/utmpin the kernelin order to hide logged in users from applications
such as wand who. This effectively hides
logged in users from network administrators and incident response members
working from userland on a live computer.

How Average Code Accomplishes it

The rootkit
hides users in its ­init_users_hide_hookfunction
(here).
It does this by calling the kernel’s path_lookupfunction on /var/run/utmpin
order to find the inodestructure of
the file. The path_lookup function
works by enumerating the filesystem’s directory structure and then locating the
file of interest. Average Coder wants
the inodestructure because its i_fopmember is a pointer to the particular
file’sfile_operations structure. Once the i_fopmember is found, it can simply be overwritten with the rootkit’s function
that filters users from utmp on
demand.

How Volatility Detects This

To detect
this part of the rootkit, we need to verify that the file operations function
pointers for /var/run/utmpare valid.
Valid in this case means that the function pointers point to a function inside
the base kernel or within a known kernel module. If the function pointers are
invalid, we report them so they can be investigated.

To do this
for /var/run/utmp, we first find its inode using the linux_find_filecommand with utmp as the parameter:

This gives us
the inode number and address of the inodestructure.
We can use the number to investigate the file on disk with tools, such as the
Sleuthkit, and we can use the address to investigate the file in memory with
Volatility.To verify that the function
pointers are correct we pass the address to linux_check_fop:

This plugin,
when given the –i/--inode option, reads the inodeat
the given address and verifies each member of its i_foppointer. As we can see, the plugin tells us that the readmember is hooked and the address of
the hooked function. This verifies what we see in the Average Coder source code
and if this was an unknown rootkit, we could then perform binary analysis of
the function if we desired.

Since we now know
for certain that wand whowere being lied to on a live system,
we want to figure out what the correct values were and who was actually logged
in. To do that, we are again going to use the linux_find_file plugin but this time we will have it recover the /var/run/utmp file from
memory for us:

This writes
the contents of /var/run/utmpfrom
memory into the local “utmp” file. We can then use the who binary on our analysis system to determine who was logged in:

# who utmp

centoslive tty12013-08-09 16:26 (:0)

centoslive pts/0 2013-08-09 16:28 (:0.0)

Ignoring the
fact that the SecondLook system had a skewed clock, we know now which user was
logged in, how they were logged in (tty1,
pts/0), and the time they logged in. Again, any IR script that ran on the
live system would have missed these logins (we will soon see that centoslive
was a user hidden by the rootkit), and we recovered this information by pulling
/var/run/utmpdirectly out of memory
and did not rely on the disk.

Hiding
Processes, Kernel Modules, and Communicating with Userland

Effect on Forensics

Average Coder
also allows users to hide processes and kernel modules from userland tools.It does this by setting up a communications
channel on which the kernel module accepts commands from userland. This has a
very negative effect on forensics as investigators working on a live machine
cannot trust the output of a majority of the tools and operating system
interfaces that they normally rely on for gathering evidence.

How Average Code Accomplishes it

Commands are taken
from userland by hooking the writemember
of /proc/buddyinfo, which is a file
that normally does not support writes, and data written to this file (commands)
can be used to hide process, users, and to elevate privileges.

Processes are
hidden by hooking the readdirmember
of the root of the /procfilesystem.
Every active process in Linux has a corresponding directory under /procwhose name is the PID of the
process (e.g inithas a directory of /proc/1/). To hide processes, the
hijacked readdirfunction simply
filters out the directories that correspond to hidden processes. This effectively
hides the process from a number of userland tools.

Kernel
modules are hidden from lsmodby
hooking the readmember of /proc/modules. The modulesfile is the only source used by lsmod to list loaded
modules, so this effectively hides it from the application.Again, we will see in a future MoVP post how
to find hidden modules on both a live system and from a memory image by leveraging
sysfs.

How Volatility Detects This

To detect
these overwritten file­_operationsstructures,
we will uselinux­_check­_fopin its
default mode. This mode checks the file_operationsstructure for a number of common filesystems and files, including all of /proc:

As we can see
from the output, Volatility was able to report the three hooks placed by
Average Coder (readdirfrom root of proc, writeof buddyinfo, and readof modules), by enumerating all the
files and directories under /procand
verifying their members. From here, the investigator knows the machine is
compromised and can begin to investigate the rootkit.

Recovering
.bash history from Memory

Effect on Forensics

.bash_historyis a file on disk that
stores all the commands run by a user directly on the bash command line. This
file is a forensics goldmine when populated as the investigator can recreate
everything done by a logged in user. Due to its importance to forensics, any
reasonable attacker is going to make every effort to avoid having commands
logged to this file. Common methods to accomplish this include:

Logging in with ssh –T, which does not allocate a pseudo terminal and therefore
does not spawn bash

Setting HISTFILE to /dev/null or unsetting it
from the process environment, which effectively stops logging

Setting HISTSIZE to 0 which prevents any logging

The
exclusion of these commands to disk makes traditional disk forensics much more
difficult and means we have to rely on in-memory information.

How Volatility Handles This

Fortunately
for investigators, no matter which of the previous anti-forensics tricks are
used (or any others), bash still stores the history of commands in memory for
the current session along with the time
executed – regardless of if timestamps are enabled for disk logging.

To recover
this information, the linux_bashplugin
can be used. This plugin finds running bash instances, walks their history
structures in memory, and pulls out the commands executed and their time ran (epoch). Running linux_bash on the Average Coder
memory sample produces the following output:

As we study
this output, we see a number of interesting results.First we see that the rootkit was inserted
into the kernel sudo insmod rootkit.ko.
We then see “hide" being sent to /proc/buddyinfowhich hides the kernel module from lsmod.
Next, we see the person checking to be sure that rootkit does not appear in
lsmod output. The user then hides the centoslive
user from utmp (we covered this earlier). The user checks the success of
this by running the w command. Finally, we see the user spawn and background the sleep command, which will print the PID of it to the command line, and then
this PID is sent to buddyinfoto be
hidden. The user then checks this to make sure sleepdoes not appear in ps auxw.

Recovering these
commands confirms much of what we have seen with the previous Volatility
plugins and also would give us a starting point in a real investigation.Due to the way this rootkit hides data, by
modifying function pointers instead of in-kernel structures, we do not have to
do anything special with Volatility to recover the hidden information, it is
just simply there. We then verify the presence of malicious hooks using methods
shown throughout the post.

Note: The -P/--printunalloc option to linux_bashtells the plugin
to print unallocated entries as well as the currently allocated ones. This has
the effect of printing overwritten entries to the terminal, so it is
recommended to redirect this output to a file. In the above output I have taken
out the unreadable unallocated entries and replaced them with a label.

Elevating
Process Privileges

Effect on Forensics

Average Coder
provides the ability for userland processes to send a command to /proc/buddyinfoin order to have their privileges
elevated to root (user ID 0).This is a
very common method used by attackers in order to maintain persistence on
compromised resources.

The sample
provided by SecondLook did not utilize this portion of the rootkit, so I made
my own sample to test it out and to verify that the created plugin, linux_check_creds, detects it. The
following shows my interaction with the rootkit to elevate my bash shell:

As can be
seen, I made my user ‘x’ go from privileges of uid 1000 to uid 0 (root) by
writing the PID of my bash shell to /proc/buddyinfo
with the root command. I then took a memory
capture of the infected VM using a version of LiME already compiled for my VM:

On older 2.6
kernels, the user ID and group ID of a process were kept as simple integers in
memory. For a rootkit to elevate the privileges of a process, it simply set
these two values to zero.This
simplicity also made it very difficult to use only the information in the
process structure itself to detect which processes had been elevated and which
were simply spawned by root.

This changed
in later versions of 2.6 as the kernel adopted a credstructure to hold all information related to the privileges of
a process.This structure is fairly
complicated and forced rootkits to adapt their process elevation methods. Although the kernel provides the prepare_credsand commit_credsfunctions to allocate and store new credentials, a number of rootkits choose not to use this functionality. Instead, they simply find another process that has the privileges of root
and that never exits, usually PID 1, and set the credpointer of the target process to that of PID 1’s. This
effectively gives the attacker’s process full control and the rootkit does not
have to attempt the non-trivial task of allocating its own credstructure.

How Volatility Detects This

The borrowing
of credstructures leads to an
inconsistency that Volatility can leverage to find elevated processes. In the normal
workings of the kernel, every process gets a unique credstructure and they are never shared or borrowed. The linux_check_credsplugin utilizes this
by building a mapping of processes and their credstructures and then reports any processes that share them. The following output shows the cred structure
running on my infected VM and showing that PID 1 has the same cred structure as
my elevated bash shell (PID 9673):

In real
investigations, we could now focus our efforts on PID 9673 - using the bash plugin
would be a good start – as we know that the attacker used that shell in
conjunction with the rootkit!

Conclusion

We have
thoroughly investigated the Average Coder rootkit, including its internals,
artifacts left on a system, and interactions with the attackers who place it on
a system. This includes finding the
module, locating its hooks throughout the kernel, finding interactions with the
rootkit by userland processes, and recovering files that it filtered on the
running machine.

In tomorrow’s
post we will analyze another kernel rootkit that requires other plugins not
covered in this post to detect. If you have any questions or comments please
use the comment section of the blog or you can find me on Twitter (@attrc).