Tuesday, May 27, 2014

Acquiring Linux Memory from a Server Far Far Away

By Dan Caban.

In the past it was possible to acquire memory from linux systems by directly imaging (with dd) psudo-device files such as /dev/mem and /dev/kmem. In later kernels, this access was restricted and/or removed. To provide investigators and system administrator’s unrestricted access, loadable kernel modules were developed and made available in projects such as fmem and LiME (Linux Memory Extractor).

In this blog post I will introduce you to a scenario where LiME is used to acquire memory from a CentOS 6.5 x64 system that is physically hosted in another continent.

LiME is a Loadable Kernel Module (LKM). LKM’s are typically designed to extend kernel functionality, and can be inserted by a user with root privileges. This sounds a little scary, and it does introduce tangible risks if done wrong. But on the positive side:

the LiME compiled LKM is rather small

the process does not require a restart

the LKM can be added/removed quickly

the resulting memory dump can be transferred over the network without writing to the local disk; and

the memory dump is compatible with Volatility

Getting LiME

Since LiME is distributed as source without any binaries you need to compile it yourself. You will find documentation on the internet suggesting that you jump right in and compile LiME on your target system. I recommend you first see if a pre-compiled LKM exists, or alternatively compile and test in a virtual machine first.

In either case, you first need to determine the kernel running on your target system, as the LKM you use must have been compiled on the the exact operating system, kernel version and architecture. Here we determine our target is running the kernel 2.6.32-431.5.1.el6.x86_64.

Conducting a test run

We can now test out our newly built LiME LKM on our virtual machine by loading the kernel module and dumping memory to a local file.

We are opting to create our memory image on the local file system, so I provide the argument path=/root/mem.img. LiME supports raw, padded and “lime” formats. Since volatility supports the lime format, I have provided the argument format=lime.

The great thing about LiME is that it is not limited to just output to a local disk or physical file. In our test run we supplied an output path with the argument path=/root/mem.img. We will instead create a TCP service using the argument path=tcp:4444.

Since in this scenario our server is on the internet, and a restrictive firewall is inline we are forced to get creative.

Remember how I downloaded the LiME LKM to the target server via HTTP (port 80)? That means the server can make outbound TCP connections via that port.

I can setup a netcat listener on my investigative laptop here in our lab, and opened it up to the internet. I did this by configuring my firewall to pass traffic on this port to my local LAN address, and you can achieve the same results with most routers designed for home/small office with port forwarding.

On the target server I can now use a local netcat connection that is piped to a remote connection in our lab via port 80 (where 60.70.80.90 is our imaginary lab IP address.)

Step 3: In another shell initiate the netcat chain to transfer the memory image to my investigative laptop at our lab.

[root@targetserver ~]# nc localhost 4444 | nc 60.70.80.90 80

Voila! I now have a memory image on my investigative laptop and can start my analysis.

Below is a basic visualization of the process:

Memory Analysis with Volatility

Volatility ships with many prebuilt profiles for parsing memory dumps, but they are focused exclusively the Windows operating system. To perform memory analysis on a sample collected from linux, we need to first create a profile that matches the exact operating system, kernel version and architecture (surprise, surprise!) So let’s head back to our virtual machine where we will need to collect the required information to create a linux profile:

the debug symbols (System.map*);

Requirements: access to the test virtual machine system running on the same operating system, kernel version and architecture

and information about the kernel’s data structures (vtypes).

Requirements: Volatility source and the necessary tools to compile vtypes running on the same operating system, kernel version and architecture.

Now let’s collect the debug symbols. On a CentOS system it is located in /boot/ directory. We will need to find the System.map* file that matches the active kernel version that was running when we collected the system memory (2.6.32-431.5.1.el6.x86_64).

One of the requirements to compile the vtypes is libdwarf. While this may be easily installed on some operating systems using apt-get or yum, CentOS 6.5 requires that we borrow and compile the source from the Fedora Project. The remaining prerequisites for compiling should have been installed when we compiled LiME earlier in the section Getting LiME.

On my investigative laptop I could drop this ZIP file in the default volatility profile directory, but I would rather avoid losing it in the future due to upgrades/updates. I instead will create a folder to manage my custom profiles and reference it when running volatility.

All content provided here is purely for educational purposes only. Review state and local laws before partaking in any activity. The views and statements here have not been reviewed, approved, or endorsed by Foundstone, McAfee, or Intel.