Hack and / - Forensics with Ext4

Learn from my mistakes as I figure out how to gather forensics data
on an ext4 filesystem.

One great thing about writing technical articles is that you have a nice
collection of documentation you can turn to. I tell people that I probably
reference my books and articles more than anyone else, because although I may
not always remember specific steps to perform a task, I do always remember
whether I wrote about how to do it.

One article I find myself referring to now and then is the "Introduction to
Forensics" article I wrote in Linux Journal back in the January 2008 issue
(my first feature article in Linux Journal). In that
article, I walk through
how to use Autopsy, a front end to Sleuthkit, to perform your own forensics
investigation on a server that has been hacked. Recently, I had to perform
an investigation on a server that fell victim to an SSH brute-force attack
(use SSH keys!) and discovered that my personal documentation no longer
worked. In this article, I walk through the symptoms of this problem and
ultimately how I was able to work around it.

The Victim

As I mentioned, recently I investigated a server (let's call it alvin to
protect the innocent) that had been compromised by a brute-force attack. I
followed the procedure I document in my forensics article to a T. I pulled
the plug from the server the moment I detected it was compromised,
immediately created an image of the entire drive, created a second copy of
the image on a separate drive that I would work from, and once my evidence
was secure, I re-installed the affected server with a clean OS.

The first big difference about this system I ran into compared to past
investigations was the sheer size of the data. It's one thing to image a
10–20GB disk and quite another when the drive is hundreds of gigabytes and
every action—creating the initial image, image duplication and
md5sums—takes
hours. This was a slow process to say the least. Once I had my working
image, I fired up Autopsy and started my investigation. Initially,
everything was fine—Autopsy saw the image and was able to detect its
partitions, and when I started to browse the filesystem, I could see the
contents of the root directory.

The Problem

The first problem showed up when I tried to navigate through the filesystem
with Autopsy—all of the directories below the root directory appeared to
be empty. I knew that couldn't be right, but I wasn't sure what the problem
was. At first, I thought I might just have a corrupted copy of the image, but
since the md5sum seemed to take almost as long as copying the image, I
started the image copy again from my gold master just to be sure. When that
still didn't work, I tried to use Sleuthkit from the command line and even
tried tools from The Coroner's Toolkit, yet I got the same result. I thought
perhaps Autopsy couldn't support such a large disk image. Before I did too
much more research, I decided to try to mount the image loopback to see if
it was a corrupt image. After all, if it were, I'd need to create a
fresh image before I went any further.

The challenge here was that I had created an image of the entire disk, not
just individual partitions. Normally, when you mount something loopback, you
mount an image of a partition, and the syntax is something like:

$ sudo mount -o loop /path/to/image /mnt/image

Then, you can browse /mnt/image like any other mounted filesystem. In this
case, since the image was of a full partition, I had to figure out how to
skip ahead in the image and mount only the partition. Mount has an option
called offset that allows you to specify how far ahead in the file to skip
before it mounts it as a filesystem, but the trick is knowing what that
offset should be set to. It turns out that if you have parted installed,
it's relatively simple to find the offset. First, I run parted against
alvin's full disk image and tell it to print out the full partition table
in bytes:

In this case, I'm interested in the first partition, and from this output, I
can see that the partition starts at byte 32256. Now I can use that offset
to mount the partition loopback:

$ sudo mount -o loop,ro,offset=32256 alvin-sda /mnt

Note that I mounted the filesystem read-only here. I didn't want to risk
writing any new data to this partition! From this point, I found that I
could browse the filesystem under /mnt just fine—the problem somehow had to
be with Autopsy and Sleuthkit. I did research, but I couldn't really
find any evidence that there was an upper limit to the image size that I
was close to, but all the same, I decided to try to create an image of just
the initial partition. A few hours later, Autopsy still couldn't use the new
image, but strangely enough, I could mount the partition loopback on my
filesystem just fine. After trying countless other things, I started to realize
that it couldn't be the size of my partition that was a problem. It had to
be something else, and it was then that I noticed that this filesystem was
ext4.

Like Ext3 Plus One More

To be honest, apart from the fact that ext4 touted faster speeds with large
numbers of files while claiming backward compatibility with ext3, I hadn't
given the filesystem much thought. I had used it on a few new systems (it
is the default on modern Ubuntu installs), and it seemed to work fine.
Because
it was supposed to be backward-compatible, I initially wrote it off as
being a cause of my problem. After more research though, I discovered not
only did other users complain about lack of support for ext4 in Sleuthkit,
but that the backward compatibility isn't as compatible as you might
think. Although it's true that you can mount ext2 and ext3 filesystems as
ext4, you can mount ext4 filesystems as ext3 only if the filesystem does
not use extents (which is a major new feature of ext4, so
it's commonly turned on). Because I couldn't treat this ext4 filesystem as
ext3, that meant neither could Sleuthkit.

So if I could mount the partition loopback, what is the big deal if
Sleuthkit and Autopsy didn't work? Although being able to browse through the
filesystem is a useful feature of forensics tools, another incredibly
valuable feature is the ability to create a filesystem timeline. A filesystem timeline organizes all the files into a giant text database where
they are ordered according to their MAC times (when the file was last
Modified or Accessed or Changed its metadata). With a filesystem timeline,
if you have a good idea when the attackers might have been on the system,
you can start to track their virtual footprints as they execute programs,
browse directories and untar their scripts onto the system. Normally, I
would have Autopsy generate this file for me. Luckily, it turned out that a
tool from The Coroner's Toolkit called mactime was able to generate the
timeline either from an image or from a mounted filesystem. Here's the
command I used to create a timeline from the filesystem mounted at /mnt:

The -d option specifies the directory where the filesystem is mounted, and -R
tells mactime to scan recursively through that directory. The -g and -p
options tell mactime to get group and user ID information from the group
and passwd files in my mounted filesystem, and finally, the date specifies
the date range to start from. The date must be after 1/1/1970 (I'll leave
why that is as an exercise for the reader), so I chose the next day.

Although I didn't have the same user experience I was used to with Autopsy,
once I could browse the filesystem and view the timeline, I could use the
same investigation techniques. Ultimately, I was able to piece together the
timeline when the attacker compromised the machine via a weak password,
changed the password so no one else could break in the same way, and then
downloaded and launched SSH brute-force attack scripts to spread onto other
servers.

I admit I was really surprised to see that some of my favorite forensics
tools no longer worked. Although it is great that Linux filesystems continue
to improve, one unintended downside (or possibly an upside if you are an
attacker) is that forensics tools must be upgraded continually to work on
modern filesystems. Although I was able to mount the partition and create a
timeline, I don't know that I would have been able to recover any deleted
files from the partition—another valuable use of forensics tools. That
said, at least if I (or you) need to analyze an ext4 image again, I now have
all of the steps documented.

______________________

Kyle Rankin is a director of engineering operations in the San Francisco Bay Area, the author of a number of books including DevOps Troubleshooting and The Official Ubuntu Server Book, and is a columnist for Linux Journal.

Hi I am workinh on system with MIPSEL,debian Lenny 5.0 on it and using alchemy 1250 processor.

I enaled ext3 and ext4 support from the kernel and compiled modules using e2fsprogram and copied that to initramfs of my kernel.

With the resulting image I booted my device and formatted all my partition with ext4 and trying to copy data to and filesystem to respectine hda partition.

The partition mention above has ext2 and vfat on it earlier. On converting them to ext3 or ext4 and copying data to them giving me problem of loss of data. the md5sum of files copied is wrong.I tried forcefully copting but on device reboot again same thing and because of that my device not working.

Rankin, did you have "PasswordAuthentication no" on the brute-force attacked compromised server? In the third paragraph you said it was brute-force attack on SSH keys, but later in the second last paragraph you said it was due to weak password. Really interested to know whether SSH servers using **only** PubkeyAuthentication can be compromised by brute-force attack. Thanks.

Thanks for the description. I had the same problem with these tools a year ago when a lot of my files were deleted by a badly written (freeware) application. Recovery of deleted files from ext4 is also more difficult. I could do it only using carving tools like foremost, but the whole directory structure was gone for good. Luckily beagle was indexing my files in the background so I managed to reconstruct the directory structure from the index.

I can absolutely recommend this method - I use it all the time with kvm images. I know - I too used to use the mount -o loop with the offset, but that is a pain with multiple partition in particular!!!

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.