SANS Digital Forensics and Incident Response Blog

Disk layouts using the Linux Logical Volume Manager (LVM2) are increasingly becoming the norm for new Linux installs. And very often the physical volume used by LVM2 has been encrypted via dm-crypt. A recent email from a Sec508 student asking for a procedure for mounting these images prompted me to codify this information into a blog posting.

Investigating the Image

When initially presented with the image, you may not know whether LVM2 or dm-crypt has been employed. So let's start from scratch:

You'll obviously want to verify the md5sum of the image before you begin your investigation. I'm also showing it here so that we can verify at the end of our procedure that we haven't modified the disk image in any way.

The mmls output shows two Linux file systems- one rather small and the other consuming most of the disk- and some unallocated regions. I'd already be suspicious that encryption was being used here, since this layout suggests the typical "unencrypted /boot and large encrypted volume" layout common to modern Linux installers. We also don't see a swap partition in this disk layout, suggesting that the swap area is hidden away inside the larger volume.

But let's check out the two Linux file systems to see if we can get more information. First, the smaller partition at the front of the disk:

Now let's see what we get when we run fsstat against the other partition:

# fsstat -o 499712 sda.ddCannot determine file system type

So the partition does not contain a file system type that fsstat recognizes. It seems unlikely that there would be no file system in this rather large partition, so the most likely explanation is LVM2 and/or a dm-crypt volume. But how can we test this theory?

The next step is to set up a loop device so that we can access the second partition more easily:

The losetup command we'll be using to create the loop device accepts file offsets, but uses bytes as the unit for the offset rather than sectors. So we multiply the starting sector from our earlier mmls output by the sector size to figure out the correct byte offset. Then we run "losetup -f" to figure out the name of the first available loop device. Finally we combine these two pieces of information into the losetup command to actually create the device. Notice the use of the "-r" flag here, which creates a read-only device.

Now that we have the loop device we can investigate things a little further. First let's see if we're dealing with an encrypted volume:

The cryptsetup command is the primary administrative interface to dm-crypt volumes. The "luksUUID" sub-command is enough to confirm that you're dealing with an encrypted volume, and "luksDump" provides more detailed information. Note that if you're scripting this process, there's also an "isLuks" command that simply returns true or false depending on whether the volume is encrypted.

Note that if you run this command on a volume that's not encrypted, you see output like this:

# cryptsetup luksUUID /dev/sda1/dev/sda1 is not a LUKS partition

But in this case, our disk image definitely contains an encrypted volume. Accessing this volume is the subject of the next section. If you get a negative result from your cryptsetup commands, then you're probably dealing with just a plain LVM2 volume. Instructions for accessing LVM2 volumes will follow in the section after we deal with our encryption problem.

But before we get into all of that, let me pre-empt a question that I know several people are going to raise. Yes, I am aware that the cryptsetup command has a "-o" option for specifying a sector offset. So why did I go to the trouble of creating the loopback device when I could just do something like this:

Why didn't this work? Trust me, I spent a great deal of time trying to answer that question myself. Ultimately, I ended up running cryptsetup via strace to see what was going on. Here are the relevant lines of output:

The ioctl() call is trying to retrieve the physical sector size (BLKSSZGET) but getting the "Inappropriate ioctl for device" error, which causes the program to fail. Apparently the Linux kernel doesn't like this ioctl() being applied directly to our image file. What's interesting is that creating a loop device binding to our file and then calling cryptsetup on the loop device works fine, as demonstrated in the example above. There's really little very difference between the two cases from a practical perspective, but apparently the loop device is necessary to fake out the kernel enough to allow the ioctl().

Getting Into The Encrypted Volume

In any event, we're dealing with an encrypted volume and we want to access the data that's inside of it. That means that we're going to have to discover the pass phrase that was used to encrypt the volume. There's no magic short-cut to getting through this pass phrase: you either need the owner of the system to tell it to you or leave it written down on a handy Post-It Note(tm). A brute-force attack is possible, but you'd probably better assume that anybody who went to the trouble of encrypting their hard drive also went to the trouble of picking a long pass phrase. Good luck with that.

But let's be optimistic and assume that you've obtained the pass phrase to decrypt the drive. With that hurdle out of the way, actually getting into the encrypted volume is a simple cryptsetup command:

"cryptsetup luksOpen ..." prompts you to enter the pass phrase. Assuming you enter the pass phrase correctly, it unlocks the appropriate encryption key and sets up a device mapping to the unencrypted device. The name of the mapped device- "unencrypted" in this case- is supplied as the last argument to "luksOpen". It doesn't really matter what you call this device. Just pick a unique name.

Now it's possible that the encrypted volume contains a Linux file system rather than a series of LVM2 volumes. Let's see what fsstat has to say:

# fsstat /dev/mapper/unencryptedCannot determine file system type

No joy. It could be an XFS, JFS, or ResierFS volume or some other file system type that fsstat doesn't recognize. But let's forge ahead under the assumption that LVM2 is going to come into play.

Dealing with LVM2 Volumes

Regardless of whether the image was encrypted or not, the process is the same for decoding any LVM2 configuration that might be present. In the case of an encrypted volume, the LVM2 commands will operate on the /dev/mapper/unencrypted device created by cryptsetup. If there is no encryption layer, then the commands will operate on the /dev/loop0 device we created with losetup. As you'll see in the examples below, you won't actually be specifying the device name, so the difference is transparent to the user.

The first step is to run vgscan ("volume group scan") to look for LVM2 volume groups attached to system devices:

# vgscan Reading all physical volumes. This may take a while... Found volume group "RAID6" using metadata type lvm2 Found volume group "VG000" using metadata type lvm2 Found volume group "Root" using metadata type lvm2

The vgscan command actually found several volume groups on my analysis machine. However, two of them were pre-existing volumes that I had configured when installing the system. You can use the pvdisplay ("physical volume display") command to show the volume group associated with a particular device:

The vgchange command makes VG000 active ("-a y"). Then lvscan ("logical volume scan") scans all active volume groups looking for logical volumes. Unfortunately, there is no way to tell lvscan to restrict the scan to a particular volume group, so it will report all logical volumes on all volume groups.

But as a result of these commands we now have a host of new device nodes associated with VG000. You can pull strings out of these devices using strings or srch_strings. You can apply your favorite forensic tools to these devices. You could even dd the devices if you wanted to make a raw image file of each partition.

And of course we can actually reassemble the file system now using the mount command. In this case, the names of the logical volumes are a clue to where each volume should be mounted. In the absence of a meaningful naming scheme, I suggest using fsstat on each partition to find out where it was last mounted. Alternatively you can try mounting each volume in turn until you locate the root file system and the /etc/fstab file, which will tell you where the other volumes should be mounted.

For the sake of completeness, let's go ahead and put the entire file system back together:

Mounting the logical volumes is as simple as calling mount on the appropriate devices. You'll notice that I'm specifying the "ro" ("read-only") option here even though the underlying loop device we created with losetup was also created as a read-only device. In my world, it never hurts to be careful. We're also using "noexec" here so that we don't accidentally end up executing any (possibly malicious) binaries from our mounted image.

If we also want to mount the /boot partition, then we need to fish that out of our original disk image. Like losetup, the mount command wants us to specify the image offset in bytes- in fact the mount command is really calling losetup under the covers and the offset option here is being handed directly to the losetup command. So we calculate the byte offset of the start of the /boot partition from the output of mmls and use that in our list of options to mount.

Great! Now we have a fully connected file system that we can investigate using standard file system forensic tools, or even simple Unix commands like find and grep. And we've gone from a raw disk image of a an encrypted LVM2 volume to a fully operational file system. And that's pretty awesome.

Tearing It All Down

But what about when the investigation is over and we don't need to look at the image anymore? How do we close things down neatly?

The first step is to unmount any file systems you may have mounted out of the image:

You'll notice that the vgchange command above is nearly identical to the command we used to activate the volume, except that we use "-a n" to mark the volume as not active.

Next we need to tear down the /dev/mapper/unencrypted device we created with cryptsetup. As you might be guessing at this point, there's a cryptsetup sub-command for this operation:

# cryptsetup luksClose /dev/mapper/unencrypted

Last but not least, we should drop the /dev/loop0 device we created:

# losetup -d /dev/loop0

And incidentally this teardown process has been a nice review of all of the different layers we had to drive through in order to accomplish our mission!

But let's also confirm that none of our operations has modified the original disk image:

# md5sum sda.ddf4c7a8d54b9b0b0b73ec03ef4cf52f42 sda.dd

You can either scroll back up to the top of this article, or just take my word for it. We're good.

Wrapping Up

Encryption and LVM2 can add substantial complexity to the task of accessing data in your Linux disk images. But the commands presented here should allow you to navigate these murky waters with no difficulty. And taking care with your use of the "read-only" option at various strategic points can prevent corruption of your image file.

Hal Pomeranz is an Independent IT/Security Consultant, a SANS Institute Faculty Fellow, and a GCFA. He can often be found in a darkened room, gazing into a monitor, cackling softly to himself, and muttering, "My god! It's full of stars!"

Hal Pomeranz

I should mark Rob's comment as spam'' just because.Oh fearless leader, I am indeed working on such a course even as we, er, type. Articles such as this one are what we call "research".But we must keep this course a secret or else the wailing of the multitudes demanding its delivery shall distract us from our glorious task.