In response to the ADD work, Jack Crook wrote the blog postForensic Analysis of Anti-Forensic Activitieswhere he analyzes the artifacts that ADD leaves behind, and the Volatility Blog published a rather confident piece TitledADD: The Next Big Threat To Memory Forensics….Or Notclaiming that anti-forensics is dead, and the author reassures us that "There is a significant amount of work that needs to be done for this to change, so while the attackers are spending their weeks and months trying to build things up to spec, rest assured that with proper training and the right tools, you won’t need to worry about future versions.". So all we need is training and tools apparently.

In my opinion, the reality is slightly less bright than the Volatility blog post would like us to believe. Anti-forensics can be very effective and being overconfident is really setting an investigator up for a fall. I think the topic of anti-forensics is one that needs wider discussion in the forensic community and to be taken seriously.

There are two goals that anti-fornsic techniques can implement:

Prevent the analyst from analyzing the image in the first place.

Plant evidence to decoy or confuse the investigation (or even to present doubt to the whole thing in a court proceeding).

We really do need to consider both goals, as currently these are both very effective. It does not pay to just "shoot the messenger" without considering the message itself.ADDis not a perfect tool - it is merely a proof of concept, but it can easily be changed to defeat Volatility or any other memory analysis tool.

Goal 1: Disable memory analysis tools.

To achieve this goal the anti-forensic tool simply needs to prevent the initial acquisition, or the subsequent analysis. We have already published how a tool can interfere with memory acquisition by manipulating the data structure that back theMmGetPhysicalMemoryRanges()function. Virtually all memory acquisition tools use this function to work out where they should be reading. But since this function is undocumented and is not normally used, it is quite safe to patch it without affecting system stability.

Memory acquisition tools also typically use a couple of well known APIs such asMmMapIoSpace()andZwMapViewOfSection()which have previously been patched as far back as 2006 inDarren Bilby’s ddfywork.

So anti-forensics is definitely not dead in the memory acquisition world.

How about the analysis world? It is easy to disable a memory analysis framework, such asVolatilityorRekallsince the tool makes many assumptions which may not be true. For example, we have run a hands on memory forensic workshop during the 2013 Open Source Digital Forensic Conference (OSDFC) workshop, titledMemory Forensics With Volatility. In that workshop, we have used theWinPmemdriver (now part of theRekall framework) to actively manipulate physical memory to examine how good the analysis tool was. (You can still view the workshop onYou-tube).

At first we corrupted the"KDBG"owner tag from theKdDebuggerDataBlockstructure. The first thing that Volatility does is to recover theKDBGstructure from memory (Rekall no longer uses the KDBG but that is the topic for another post). Without a validKDBGit is very hard to use Volatility since it uses it to locate thePsActiveProcessHeaddata structure so it can list processes etc. In fact the Volatility blog postThe Secret to 64-bit Windows 8 and 2012 Raw Memory Dump Forensicsstresses the importance of finding the correct KDBG structure. The important thing, however, is that the KDBG structure is not really used in Windows under normal circumstances (its used by the kernel debugger and to produce a crash dump). So removing it does not affect system stability.

It is quite trivial to remove the KDBG signature, in which case Volatility is unable to proceed without a lot of hand holding. It is also possible to add false positive (sincekdbgscanis just a simple scan for a known signature), which can force Volatility to present whatever results one wishes when traversing lists such as the process list or the module list (i.e. one can create entirefakelists which are not even used by the kernel at all).

In fact quoting from the Volatility blog post "In other words, tools that use object carving (i.e. pattern matching, scanning) as an analysis technique are implicitly susceptible to attacks that create objects that look like the ones being carved.". One wonders what that statement means to the analysis process, since the entire Volatility analysis process begins with a fuzzy signature matching carve for theKDBGstructure.

Goal 2: Plant or change evidence.

This goal for an anti-forensic tool is to confuse the analysis process. It is actually much easier than it sounds since the analysis process is usually focused around the output of the tool. So all one needs to do is find data that the tool presents (but is not essential for system stability) and change it to say whatever they want. Without prior knowledge most investigators will simply trust the output of their tool.

For example, in ourOSDFCworkshop, we illustrated how one can simply overwrite the filename strings presented in thevadplugin output to whatever you want (One can similar change protection flags). Since the filename itself is not used to read from the file once the handle is created, it does not matter what is written there. This can be used to influence the output of thevadinfoplugin (either to hide or plant evidence). Similarly one can alter process names, pid-ppid relations, driver names etc, and ever remove entire VAD nodes.

In fact, during that workshop we used the actual Volatility tool to tell us where the offset of the strings were in physical memory. This way we knew exactly which bytes to change in the physical memory, so that the tool output said what we wanted. The only caveat was that whatever we changed was not critical for stable system operation. Since memory forensic tools are designed to find the evidence and tell you where it is, they can all be easily misused to tell you which bytes in memory you should alter to reflect in the tool output.

Many Volatility plugins rely on pool scans which use fixed signatures. A good anti-forensic technique is to simply overwrite the pool tags (which are just known strings used for debugging) - this renders all the volatility pool scanning plugins worthless. Most other scanning plugins also use artifacts which can easily be changed to avoid signatures matching.

Conclusions

As much as I would love to live in a world where anti-forensics is not effective, the real world is less perfect. As investigators we need to acknowledge the fact that we are not perfect and can easily be fooled by anti-forensic techniques. It is an arms race after all. A little bit of training can improve your ability to detect these techniques, but you can never be sure.

Just a quick word on Jack Crook’s blog post. It is important to realize that Jack went about his analysis with 20/20 vision knowing exactly what he was looking for. If you re-read he analysis without the knowledge that ADD was at play you would see:

A process inpsscanwhich does not appear in any of the other lists. (This is perfectly plausible for existed processes).

Some command line output extracted with strings that mention the suspicious process (OK Maybe the authors of ADD did not have great opsec, they could have scripted the tool better.).

Netscanoutput which does not agree with the output of the apparently unpublishedtcpscanplugin. This is still consistent with the theory that the process has existed or closed all of its connections, but we don’t really know since thetcpscanmodule is not published so we don’t know how it works.

The file paths start with "\private\" instead of "\Device\HarddiskVolume0\". Maybe an oversight on behalf of the ADD authors - I am sure the string can say whatever you want.

Finally Jack finds the string "ADD" in various forms in the strings output (e.g.add.exeand the bat script which runs the add binary with all the parameters). Had he not known in advance the tool was calledadd.exeI don’t think this would have been that suspicious. (We still resort to memory analysis using strings in 2014?)

It is not constructive to pick the faults in a specific implementation of an anti-forensic POC, ignoring the overall lesson - which is that anti-forensics is actually very effective, especially when you don’t know it is there. If one re-reads Jack Crook’s blog post without the knowledge that there is anti-forensics employed, one can see that most investigators will probably fall for the story that the tool is trying to sell: "There was an exited process which had these sensitive files open and also had network connections.".

Update: An earlier version of this post erroneously attributed the blog post to Brent Muir, my apologies.

Monday, December 10, 2012

Volatility uses a profile to encapsulates information about a specific version of an operating systems. As internal kernel data structures change from release to release, so too must volatility adapt to these changes. The profile is usually automatically generated from debugging symbols taken from a specific version of the operating system. For analyzing windows systems, Volatility comes with a number of pre-generated profiles for the common windows releases out there. For Linux, a profile must almost always be generated for the system under examination, since the layout of Linux data structures change, even depending on local configuration changes. For that reason we wont be discussing Linux profiles in this post.

Previous versions of Volatility required the profile to always be explicitly specified. This is fine if you know in advance what version of windows you have, but sometimes you receive an image taken by a third party which has no context - you don’t exactly know the version or patch level of the image. In previous versions of volatility you would need to run the imageinfo plugin. This plugin uses difference in the KDBG structure between windows versions to guess the correct profile. You would then need to copy the profile detected and provide it to all future invocations with the --profile options:

The old imageinfo plugin takes a long time to run since it instantiates every profile in turn. It is OK to use this plugin to discover the profile name initially, but then we still need to provide it again.

Previous versions of Volatility used incidental information in the kernel debugger block to guess the profile (for example, the size of the KdDebuggerDataBlock struct). This information needed to be hand maintained for each profile since it is not present in debug symbols. In the next Volatility version, this information is not used, and even the KDBG scanner does not use profile specific information (The kernel debugger block layout is mostly identical for all versions of Windows).

For the next version of Volatility, I wanted to have an automatic profile selection system, so users just do not need to think about it. It is not necessary to even provide the profile at all. To make this work we need to have a very quick but very reliable method of detecting the profile.

Finding the kernel DTB

One of the first things Volatility does when opening the image is to detect the kernel Directory Table Base. This is the physical address of the kernel’s base of the page tables. Without a valid DTB it is impossible to create the kernel’s virtual address space - and hence this is the first thing we need.

Volatility finds out the address of the DTB by scanning for a well known _EPROCESS signature corresponding to a kernel process. Volatility 2.x scans for theIdle process by searching to the string "Idle". Once the Idle process is found, we can deduce the kernel DTB by following the _EPROCESS.Pcb.DirectoryTableBase member. In order to follow this member we need to know about the memory layout of the _EPROCESS struct - which we get from the profile. Therefore in the old 2.x releases of Volatility, we needed to know the profile before determining the DTB.

Guessing the profile

With the next generation of Volatility, profile auto-selection is combined with the DTB scan in order to kill two birds at once. The guess_profile plugin is automatically run by the framework when the profile is not provided. This plugin scans for both the DTB and profile parameters at the same time. Deciding on the correct profile to choose comes down to the differences in the _EPROCESS struct layout alone (i.e. we no longer look at differences in the KDBG structure). In the new version, we search for the "System" process instead of the "Idle" process since it is more robust for verification (The Idle process does not always have valid threads, and is not always in the PsActiveProcessList list).

Since profiles are large and rather expensive to load and compile (There are now over 20 different Windows profiles), we pre-generate a concise and very small test profile containing important _EPROCESS member offsets for all the currently supported profiles. We then load this very small concise profile and overlay each version of the _EPROCESS struct over the System process signature, ensuring to sanity check the match:

Search for a string "System" - this should be the ImageFileName member.

For each test profile in the concise look-up map, perform the following:

Go back to the start of the _EPROCESS and get the DirectoryTableBase member. Validate the DTB to eliminate obviously wrong values.

Instantiate a virtual address space using the DTB value, and use this address space to reflect through both the ActiveProcessLinks and ThreadListHead linked lists. This gives a pretty strong signal that the _EPROCESS address is valid.

When we get here we are pretty certain to have found the correct profile and DTB.

When the System _EPROCESS is identified, we set both the DTB and the profile. The above process is pretty fast and actually does not add any more processing than in previous versions. We are still scanning the image once, looking for the System process signature. This algorithm is also very robust.

Due to the low cost of this plugin in comparison to the previous imageinfo plugin, we can afford to run this by default at every run. This improves usability since the user doesn’t even need to think about a profile - they just run Volatility as normal:

Guessing a profile from KDBG

Sometimes we already know a lot of information about the image before we start. For example, when analyzing a crash dump, we already have the DirectoryTableBase member, as well as the KdDebuggerDataBlock address provided for us in the crash dump _DMP_HEADER64 header.

Similarly when using the WinPmem driver for analyzing live systems, we can obtain these structures directly from the driver which collects the kernel’s CR3 and the KdDebuggerDataBlock address.

In these cases, profile auto-selection does add a significant amount of work to startup, since we would not normally need to perform a DTB scan at all, but without a profile we still need to scan for the System process as described above.

To avoid this we need to implement a slightly different algorithm for profile selection when the KDBG and DTB are already known:

Locate the _KDDEBUGGER_DATA64.PsActiveProcessHead member for the head of the process list. We also instantiate a virtual kernel address space using the provided DTB.

For each _EPROCESS in the test profile, attempt to get the next process in the list from the PsActiveProcessHead. We can then verify this process in the same way as above (i.e. check that we can properly reflect through its lists etc). The profile which works is the correct profile to select.

This method is extremely fast and does not actually add any time to startup time.

So now we can avoid having to specify the profile altogether, making the user experience much smoother. As a result the imageinfo plugin is now deprecated (in its current form) in the Tech Preview branch since it is no longer useful.

Sunday, November 18, 2012

The kernel debugger block (named KdDebuggerDataBlock of the type _KDDEBUGGER_DATA64) is important for many things that Volatility and debuggers do. For example, it has a reference to the PsActiveProcessHead which is the list head of all processes required for process listing.

Because it is so useful the location of the kernel debugger block is also stored in the crash dump header. This assists the kernel debugger to find the _KDDEBUGGER_DATA64 quickly.

I wanted to add the ability to write crash dump files to the WinPmem memory acquisition tool. Many commercial acquisition tools also offer this useful feature. Storing the image in a crash dump format is convenient since it is possible to open the image using a variety of tools such as volatility and Microsoft’s own kernel debugger Windbg. (Of course its possible to convert a raw image to a crash dump using volatility’s raw2dmp plugins but its more convenient to take the image using a crash dump in the first place).

The first method involves finding the KPCR (Which I talked about previously) and following its KdVersionBlock member. This method is ideal for performing on the running system since the KPCR is simply stored in the fs/gs register of the running thread.

The second method which is used by Volatility itself is to scan for KdDebuggerDataBlock using a specific signature for a valid _KDDEBUGGER_DATA64.

Both of these methods are not ideal. The first method was used by win32dd for a time and Matthew Suiche wrote about it here. Unfortunately this method stopped working in recent versions of Windows. Recently the KdVersionBlock member is always 0 and does not link to the kernel debugger block.

The second method would work but would be quite slow as it scans the entire kernel address space for the KDBG signature. It is also less reliable since there could be a number of hits for the KDBG signature and we might get the wrong hit.

There has to be an easier way :).

The first thing I asked myself was "What exactly is the kernel debugger block?". I used google to locate a definition for it in the sources for ReactOS (an open source reimplementation of windows). The relevant snippet shows:

So according to the ReactOS sources the debugger block is a statically allocated structure which seems to be filled in by KdInitSystem.

This is quite interesting since static structs are always located in the same position relative to the PE executable’s base address. The compiler usually places static variables in the .data section of the PE binary. This means that in practice, the KdDebuggerDataBlock can only exist within the .data section of the kernel binary.

The problem boils down to finding the kernel base address. Once we know that, we can easily find the valid ranges for the data section. In most kernel’s the .data section is very small (less than 100kb) and is super fast to scan. Also since there is only deterministic data there, there will never be another struct with the same signature in such a small region.

So how can the WinPmem driver locate the kernel base address? The base address itself is not actually exported, but many other symbols are exported from the kernel’s Export Address Tables (pretty much any kernel API). We know that the kernel export address table is usually located at a higher address than the kernel base, and that the kernel base address is page aligned. We now can scan for it:

Figure 1. KDBG Scan algorithm

We choose to use NtBuildNumber as the exported symbol to look for (just in case a real API function is hooked). We then round down to page align and step backwards looking for the PE header. In Windows 7, there are unmapped guard pages between the mapped kernel sections - trying to read from these will blue screen the system. We therefore check first that the pages are mapped using the MmIsAddressValid() API.

Once the PE header is found, we calculate the region for the .data section. Since this region is very small and we do not need to worry about false positives, we can relax the search criteria and just search for the OwnerTag "KDBG".

The result is an extremely fast and reliable way for locating the kernel debugger block in a live system. This allows the WinPmem kernel driver to report this address so that the user space component can write the crash dump files correctly.

The added benefit of the kernel driver locating the debugger block itself is that Volatility does not need to scan for it while it is analysing the live system. This saves a rather expensive call for the kdbgscan plugin, which would otherwise need to be made before running most plugins.

Saturday, November 17, 2012

Memory acquisition is the first step in memory analysis. Before any analysis can be done, we need to acquire the memory in the first place. There are a number of commercial solutions to acquire memory, but sadly open source solutions have been abandoned or not maintained (For example win32dd has been a popular solution many years ago but has now been commercialized and is no longer open source).

We believe in open source forensic tools to make testing and transparency easier. We also believe that the availability of open source solutions spurs further development in the field and enables choices.

That is the reason we feel an open source, well tested and capable forensic memory acquisition tool is essential - we call it the Pmem suite of tools. The pmem acquisition tool aims to provide a complete imaging solution for Windows, Linux and OSX.

The following is a quick overview of how to use the pmem tools. For detailed information consult the source.

1. WinPmem

The windows memory acquisition tool is called WinPmem.

These are the features it supports:

Supports all windows versions from WinXP SP2 to Windows 8 in both i386 and amd64 flavours.

Output formats include:

Raw memory images.

Microsoft Crashdump files for use in windbg and volatility.

Output to stdout (in both the above formats) for piping through other tools (e.g. ssh, ewfacquirestream etc).

Optional Write support for manipulating kernel data structures from Volatility.

1.1. Download

The latest version can be found here or on the Volatility download page. You will find the tool released in two versions:

winpmem-1.3.1.exe: is the recommended binary for general use. This binary contains signed drivers so it can load on any windows system (even 64 bit ones). This binary does not include write support for memory.

winpmem-1.3.1-write.exe: is the binary with write support enabled. It is not signed so it will only work on 32 bit windows or 64 bit windows with special preparation (see below).

The recommended version for regular use is the one without write support. The version with write support can not be used on a regular system.

1.3. Experimental write support

As from Version 1.1, the winpmem drivers support writing to memory as well as reading. This capability is a great learning tool since many rootkit hiding techniques can be emulated by writing to memory directly. For example the following Volatility session illustrates changing the name of the binary:

Since this is a rather dangerous capability, the signed binary drivers have write support disabled. The unsigned binaries (really self signed with a test certificate) can not load on a regular system due to them being test self signed. You can allow the unsigned drivers to be loaded on a test system by issuing (seehttp://msdn.microsoft.com/en-us/library/windows/hardware/ff553484(v=vs.85).aspx):

Bcdedit.exe -set TESTSIGNING ON

and reboot. You will see a small "Test Mode" text on the desktop to remind you that this machine is configured for test signed drivers.

Alternatively you can test this on XP or Vista32 which have no driver signing restrictions.

Once the correct driver is loaded, Write support must also be enabled at load time using the -w switch:

winpmem_1.3-write.exe -w -l

This will load the drivers and turn on write support. Then we can run volatility interactively, as usual on the raw device:

Thursday, October 11, 2012

I was looking into finding the KPCR structure in memory images. The KPCR is used to store a lot of information about the running CPU and in windows version prior to Windows 7 also contains a link to the kernel debugger block.

Volatility already contains a scanner for KPCR (in this example the image is 821676355 bytes big):

So the scanner compares each 4 byte position in the image by overlaying the _KPCR struct over it and checking that _KPCR.Self points at the address of _KPCR.

This is extremely slow since it will perform millions of comparisons in python as well as many very small read operations from the raw image (IO operations are very expensive on windows systems making this even slower).

There must be an easier way to find KPCR!

The solution is to examine the _KPCR object and see what we can leverage so we do not need to resort to an exhaustive search:

nt!_KTHREAD
The WaitListEntry field is used to add the KTHREAD structure to the list of
threads that have entered into a wait state on a particular CPU. The
WaitListHead field of the Kernel Processor Control Region (KPRCB) structure for
every CPU links such threads together via the KTHREAD.WaitListEntry
field. Threads are added to this list by the function KiCommitThreadWait() and
removed from this list by KiSignalThread().

So the trick is to enumerate all threads and see which ones are waiting to run on this CPU. In order to find the kernel DTB we already are searching for the System process so we usually already have one _EPROCESS object we know about.

00001:for kthread in task.Pcb.ThreadListHead.list_of_type(00002:"_KTHREAD","ThreadListEntry"):00003:00004:# Look for threads in the Wait state. If this thread is in the Wait00005:# state, the WaitListEntry will belong to the list of all waiting00006:# threads. By following this list we should get to the list head00007:# which lives inside the _KPCR object.00008:for kwaiter in kthread.WaitListEntry.list_of_type(00009:"_KTHREAD","WaitListEntry"):00010:00011:# Assume the kwaiter is actually the KPRCB.WaitListHead.00012: possible_kpcr = self.profile._KPCR(00013: kwaiter.WaitListEntry.obj_offset - offset)00014:00015:# Check for validity using the usual condition.00016:if possible_kpcr.Self == possible_kpcr.obj_offset:00017:if possible_kpcr.obj_offset notin seen:00018: seen[possible_kpcr.obj_offset]= possible_kpcr

Unfortunately this only works on later versions of windows than XP right now since the WaitListHead was introduced with Windows 2003.

PS - I originally wanted to examine the KPCR as a way of quickly retrieving the KDBG but this does not work as recent windows versions set KdVersionBlock to 0. We can not use the KPCR as a substitute for the DTB scan either since all this work is done in the kernel virtual address space (so we need a DTB already). In practice DTB scanning is very quick and KDBG scans are also not too bad so its not a huge problem.