Unpacking a suspicious dll — top to bottom

Fyi, this writeup is geared to satisfy curiosities about technical stuff, to start responding to some of the interest expressed over at our forum. You have been warned…

We use Ollydbg for all sorts of things around here. It’s an outstanding tool. In fact, Olly himself found some spare time and is releasing a new version soon. He’s got the pre-alpha version 2 code available on his website.

His debugger is a very useful tool for reversing user-mode software. When we’re looking to get to the bottom of a suspicious component, one way is to fire up olly and get started. Unfortunately, there are challenges to that approach. Sometimes, we need to understand what a dll or other component is doing as well, and sometimes those dlls and other components are packed.

There are other tools that we use, and this post will survey the steps for using them while unpacking a dll…you can find this sort of information all over the web, but the writing styles sometimes make understanding the content very difficult.
Some of the fine reverse engineering tools available are
Ollydbg
LordPE
Import Reconstructor
IDA Pro

In our labs, we have a suspicious dll to examine. Apparently, it was installed as a bho into Internet Explorer:

When you load this dll into Olly, the tool reports that its listing of the binary’s instructions are most likely inaccurate. IDA Pro can’t disassemble the binary either.
So we can use a couple of tools to help identify if this executable has been tampered with. One popular tool is PEiD. PEiD detects “Upack” as the packer used here, and usually is pretty accurate. You can also take a peek with ProtectionID.
Upack is a very simple packer, used to compress executables, and can make file examination only somewhat difficult. There are no antidebugging tricks that it employs to be concerned with. Here is PEiD in action, identifying the file as packed with UPack by Dwing:

If we want to load it into olly and dump it for full unpacking, one way to start the unpacking process is to simply rename the file extension to “exe” and modify a flag in its PE header so that windows loads the file as an exe, not a dll. You can take a course from a reverser like Jason Geffner on deobfuscation and read all the PE documents, then perform the math, pop open Ultraedit or hexedit and manually edit the file’s PE header. Or you can run LordPE on the file and simply deselect the “Dll” checkbox under its file characteristics:

After you save your modification, load up the file into Olly and identify the program’s original entry point, or OEP. This work can be time consuming when learning about a new packer. But Upack is a simple packer. It’s much like UPX, the industry standard, but it uses the LZMA compression algorithm. A reverser might notice that the first instruction of the unpacker is “pushad”, followed by a call instruction:

The easiest thing to do would be to scan the rest of this section for a matching “popad” instruction followed by a jmp to the beginning of the lzma decompressed code. When we do that, we find a popad (a restore of all the register values that were pushed onto the stack at the beginning of the unpacker stub) followed immediately by a jmp to the .Upack section that was previously empty:

At this point, we can hit “F7″ to step into this new code section, use Olly’s “Analyse” function and voila, we see
push ebp
mov ebp, esp
and we are most likely at the dll’s original entry point (OEP):

Great! Now, using LordPE again we can dump the file to disk and fix up the Imports with ImpRec. Here’s a view of LordPE options for attaching to a process and dumping an individual module to disk:

Now that we have the image dumped to disk, we can use Import Reconstructor to attach to the dll’s process as it is suspended at its OEP, find the import address table in memory and then fixup the dumped image on disk:

We have to provide ImpRec with the OEP. Hopefully it then can find the Import Directory and IAT for us, and with UPack, it reliably completes the fixup for us. Clicking on “Fix dump” and selecting the image dumped by LordPE will provide us with an unpacked file that we can next throw into IDA Pro for disassembly and analysis, which will be another post:

Hope that satisfies some of the curiosities of our forum readers, next we’ll take a look at some of the malicious behaviors this dll performs.

Note- This example worked through one of the simplest packers out there, Upack. For more information on unpacking tricks, you can find a couple of awesome lists of tips and tricks related to anti-debugging/anti-reversing and at openrce and Mark Vincent Yason’s Blackhat paper.