Malware represents one of the greatest threats that organizations face today and IT departments are coming to understand that their AV tools can only do so much to protect them. When malware is discovered on their systems, they want to know what it might have done, if the threat is still ongoing, and what they might have lost to the infection. Answers can be very tough to find, but reverse engineering the malware might just be the way to provide them.

ITworld|February 17, 2013

While reverse engineering malware may not be something you've ever even considered, after a three day class, I can say that, yes, it's difficult, but also that it's well within the grasp of ordinary mortals (at least those ordinary mortals who can wrap their brains around assembly language and system calls) -- especially if they take the right class, get the right tools, and practice.

What is it?

Reverse engineering malware is the process of taking a captured executable (a stand-alone executable or a library file, such as a DLL) and doing the computer equivalent of an MRI. You should always take some care in where and how you do this kind of work. Analyzing malware in a "safe" environment -- such as a virtual system on a machine that isn't connected to your network -- would be the option of choice. At least this would allow you to snapshot your OS and revert to it whenever you inadvertently kick the malware under your microscope into action.
Unix admins who have some experience using the strings command to identify strings defined within binaries or hex dumps provided by commands such as od will have some rudimentary knowledge of the contents of executables, but there is lot more that you can see if you examine the malware file using a debugger (a tool that allows you to step through a program, watching for the effects that each step makes) or a disassembler (a tool that turns the machine language of your malware into assembly language).
With a disassembler, you would perform what is called "static analysis". With some experience, you will come to see how the malware you are analyzing was intended to work. Maybe it inserts itself into another process, maybe it creates a file, maybe it changes user settings ...
With a debugger, you can break and change the code "on the fly". You can exploit test conditions that may have been designed to make the malware appear benign under certain conditions. You can jump over code that isn't worth your time to analyze. This process is often referred to as "dynamic analysis".

The tools

Some debuggers and disassemblers are free while others cost a bundle, but if you are going to be doing serious work with malware analysis, you should keep in mind that any tool that helps you get answers in hours instead of days should be considered a good investment (do the math!), especially if you will be using it often.
Some of the tools that you are likely to learn about if you take a class or do some online research are IDA Pro, Immunity Debugger and Olly Debugger. The IDA tools are available on Mac OS X as well as Linux. In the AoE class that I took, I had opportunities to work with all of these tools and much appreciated that I had someone walking around the classroom willing to answer all kinds of questions. While these tools all provide options that will help you delve into your malware, I don't think I would have wanted to learn any of them without some start-up help to guide me on how to make the best use of them.

The class

The class that I took was offered by the Art of Exploitation (http://www.artofexploitation.com/academics_malware_rev_eng.aspx) and was both eye-opening and chock full of tips designed to help the new malware analyst past the stumbling blocks and into profitable analysis fairly quickly.
Having a chance to become familiar with some of the better tools and getting tips on how to make the best use of them will help you make good decisions when it comes time to acquire your own.
We spent time ...

looking for system calls that clue you into what the malware is doing

watching out for techniques that malware architects use to make it difficult for us to reverse engineer their creations

understanding how the system stack is used -- or not -- in passing arguments

turning what first appeared to be arbitrary data into clearly articulated code (de-obfuscation)
and

identifying lots of malware "indicators"

This class provided me with ...

surprising insights both into how malware works and how I can analyze it

super tips on what to watch out for and what to ignore

surprising insights into how malware authors try to make their agents of evil hard to analyze -- more than I would ever have imagined

ways to make the process of recognizing malware and identifying what it does considerably faster

how to focus on the "deliverables" -- the answers that your management is waiting to hear

This AoE (Reverse Malware Engineering) class ranks in my eyes as best of breed, top notch, and highly recommended.

Sandra Henry-Stocker has been administering Unix systems for over 25 years. She describes herself as "USL" (Unix as a second language) but remembers enough English to write books and buy groceries. She currently works for TeleCommunication Systems -- a company that builds innovative technologies to make critical connections happen -- where no one else necessarily shares any of her opinions.

The opinions expressed in this blog are those of the author's and do not necessarily represent those of ITworld, its parent, subsidiary or affiliated companies.