Antivirus analysts and security testers have to deal with a fundamental question every day: Is obfuscated code good or bad?


Sun Tzu counseled a strategy of maneuver warfare, and that is the doctrine followed by modern militaries. We need to find something different than the attrition warfare that sustains the malware ecosystem in the state it is in today.


Obfuscation, the deliberate hiding of the software's behavior, is used by malware authors as well as legitimate software developers. They both use code obfuscation techniques to keep curious souls from understanding how their software works and what it is doing to the computer on which it runs.

As used by both malware and legitimate software, obfuscation benefits the software writer at the expense of the software user who is left in the dark about what is happening on their computer. Many software products try to hide how they work to block attempts at reverse engineering the code. Legitimate reasons exist for doing this, including content protection, license enforcement, protection of the intellectual property of the software itself, or to hide behavior the user might not appreciate.

On the other hand, malware authors use a variety of techniques to obfuscate their code's malicious behavior and to make sure the program is not recognized by antivirus scanners.

Obfuscation by itself is not good or bad but it has led to the situation where users cant determine if the behavior of obfuscated software is good or bad. While it would be a controversial measure, perhaps it's time to treat any code that exhibits unknown behavior as bad.

If obfuscation technology was ever perfected we would have perfect DRM and perfect malware. Yet, that outcome is unlikely. The computer ultimately has to decipher and follow a software programs true instructions. Each new obfuscation technique has to abide by this requirement and, thus, will be able to be reverse engineered.

The battle between anti-reverse engineering techniques and the efforts of competitors, software pirates or anti-malware researchers has led to malware and legitimate software that is designed to be updated after a period of time. Most DRM systems will force an update once an adversary has broken the system, thus restarting the reverse engineering clock. Malware, especially botnets, are moving to a model where the software updates itself on the command of its master. By the time malware researchers determine how to detect or remove a bot, many existing bots will be updated to use new software, once again starting the reverse engineering clock.

In warfare, such a battle is known as a war of attrition. Both the reverse engineer and the software writer are attempting to wear the other down. As far as I can tell neither side is worn down and it is simply the user that loses. Sun Tzu, the famous military theorist, warned against getting into a war of attrition — even if you could win, it would be a pyrrhic victory as the cost would be very high.

Sun Tzu counseled a strategy of maneuver warfare, and that is the doctrine followed by modern militaries. We need to find something different than the attrition warfare that sustains the malware ecosystem in the state it is in today. What if computer users were to take the stance that obfuscated software, by its very nature of hiding its behavior, was to risky to run on their machines? Why should the user trust software with behavior that cannot be verified? The software could be malware. It could have a back door. It could leak information, violating the users privacy. It could shutdown, if it is used in a way the software author didnt intend.

Reverse engineering is clearly of benefit to both defenders and attackers, if both legitimate software and malware are trying to protect themselves from it. Reverse-engineered software cant hide its behavior from the user. The user is free to see how the software works, make decisions to either use or discard the software, or perhaps even modify the software.

Chris Wysopal is co-founder and CTO of Veracode, a provider of on-demand software security testing services. Chris co-authored the password auditing tool L0phtCrack and was a researcher at the security think tank, L0pht Heavy Industries. He has held key roles at @stake and Symantec and is the author of The Art of Software Security Testing: Identifying Security Flaws.