Detecting Polymorphic Malware

Malware creators constantly look for new techniques to stay one step ahead of anti-malware researchers in an effort to avoid detection by antivirus programs. The technique we are going to discuss here is a frequently employed trick that is widely used by web exploits and well-known botnets - server-side polymorphism.

Examples of this technique include Shiz, Carperb and Nrgbot(Dorkbot). The main purpose of these backdoors is to steal credentials for Internet-banking, trade platforms and RBS (remote banking services).

Once released, it is extremely common that the new copy of the malware is not detected by the majority of AV file scanners:

Figure 1 - VirusTotal scan result for the Shiz sample

Thus, it makes detecting malware created using server-side polymorphism more challenging for the traditional signature-based approach.

The idea of polymorphic encryption is not new and lies in re-encrypting a malicious file on the attacker’s servers every time it is requested by an infected bot machine. Let us consider the scheme of polymorphic infection (Figure 2).

Figure 2 - A server-side polymorphism model

Once infected, the user‘s computer sends registering information to a C&C server. The C&C server then replies with a set of commands to execute on the victim’s computer.

A new piece of malware is generated by a “Polymorphic Generator” that re-packs or re-encrypts it with a randomly generated key. This technique ensures that the malware is unique giving it a significant advantage – it will never have been caught and analysed by malware researchers. This vastly increases the likelihood that it will not be detected. The attacker can choose to scan the newly-created copy with popular antimalware products to verify that no detection occurs. Although the file can be scanned by online services, such as Virus Total, malware authors tend not to take this route since the sample will be shared amongst the AV community leading to the file being analysed and added to detection databases. Once the copy is generated and verified as not being detected, it is stored on a “Download Server” and the link is sent to the victim.

Let us take a look at a real-life example. The bot receives a URL to update its original file. A command to reboot the PC follows:

Figure 3 - Nrgbot bot-server communication

The bot then downloads a new backdoor instance:

Figure 4 - Updating Nrgbot

After the “update”, the backdoor becomes invisible to AV signature-based scanners. Moreover, such backdoors often block access to AV websites stopping the user’s security application from downloading new detection database updates.

If we compare two polymorphic instances of the same bot, we will see the following picture:

Despite the significant differences in file content, both samples have identical functionality shown in malicious injections (Figure 6). If antivirus scanners were able to run a sample in a sandbox during scanning, they wouldn’t be tricked by the polymorphic encryption and give an exact family verdict.

Unfortunately, such technology is generally resource consuming and can be implemented on the server side only. However, this fact highlights a possible fusion of cloud and client technologies for better user protection against polymorphic malware.