3 Answers
3

It is one of the obfuscation techniques employed by some viruses to mask their presence and avoid detection and facilitate easier spreading (affecting other systems).

Such viruses would encrypt (self-encrypt in case of metamorphic or polymorphic viruses that can mutate with spreading) their payload and as such prevent direct examination, which can result in reduced ability of antivirus software to detect, and/or clean them. They are difficult to detect by fast antivirus detection methods such as checking file signature against virus definitions tables, if the payload moves position within it, effectively changing file signature (metamorphism). Metamorphic viruses don't necessarily use encryption though.

Out of the self-decrypting bunch of the nasties, the hardest to detect are polymorphic viruses, which can change both the encryption scheme (and/or use a random encryption key) for the payload, as well as mutate decryption code itself (the part of the file that is not encrypted and decrypts the payload). Only reliable way to detect such viruses with antivirus software is by deep heuristics, inspecting signatures of smaller chunks of their structure, and/or by running them in a sandbox and inspecting what they actually do. Such scanning techniques are of course compute expensive, disabled by default for speed concerns (and possibly to avoid false positives) on consumer-grade software, and not many users ever enable this on distributions that are actually capable of such advanced detection methods.

Apart from the standard self-encrypting/decrypting virus, there is a new virus encryption technique in use. Last year a high-tech professional grade virus named Gauss was discovered with an encrypted payload. The clever part of this scheme is that the encrypted payload uses data from the intended victim's computer as the decryption key. Until the virus is installed on a very specific victim's machine, no analyst can decrypt what the payload will do.

In the case of Gauss, the key is made up of a combination of two values. The virus builds a list entries in the PATH environment variable, and a list of folder names in the Program Files directory. Each pair from the two lists are combined, a salt is added, it's run through 10,000 iterations of MD5 (fairly similar to the operation of PBKDF2) and then a decryption is attempted using the hash as the key. If it fails, the next folder name is paired, and the cycle is repeated. If it is successful, the payload will be decrypted and executed. To date, nobody has yet announced the discovery of the pair of terms that will unlock the payload.

Gauss already has unencrypted malware. It can monitor keystrokes and steal money from accounts in a certain Lebanese bank. The nature of the encrypted payload is expected to be far more dramatic, something along the lines of Stuxnet, famous for destroying Iranian centrifuges.

An encrypted virus consist of a decryption routine and encrypted virus payload. If a user launches such a virus then:

The virus decryption routine executes which decrypts the virus payload.

Then the decryption routine executes the encrypted virus payload.

Each time a new file is infected, it make a copy of both decrypted virus payload and decryption routine. After infection it encrypts the virus with payload with the new encryption key and attach itself along with decryption routine to the target file.

As encryption key changes from infection to infection, because of which the virus payload body changes, making the virus payload appearing different from infection to infection. This makes it extremely difficult for anti-virus software to search for a virus signature extracted from a consistent virus body.

The decryption routines remain constant from infection to infection a weakness that
anti-virus software can exploit.

But this is a very basic technique and more and more sophisticated techniques are used by malware writers to bypass AV detection.