This task is fulfillled by the application of polymorphic and metamorphic techniques, which essentially - without getting into the technological nitty-gritty – enable a malicious program to mutate at byte level when the program creates a copy of itself. Meanwhile, the program’s functionality remains unchanged. Encryption and obfuscation are primarily used to hinder code analysis, but when they are implemented in a certain way, the result can be a variation of polymorphism – an example here is again Cascade, where every copy of the virus was encrypted with a unique key. Obfuscation may just hinder analysis, but when it is applied in a different way to every copy of a malicious program, it hinders the effective use of signature-based detection methods. However, it cannot be said that any one of the abovementioned tactics is more effective than any other in terms of malware self-defense. It would be more correct to say that the effectiveness of these techniques depends on the specific circumstances and how the techniques are implemented.
The use of polymorphism only became relatively widespread in terms of DOS file viruses. There's a reason for this. Writing polymorphic code is a highly time-consuming task that is really only justified in cases when a malicious program is self-replicating: then each new copy contains a more or less unique byte sequence. The majority of contemporary Trojans aren't able to self-replication, and polymorphism is therefore irrelevant. That’s why since the end of the DOS file virus era, polymorphism has been seen less, and it was used mostly by virus writers who wanted to show off their skills rather than to create a particularly useful malicious function.
Figure 2. The polymorphic code of P2P-Worm.Win32.Polip
In contrast, obfuscation continues to be used today, as are other code modification methods that, to a large extent, make it more difficult to analyze code as opposed to hindering detection.
Figure 3. A diagram of the obfuscated code in Trojan-Dropper.Win32.Small.ue - Click for larger version
By subscribing to our early morning news update, you will receive a daily digest of the latest security news published on Help Net Security.
With over 500 issues so far, reading our newsletter every Monday morning will keep you up-to-date with security risks out there.