Until recently, antivirus programs worked exclusively by analysis file code. The earliest signature-based detection methods focused on searching for exact byte sequences, often at a fixed offset from the beginning of the file, in a malicious program's binary code. Later heuristic detection methods also used file code, but with a more flexible, probability based approach to searching for common malware byte sequences. Obviously, it’s not difficult for malicious programs to get around that kind of protection if each copy of the program includes a new byte sequence.

This task is fulfillled by the application of polymorphic and metamorphic techniques, which essentially - without getting into the technological nitty-gritty – enable a malicious program to mutate at byte level when the program creates a copy of itself. Meanwhile, the program’s functionality remains unchanged. Encryption and obfuscation are primarily used to hinder code analysis, but when they are implemented in a certain way, the result can be a variation of polymorphism – an example here is again Cascade, where every copy of the virus was encrypted with a unique key. Obfuscation may just hinder analysis, but when it is applied in a different way to every copy of a malicious program, it hinders the effective use of signature-based detection methods. However, it cannot be said that any one of the abovementioned tactics is more effective than any other in terms of malware self-defense. It would be more correct to say that the effectiveness of these techniques depends on the specific circumstances and how the techniques are implemented.

The use of polymorphism only became relatively widespread in terms of DOS file viruses. There's a reason for this. Writing polymorphic code is a highly time-consuming task that is really only justified in cases when a malicious program is self-replicating: then each new copy contains a more or less unique byte sequence. The majority of contemporary Trojans aren't able to self-replication, and polymorphism is therefore irrelevant. That’s why since the end of the DOS file virus era, polymorphism has been seen less, and it was used mostly by virus writers who wanted to show off their skills rather than to create a particularly useful malicious function.




Figure 2. The polymorphic code of P2P-Worm.Win32.Polip

In contrast, obfuscation continues to be used today, as are other code modification methods that, to a large extent, make it more difficult to analyze code as opposed to hindering detection.



Figure 3. A diagram of the obfuscated code in Trojan-Dropper.Win32.Small.ue - Click for larger version