Sources: polymorphism, obfuscation and encryption
It makes sense to examine polymorphism, obfuscation and encryption together, as they all fulfill the same end, albeit to different degrees. Initially, modification of malicious code had two goals: to make it more difficult to detect files and to make it more difficult for virus analysts to examine the code.
The history of malware began in the 1970s, but the history of malware self-defense didn’t start until the late 1980s. The first virus that attempted to defend itself from the antivirus utilities then in existence was the DOS virus Cascade (Virus.DOS.Cascade). It defended itself by partially encrypting its own code. This wasn’t very successful, however, since each new copy of the virus - despite being unique from previous copies - still contained an unaltered piece of code that gave it away every time. As a result, antivirus programs could still detect it. Nevertheless, virus writers were turning in a new direction, and in two years the first polymorphic virus appeared: Chameleon (Virus.DOS.Chameleon). Chameleon, also known as 1260, and its contemporary Whale, used complex encryption and obfuscation methods to protect their code. Two years later, we saw the emergence of so-called polymorphic generators, which could be used as out of the box defense solutions for malicious programs.
Why code modification can be used to hinder file detection, and how file detection works, needs some explanation.
Until recently, antivirus programs worked exclusively by analysis file code. The earliest signature-based detection methods focused on searching for exact byte sequences, often at a fixed offset from the beginning of the file, in a malicious program's binary code. Later heuristic detection methods also used file code, but with a more flexible, probability based approach to searching for common malware byte sequences. Obviously, it’s not difficult for malicious programs to get around that kind of protection if each copy of the program includes a new byte sequence.
This task is fulfillled by the application of polymorphic and metamorphic techniques, which essentially - without getting into the technological nitty-gritty – enable a malicious program to mutate at byte level when the program creates a copy of itself. Meanwhile, the program’s functionality remains unchanged. Encryption and obfuscation are primarily used to hinder code analysis, but when they are implemented in a certain way, the result can be a variation of polymorphism – an example here is again Cascade, where every copy of the virus was encrypted with a unique key. Obfuscation may just hinder analysis, but when it is applied in a different way to every copy of a malicious program, it hinders the effective use of signature-based detection methods. However, it cannot be said that any one of the abovementioned tactics is more effective than any other in terms of malware self-defense. It would be more correct to say that the effectiveness of these techniques depends on the specific circumstances and how the techniques are implemented.