The second criterion is the degree to which a malicious program's self-defense mechanism is dedicated. The most narrowly dedicated forms of self defense are found in malicious programs that somehow disrupt the function of a specific antivirus program. More general self-defense mechanisms are designed to defend malicious programs against everything by making the virus presence in the system as undetectable as possible in every way.

We have used a scatterplot to present the different kinds of malware self-defense mechanisms. This diagram is merely a simple example that we can use as a guide to categorize different means of malware self-defense. This model is based on a careful analysis of malware behaviors, but is, necessarily, subjective.



Figure 1. A scatterplot of malware self-defense technologies

Malware self-defense mechanisms can fulfil one or more tasks. These include:

1. hindering detection of a virus using signature-based methods;
2. hindering analysis of the code by virus analysts;

3. hindering detection of a malicious program in the system;
4. hindering the functionality of security software such as antivirus programs and firewalls.

This article will only examine malicious programs written for the Windows operating system (and its predecessor, DOS) due to the rarity and relatively small number of malicious programs for other platforms. All of the trends examined in this article that apply to executable malware files (EXE, DLL and SYS), and to some extent also apply to macro viruses and script viruses, which is why I will not be addressing the latter separately.

Sources: polymorphism, obfuscation and encryption

It makes sense to examine polymorphism, obfuscation and encryption together, as they all fulfill the same end, albeit to different degrees. Initially, modification of malicious code had two goals: to make it more difficult to detect files and to make it more difficult for virus analysts to examine the code.

The history of malware began in the 1970s, but the history of malware self-defense didn’t start until the late 1980s. The first virus that attempted to defend itself from the antivirus utilities then in existence was the DOS virus Cascade (Virus.DOS.Cascade). It defended itself by partially encrypting its own code. This wasn’t very successful, however, since each new copy of the virus - despite being unique from previous copies - still contained an unaltered piece of code that gave it away every time. As a result, antivirus programs could still detect it. Nevertheless, virus writers were turning in a new direction, and in two years the first polymorphic virus appeared: Chameleon (Virus.DOS.Chameleon). Chameleon, also known as 1260, and its contemporary Whale, used complex encryption and obfuscation methods to protect their code. Two years later, we saw the emergence of so-called polymorphic generators, which could be used as out of the box defense solutions for malicious programs.

Why code modification can be used to hinder file detection, and how file detection works, needs some explanation.