We have used a scatterplot to present the different kinds of malware self-defense mechanisms. This diagram is merely a simple example that we can use as a guide to categorize different means of malware self-defense. This model is based on a careful analysis of malware behaviors, but is, necessarily, subjective.
Figure 1. A scatterplot of malware self-defense technologies
Malware self-defense mechanisms can fulfil one or more tasks. These include:
1. hindering detection of a virus using signature-based methods;
2. hindering analysis of the code by virus analysts;
3. hindering detection of a malicious program in the system;
4. hindering the functionality of security software such as antivirus programs and firewalls.
This article will only examine malicious programs written for the Windows operating system (and its predecessor, DOS) due to the rarity and relatively small number of malicious programs for other platforms. All of the trends examined in this article that apply to executable malware files (EXE, DLL and SYS), and to some extent also apply to macro viruses and script viruses, which is why I will not be addressing the latter separately.
Sources: polymorphism, obfuscation and encryption
It makes sense to examine polymorphism, obfuscation and encryption together, as they all fulfill the same end, albeit to different degrees. Initially, modification of malicious code had two goals: to make it more difficult to detect files and to make it more difficult for virus analysts to examine the code.
The history of malware began in the 1970s, but the history of malware self-defense didn’t start until the late 1980s. The first virus that attempted to defend itself from the antivirus utilities then in existence was the DOS virus Cascade (Virus.DOS.Cascade). It defended itself by partially encrypting its own code. This wasn’t very successful, however, since each new copy of the virus - despite being unique from previous copies - still contained an unaltered piece of code that gave it away every time. As a result, antivirus programs could still detect it. Nevertheless, virus writers were turning in a new direction, and in two years the first polymorphic virus appeared: Chameleon (Virus.DOS.Chameleon). Chameleon, also known as 1260, and its contemporary Whale, used complex encryption and obfuscation methods to protect their code. Two years later, we saw the emergence of so-called polymorphic generators, which could be used as out of the box defense solutions for malicious programs.
Why code modification can be used to hinder file detection, and how file detection works, needs some explanation.
By subscribing to our early morning news update, you will receive a daily digest of the latest security news published on Help Net Security.
With over 500 issues so far, reading our newsletter every Monday morning will keep you up-to-date with security risks out there.