The importance of data normalization in IPS
by Darren Suprina - Security Architect, Stonesoft - Monday, 7 January 2013.
Packet-oriented pattern matching is insufficient as a means of invasion detection due to the need for a 100% pattern match for blocking or detection. Advanced Evasion Techniques (AETs) possess the ability to utilize a vast multitude of combinations to infiltrate a system, rendering the likelihood of a 100% pattern match for every possible combination nonexistent. It is simply impossible to create enough signatures to be effective.

AETs exploit the weaknesses in the system, often being delivered in a highly liberal manner that a conservatively designed security device is incapable of detecting. In addition to using unusual combinations, AETs also focus on rarely used protocol properties or even create network traffic that disregards strict protocol specifications.

A large number of standard IPS devices fail to detect and block AETs, which have therefore effectively disguised a cyber attack that infiltrates or even decimates the network. Standard methods used to detect and block attacks generally rely on protocol anomalies or violations, which is no longer adequate to match the rapidly changing and adaptable AETs. In fact, the greatest number of anomalies occurs not from attacks, but rather from flawed implementation in regularly used Internet applications.

An additional issue that arises with many IPS devices is the environment in which they are optimized. Optimization typically takes place in a clean or simulated network that has never suffered a complex and highly elusive attack.

Resistance to normalization

Resistance to data normalization does not typically arise from the advanced security it promises, but rather the impact it may have on a network. When the security design flaw is found in hardware-based products, network administrators may resist the upgraded security measure due to the necessity of significant research and development for redesign. Additional memory and CPU capacity are also required to properly implement a data stream inspection that comprehensively protects against AETs.

When vendors decide that the required changes are impossible to implement, they leave their networks highly vulnerable for exploits and attacks. Focusing on the cost of the cleanup required for all infected computers in the network, and the even higher cost of network downtime, can help change the minds of vendors who continue to resist the necessary adaptations.

How the most effective IPS devices use data normalization

Instead of analyzing data as single or combined packets, effective IPS devices analyze data as a normalized stream. Once normalized, the data is sent through multiple parallel and sequential machines. All data traffic should be systematically analyzed by default, regardless of its origins or destination.

The most effective way to detect infiltration is to systematically analyze and decode the data, layer by layer. Normalization must occur at every layer simply because attacks can be hidden at many different layers. In the lower protocol layers, the data stream must be reconstructed in a unique manner. Modifications should generally be very slight or nonexistent, although any fragments or segments containing conflicting and overlapping data should be dropped.

Normalizing traffic in this manner ensures there is a unique way to interpret network traffic passing through the IPS. The data stream is then reassembled for inspection in the upper layers. Inspection of constant data stream in this manner is a must for correcting the flaws and vulnerabilities left open by many IPS devices. This process also removes the possibility of evasion of attacks that span over segment boundaries.

Higher levels are subjected to inspection of separate data streams that are normalized based on the protocol. In compressed HTTP, for instance, the data can be decompressed for inspection. In another example, MSRPC-named pipes using the same SMB connection would be demultiplexed and inspected separately.


101,000 US taxpayers affected by automated attack on IRS app

The IRS has revealed more details about an attack it suffered last month, mounted by unknown individuals with the aim to file fraudulent tax returns and funnel the returned money to their own bank accounts.

Weekly newsletter

Reading our newsletter every Monday will keep you up-to-date with security news.

Daily digest

Receive a daily digest of the latest security news.

Wed, Feb 10th