Introduction The threat: modified malware frequently remains undetected. Background Dangers resutling from run-time compression and other modifications. Example Two widely-used AV scanners are outfoxed. AV Reviews Sense and non-sense of AV reviews Scanners We examine the modes of operation and detection rates of several AV & AT scanners Forum ... and you can participate

 

-- Test Procedure --

1. Test Environment


Operating System: Windows XP Pro SP1 (in connection with HDD KING).

Computer System: Generally, the scanners were installed on several different computer systems (including virtual machines).

In order to examine the scanner's workability under realistic conditions we did not reinstall the OS for each test run. On the contrary, we performed additional tests in order to examine possible hardware or software conflicts. In particular, we tried to install several AV/AT scanners simultaneously since we believe that there is currently no stand-alone scanner which offers sufficient protection against all kinds of malware. (We avoided to enable two on-access scanners at the same time.)

Each scanner had the chance to run on at least one computer system with a freshly installed OS. Each scanner was installed on at least one real computer.


2. Malware Archive


Starting from November 2003, the following malware archive (containing about 530 samples) has been used for our test reports: see here.

Older reports rely upon a different test procedure: see here (Altes Testverfahren, deutschsprachig).


3. Why Is The Malware Archive So Small?


(i)
The malware archive is handmade. The malware samples' functionality has been tested in every single case (by self-infection). In other words, we did not automatically compress or crypt a multitude of trojans because such a timesaving approach could seriously distort the test results. (Frequently, malware samples, which are compressed or crypted, get corrupted and cannot be detected anymore. However, because such non-working samples are not dangerous it is irrelevant whether a scanner detects them or not. The same applies to malware samples which are still working but whose configurations settings were lost during the packing/crypting process.)

(ii)
The tiny size of the archive allows you and us to analyze the test results in an easy way. For this reason, we publish not only a summary report for each test but allow you to examine the entire scanlog (i.e., you are not required to believe in our findings but can make up your own mind instead).

(iii)
Please keep in mind that the primary purpose of our tests is not to determine whether a scanner has a comprehensive signature database. We rather want to examine whether a scanner can handle known malware that has been compressed, crypted or otherwise modified.


4. How Can The Test Results Be Analyzed?


The malware archive consists of several sections: see here. Each section has a different function which is explained in the following...


-- ORG_CC (custom configuration) / ORG_DC (default configuration) --

Both sections contain original malware samples which are not compressed, crypted or otherwise modified. It is important to include the original malware samples into the test archive because they constitute the basis for their modified derivatives which are part of the other sections. If a scanner is unable to detect the original malware samples it will usually fail to detect the modified derivatives as well. Consequently, we will rate the quality of a scanner's unpacking engine solely on the basis of the derivatives which are detected in their original form. (Note that a good scanner should detect all samples contained in the sections ORG_CC and ORG_DC. If a scanner fails to detect trojans contained in these section there is definitely something wrong.)

ORG_CC contains non-modified trojan samples that have been created with the editor application which is part of the trojan package. We did not use the editor's default settings.

ORG_DC contains samples that have been created in the same way as described above. However, the default configuration was used. If an AV/AT scanner detects the malware samples contained in ORG_DC but fails to detect the samples contained in ORG_CC it probably uses ineffective signature. Such scanners cannot be trusted.

Due to the small number of trojans contained in ORG_DC and ORG_CC there is no performance rating. We do not want to create the impression that we are trying to determine whether a scanner has a comprehensive signature database or not.


-- ORG_DLL --

The section contains original DLL trojans which constitute the basis for the modified derivatives that are part of the other sections. We have not included the standard injector (part of the trojan package) because it may be easily replaced by another injector. We believe that the detection of a injector is no reliable way to detect DLL trojans.

Due to the small number of trojans contained in ORG_DLL there is no performance rating. We do not want to create the impression that we are trying to determine whether a scanner has a comprehensive signature database or not.


-- BytesAdded --

The section deals with malware samples whose file size has been increased (by adding bytes). Only the worst file scanners can be fooled by this trick.


-- DoublePack --

The malware samples contained in this section were compressed or crypted twice. In principle, a scanner with decompression support should be able to identify and unpack two or more known compressors/crypters successively. (Please note that a scanner will not be able to decompress double-packed samples unless it is able to separately identify each single compressor/crypter applied. Therefore, if a scanner fails to unpack a compressor/cryper in a stand-alone situation we will not take into account that it also misses the double-packed sample.)


-- HeaderFaked --

The section contains compressed malware samples whose file header (containing a text string with the name of the compressor) has been faked. Because nowadays the identification of compressors/crypters usually relies upon signatures of the packer's/crypter's unpacking stub we do not expect many scanners to fail this test.


-- Hexedited --

If you open a trojan file with a hex editor and modify a number of bytes there is a good chance that you will change the executable code of the trojan and corrupt it (unless you exactly know what you do -- see section "Patched"). However, certain parts of a trojan executable can be easily changed without corrupting the file. In particular, this may apply to plain text strings which do not contain any instructions. Such text strings may, for example, contain the name of the trojan.

A text string containing the trojan's name facilitates the identification and removal of a trojan. Therefore, an AV/AT software producer may decide to use such a text string as a signature. In general, we consider text-based signatures unsafe and call them "weak" signatures. This is because even the most inexperienced attacker can identify and manipulate such text strings.

Consequently, text-based signatures should only be used as supplementary means of detection. (An exception applies to special text strings which cannot be easily modified, e.g., text strings being part of the client-server protocol.)

In order to figure out whether a scanner exclusively uses "weak" text-based signatures we have manipulated the trojan files contained in section "Hexedited" as follows: with the help of a hex editor we changed text strings like the trojan's name, the word "victim" etc. (i.e., text strings which are simply too obvious to use them as a reliable signature).

If a scanner fails this test you should not use it as an AT scanner.


-- OEP --

Most AV/AT scanners which are able to deal with compressed/crypted malware use a static unpacking engine (so-called "passive unpacking"). A static unpacking engine tries to identify the compressor, crypter or protector that has been applied. The identification is generally based on a checksum or signature which matches the unpacking stub (that is usually located immediately after the original entry point (OEP) of a compressed/crypted executable). After a compressor, crypter or protector has been identified a specific static unpacking routine comes into play. After a malware sample has been correctly unpacked it can be processed as usual by the scanner's scan engine.

Static unpacking engines are unable to handle unknown compressors, crypters or protectors. Moreover, they may be vulnerable to modifications of a known compressor's unpacking stub. If the unpacking stub is modified there will be no signature match and a (correct) identification of the compressor etc. may not be possible (i.e., the correct static unpacking routine will not be applied). A scanner may try to improve the identification of compressors by filtering out certain modifications like the inclusion of NOPs or useless jumps.

The OEP section tries to determine whether a scanner's unpacking engine is easily vulnerable to modifications of the unpacking stub. In addition, we will check whether a scanner suffers from a material design flaw which is not related to the unpacking engine but to the scan engine itself (see ii). The samples contained in OEP have been modified in the following ways:

(i) NOP
Meaningless "no operation" instructions (NOP = 90) were inserted with a hex editor into the unpacking stub. A corresponding number of NOPs was deleted after the unpacking stub. Any jumps contained in the unpacking stub were adjusted.

(ii) EPChange

The entry point of a non-packed trojan is moved with the help of a PE Editor. Note: It is very easy to do this. The following screenshot shows a non-packed Bionet 3.18 server. The original entry point is 9F20C. We changed it to 9F20D. (The trojan will not be corrupted by this procedure.)





If a scanner can be fooled by this simple trick it uses the entry point as a reference. This is simply embarassing and indicates that the scan engine is conceptually flawed.

(iii) OEP.Changed
The entry point was changed. At the new entry point a jump directs the execution flow to the original entry point.

(iv) OEP.Changed + NOP
A combination of method (i) and (ii) was used.

(v) JMP
Meaningless forward/backward jumps were included into the unpacking stub.

(vi) CEP
An automatic tool called "Change Entry Point" was applied.

(vii) AntiAVP
An automatic tool called "AntiAVP" was used.

(viii) HidePX14
An automatic tool called "HidePX 1.4" was used.

(ix) EPP
An automatic tool called EP Protector 0.3 was used.

(x) UPXREDIR
An automatic tool called UPX Redir was used.

(xi) HPE / SPE
The abbreviations refer to three automatic tools which use relatively complex obfuscation methods. For the time being, we do not want to disclose the names of these tools. (They will be submitted to AV/AT software producers upon request.)

By comparison we also modified a few non-packed samples. If a scanner fails to detect the non-packed samples the scan engine itself (and not the unpacking engine) must be considered faulty.


-- Patched --

The section contains "patched" malware samples (see "The Art of Patching" -- German language). The patched samples demonstrate that AV/AT scanners can be circumvented regardless of whether they are using "weak" or "strong" signatures provided the signature database becomes known to an attacker who has a rudimentary understanding of assembler language.

The samples PATCH.KAVSignature show whether "Kaspersky clones" like F-Secure, AVK, etc. use identical signatures (i.e., whether they are equally affected by the vulnerability which is caused by SennaSpy's AVP Offset Generator, a hacker tool which helps to extract signatures from Kaspersky's signature database.)

Several other patched samples illustrate that it is a bad idea if a scanner (like Trojan Hunter) does not encrypt its signature database.

AV/AT scanners would be less vulnerable against patching if they were using "rotating signatures" for popular malware. Rotating signatures require a signature pool with several different signatures for each malware sample. With each signature update the signatures contained in the signature database are replaced with another random signature from the pool.


-- Rebased --

See section contains "rebased" malware samples. See here for further details.


-- RePacked --

See section -- UnPacked -- first. We have taken unpacked samples from the UnPacked section and compressed them again (with a different compressor). Sometimes this procedure affects a scanners ability to detect known malware samples.

If a scanner misses a sample contained in the UnPacked section we will not take this sample into account for determining the scanner's performance under this section.


-- Resources --

A trojan file consists of several sections including a so-called "resource section". Inter alia, the resource section may include an icon and the file's version information. If a trojan file is compressed or crypted the resource section frequently remains untouched, at least parts of it (especially the icon and the version information). That's why several AV/AT software producers, who are reluctant to develop a sophisticated unpacking engine, take signatures from the resource section allowing the detection of compressed trojans.

Unfortunately, signatures taken from the resource section are usually not safe. This is because several parts of the resource section (e.g., the icon and the version information) can be easily modified with a tool like Resource Hacker. After such modification, the signature will not be valid anymore (i.e., the trojan will remain undetected). Therefore, it is not a good idea if a scanner detects (compressed/crypted) trojan exclusively on the basis of "weak" signatures taken from the resource section.

With the help of the malware samples contained in this section we are examining whether a scanner uses such "weak" signatures. The section contains trojans taken from the other sections (including compressed trojans) that have been modified in the following ways: (i) the icon was replaced or deleted, (ii) Version Info was replaced or deleted, (iii) RCData was changed. Please note that not every signature taken from the resource section is a "weak" one. Sometimes you cannot easily modify the signature without destroying the trojan. We have not attempted to circumvent "strong" signatures because such signatures do not endanger the user.


-- UnPacked --

The section includes formerly compressed trojans which were unpacked with static unpackers or memory dumpers. We want to examine whether the unpacking procedure affects the detection of a trojan. (As always, we have verified that the unpacked samples are still functional.)


-- Variants --

The section includes several non-modified variants of the same trojan. Trojan coders sometimes release newly compiled variants under the same version number in order to mislead AV/AT software producers and prevent them from adding a new signature for every new variant. An AV/AT software producer who does not know about this practice will be unable to protect its customers.

We have decided to include variants of the Beast family into this section because this popular trojan has been downloaded more than 10.000 times from the developer's website (i.e., it is really important that an AV/AT scanner reliably detects all variants).

-- www --

This is an experimental section which is still under development. It contains trojans collected from the world wide web (including samples from asia, eastern europe and other non-english speaking countries). The collection also includes less frequently used trojans. The idea is to figure out whether an AV/AT software producer actively searches the web for trojans instead of relying on trojan submissions by its customers.

The section must be considered experimental because it does not contain enough malware samples for a reliable evaluation of a malware scanner's detection rate (i.e., minor differences between the detection rates of two scanners are meaningless). However, if the differences between the detection rates were huge this could support the claims of certain AT software producers who believe that many AV software producers do not actively search the web for new trojans.

Because the section is still under development we do not want to include a performance rating yet.

-- RED SECTIONS: Compressors, Crypters & Commercial Protectors --

The red sections include samples which are treated with a selection of compressors, crypters and commercial protectors (featuring compression, encryption and various anti-debugging/anti-dumping techniques) that we consider of particular importance. Compressors, crypters and protectors are considered important if they are stable and easily available to trojan users (e.g., because they are freeware or have been cracked/keygened by reverse engineers). A good AV/AT scanner should be able to detect the malware samples contained in the red sections. With respect to the red sections our performance ratings have the following meaning: ( - ) = decompression not supported, ( 0 ) = unclear or no reliable decompression support, ( + ) = decompression supported.

Currently, the red sections include the following compressors, crypters and protectors: ACProtect, Armadillo, ASPack, ASProtect, Crunch (Bit-Arts), DBPE (Ding Boy), ExeShield, ExeStealth, EZIP, FSG, JD Pack, Krypton, Morphine, Netwalker, PC Guard, PE Compact, PE Crypt, PEncrypt, PE Shield. Petite, PeX, PKLite32, tElock, UPX, WinKripT, WWPack32 and Yoda Crypt.

We have tried to make sure that a scanner reliably supports the decompression of a specific compressor, crypter or protector. Sometimes this is difficult to determine because AV/AT software producers may include special signatures for compressed/crypted malware samples into their signature database (i.e., a malware sample is detected although the scanner does not offer decompression support). In order to make sure that a scanner offers reliable decompression support we have treated several malware samples with each compressor, crypter or protector. In addition, we have experimented with different versions and settings of several compressors and protectors.

A special note on Armadillo: It is quite difficult to automatically unpack this commercial protector. Allegedly, there are well-known AV software producers who say that it is not possible at all. While we believe that such statements are wrong we must realise that it is currently quite easy to circumvent almost every AV scanner by protecting non-viral malware with Armadillo. Therefore, AV software producers should consider the supplementary use of a sophisticated memory scanner in order to facilitate the detection of trojans (and certain worms). The general effectiveness of a memory scanner has been demonstrated by AT software producers like DiamondCS (TDS-3) and Mischel Internet Security (Trojan Hunter). In respect of Armadillo a memory scanner must be able to deal with Silicon Realms' CopyMem-II technology which encrypts pages of memory.


-- GREY SECTIONS: Commercial Protectors --

The grey sections deal with commercial protectors which we expect to be less frequently used by attackers. An excellent AV/AT scanner should nevertheless provide unpacking support for them. If one of the "grey" protectors becomes easily available we will move it from the grey to the red section.

With respect to the grey sections our performance ratings have the following meaning: ( - ) = decompression not supported, ( 0 ) = unclear or no reliable decompression support, ( + ) = decompression supported.

Currently, the grey section encompasses the following protectors: JD Protect, Obsidium, Peetles, PE Lock, SVK Protect, Thinstall, UltraProtect (the predecessor of AC Protect), Xtreme Protector.


-- TOTAL --

This number is included for verification purposes. In addition to scanning individual sections we also perform a scan of the entire malware archive. The total score, which results from the scan of the entire archive, must be equal to the sum of the detected malware samples contained in each individual section.

A very low total score will indicate at first glance that a scanner is unsuitable for detecting non-replicating malware.

Please note, however, that a high total score does not necessarily indicate that a scanner performs well. For example, a scanner which uses insecure signatures taken from the resource section may archieve a high total score because our malware archive contains only a few samples whose resource section was modified. A scanner which uses a lot of special signatures for compressed/crypted trojans may also achieve a relatively high total score although it does not feature a sophisticated unpacking engine. Therefore, it is necessary to analyze the test results in detail.


5. Additional Tests


Last but not least, we performed additional tests with special test archives (containing different malware samples) if we deemed such tests necessary to analyze the special characteristics of a scanner or to get to the root of a problem. For instance, we performed tests with outdated signatures if we had the feeling that a scanner uses special signatures for compressed/crypted malware samples or if we wanted to examine the performance of a scanner's heuristic engine. The exact procedure of any additional tests will be described in the individual test reports.


6. Disclosure Policy

This site is a security site and not a hacker site. Therefore, we generally do not disclose any information which has not already been discussed in the VX/trojan scene. If we disclose information which may be new to members of the VX/trojan scene we do not provide any details. (The details can be made available to AV/AT software producers upon request.)


7. Submission Policy

We do not submit compressed, crypted, patched or otherwise modified malware samples to AV/AT software producers anymore. This is because we made the experience that most software producers do not use the samples to add decompression support, OEP filters etc. They simply add special signatures for our zoo samples which ameliorate the test results but do not increase security. We still submit tools (compressors, crypters, OEP tools etc.) and non-modified malware samples to AV/AT software upon request.