-- Sense and Non-Sense of AV/AT Reviews --
1. Ordinary Reviews
AV/AT software testers usually require a scanner
to scan through huge trojan and virus archives and believe that this is an
adequate way of determining the scanner's "detection rate". Neither
the fact that a significant amount of the archived malware samples is not used
(anymore) "in the wild", nor the possibility to use compressors
or crypters for making well-known trojans undetected seems to be a matter
of particular interest.
There are a few reviews that illustrate
(to some extent) how badly the majority of AV/AT scanners fails to detect
compressed or encrypted trojans.
See, for example, the reviews of AV-Test.org ( http://www.av-test.org ) that show the inability of many scanners to detect compressed malware.
Below you will find several extracts from the reviews:
Review for ComputerBild (Dec.
2001)
Review for PC-Welt (dated 11 October
2001) -- (The complete review can be found here.)
Review for c't (13/2002) -- See here (test
report) and here (test
results).
Review for PC-Welt (dated 31 October
2002) --
(The complete review can be found here.)
and ...
Review for c't (1/2005):
Note:
We consider the highlighted (rubricated) test results of particular
importance. The test results relate to malware which was packed with various
Windows (32Bit) runtime compressors. By contrast, the tests relating to self-executing
archives are of minor importance because archived malware can generally
be detected by the real-time monitor of an AV/AT scanner after the malware
has been extracted from the archive (i.e., the malware can be detected before it is executed).
Runtime compressed malware behaves in a completely different manner because
it is
unpacked directly
into the computer's memory.
Also the reviews "Antivirenprogramme
Mai 2002" dated 26 Mai 2002 and "Antitrojanerprogramme
Juli 2002" dated 16 July 2002, both published by Rokop Security,
deal with the problem of runtime compressed malware.
On 16 February 2003, Rokop Security published another review (in English language) --> see here.
Generally, the above-mentioned reviews do not examine whether a scanner can handle complex compressor/protectors like Armadillo, AC Protect etc. The high detection rates of scanners using Kaspersky technology (e.g., AntiVirenKit, F-Secure, Kaspersky Anti-Virus) suggest that it is not difficult to look through a compressor, crypter or protector. In practice, this is not true. Many attackers are adept in using those compressors that are not supported by AV/AT scanners. Moreover, the above test results fail to reveal that certain scanners do not feature a working unpacking engine but rely on signatures taken from uncompressed parts of a file (e.g. the resource section) in order to detect compressed malware. Such signatures are generally not safe because (i) they can be easily bypassed by an attacker and (ii) there are many malware samples that do not have a resource section. In addition, the above reviews do not deal with the actual signature quality of a scanner. Last but not least, the design flaws of many scanners are not disclosed.
2. Scheinsicherheit Reports
While ordinary AV/AT reviews are mainly a quantitative evaluation of a scanner's signature database the focus of our reports lies with the quality of a scanner. In particular, we examine the quality of a scanner's unpacking engine and signature database as well as the quality of the scan engine and the scanner's design. Our reports can provide you with additional information and support your choice of a scanner.
In principle, we do not perform quantitative tests.
Last update: January 2005