Antivirus Proactive Protection Test (March 2009)
Serious efforts of the antivirus industry are focused on proactive methods of antivirus protection, which allow antivirus software to combat malicious programs that have undergone modifications and those that are still unknown. This development trend is the most promising on the market and almost every developer likes to emphasize just how good their proactive defense is.
Table of Contents:
- Intoduction
- Evaluating the effectiveness of proactive antivirus protection
- Results and Awards
- Changes to the effectiveness of proactive antivirus protection over time
- Signatures or Heuristics?
- Changes in comparison with previous tests
It should be noted that proactive technologies encompass a wide range of concepts and approaches, and it is impossible to take all of them into account within a single test. In this test we will only compare the heuristic components of antivirus protection (heuristic + generic detection, i.e., extended signatures), without taking into account an analysis of system events (behavior blockers or HIPS).
The results of the test make it possible to conclude how effective a heuristic analyzer is and in which antivirus product this component works better.
As an addendum, a final measurement of the detection level for the collection of malware samples was performed on the updated antivirus software a week after the main test. As a result, the quality of detection for new viruses, as well as the effectiveness of the classical signature-based method of each antivirus program was ascertained in addition to their heuristics.
- Methodology for Proactive Antivirus Protection Test »
- Test Results and Awards
Intoduction
Fourteen popular antivirus programs participated in the testing of proactive antivirus protection, including:
- Agnitum Outpost Antivirus Pro 2009
- Avast! Professional Edition 4.8
- AVG Anti-Virus 8.0
- Avira AntiVir Premium 8.2
- BitDefender Antivirus 2009
- Dr.Web 5.0
- Eset Nod32 Anti-Virus 3.0
- F-Secure Anti-Virus 2009
- Kaspersky Anti-Virus 2009
- Panda Antivirus 2009
- Sophos Anti-Virus 7.6
- Symantec Anti-Virus 2009
- Trend Micro Internet Security 2009
- VBA32 Antivirus 3.12
The antivirus testing was performed on Windows XP SP3 operating system from 3 December 2008 to 18 January 2009 strictly in line with methodology. The special conditions for checking of the heuristics effectiveness were created by means of this methodology (updates were turned off, therefor antivirus databases were frozen by the date of the beginning of the test).
The collection of 5166 unique samples of the newest malicious programs and the collection of 15121 pure files were created especially for this test during the freezing of the antivirus databases.
Evaluating the effectiveness of proactive antivirus protection
The detection rate of unknown malicious programs by various antiviruses and their level of false alarm (as a side-effect of any proactive technology) are illustrated by diagram 1 and table 1.
Diagram 1: Test results of heuristics effectiveness
Table 1: Effectiveness of heuristic analyzers
|
Antivirus product
|
Number of undetected viruses | Detected viruses (%) |
| Avira | 71,0% | 0,13% |
| Sophos | 61,7% | 2,24% |
| DrWeb | 61,0% | 0,20% |
| Kaspersky | 60,6% | 0,01% |
| Eset | 60,5% | 0,02% |
| BitDefender | 60,1% | 0,04% |
| AVG | 58,1% | 0,02% |
| Avast | 53,3% | 0,03% |
| Norton | 51,5% | 0% |
| VBA32 | 44,9% | 0,07% |
| F-Secure | 43,5% | 0,04% |
| Panda Security | 37,9% | 0,02% |
| Trend Micro | 37,2% | 0,04% |
| Agnitum | 33,4% | 0,07% |
Important! As opposed to the previous test, we changed the system of evaluation of final results in the following way: we introduced a maximal level of the thresholds of false alarms for each award. Thereby not only the high detection level is required for the receipt of awards, but also a false alarm minimal level.
The leader of the effectiveness of heuristics component protection is Avira AntiVir Premium, its detection rate of malicious programs proved to be very high – 71%. However a high level of the false alarm allowed this product to receive only the Silver in the Proactive Protection Award.
DrWeb suffered the same fate, the new version 5.0 achieved a very high detection result – 61%, but false alarm high level allowed to receive only Silver Proactive Protection Award.
The main disappointment of the test became Sophos Anti-Virus, which had achieved an impressive level of proactive detection — more than 61% — with a monstrous number of false alarms – 2.24%. Unfortunately this result does not leave any chance for this antivirus to get only the gold, but any other award too.
Kaspersky Anti-Virus, Eset Nod32 Anti-Virus and BitDefender Antivirus achieved the best total results the proactive detection and in the false alarms. Their results are nearly identical — the heuristics detection level is 60% and the false alarm level is 0.01-0.04%. According to the adopted system of evaluation this antivirus software received Gold Proactive Protection Award.
A considerable group of antivirus products which received Silver Proactive Protection Award, achieved a high detection rate of the heuristics component. In addition to the listed products, it includes AVG Anti-Virus, Avast! Professional Edition, Norton Anti-Virus, VBA32 Antivirus and F-Secure Anti-Virus. Note that Norton Anti-Virus appeared to be the only antivirus which has demonstrated the null level of the false alarm.
Panda Antivirus, Trend Micro Internet Security and Agnitum Outpost Anti-Virus Pro achieved a satisfactory result and received Bronze Proactive Protection Award.
Results and Awards
| Award | Products |
 Gold Proactive Protection Award |
Kaspersky Anti-Virus 2009 (61% - 0.01%) Eset Nod32 Anti-Virus 3.0 (61% - 0.02%) BitDefender Antivirus 2009 (60% - 0.04%) |
 Silver Proactive Protection Award |
Avira AntiVir Premium 8.2 (71% - 0.13%) Dr.Web 5.0 (61% - 0.2%) AVG Anti-Virus 8.0 (58% - 0.02%) Avast! Professional Edition 4.8 (53% - 0.03%) Norton Anti-Virus 2009 (52% - 0%) VBA32 Antivirus 3.12 (45% - 0.07%) F-Secure Anti-Virus 2009 (44% - 0.03%) |
|
|
Panda Antivirus 2009 (38% - 0.02%) Trend Micro Internet Security 2009 (37% - 0.04%) Agnitum Outpost Anti-Virus Pro 2009 (33% - 0.07%) |
|
Failed
|
Sophos Anti-Virus 7.0 (61% - 2.24%) |
Changes to the effectiveness of proactive antivirus protection over time
Since according to the methodology, the test used a monthly collection of new malicious programs (collected from 18 Dec. 2008 to 18 Jan. 2009), one could check how the effectiveness of various heuristics changes in time. For that reason, the collection was divided into four equal parts according to the time of getting samples.
So the data of the effectiveness of heuristics through weekly collections is represented at diagram 2 and table 2.
Diagram 2: Test results of the heuristics effectiveness (over weeks)
Table 2: Test results of the heuristics effectiveness (over weeks)
| Antivirus product | Detected viruses (%) | |||
| Week 1 | Week 2 | Week 3 | Week 4 | |
| Avira | 95,4% | 66,4% | 54,4% | 65,2% |
| Sophos | 80,0% | 57,7% | 47,4% | 58,7% |
| DrWeb | 91,0% | 49,6% | 32,9% | 61,9% |
| Kaspersky | 89,8% | 35,6% | 44,2% | 66,1% |
| Eset | 89,0% | 53,1% | 37,4% | 56,9% |
| BitDefender | 95,8% | 51,8% | 35,7% | 52,6% |
| AVG | 95,8% | 34,6% | 31,6% | 61,2% |
| Avast | 93,3% | 35,7% | 24,6% | 51,9% |
| Norton | 83,7% | 39,5% | 28,7% | 48,4% |
| VBA32 | 80,4% | 20,8% | 23,9% | 47,5% |
| F-Secure | 85,9% | 18,5% | 14,3% | 45,7% |
| Panda Security | 44,0% | 39,9% | 42,7% | 29,2% |
| Trend Micro | 64,0% | 29,9% | 18,3% | 32,8% |
| Agnitum | 51,2% | 33,0% | 15,5% | 29,8% |
Table 2 shows that nearly all antivirus had a decrease in the effectiveness on the second and third week. The only exception is outsiders of the test whose detection level is constantly low.
Signatures or Heuristics?
In addition, one week after the main test (on 25 January, 2009), a final measurement of the detection level for the collection was performed on updated versions of the antivirus software. It appeared that classic signature-based methods of each antivirus program are efficient together with a heuristic. This allowed to figure out the role of each component in the overall detection level (cf. diagram. 3).
Diagram 3: Effects of different antivirus protection components on the overall level of malware detection
The diagonal line in diagram 3 denotes a 100% level of detection for new malicious programs. Getting close to the line can be achieved through the effective functioning of one of the components of antivirus protection or a combination of the two.
Products in the dark orange (80-100%) and light orange (60-80%) zones demonstrated excellent and good detection levels of new viruses (aged from 1 to 5 weeks, see methodology).
The majority of them (Avira Antivir Premium, Sophos Anti-Virus, Dr.Web, Kaspersky, Eset Nod32, BitDefender Antivirus, AVG Anti-Virus, Avast Professional Edition and Norton Anti-Virus) attained that level based on the contribution of their proactive component.
F-Secure Anti-Virus and Panda Antivirus achieved similar results based on their signature component.
VBA32 Anti-Virus proved to be the most balanced antivirus in this respect. Both components of it were equally efficient (they appeared in the bottom left square on diagram 3, and the overall level of detection of malicious software proved to be perfect.
Trend Micro Internet Security and Agnitum Outpost Anti-Virus Pro appeared to be the weakest with respect to the detection of the new malware, and absolutely inefficient against the new threats.
Table 3: Quality of new virus detection
|
Antivirus product
|
Detected viruses before update (%) (4 weeks) |
Detected viruses after update (%) |
Total % of detected viruses |
| Avira | 71,0% | 27,0% | 98,0% |
| Sophos | 61,7% | 27,0% | 88,7% |
| DrWeb | 61,0% | 27,6% | 88,6% |
| Kaspersky | 60,6% | 39,1% | 99,7% |
| Eset | 60,5% | 29,6% | 90,1% |
| BitDefender | 60,1% | 37,7% | 97,8% |
| AVG | 58,1% | 38,7% | 96,6% |
| Avast | 53,3% | 41,7% | 95,1% |
| Norton | 51,5% | 39,4% | 90,9% |
| VBA32 | 44,9% | 37,5% | 82,4% |
| F-Secure | 43,5% | 56,1% | 99,6% |
| Panda Security | 37,9% | 53,9% | 91,8% |
| Trend Micro | 37,2% | 21,3% | 58,5% |
| Agnitum | 33,4% | 6,2% | 39,6% |
Diagram 4: Quality of new virus detection - contribution of different components
As it was one year earlier, the best software for the detection of malware proved to be Kaspersky Anti-Virus (99.7%), F-Secure Anti-Virus (99.6%), Avira AntiVir Premium (98%), BitDefender Antivirus (97.8%), AVG Anti-Virus (96.6%) и Avast Professional Edition (95.1%).
Changes in comparison with previous tests
We decided to analyze the results of all our tests with respect to the effectiveness of the proactive antivirus protection for the period from 2007 until 2009. Therefore we added the results of the Dec. 2007 test to the results of this test.
Diagram 5: Changes in the heuristics effectiveness
Diagram 5 shows that heuristic components of the most antivirus products were considerably enhanced. Most vendors took essential steps to improve their software, and were able to achieve very high results. As opposed to the last year, today we have a whole galaxy of vendors with mature heuristics technologies. It implies that the quality of protection of users will enhance considerably.
Diagram 6: Changes in the detection rate of new viruses (overall detect)
As to the overall level of the detection rate of new malware, the change is not so drastic. Despite of the fact that a lot of vendors try to enhance their indexes, the leaders remain the same.
First of all it is due to the fact that to improve this metric it is not sufficient to have only the technology. It is also necessary to maintain a high reaction speed of virus labs, and only some vendors manage to do so.
Vasiliy Berdnikov, Anti-Malware Test Lab expert, says:
“One can conclude from the test results that a lot of the antivirus products have a perfect heuristics detect (about 60%) together with a low percent of false alarm (up to 0.1%). Heuristics represent a good instrument to increase a general effectiveness of the protection, but as we can see from the test results, some products using it have a high level of false alarm. From this point of view, every vendor seeks a balance between the heuristics detect and the level of false alarm. Some deliberately choose to increase the false alarm level for the sake of high results. Furthermore, modern heuristics technologies, as showed the test, cannot be the panacea for infections and cannot efficiently protect users from the newest forms of threats”.
- Login to post comments
