Antivirus Proactive Protection Test (January 2008)

The industry has recently witnessed a shift in emphasis to so-called proactive methods of antivirus protection, which allow antivirus software to combat malicious programs that have undergone modifications and those that are as yet unknown. This development trend is the most promising on the market and almost every developer likes to emphasize just how good their proactive defense is.


Table of Contents

Intoduction
Evaluating the effectiveness of proactive antivirus protection
- Results and Awards
Changes to the effectiveness of proactive antivirus protection over time
Signatures or Heuristics?

There are even attempts to contrast the newer proactive technologies with the older reactive technologies that use signature-based methods to detect malware and that require continuous and rapid updates of antivirus databases.

The concept of proactive protection is, of course, extremely attractive: a virus hasn’t even appeared and already there is protection against it. But the question arises as to just how effective these technologies are.

It should be noted that proactive technologies encompass a broad range of concepts and approaches, and including them all within the framework of a single test is simply not feasible. In this test we will only compare the heuristic components of antivirus protection (heuristic + generic detection, i.e., extended signatures), without taking into account an analysis of system events (behavior blockers or HIPS).

The results of the test make it possible to say how effective a heuristic analyzer is and in which antivirus product this component performs the best.

As an addendum, a final measurement of the detection level for the collection of malware samples was performed on the updated antivirus software a week after the main test. As a result, the quality of detection for new viruses, as well as the effectiveness of the classical signature-based method of each antivirus program was ascertained in addition to their heuristics. 

 

Introduction

The industry has recently witnessed a shift in emphasis to so-called proactive methods of antivirus protection, which allow antivirus software to combat malicious programs that have undergone modifications and those that are as yet unknown. This development trend is the most promising on the market and almost every developer likes to emphasize just how good their proactive defense is.

There are even attempts to contrast the newer proactive technologies with the older reactive technologies that use signature-based methods to detect malware and that require continuous and rapid updates of antivirus databases.

The concept of proactive protection is, of course, extremely attractive: a virus hasn’t even appeared and already there is protection against it. But the question arises as to just how effective these technologies are.

It should be noted that proactive technologies encompass a broad range of concepts and approaches, and including them all within the framework of a single test is simply not feasible. In this test we will only compare the heuristic components of antivirus protection (heuristic + generic detection, i.e., extended signatures), without taking into account an analysis of system events (behavior blockers or HIPS).

The results of the test make it possible to say how effective a heuristic analyzer is and in which antivirus product this component functions the best. 

 

Evaluating the effectiveness of proactive antivirus protection

Fifteen of the most popular antivirus programs participated in the test to measure the effectiveness of proactive antivirus protection:

  1. Agnitum Outpost Security Suite 2008
  2. Avast! Professional Edition 4.7
  3. AVG Anti-Virus Professional Edition 7.5
  4. Avira AntiVir Personal Edition Premium 7.0
  5. BitDefender Antivirus 2008
  6. Dr.Web 4.44
  7. ESET NOD32 Anti-Virus 3.0
  8. F-Secure Anti-Virus 2008
  9. Kaspersky Anti-Virus 7.0
  10. McAfee VirusScan Plus 2008
  11. Panda Antivirus 2008
  12. Sophos Anti-Virus 7.0
  13. Symantec Anti-Virus 2008
  14. Trend Micro Antivirus plus Antispyware 2008
  15. VBA32 Antivirus 3.12

Testing of the antivirus programs was performed on a Windows XP SP2 operating system from 21 October to 8 December 2007 strictly in line with the methodology for setting up the conditions to check the effectiveness of heuristic analyzers (the update function of all the antivirus products was switched off, i.e., the antivirus databases were frozen on the date the test was initiated).

Diagram 1 and table 1 show the detection results for unknown malware by the different antivirus programs.

 

Diagram 1: Effectiveness of heuristic analyzers


Table 1: Effectiveness of heuristic analyzers

Antivirus product Number of undetected viruses Detected viruses (%)
Avira 1210 71%
BitDefender 1560 63%
Eset 1739 59%
DrWeb 1793 57%
Sophos 1855 56%
Avast 2029 52%
VBA 2175 48%
Kaspersky 2289 45%
McAfee 2381 43%
Symantec 2583 38%
AVG 2637 37%
F-Secure 2685 36%
Trend Micro 2927 30%
Panda Security 3165 24%
Agnitum (VirusBuster) 3679 12%
Total samples in collection: 4191  


The out-and-out leader in terms of the effectiveness of the heuristic protection component was Avira AntiVir Personal Edition Premium, whose detection level for unknown malicious programs was very high (71%).

The BitDefender Antivirus heuristic analyzer was also very effective, detecting 65% of the unknown malware. Their results meant that both of these products received the Gold Proactive Protection Award.

The heuristic components in ESET NOD32 Anti-Virus, Dr.Web, Sophos Anti-Virus, Avast! Professional Edition, VBA32 Antivirus, Kaspersky Anti-Virus, and McAfee VirusScan Plus all demonstrated a high level of effectiveness, ranging from 59% to 43% respectively. All those products received the Silver Proactive Protection Award. The first two antivirus programs from this group only missed out on the Gold Proactive Protection Award by 1-3%.

Another 5 products – Symantec Anti-Virus, AVG Anti-Virus Professional Edition, F-Secure Anti-Virus, Trend Micro Antivirus plus Antispyware, and Panda Antivirus – all achieved satisfactory results and received the Bronze Proactive Protection Award.

The only product to fail the test was Agnitum Outpost Security Suite, which appears to lack a heuristic analyzer. It managed a result of only 12%. 

 

Results and Awards

 

Award Products

Gold Proactive
Protection Award
Avira AntiVir Personal Edition Premium 7.0 (71%)
BitDefender Antivirus 2008 (65%)
Silver Proactive Protection Award
Silver Proactive
Protection Award
ESET NOD32 Anti-Virus 3.0 (59%)
Dr.Web 4.44 (57%)
Sophos Anti-Virus 7.0 (56%)
Avast! Professional Edition 4.7 (52%)
VBA32 Antivirus 3.12 (48%)
Kaspersky Anti-Virus 7.0 (45%)
McAfee VirusScan Plus 2008 (43%)

Bronze Proactive Protection Award
Bronze Proactive
Protection Award

Symantec Anti-Virus 2008 (38%)
AVG Anti-Virus Professional Edition 7.5 (37%)
F-Secure Anti-Virus 2008 (36%)
Trend Micro Antivirus plus Antispyware 2008 (30%)
Panda Antivirus 2008 (20%)

 

Failed

 

Agnitum Outpost Security Suite 2008 (12%)

 

Changes to the effectiveness of proactive antivirus protection over time

Due to the fact that the new malicious programs were collected over a four-week period (from 5 November to 2 December 2007), it became possible to evaluate how the effectiveness of the various heuristic analyzers changed over time. This entailed the sample collection being split up into four equal parts, each representing a week.

Diagram 2 and table 2 show data on the effectiveness of the heuristic analyzers for the weekly collections.

 

Diagram 2: Effectiveness of heuristic analyzers (broken down into weeks)

 

Table 2: Effectiveness of heuristic analyzers (broken down into weeks)

Antivirus product Detected viruses (%)
Week 1 Week 2 Week 3 Week 4
Avira 72% 72% 65% 74%
BitDefender 66% 63% 56% 65%
Eset 72% 58% 47% 56%
DrWeb 58% 53% 53% 63%
Sophos 64% 60% 44% 55%
Avast 54% 51% 40% 59%
VBA 57% 45% 48% 43%
Kaspersky 49% 40% 42% 49%
McAfee 45% 42% 33% 50%
Symantec 42% 36% 30% 44%
AVG 47% 32% 35% 34%
F-Secure 38% 29% 32% 43%
Trend Micro 41% 23% 21% 33%
Panda Security 39% 23% 20% 16%
Agnitum (VirusBuster) 17% 9% 11% 12%
Total samples in collection: 4191      

 

From table 2 it can be seen that the heuristic component in Avira AntiVir Personal Edition Premium, BitDefender Antivirus, Dr.Web and Kaspersky Anti-Virus demonstrated the most stable levels of effectiveness throughout the period when there were no updates, while the effectiveness of the component in ESET NOD32 Anti-Virus, Panda Antivirus and AVG Anti-Virus Professional Edition fell significantly. The drop in the heuristic effectiveness of ESET NOD32 Anti-Virus over the test period can be singled out in particular.

 

Signatures or heuristics?

As an addendum to the test, a final measurement of the detection level for the collection of malware samples was performed on the updated antivirus software a week after the main test (8 December 2007). As a result, the effectiveness of the classical signature-based method of each antivirus program was ascertained in addition to their heuristic analyzer. This made it possible to see which role each of the components plays in the detection level for each antivirus program (see diagram 3).


Diagram 3: Effects of different antivirus protection components on the overall level of malware detection

 

 

 

The diagonal line in diagram 3 denotes a 100% level of detection for new malicious programs. Getting close to the line can be achieved through the effective functioning of one of the components of antivirus protection or a combination of the two.

Products in the dark orange (80-100%) and light orange (60-80%) zones demonstrated excellent and good detection levels of new viruses (aged from 1 to 5 weeks, see methodology).

The majority of them – Avira AntiVir Personal Edition Premium, Dr.Web, BitDefender Antivirus, ESET NOD32 Anti-Virus, and Sophos Anti-Virus – attained that level based on the contribution of their proactive component.

Kaspersky Anti-Virus and F-Secure Anti-Virus achieved similar results based on their signature component.

The most balanced in this sense were AVG Anti-Virus Professional Edition, VBA32 Antivirus, Symantec Anti-Virus and McAfee VirusScan Plus, where both components functioned equally effectively (ending up in the lower left-hand square on diagram 3, while the overall detection level for new malware was good or excellent).

Panda Antivirus achieved a satisfactory overall detection level for new malicious programs, but Trend Micro Antivirus plus Antispyware and Agnitum Outpost Security Suite were completely ineffectual against new threats.

 

Table 3: Quality of new virus detection

Antivirus product Detected viruses before update (%)
(4 weeks)
Detected viruses after update (%) Total % of detected viruses
Avira 71.1% 21.5% 92.7%
BitDefender 62.8% 23.1% 85.9%
Eset 58.5% 18.1% 76.6%
DrWeb 57.2% 23.8% 81%
Sophos 55.7% 12.6% 68.3%
Avast 51.6% 24.6% 76.2%
VBA 48.1% 31% 79.1%
Kaspersky 45.4% 52.2% 97.6%
McAfee 43.2% 20% 63.2%
Symantec 38.4% 26.2% 64.6%
AVG 37.1% 49.2% 86.3%
F-Secure 35.9% 62.2% 98.2%
Trend Micro 30.2% 0.3% 30.4%
Panda Security 24.5% 33.6% 58.1%
Agnitum (VirusBuster) 12.2% 1.3% 13.5%


 Diagram 4: Quality of new virus detection - contribution of different components

 


F-Secure Anti-Virus (98.2%), Kaspersky Anti-Virus (97.6%), Avira AntiVir Personal Edition Premium (92.7%), BitDefender Antivirus (86%) and AVG Anti-Virus (85.9%) turned out to be the best at detecting new malicious programs.


Comments

#

i note your comment about the suite not having a heuristic option, it does, it's under anti-malware settings, use the customize button.
Photobucket
what version did you use? must have been an early beta, please test current versions, not old obsolete ones. this is from the current v. 6.0.2284 build, tho i think the heuristics option was there in the christmas edition released in early december. please check the option is turned on, hard to test it if it is not.

#

kronckew, you didn't understand the conclusion from testing results. I meant Outpost security suite 2008 has heuristic but it's absolutely useless. It looks like a fake option.

This fact is equal to they don't have it at all :-)

#

ouch, "its like a fake option"

funny, so i guess people should not rely on catching those new virii with outpost suite.

i must admit, i have tried it and it wasnt a favourite of mine.

some companys should stick to what they know, Firewalls.

Post new comment

The content of this field is kept private and will not be shown publicly.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Fill in the blank