Methodology for Antivirus Performance Test (February 2010)

Introduction

This testing methodology is the result of co-work and an intellectual property of an expert community Anti-Malware.ru. Our forum offered an open discussion of this test methodology long time before the test and everyone who wanted could bring in his or her own suggestions concerning the future test.

The following antivirus programs took part in the test:

• Avast Antivirus Professional 4.8.1368
• AVG Anti-Virus & Anti-Spyware 9.0.716
• Avira Antivir Preminum 9.0.0.75
• BitDefender Anti-Virus 2010 (13.0.18.345)
• Dr.Web 5.01.1.11171
• Eset Nod32 4.0.4680
• F-Secure Anti-Virus 2010 (10.10 build 246)
• Kaspersky Anti-Virus 2010 (9.0.0.736 (a,b))
• McAfee VirusScan Plus 2010 (13.15.113)
• Microsoft Security Essentials 1.0.1611.0
• Norton Anti-Virus 2010 (17.1.0.19)
• Outpost Antivirus Pro 2009 (6.7.1.2983.450.0714)
• Panda Antivirus 2010 (9.01.00)
• Sophos Anti-Virus 9.0.0
• Trend Micro Antivirus plus Antispyware 2010 (17.50.1366)
• VBA32 WinNT Workstation 3.12.12.0

NB! Antivirus programs were updated after installation till they were actual for the moment of performing the test. The system was forcedly restarted and update function was disabled after that to avoid the influence on measurement results.

Also, all the scheduled scanning tasks were disabled to avoid the test results misrepresentation. If it was impossible, for example in case of autorun files checking, then a special technical pause was done to shutdown (controlled through Process Explorer) right before the test measurements.

Test environment configuration

The test environment was prepared before the beginning of the test. To do that, Microsoft Windows XP Pro Rus SP3 was installed on the clean computer with all updates available for that moment as well as additional software necessary for that test.

The basic characteristics of the computer where the test was performed (see Table 1) as well as the software installed (see Table 2) are given below.

Table 1: The test platform description

Table 2: The list of software installed

After operation system and all the necessary software have been installed, the following actions were undertaken:

1. Disabling all operation system functions that can influence the test results (update, screen saver, Windows Security Notification, Windows Firewall, Windows Defender, Prefetching).
2. Setting Windows Boot-Timer to activate with the system restart.
3. Creating the system snapshot with Acronis True Image.
4. Forming the collection of clean files of different type and size (Windows system files, installed program files, distribution files, documents, media-files and archives).

Thus, the source system spanshot was created and used as an initial one.

The license for Acronis True Image was provided specially for this test by Aflex Software, Acronis, Parallels, ASPLinux representative in Russia and CIS.

Windows Boot-Timer is a special tool for measuring the Widows start up time. The program starts measuring the time after BIOS initialization and till the system loading is completed (with all automatic start up processes).

AppTimer is a special PassMark Software tool for measuring applications start up time. This tool was used with the default settings.

Initial system performance measurement (without installing an antivirus)

To calculate an antivirus system slowdown it is necessary to measure all the initial system characteristics. The following things were done for that purpose:

1. Restoring the system for the initial snapshot.
2. Writing the system boot-time (using Windows Boot-Timer).
3. Measuring the files copying time (using a special BAT-file).
4. Measuring five office applications boot-time (Microsoft Internet Explorer, Microsoft Office Word, Microsoft Outlook, Adobe Acrobat Reader, Adobe Photoshop).
5. Returning to Step 1 – repeating all the measurements (5 cycles).

NB! All the performance and resource-intensiveness characteristics were measured 5 times and the results were averaged less the limit values. At the same time, the system was restored from the initial snapshot with any iteration.

Measuring the boot-time for the system with antivirus software

The system boot-time is one of the most important system characteristics. To measure the system boot-time we used Windows Boot-Timer tool allowing to measure the time from the BIOS initialization finish to the complete system boot (with all automatic start-up processes).

Every antivirus was installed into a separate snapshot of the initial system that provided absolutely the same conditions for every tested product.

After each antivirus installation, update and several technical restarts, the system image was fixed (saved another snapshots) that allowed to save a lot of testing time.

The measurement algorithm for this part of the test is as follows:

1. Restoring the system from the initial snapshot.
2. Antivirus installation and the system restart.
3. Antivirus update.
4. Writing the default settings into a summary table (checking the file types, installed modules, etc.).
5. System restart.
6. Disabling the antivirus automatic update other scheduled default tasks.
7. Windows Boot-Timer launch (system restart delay).
8. Creating the image for the system with installed antivirus software using Acronis True Image (further on mentioned as "antivirus image").
9. Restoring the system from the new image (Step 8).
10. Writing the boot-time (using Windows Boot-Timer).
11. Returning to Step 9 – repeating all the changes (5 cycles).

NB! The system boot-time was measured 5 times and the system was restored to its initial snapshot with an installed antivirus every time before the measurements (returning to Step 9). The results were averaged less the limit values.

Measuring of on-access and on-demand scanners performance and an antivirus influence on the office applications boot-time

At this stage of the test, we checked on-access monitor influence on the file collection copying (clean files folder) from one drive into another. Besides, we checked the time spent on on-demand scanner checking of the same file collection. See the files characteristics in Supplement 1.

For estimating the antivirus software influence on the office applications work, we measured the boot-time of the five top applications (AppTimer tool was used):

1. Microsoft Internet Explorer 8.0.6001.18702;
2. Microsoft Word 2003 (11.8307.8221);
3. Microsoft Outlook 6 (6.00.29.00.5512);
4. Adobe Acrobat Reader 9.2.0;
5. Adobe Photoshop CS4 Extended.

To receive the full and objective picture, we measured CPU and memory utilization at system idling and during the on-demand test scanning (controlled by Process Explorer).

The following characteristics were measured in this part of the test:

1. Free idle system resources (5 cycles for each antivirus).
2. Test files copying time, time measurements (5 cycles for each antivirus). System restoring after every scanning.
3. On-demand test files scanning (5 cycles for each antivirus). System restoring after every scanning.
4. Free system resources during every testing copying and scanning (5 cycles for each antivirus).
5. Test files rescanning time for determining the efficiency of checking optimization technologies used in antivirus products. Rescanning is performed only after the system restart when the first scanning has been completed (1 cycle for each antivirus).
6. Office applications boot-time (5 cycles for each antivirus). System restoring after every set.

The algorithm for this part of the test is as follows:

1. System restoring form the image with antivirus.
2. Idle free system resources measurement.
3. Measuring test files copying time (using a special BAT-file) and free resources.
4. Returning to Step 1 – repeating all the measurements (5 cycles).
5. System backup from the image with antivirus.
6. Measuring test files scanning time and free system resources.
7. * Single measurement of the files re-scanning time.
8. Measurement of five office applications boot-time (Microsoft Internet Explorer, Microsoft Office Word, Microsoft Outlook, Adobe Acrobat Reader, Adobe Photoshop).
9. Returning to Step 5 – repeating all the measurements (5 cycles).

NB! The measurements of test files copying and scanning time, office applications boot-time and idle resources were performed 5 times and every time the system was restored to its primary image with an antivirus installed (returning to Step 1 or Step 5). The results were averaged less the limit values.

Test results processing

All the test data needed special processing that can be divided into several stages:

1. Filtration and averaging of the results of five measuring iterations (the function was chosen when two limit values were not taken into consideration and the remaining three were averaged).
2. Assessment of the system slowdown by each parameter depending on the antivirus software and as compared to the initial system.
3. Results normalization in percentage as compared to the initial system values.
4. Compiling the antivirus performance rating for key characteristics.
5. Awarding the best antiviruses.

Supplement 1: File collection

The collection included the files of the following type and size: Windows system files, installed program files, distributives, documents and, media-files. Total: 5031 files of 67 types (by extension); the total collection size is 5.7 Gb.