Methodology for Zero-Day Threats Detection Test (November 2009)

The test was performed from July 7 up to October 22, 2009. Before the test we prepared the testing environment – a set of clean virtual machines under VMware Workstation 6.0 with Microsoft Windows XP Pro SP3 installed (it was not updated on purpose). Each computer has its own separate protection software installed from the list given below.

If applicable, we used the integrated “Internet Security” software, but if they were not available in the vendor's product line we used classic antiviruses. Eighteen antivirus popular antivirus programs participating in this testing included:

1. Avast Antivirus Professional 4.8-1335
2. AVG Internet Security 8.5.386
3. Avira Premium Security Suite 9.0.0.377
4. BitDefender Internet Security 2009 (12.0.12)
5. Comodo Internet Security 3.9.95478.509
6. Dr.Web Security Space 5.0.1.06018
7. Eset Smart Security 4.0.437
8. F-Secure Internet Security 2009 (9.00 build 149, он же СТРИМ.Антивирус)
9. G DATA Internet Security 2010 (20.0.2)
10. Kaspersky Internet Security 2010 (9.0.0.459)
11. McAfee Internet Security Suite 13.11
12. Microsoft Security Essential 1.0.2140.0
13. Norton Internet Security 2009 (16.5.0.135)
14. Outpost Security Suite 2009 (6.5.5.2535.385.0692)
15. Panda Internet Security 2010 (15.00.00)
16. Sophos Anti-Virus 7.6.9
17. Trend Micro Internet Security 2009 (17.1.1250/8.913.1006)
18. VBA32 Workstation 3.12.10.10

Also in the test participated two specialized HIPS (Hosted Intrusion Prevention System):

1. DefenceWall HIPS 2.56
2. Safe'n'Sec Personal 3.5.0.490

Unfortunately, some vendors issued their products updates during the long testing period and it could not be reflected in the final test results.

It is worth mentioning that all the antiviruses were tested with default settings and updates received in automatic mode. In fact, was created a situation as if an ordinary user having one of the tested protection applications installed on his computer used the Internet and clicked some interesting links he got this or that way (see above).

Malware selection

For the test we selected the links to the sites infected by Zero-day malware samples. What does "zero-day" mean? It means that all the malware downloaded via these links must not be detected by file antivirus software from more than 20% of the tested products link that was checked through VirusTotal service (a 41-antivirus engine is plugged in at the server). If the samples selected were even detected the verdicts as a rule were uncertain (an infection suspect or a package).

The number of samples meeting all the requirements was not big and that influenced the final selection size and testing schedule. 36 working links to the Zero-day malware were selected within several months of the test and they were used in the testing process.