Firewall Outbound Attacks Protection Test (July 2013)

Comparative testing of 21 popular firewalls to check  the quality of protection from attacks, coming from inside the system. In the test we checked the protection on 64 specially developed utilities, checking the protection of processes from quitting, protection from standard outbound attacks, protection from non-standard leaks and protection from non-standard techniques of penetration inside the kernel-mode.

Just like an antivirus, firewall is one of the main components of computer safety system. However, compared with antiviruses, objective tests of firewall functioning are rarely done. We tried to close up this gap doing firewall outbound attack testing and testing personal IDS/IPS for attack protection from vulnerable applications in 2011 and 2012. This year we've decided to expand the list of used methods and repeat the firewall test for outbound attack protection to see how the results have changed over the time that went by for popular products with this criteria.

What is this test directed to or what functions does the firewall fulfill? Based on the online standard definition [RFC3511] (2003), firewall is the system, implementing filtration functions for network packages in accordance with the specified rules in order to separate traffic between the network segments. However, as malicious programs' complexity and hacker attacks grow, the initial firewall tasks have been complemented by new functional modules. It's practically impossible to imagine the complete firewall system now without HIPS module (monitoring system events, controlling system integrity, etc).

The main task of a modern firewall is to block unauthorized outgoing and incoming network communications (hereafter — attacks). These include:

Outbound attacks on the firewall protected system:

  • initiated by hackers;
  • initiated by malicious code.

Unauthorized outgoing network connections:

  • initiated by unreliable applications (malicious code);
  • initiated by the applications whose network activity is evidently prohibited by the rules.

Other than that, the products, that can be clearly considered personal firewalls in the classical 2003 definition, practically disappeared from the market. Instead of them we are using complex products for personal computer protection that always include a firewall component.

A firewall test for external attack protection contains studying the attack quality protection from attacks coming from inside the system. The test was done in the following directions:

  1. Checking process protection from finishing.
  2. Protection from standard outbound attacks.
  3. Testing protection from non-standard leaks.
  4. Testing protection from non-standard techniques of kernel mode penetration.

Compared to the previous test there are quite a lot more attacks used now – from 40 to 64. The operating system has also changed, and that's what the tested products have to be able to do. In the past test this was Windows XP, and in this test it's Windows 7 x32. Also, at the end of the year we are planning to have a similar test for Windows 7 x64 OS.

Test Methodology
Test Result and Awards

 

1. Introduction
2. Testing Firewalls for Outbound Attacks Protection
3. Comparing Quality of Protection with Standard and Maximum Settings
4. Final Test Results
5. Anti-Malware.ru Partner Comments

 

Introduction

The testing included 21 popular complex protection program (Internet Security class, if the product line didn't include this product, specifically firewall was chosen) by different vendors relevant at the moment of testing start (May 2013) for the product versions working on Windows 7 x32 platform:

  1. Avast! Internet Security (8.0.1488).
  2. AVG Internet Security (2013.0.3272).
  3. Avira Internet Security (13.0.0.3499).
  4. Bitdefender Internet Security (16.29.0.1830).
  5. Comodo Internet Security (6.1.276867.2813).
  6. Dr.Web Security Space (8.0).
  7. Eset Smart Security (6.0.316.0).
  8. F-Secure Internet Security (1.77 build 243).
  9. G DATA Internet Security (1.0.13113.239).
  10. Jetico Personal Firewall (2.0).
  11. Kaspersky Internet Security (13.0.1.4190(g).
  12. McAfee Internet Security (11.6.507).
  13. Kingsoft Internet Security (2009.05.07.70).
  14. Microsoft Security Essentials (4.2.223.0) + Windows Firewall.
  15. Norton Internet Security (20.3.0.36).
  16. Online Armor Premium Firewall (6.0.0.1736).
  17. Outpost Security Suite Pro (8.0 (4164.639.1856).
  18. Panda Internet Security (18.01.01).
  19. PC Tools Internet Security (9.1.0.2900).
  20. Trend Micro Titanium Internet Security (6.0.1215).
  21. TrustPort Internet Security (2013 (13.0.9.5102).

Before starting the testing we have prepared the testing environment. For that purpose we have installed on a clean computer Windows 7 Enterprise SP1 x86 with all available updates at that moment, as well as the software necessary for the test.

The testing was performed at two setting stages: the recommended by the vendor standard settings (default settings) and the maximum settings. In the first case we used the default settings recommended by the vendor and we followed all the recommended steps.

In the second case we turned on and/or put on highest mode (the strictest settings) all the settings that were turned off in the default mode, but that could impact the testing results. In other words, by setting up the highest settings we mean putting all available from the graphical user interface settings for all modules related to detecting malicious file or network activity to the strictest mode.

The firewall testing was done based on the following outbound attacks broken down for ease to the following difficulty levels:

1. Basic difficulty level (56 possible attacks):

1. checking protection from process interrupting (41 possible attacks);
2. protection from standard outbound attacks (15 possible attacks).

2. Elevated level of difficulty (8 possible attacks):

1. testing protection from non-standard leaks (3 possible attacks);
2. testing protection from non-standard techniques of kernel mode penetration (5 possible attacks).

A detailed description of all attack methods, used in the test, can be found in the testing methodology.

 

Testing Firewalls for Outbound Attacks Protection

Testing firewalls was done on a set of 64 test cases for each product separately (detailed results for each of the products can be found in an Excel report).

To remind you, in accordance with the used award scheme, we gave 1 point (+) if the attack was automatically blocked, the protection of the program tested was not disarmed. 0.5 points (or +/-) was given if the attack was blocked only in some circumstances (for example, when the user would choose the right action by a question of the tested program). And, in the end, in case if the attack was completely successful, or if it partially disabled the protective shield, the points were not awarded. The maximum number of possible collected points in this test was 64.

Table 1-2 and picture 1-2 show firewall testing results separately from the standard and maximum settings. For a better picture, the results for each firewall are broken down into two groups: protection from basic difficulty level attacks and protection from more advanced level attacks.

 

Table 1: Test Results of Firewalls on Standard Settings

Firewall Basic Level Attacks (max. 56 points) Advanced Level Attacks (max. 8 points) Totally Points (max. 64)  
%
Points % % of the amount
Points % % of the amount
Comodo 53 95% 82,8% 6 75% 9,4% 59 92%
Online Armor 50 89% 78,1% 7,5 94% 11,7% 57,5 90%
Norton 45 80% 70,3% 6 75% 9,4% 51 80%
Jetico 46 82% 71,9% 4,5 56% 7,0% 50,5 79%
Outpost 45 80% 70,3% 2,5 31% 3,9% 47,5 74%
Trend Micro 42 75% 65,6% 3 38% 4,7% 45 70%
Kaspersky 42 75% 65,6% 2,5 31% 3,9% 44,5 70%
Dr.Web 42,5 76% 66,4% 2 25% 3,1% 44,5 70%
TrustPort 43 77% 67,2% 0,5 6% 0,8% 43,5 68%
G DATA 42 75% 65,6% 1 13% 1,6% 43 67%
Avast 41 73% 64,1% 1 13% 1,6% 42 66%
Eset 41 73% 64,1% 1 13% 1,6% 42 66%
Bitdefender 41 73% 64,1% 1 13% 1,6% 42 66%
AVG 41 73% 64,1% 0 0% 0,0% 41 64%
McAfee 41 73% 64,1% 0 0% 0,0% 41 64%
PC Tools 41 73% 64,1% 0 0% 0,0% 41 64%
Avira 40 71% 62,5% 0 0% 0,0% 40 63%
Microsoft 40 71% 62,5% 0 0% 0,0% 40 63%
F-Secure 31,5 56% 49,2% 1 13% 1,6% 32,5 51%
Panda 30 54% 46,9% 0 0% 0,0% 30 47%
Kingsoft 27 48% 42,2% 1 13% 1,6% 28 44%

 

Picture 1: Test Results of Firewalls on Standard Settings

 Test Results of Firewalls on Standard Settings

 

Protection from outbound attacks on the recommended by the vendor settings leaves much to be desired. Only three firewalls were able to overcome the threshold of 80% on standard settings. That's Comodo, Online Armor and Norton. Jetico products (79%) and Outpost (74%) are scoring quite close to them. The results of the other firewalls turned out to be significantly worse. 

Compared to the results of the past testing all leaders confirmed their high results, there were only small movements within the leading group, for example, Outpost and Jetico switched their positions. The only surprise was Norton product that showed the results of 45% in the past testing and stayed in the bottom part of the table, and took the third place with 80% in this testing.

The results achieved are related to the fact that many vendors tweak the standard settings in such a way that the number of messages a user reacts to is lower. This is also confirmed by the test results: firewalls asked users questions only in 5.4% of attacks with standard settings and in 9.2% of attacks with the maximum settings. This, however, impacts the protection level that stays silent in a situation when a malicious program will imitate doing very legitimate actions in the system.

It's also important to note two trends. First, the percentage of preventing complex types of attacks overall is much worse than for the attacks at the basic complexity level. More than half of these attacks have been deflected only by four products - Comodo, Online Armor, Norton and Jetico. Four more products have been included in the borderline group having deflected from 25% to 38% of these attacks - that's Outpost, Trend Micro, Kaspersky and Dr.Web. All other products shielded from not more than one complex attack. Secondly, the indicators of basic attack protection improved. If in the past test 11 (50%) products shielded from fewer than 50% of attacks, then there were only 3 (14%) such products in this test.

 

Table 2: Test Results of Firewalls on Maximum Settings

Firewall Basic Level Attacks (max. 56 points) Advanced Level Attacks (max 8 points) Totally Points (max. 64) Total
%
Points % % of the amount Points % % of amount
Comodo 56 100% 87,5% 8 100% 12,5% 64 100%
Bitdefender 56 100% 87,5% 8 100% 12,5% 64 100%
Online Armor 53 95% 82,8% 8 100% 12,5% 61 95%
Kaspersky 53 95% 82,8% 7 88% 10,9% 60 94%
Norton 50,5 90% 78,9% 8 100% 12,5% 58,5 91%
PC Tools 49,5 88% 77,3% 5,5 69% 8,6% 55 86%
Outpost 49 88% 76,6% 5,5 69% 8,6% 54,5 85%
Eset 49 88% 76,6% 5,5 69% 8,6% 54,5 85%
Dr.Web 46,5 83% 72,7% 5 63% 7,8% 51,5 80%
Jetico 46 82% 71,9% 4,5 56% 7,0% 50,5 79%
Trend Micro 43 77% 67,2% 3 38% 4,7% 46 72%
TrustPort 43 77% 67,2% 2,5 31% 3,9% 45,5 71%
G DATA 42 75% 65,6% 3 38% 4,7% 45 70%
Avira 41,5 74% 64,8% 2 25% 3,1% 43,5 68%
Avast 41 73% 64,1% 1,5 19% 2,3% 42,5 66%
AVG 41 73% 64,1% 0 0% 0,0% 41 64%
McAfee 41 73% 64,1% 0 0% 0,0% 41 64%
Microsoft 40 71% 62,5% 0 0% 0,0% 40 63%
F-Secure 31,5 56% 49,2% 1 13% 1,6% 32,5 51%
Panda 30 54% 46,9% 0 0% 0,0% 30 47%
Kingsoft 27 48% 42,2% 1 13% 1,6% 28 44%

 

Picture 2: Test Results of Firewalls on Maximum Settings

 Test Results of Firewalls on Maximum Settings

 

When the maximum settings are on, the protection quality from outbound attacks for many tested firewalls has significantly gone up. This is especially noticeable for the products showing very average results. All leaders of the previous testing showed high results in this test as well. We should note the Bitdefender product for the changes made on it, and, together with Comodo, it showed 100% results and the Norton product that moved to the leading group.

The results for a number of products testing for the standard and the maximum settings turned out to be the same. This is related to the fact that these products don't have any special settings that can impact the results of our test.

 

Comparing Quality of Protection with Standard and Maximum Settings

In accordance with the logic of this test we are not going to add or average out the results of one and the same product with different settings. To the contrary, we want to compare them and show significant differences in the quality of protection for the tested products depending on the settings used.

For illustration we are giving end test results for firewalls with the standard and the maximum settings in table 3 and in picture 3.

 

Table 3: Overall results for the firewall testing with standard and the maximum settings

Product

Standard Settings
Maximum Settings
Comodo 92% 100%
Online Armor 90% 95%
Norton 80% 91%
Jetico 79% 79%
Outpost 74% 85%
Trend Micro 70% 72%
Kaspersky 70% 94%
Dr.Web 70% 80%
TrustPort 68% 71%
G DATA 67% 70%
Avast 66% 66%
Eset 66% 85%
Bitdefender 66% 100%
AVG 64% 64%
McAfee 64% 64%
PC Tools 64% 86%
Avira 63% 68%
Microsoft 63% 63%
F-Secure 51% 51%
Panda 47% 47%
Kingsoft 44% 44%

 

Picture 3: Summary Firewall Test Results on Standard and Maximum Settings

 Overall results for the firewall testing with standard and the highest security settings

 

Picture 3 very clearly illustrates the difference in test results depending on the chosen settings.

First of all, only two products – Comodo and Online Armor show close to the highest security protection indicators with both standard and maximum settings.

Secondly, when changing the standard settings, suggested by the vendor, a number of products show a significantly better level of protection. This is best seen on such products as Bitdefender, Kaspersky, Eset, F-Secure and PC Tools.

Thirdly, as we have noted above, some of the tested products don't have any settings that could in any way impact the results of the test. That's why their results at all types of settings are the same in this test. This group includes Jetico, Avast, AVG, McAfee, F-Secure, Panda, Kingsoft and Microsoft.

The resulting number of points doesn't take into account the situations when the attack has been rejected, but there were issues with the product user interface. In the majority of the cases the issues were in «falling» out of the interface for a short period of time (from 2 to 10 seconds) or until the next loading of the operating system. Despite the fact that when there were problems with the user interface the products continued to provide protection, the fact that these problems existed is perceived negatively and can impact the preferences in the product choice. The number of problems with the user interface are shown in table 3 and in picture 3. The errors analyzed were the ones appearing during the 1st level attacks, and the total number of them was 41.

 

Table 4: The Number of Errors with Firewall's GUI on Standard and Maximum Settings

Product Standard Settings
Maximum Settings
Errors % Errors %  
McAfee 34 83% 34 83%
Microsoft 33 80% 33 80%
Kingsoft 20 49% 20 49%
F-Secure 19 46% 19 46%
Panda 17 41% 17 41%
Jetico 16 39% 16 39%
PC Tools 13 32% 13 32%
Trend Micro 12 29% 12 29%
AVG 10 24% 9 22%
TrustPort 9 22% 9 22%
G DATA 9 22% 9 22%
Bitdefender 8 20% 8 20%
Norton 6 15% 6 15%
Avast 5 12% 5 12%
Outpost 5 12% 5 12%
Eset 5 12% 4 10%
Comodo 5 12% 0 0%
Avira 2 5% 2 5%
Dr.Web 2 5% 2 5%
Kaspersky 1 2% 1 2%
Online Armor 1 2% 1 2%

 

Picture 4: The Number of Errors with GUI on Standard and Maximum Settings 

 The number of errors with the user interface on standard and highest security settings

 

The results we got show that the problems with the user interface with McAfee and Microsoft products have occurred in the majority of attacks (more than 80%). This can be called an unacceptable level, because practically any deflected attack will lead to problems. Pretty bad results, that are in the range of 30% to 50% are shown by products Kingsoft, F-Secure, Panda, Jetico and PC Tools. When they are being used, each second-third attack will lead to problems with the interface. Another whole lot of products show the results from 10% to 30%, that can be called satisfactory. The products Avira, Dr.Web, Kaspersky and Online Armor showed good results, the problems for them occurred in the range of 2-5% of attacks. The only product that hasn't had any issues with user interface was Comodo on maximum settings, that can be considered to be an excellent result. However, with the standard settings, Comodo results get worse (12%), which tells about the fact that using this product requires certain knowledge about its settings.

 

Final Test Results and Awards

Just like in the previous testing we haven't averaged out the results of one and the same product with different settings, but looked at the results independent of each other. Therefore, each of the tested products can get two awards, one for each setting type.

In accordance with the award plan, the best firewalls get the award specifying the settings used for that purpose, see table 4.

 

Table 5: Final Firewall Test Results on Standard and Maximum Settings

Product Settings Type Blocking Attacks [%] Total
[%]
Award
Basic Level Advanced Level
Comodo Max 100% 100%

100%

Platinum Firewall Outbound Protection Award 
Platinum Firewall Outbound
Protection Award
Bitdefender Max 100% 100%

100%

Online Armor Max 95% 100%

95%

Gold Firewall Outbound Protection Award 
Gold Firewall Outbound
Protection Award
Kaspersky Max 95% 88%

94%

Comodo Standard 95% 75%

92%

Norton Max 90% 100%

91%

Online Armor Standard 89% 94%

90%

PC Tools Max 88% 69%

86%

Outpost Max 88% 69%

85%

Eset Max 88% 69%

85%

Norton Standard 80% 75%

80%

Dr.Web Max 83% 63%

80%

Jetico Max 82% 56%

79%

 Silver Firewall Outbound Protection Award
Silver Firewall Outbound
Protection Award
Jetico Standard 82% 56%

79%

Outpost Standard 80% 31%

74%

Trend Micro Max 77% 38%

72%

TrustPort Max 77% 31%

71%

Trend Micro Standard 75% 38%

70%

Kaspersky Standard 75% 31%

70%

Dr.Web Standard 76% 25%

70%

G DATA Max 75% 38%

70%

TrustPort Standard 77% 6%

68%

Bronze Firewall Outbound Protection Award  
Bronze Firewall Outbound
Protection Award
Avira Max 74% 25%

68%

G DATA Standard 75% 13%

67%

Avast Max 73% 19%

66%

Avast Standard 73% 13%

66%

Eset Standard 73% 13%

66%

Bitdefender Standard 73% 13%

66%

AVG Max 73% 0%

64%

AVG Standard 73% 0%

64%

McAfee Max 73% 0%

64%

McAfee Standard 73% 0%

64%

PC Tools Standard 73% 0%

64%

Microsoft Max 71% 0%

63%

Microsoft Standard 71% 0%

63%

Avira Standard 71% 0%

63%

F-Secure Max 56% 13%

51%

No Award
F-Secure Standard 56% 13%

51%

Panda Max 54% 0%

47%

Panda Standard 54% 0%

47%

Kingsoft Max 48% 13%

44%

Kingsoft Standard 48% 13%

44%

 

Comodo and Bitdefender showed the best test results, and they got 100% on the maximum settings. These two products are getting the Platinum Firewall Outbound Protection Award.

Online Armor, Kaspersky, Comodo, Norton, PC Tools, Outpost, Eset and Dr.Web firewalls also showed very high results in the test (over 80%), and they are receiving the Gold Firewall Outbound Protection Awards. It's important to note that Comodo got this award for the standard settings, Online Armor and Norton for the standard and the maximum settings, and all the rest only for the maximum settings.

Further down the list is the group of seven firewalls whose results get in the range from 60% to 70%. That's Outpost, Kaspersky and Dr.Web with the standard settings; TrustPort and G DATA on the maximum settings, as well as Jetico and Trend Micro at the same time with the standard and the maximum settings. They all get the Silver Firewall Outbound Protection Awards.

There is a fairly large group of products that got in the range from 60% to 70%, are getting the Bronze Firewall Outbound Protection Award. We should note in this group the products Eset and Bitdefender with the standard settings, as at the maximum settings they managed to reject a significantly larger number of attacks.

Studying the detailed test results and making sure that the final calculations are correct can be done by downloading the test results in  Microsoft Excel format.

 

Ilya Shabanov, Anti-Malware.ru managing partner:

«I have been very happy with the fact that a lot of vendors have significantly improved their proactive protection from outbound attacks and self-defence in their products. We even had to review the award plan to raise the requirement benchmark. The result of less than 51% of points is now considered to be a complete failure.

Bitdefender gave us a nice surprise after it has paranoidly rejected all 100% of attacks, Eset and Dr.Web with the results at the maximum settings of 85% to 80% correspondingly, as well as a newbie in our testing TrustPort. The «golden group» of products based on the results of this test got firewalls from Comodo, Norton and Online Armor, that got the result of more than 80% on the standard and the maximum settings. The stable high results are in the tests that touch upon the proactive protection, and it has been demonstrated by Kaspersky, Outpost, PC Tools.

However, in the case of a whole number of tested products the logic used for standard settings is not clear. As a result, the level of protection is significantly lower for the majority of users, accustomed to using the protection with the standard settings. This, first of all, related to Bitdefender, Kaspersky, Eset and PC Tools products».

 

Mikhail Kartavenko, Anti-Malware.ru testing lab manager:

«Looking at this test as a continuation of the previous similar test, we can specify a number of main tendencies and issues firewall functions.

First of all, on average, the majority of products showed better results than 1.5 years ago, however, they did it mainly due to rejecting the simplest level 1 attacks. More complex attacks are only doable for a limited number of products.

Secondly, even if the protection of processes from halting worked (first level attacks), the user interface stops for many products. This places the user in an uncomfortable position, in which he doesn't realize whether the protection is working or not.

Thirdly, there is quite a large spread in firewall working on the standard and the maximum settings. As a result, a reasonable protection level can only be achieved for the experienced users who know and can correctly set up firewalls to work.

Therefore, the test has uncovered the «weak» points for modern firewalls, resolving which can improve their protection».

 

Anti-Malware.ru Partner Comments

 

Konstantin Kutikov, AV Lab director and Agnitum company testing department manager:

«As the testing showed, Outpost product places first based on the number of attacks found by the product and that the product was able to prevent. This is another proof for us that the proactive protection technology works well, and this has been noted multiple times by the world leading testing labs.

However, having done a large, thorough testing, and having received exact results, our colleagues from Anti-malware.ru have interpreted the results in a somewhat strange way in our opinion. Our products have lost quite a few points for asking users questions about banning potentially malicious activity. Other products, that blocked the same actions without asking the user about it, haven't been fined for that.

Based on our more than ten years of experience working with Windows kernel we think that blocking these actions globally for unknown applications for a security product is almost a 100% probability to cause a failure and disable the system from working. If the lab tried to use the system or other applications with these settings, testers would definitely come across this.

That's why we respect the user's decision and give an opportunity to set up blocking of this activity at the application level. Our product gives an opportunity to determine and block the largest number of attacks based on Anti-malware.ru data, and do it in the safest for the system stability way, at the application level».

 

Elena Toll, Head of the department for technical marketing of the Russian ESET headquarters:

«We are glad that Anti-Malware.ru research has confirmed ESET NOD32 Smart Security firewall's high effectiveness, and that's our personal line flagship product. In the next seventh generation of home solutions for ESET NOD32 we are planning to widen the system of ESET proactive protection, having implemented new technologies to prevent infection for the known vulnerabilities, as well as for detecting deeply encoded malicious files.

By all means, firewall is an indispensable part of the whole system of proactive protection and one of the key components of a full information security product. But we cannot lose sight of modern anti-virus products like ESET NOD32 Smart Security, it's not a set of separate functions, all of the components of which (anti-virus, anti-spam, anti-phishing, HIPS, etc)  complement each other, providing multi-step protection from the newest information threats».

 

Vadim Volkov, DinaSoft company project manager, which is the official distributor of TrustPort in Russia:

«I think that it's a positive fact that Anti-Malware.ru has finally included TrustPort products in its tests. As a result of the tests, Russian users were able to see that TrustPort Internet Security provides quite decent protection, surpassing a number of more celebrated competitors. In combination with one of the highest detection rates of malicious programs, that a multi-engine scanner provides, you get quite good choice for users who are looking for something to exchange the used antivirus for.
I hope that TrustPort will take into account the test results in its future creations and they will be better protected from outbound attacks».

 

Ilya Rabinovich, SoftSphere Technologies founder and general director:

«It strikes you right away that vendors lower the user protection level to simplify their products. And in some cases it's a pretty serious issue (for example, BitDefender). I also think that in many ways this has to do with «polishing» products for Matousec tests, where they are testing only the highest protection product settings without checking the default settings. As a result, a product gets very high protection ratings based on Matousec tests, but in actual use for regular users who will only use the default settings without having everything at the highest security level, probably being afraid of breaking something, it is not capable of providing a reasonable protection level».