SonicWALL uses Ixia for Next-Gen Firewall Shootout at Interop

Posted on May 7, 2012

SonicWALL uses Ixia for Next-Gen Firewall Shootout at Interop

Overview
At Interop 2012, in SonicWALL’s booth (#751), we will be conducting a live network security effectiveness face-off using six of the top next generation firewall (NGFW) products. The following solutions will take place in the competitive comparison:

• Check Point UTM-1 Total Security 138
• Fortinet FortiGate-40C
• Juniper Networks SRX210
• Palo Alto Networks PA-200
• SonicWALL NSA 250M
• WatchGuard XTM 21

The  security services testing objectives for the shootout are divided into two areas:

1. Security service effectiveness using pure network security attacks
2. Security services effectiveness  testing along clean network traffic

We used Ixia’s IxLoad-Attack tool due to generate attack traffic combined with other network traffic, as well as generate real-time individual report for all the products. In addition we found the Ixia attack library to have theindustry’s most virulent malware attacks. Ixia also has the ability to provide real-time statistics on every product for both attacks and good traffic. The real-time statistics are imperative since they indicate why a product missed or blocked attacks.
The detailed statistics further highlight a product’s behavior under attack – i.e. TCP resets, TCP FIN, TCP time out retry etc. We used Ixia’s real-time statistical capability while running tests on all the products in parallel, showing that the feature’s effectiveness does not change based on product size. The attack terminology used here is equivalent to published vulnerabilities.
Test Topology
Ixia simulated multiple client PC on the trusted side (LAN) and servers on the untrusted side (WAN). We deployed a total of six competitive boxes for this demo as outlined below: 

Test Configuration
On each product we used ten simulated client PC’s on the LAN side with ten unique IP addresses communicating with ten servers with unique IP address on the WAN side. We selected all 534 CRITICAL attacks within the Ixia attack library for a total of 8777 attacks. The test objective was 10 concurrent attacks, meaning that 534 total attacks will be divided by ten and each group use a unique IP’s on the client side – sending about 53 attacks each in parallel. The time to complete the test is determined via how fast a product can close the TCP connection/session.  All the products were configured for maximum protection, and we synced with the latest available signature update prior to the test.  The attacks were initiated both from trusted and untrusted sides.

Below you can see an IxLoad-Attack screen capture for the test configurations used. 

Test results
The test result for effectiveness shows the difference in block rate as well as the time each product takes to kill attacks.

Product	Block ratio %	Attack Kill time in Seconds	Total attacks
SonicWALL NSA 250M	97%	20	534
Palo Alto PA 200	67%	93	534
Juniper SRX 210	59%	74	534
Fortinet  FortiGate-40C	93%	357*	534
CheckPoint UTM-1 138	57%	55	534
WatchGuard XTM 21	51%	56	534

*In our test for this device we noticed that the Fortinet does not respond to some of the attacks after the TCP session was established, and IxLoad-Attack needs to wait and go through RFC defined timeout and retry for TCP. This resulted in increased duration for kill time to complete the test.

The charts below illustrate the data: 

The fastest product to kill/stop the attack was SonicWALL NSA 250M. The appliance managed to reset all the incoming attacks as they were coming in. The test showed other products to use reset, FIN while some just timeout on the TCP connection /session created via an attack.
The second test followed the same setup but added a HTTP file transfer of 10Mbytes per second for a 50MB file. The below table illustrates the change in effectiveness for Fortinet’s appliance while the remaining appliances show hardly any change in performance. Under attack the Fortinet Fortigate-40C also showed fluctuating throughput.

Product	Block ratio %	Block ratio % with HTTP file transfer
SonicWALL NSA 250M	97%	97%
Palo Alto PA 200	67%	67%
Juniper SRX 210	59%	59%
Fortinet FortiGate- 40C	93%	78%
CheckPoint UTM-1 138	57%	57%
WatchGuard XTM 21	51%	51%

Summary
Using the same methodology and attack vectors, we establish a product’s effectiveness ratio. However there could be updates to both attacks and signatures from all the vendors.  In our test we identified SonicWALL NSA 250M as the appliance with the highest block rate and WatchGuard XTM 21 as the appliance with the lowest rate of protection when exposed to pure attacks. The test also showed a change in protection effectiveness of some appliances under additional clean traffic. Here the effectiveness of Fortinet’s FortiGat-40C dropped significantly and allowed attacks to pass the protection. Finally, the test showed variations in the attack kill time for different appliances – with some devices leaving the TCP connection open until timeout occurs.
These results show that test labs should pay special attention to attack kill time, an as increase of this value results in higher CPU and memory usage to keeping connections open. We believe the most successful way to test the protection effectiveness of next-generation firewalls is to conduct  correlating attacks that directly produce logs on the product under testing conditions. The correlation between log generated on the product and attacks log generated by Ixia are very important, as the product can experience a reliability issue under testing conditions and enter a no response state where test tools could falsely mark the attack as successfully blocked.