Security Now
Security is top-of-mind everywhere, especially right here where Juniper experts share their thoughts on the latest security breakthroughs and product advancements
Showing results for 
Search instead for 
Do you mean 

BlackNurse in review: Is your NGFW vulnerable?

by Juniper Employee ‎11-28-2016 12:15 PM - edited ‎12-02-2016 06:30 AM

This is not an official SIRT announcement. Juniper SIRT does not comment on issues in which Juniper products are not vulnerable. SIRT is still reviewing our broad portfolio and, if an issue is discovered, they will issue an appropriate advisory here

 

"This is the best article and test we have to date on the BlackNurse attack. The article provides some answers which are not covered anywhere else. The structure and documentation of the test is remarkable. It would be nice to see the test performed on other firewalls – good job Craig ” 

Lenny Hansson and Kenneth Bjerregaard Jørgensen, BlackNurse Discoverers

 

On November 10th, 2016, Danish firm TDC published a report about the effects of a particular ICMP Type+Code combination that triggers resource exhaustion issues within many leading Firewall platforms. The TDC SOC has branded this low-volume attack BlackNurse, details of which can be seen here, and here

 

Over the past few weeks, Firewall vendors have been coming out with their own unique perspective on the issue. Out of those vendors who have made public statements, none have provided any information as to exactly what can be expected when this attack targets a host behind their security device. 

 

As we were curious ourselves, we went ahead and created a lab environment to test both our products and the products of a few of our competitors. With regards as to why Fortinet and Cisco were omitted from our list, the community has taken care of testing for us (as you can see here under the list of affected products). Fortinet has posted a blog in response which recommends deploying a separate DDoS appliance to defend against these types of attacks.

 

For those of you interested in the makeup of a "BlackNurse" packet, I have included a screenshot from Wireshark below. These packets include a full 5-Tuple about the supposed session they're referencing: Source IP, Source Port, Destination IP, Destination Port, and Protocol. We will discuss why this is important later on.

 ICMP_Embedded.png

 

To ensure we were comparing "apples-to-apples" between our product and our competitors, we turned to virtualization. Virtualizing our test bed provided us with the ability to abstract hardware advantages of one vendor from another. If each device has an identical amount of compute assigned, one can easily determine whose software is superior for a given task.

 

I will be including the exact makeup of our test bed below for those of you who wish to replicate and/or deeply analyze the results.

Screen Shot 2016-11-26 at 12.54.45 PM.png

The ESXi 5.5 host used in our lab had the following components:

  • 2x Intel Xeon L5520 CPUs
  • 82GB of DDR3-1333 ECC Memory

 The Attacker, Benign User, and Protected Server were configured with relatively low specifications:

  • 2x vCPU
  • 2GB RAM
  • 1x VMXNET3 NIC 
  • Running Ubuntu Server 16.04 LTS with tcpdump, hping3, and wireshark-common (for capinfos) installed. 

 Each NGFW under test had the following resources assigned:

  • 2x vCPU
  • 4GB RAM
  • 3x VMXNET3 NIC's (Management/Outside/Inside)
  • Latest Operating System:
    Juniper vSRX was tested on Junos 15.1X49-D60
    Palo Alto PA-VM was tested on PAN-OS 7.1.6
    Check Point vSEC was tested on R77.30 with the latest Jumbo HFA and deployed as a standalone gateway.
    Check Point's dedicated management station was powered off during testing to avoid skewing results due to increased load on the ESXi host.  

Each VM is utilizing paravirtualized VMXNET3 interfaces to maximize their potential performance and interrupt handling. As our competitors do not have the capability to utilize SR-IOV, this was not leveraged by the vSRX during testing. 

 

The Test Plan

For the purpose of our experiment, we created ideal circumstances under which to identify the maximum amount of BlackNurse traffic a given solution can support. To elaborate, each NGFW had:

  1. A single rule which permits any source to any destination with any application
  2. No Network Address Translation (NAT)
  3. No Logging
  4. No advanced or CPU-intensive features such as IPS

The test was run in two stages, each with two degrees of flooding (Low and High)

A simple PASS or FAIL is assigned based on the Benign User's ability to fetch the index.html from the Protected Server's NGINX server within a generous fifteen seconds. This fetch is only executed once the flood has been active for one minute. The test attempts validate resource availability of the protected host during the Denial-of-Service (DoS) attack. Failing this test indicates that the protected host has become unavailable and that the DoS attack can be considered successful. 

 

Prior to executing the attack, each test bed is validated to ensure a successful fetch is able to complete in less than 0.01 seconds:

$ time wget 10.1.1.100
--2016-11-27 15:30:40--  http://10.1.1.100/
Connecting to 10.1.1.100:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 612 [text/html]
Saving to: ‘index.html’

100%[======================================>] 612         --.-K/s   in 0s

2016-11-27 15:30:40 (74.8 MB/s) - ‘index.html’ saved [612/612]


real    0m0.005s  <----
user    0m0.000s
sys     0m0.000s

 

A strict timeout is enforced during the test utilizing the 'timeout' command. If the request has not completed within the time specified, the timeout utility will kill the attempt and the process will exit.
Example of a failed attempt:

$ timeout 15 wget 10.1.1.100
--2016-11-27 15:34:32--  http://10.1.1.100/
Connecting to 10.1.1.100:80... 
$

 

For those of you following along at home, if you do not have a packet-generator at your disposal, you can use hping3 to generate the required ICMP Type 3 Code 3 packets. A word of caution: You will notice that if a device is more adept at handling an arbitrary flood, hping3 will be able to send more traffic to the target due to the increased availability of resources on the ESXi host. The numbers included in the results below have been normalized to reflect this. 

 

Running the following string on the Attacking Machine generated 40 Megabits-per-second (Mbps) or 75,000 Packets-per-second (PPS) of BlackNurse traffic destined to the Protected Server. Each packet is 70-bytes in length. 

hping3 --icmp -C 3 -K 3 -i u1 10.1.1.100

 

For the full flood test, this string generated 165 Mbps / 295,000 PPS from the Attacking Machine

hping3 --icmp -C 3 -K 3 --flood 10.1.1.100

 

The Results

 

Screen Shot 2016-11-26 at 1.04.11 PM.png

 

At first glance, these results may surprise some of you. Why does Palo Alto suffer significantly more than others during the Low Volume attack?

The answer is relatively simple. Remember the 5-Tuple we discussed earlier? Both Juniper SRX and Check Point analyze the inner header of the ICMP packet before making a forwarding decision, while Palo Alto does not. As a result, both the Check Point and the SRX Firewalls drop the BlackNurse packets, while Palo Alto is forced to consume resources forwarding the traffic to the internal server. 


Check Point looks to see if the ICMP destination matches the inner source IP defined within the packet.
You can see that here from a kernel debug:

;[cpu_1];[fw4_0];fw_log_drop_ex: Packet proto=1 1.1.1.50:771 -> 10.1.1.100:3379 dropped by fw_icmp_stateless_checks Reason: ICMP destination does not match the source of the internal packet;

 

Juniper SRX take this one step further by analyzing whether or not there is an established session for the embedded 5-Tuple within the packet. Using this technique, the SRX ensures that the traffic is matching a legitimate session and not being spoofed by an attacker. 
From flow traceoptions:

Nov 24 22:35:26 22:35:23.626086:CID-0:THREAD_ID-01:RT:  ge-0/0/0.0:1.1.1.50->10.1.1.100, icmp, (3/3)
Nov 24 22:35:26 22:35:23.626090:CID-0:THREAD_ID-01:RT:  packet dropped, no session found for embedded icmp pak

Check Point's solution went unresponsive under high-load due to their lack of resource separation between management and forwarding planes. The separation of Routing-Engine and Packet-Forwarding Engine is a fundamental tenet of Juniper platforms. We strongly believe that at no point should the processing of transit traffic affect the overall stability and manageability of the device in question. 

 

vSRX included with both mitigation optionsvSRX included with both mitigation options

These results are more in line with what we would expect to see from top-tier vendors, with Palo Alto being the notable exception once again. It appears as though their software is not as efficient as others when dealing with high packet-rate scenarios. My current assumption is that their poor performance is due to an inefficient softIRQ handler which may not be exposed in their physical solutions. Because Palo Alto implements anti-DoS mechanisms in hardware when available, just like Juniper, it is possible these software routines have not yet been optimized to function effectively when said hardware is absent. 

 

Check Point's anti-DoS mechanism was shown to be very similar to the SRX's ICMP Screen feature in terms of raw performance. While not an easy feature to deploy correctly (unlike Screens), it is quite effective. For those interested in seeing how complex managing these rules can be, please see here for some examples. Unfortunately for Check Point, they also lack the ability to push these rules into an ASIC or FPGA to handle these attacks at large scale. They are inherently limited by the performance of the x86 architecture.

 

On that note, for those curious as to why Packet-Filters are the most efficient solution at handling these types of attacks in software-based platforms, consult the SRX's flow diagram to see when packet-filters take effect on a given flow:

 

Screen Shot 2016-11-28 at 12.08.25 PM.png

As you can see above, both Policers and Packet-Filters take effect before a session-lookup occurs, saving precious time and system resources when dealing with large DDoS attacks. The added benefit with our physical SRX is that flood-based Screens are pushed down into our ASIC-based Network Processors (NP's) to handle attacks like this at line-rate (up to 240Gbps per line-card) with no impact to system resources. 

 

Now that you have seen the results, this is the baseline configuration which was used on the vSRX:

set version 15.1X49-D60.7
set system host-name vSRX
set system root-authentication encrypted-password "$5$CM/Zzr55$kET7anrM32v0KxpwOVyPJcT3HTALY9pNcdx.Rw8K6U3"
set system services ssh
set security policies global policy Permit_Any match source-address any
set security policies global policy Permit_Any match destination-address any
set security policies global policy Permit_Any match application any
set security policies global policy Permit_Any then permit
set security zones security-zone Outside interfaces ge-0/0/0.0
set security zones security-zone Inside interfaces ge-0/0/1.0
set interfaces ge-0/0/0 unit 0 family inet address 1.1.1.1/24
set interfaces ge-0/0/1 unit 0 family inet address 10.1.1.1/24
set interfaces fxp0 unit 0 family inet address 192.168.0.5/24

 To enable Screens on the Outside Zone and to enable ICMP Flood protection, these two lines were added:

set security screen ids-option BlackNurse icmp flood threshold 100
set security zones security-zone Outside screen BlackNurse

While Policers would also be highly effective for this purpose, during this test we used a simple Packet-Filter to discard all ICMP Type-3 Code-3 packets ingressing into our Outside interface:

set firewall family inet filter BlackNurse term 1 from protocol icmp
set firewall family inet filter BlackNurse term 1 from icmp-type unreachable
set firewall family inet filter BlackNurse term 1 from icmp-code port-unreachable
set firewall family inet filter BlackNurse term 1 then discard
set firewall family inet filter BlackNurse term 2 then accept

set interfaces ge-0/0/0 unit 0 family inet filter input BlackNurse

 

Appendix

This section will include Answers to common questions not addressed above. 

 

Q) Will you be sharing the device configurations used during testing?

You can find the vSRX config above, but Check Point and Palo Alto configurations are included below for posterity. The DoS-prevention mechanisms for each platform were configured to take action after 100 packets-per-second of ICMP flooding. 

 

Q) How was CPU utilization determined?

    • Juniper: 'show security monitoring'

    • Check Point: Utilizing 'ps aux' and 'top', tracking the assigned fw_worker core. Reaching 100% on this core creates a DoS scenario as seen in Test 1.

    • Palo Alto: There was some discrepancy with regards to dataplane CPU reporting within the VM. Generally, one would run 'show running resource-monitor' to display the current utilization for the dataplane. However, this was not reporting figures in line with device behaviour. Metrics from the 'pan_task' process proved to be much more reliable when taken from 'show system resource follow'.

      Example: The PA-VM is suffering from 100% packet loss during Test 1 but only reporting 50% dataplane utilization, whereas pan_task is reporting 100%+.

show running resource-monitorshow running resource-monitorshow system resources followshow system resources follow

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Comments
by MiteshDalal
on ‎12-02-2016 03:50 PM

Extremely informative and insightful . Good job Craig!

by
on ‎12-08-2016 03:44 PM
Good blog article but I would like to see the same comparisons made with similar hardware based platforms, i.e. CP 21800, SRX3600/4000, PA-5060 for example for added credence.
by a2X4_TXn
on ‎01-03-2017 05:56 PM

Palo Alto Networks updated their recommendation the day after this article was edited, Dec 3 and Dec 2 relatively. It would be nice to see the results with the new recommendation. And I second perry.young, if anyone has the equipment.

Announcements
Juniper Networks Technical Books
Labels
About the Author
  • Amy James is Product Marketing lead for Security at Juniper Networks. She brings her knowledge of cyber security from companies like FireEye, Cisco and Cloudmark with deep roots in technology storytelling.
  • Andrew is a Juniper Distinguished Engineer responsible for the architecture of Juniper's network management user interfaces.
  • Asher Langton is a senior software engineer and malware researcher on Juniper's Sky ATP team.
  • Aviram Zrahia is a consulting engineer at Juniper Networks and an industry researcher of cyberspace. He holds a CISSP and GCIH certifications, as well as a bachelor's degree in computer science and MBA in management of technology, innovation, and entrepreneurship. He is also a research fellow in the Blavatnik Interdisciplinary Cyber Research Center (ICRC) at Tel Aviv University, currently focusing on the domain of threat intelligence sharing.
  • Bill is the Director of Federal Certifications and Policy at Juniper Networks. In this role, Bill focuses on several areas unique to the needs of Federal Government customers, including product certifications, IPv6, and security. Bill came to Juniper Networks in January 2008 after more than 20 years in the IT community working with commercial enterprise customers, service providers, and the US Federal Government. Bill started his career as an engineering officer in the US Air Force after graduating with a Bachelor of Aerospace Engineering from the Georgia Institute of Technology. Bill has an MBA from the Wharton School at the University of Pennsylvania.
  • Brad Minnis, CPP is the Senior Director of Corporate Environmental, Health, Safety & Security for Juniper Networks, Inc. based in Sunnyvale, CA, where he is responsible for strategic design, implementation and management of the company’s security, safety, environment, crisis management and business continuity functions. He also leads the company’s efforts in corporate citizenship and sustainability, and manages the Corporation’s government-related security programs. Mr. Minnis has over 30 years experience in the Silicon Valley and has managed EHSS operations for a number of high tech companies, including Juniper Networks, 3Com Corporation, and National Semiconductor Corporation. Mr. Minnis’ specialties include security management, supply chain and product integrity, anti-counterfeit, occupational health and safety and crisis management. In his role as Cyber Incident Response Team Leader for Juniper, Mr. Minnis has managed numerous high impact cyber-related incidents and cross-functional responses. Mr. Minnis served for ten years in the United States Navy and has served in leadership positions the International Security Management Association (ISMA) and ASIS International, serving as Chairman of the San Francisco Chapter in 2003. He has also co-written several publications on software integrity assurance and supply chain security with organizations such as SAFECode. Mr. Minnis is certified as a Protection Professional by the Professional Certification Board of ASIS International and attended the University of Connecticut, where he received two certificates in Environmental, Health and Safety
  • Craig Dods is the Chief Architect for Security within Juniper Networks' Strategic Verticals. He currently maintains multiple top-level industry certifications including his JNCIE-SEC, holds multiple networking and security-related patents, as well as having disclosed multiple critical-level CVE's in a responsible manner. Prior to joining Juniper, Craig served as IBM's Managed Security Services' Chief Security Architect, and held previous security roles at Check Point Software Technologies and Nokia.
  • François Prowse is a Senior Systems Engineer for Juniper Networks, based in Brisbane Australia. Francois joined Juniper in 2006 as part of the New Zealand SE team, subsequently relocating to Australia. Prior to Juniper, Francois worked for four years at Alcatel in both operational and architectural roles, being jointly responsible for the construction of New Zealands' largest MPLS core network. Prior to Alcatel, Francois worked at UUnet, focusing on core network expansion in Europe. In all previous roles JUNOS has been the driving factor behind day to day operations, providing him with over 8 years of operational experience. Francois is a Juniper Networks Certified Internet Expert (JNCIE #144) which he obtained prior to joining Juniper Networks.
  • Greg Sidebottom is a Senior Engineering Manager in the Identity and Policy Management business unit at Juniper Networks. Greg has spent the last decade plus conceptualizing, architecting, designing, and leading the implementation of Juniper's SDX and SRC families of policy based service management applications. Previous to this, Greg held positions in the software and networking industries at Siemens, Cognos, Nortel, GTE labs subsidiary MPR Teltech, and the Alberta Research Council. Greg is an author of eight invention disclosures resulting in two patents issued and three pending. Greg holds a B.Sc. in Computer Science for the University of Calgary and an M.Sc. and Ph.D. in Computing Science from Simon Fraser University.
  • Jennifer Blatnik is vice president of cloud, security and enterprise portfolio marketing at Juniper Networks with focus on enterprise deployments of security, routing, switching, and SDN products, as well as cloud solutions. She has more than 20 years of experience helping enterprises solve network security challenges. Before joining Juniper, Jennifer served multiple roles at Cisco Systems, Inc., including directing product management for security technologies aimed at small to medium enterprises, as well as supporting managed services, cloud service architectures and go-to-market strategies. She holds a B.A. in Computer Science from University of California, Berkeley.
  • Jim Kelly, Senior Product Line Manager – CTP Products Juniper Networks. Jim Kelly is the senior product line manager for the CTP products where he is responsible for the CTP product direction, marketing and circuit emulation applications within Juniper Networks. Mr. Kelly has more than 28 years of experience in the networking industry in technical roles, sales, marketing, and product management positions. He started his career in the United States Air Force. He has worked for Wang, Digital Telecom Systems, American Airlines, Network Equipment Technologies, Carrier Access, and Nortel Networks. He started Juniper Networks federal DoD sales in July 2000 and joined Juniper Networks again in October 2005 through the acquisition of Acorn Packet Solutions where he was the director of sales and marketing.
  • I have been in the networking industry for over 35 years: PBXs, SNA, Muxes, ATM, routers, switches, optical - I've seen it all. Twelve years in the US, over 25 in Europe, at companies like AT&T, IBM, Bay Networks, Nortel Networks and Dimension Data. Since 2007 I have been at Juniper, focusing on solutions and services: solving business problems via products and projects. Our market is characterized by amazing technological innovations, but technology is no use if you cannot get it to work and keep it working. That is why services are so exciting: this is where the technology moves out of the glossy brochures and into the real world! Follow me on Twitter: @JoeAtJuniper For more about me, go to my LinkedIn profile: http://fr.linkedin.com/pub/joe-robertson/0/4a/34a
  • Kevin Walker is the Security Chief Technology and Strategy Officer for Juniper’s Development and Innovation (JDI) organization. He is responsible for driving the security strategy both internally within Juniper, and externally with investors, partners, influencers, and customers. He provides the guidance required for JDI to conceive, develop and create momentum for industry-leading security solutions. Working closely with the Security Engineering team, Walker identifies the opportunities for improved security, growth, and innovation to deliver the scalable, reliable, and compliant security architecture needed in today’s security landscape. Before joining Juniper, Walker was VP and Assistant Chief Information Security Officer (CISO) at Walmart.com. He has served as a Chief Information Security Officer (CISO), Chief Security Strategist and Director of Information Security across a number of notable companies including Intuit, Cisco, Symantec and VERITAS Software. With over twenty-five years in various computer science and information technology disciplines, focusing on enterprise applications, network design, and information security, Walker possesses research and engineering expertise across of range of technologies including networking protocols, securing applications at the atomic level, cryptography, and speech biometrics.
  • Laurence is passionate about technology, particularly cyber security. His depth and breadth of knowledge of the dynamic security landscape is a result of over twenty years’ experience in cyber security. He understands the security concerns businesses face today and can bring insight to the challenges they will face tomorrow. Laurence joined Juniper Networks in 2016 and is our senior security specialist in EMEA. Security throughout the network is a key area where Juniper Networks can help as business moves to the cloud and undertakes the challenge of digital transformation.
  • Security Life timer, who has been described as a true IT security ‘guru’. It is certainly apt: his knowledge and expertise developed over the course of more than 20 years in IT have helped many customers implement a security strategy that not only safeguards their business and information, but enables Digital Transformation. A noted public speaker on security issues, Lee’s passion and style stand out in the sometimes staid world of network security. Prior to joining Juniper Networks, Lee held a number of business and technical roles at Dr Solomon’s, McAfee, Hewlett Packard, Nokia Siemens Networks and Citrix. Lee leads the Juniper Networks security business across Europe, Middle East and Africa. In this role, Lee is responsible for the company’s commercial development in the field.
  • Michel Tepper is a Juniper consultant and instructor working for Westcon Security in the Netherlands. He started working in ICT in 1987. Michel is also is a Juniper Ambassador. Currently he holds three Junos Professional certifications and a number of specialist and associate certifications on non-Junos tracks. Michel is an active member of J-Net and juniperforum.com, where he uses the nickname screenie referring to the ScreenOS with which he started his Juniper Journey.
  • 30 Years in Book Publishing, 20 years in Technical Book Publishing, including Apple Developer Press, Adobe Press, Nokia Developer Books, Palm Books, and since 2001, almost 10 years as consulting editor/editor in chief for Juniper Networks Book. Joined the company and started the Day One book line and in 2011, the new This Week book line.
  • Solutions Marketing Sr Manager
  • Scott is the Director of Product Marketing for Mobile Security at Juniper Networks. In his 20+ years in high tech, Scott has worked on Mobile and Endpoint Security, Network Security, IPS, Managed Services, Network Infrastructure, Co-location, Microprocessor Architecture, Unix Servers and Network Adapters. He has held leadership roles at Check Point, McAfee, Symantec, Exodus Communications, Cable & Wireless, Savvis, and HP.
  • Sherry Ryan is IT Vice President and CISO of Juniper Networks. Previously, Sherry held similar positions at Blue Shield of California, Hewlett-Packard, Safeway and Levi Strauss where she established and led their information security programs. Sherry holds the Certified Information Security Manager (CISM) certification from ISACA and the Certified Information Systems Security Professional (CISSP) certification from ISC2. She is a member of the High Tech Crime Investigation Association (HTCIA) and the Information Systems Security Association (ISSA). Sherry has a bachelor's degree in Business Administration from the University of Redlands, and earned her MBA from the College of Notre Dame.
About Security Now

Discussing a wide range of topics impacting enterprises and
data center security.

Subscribe RSS Icon


Our Bloggers

Kevin Walker
Vice President
Security CTSO, Engineering

Profile | Subscribe

Ritesh Agrawal
Director
Software Engineering

Profile | Subscribe

Scott Emo
Director
Product Marketing

Profile | Subscribe

Bill Shelton
Director Field Sales

Profile | Subscribe