Archive
Juniper Employee , Juniper Employee Juniper Employee
Archive
DevOps against DDoS I: Programming BGP FlowSpec with Junos PyEz
Aug 13, 2018

Introduction

 

Distributed Denial of Service (DDoS) attacks are increasingly important in the networking industry. Their sophisticated magnitude, crafted impact and widely spread side-effects beyond a specific objective are leading to unexpected and severe economical consequences for both enterprises and service providers.

 

As proof of evidence, DDoS attacks and mitigation discussions were covered in 2 presentations in latest NANOG63 forum and diverse APRICOT 2015 Lightning Talks and videos. Probably, the most relevant and popular common topic.

 

In previous articles at #TheRoutingChurn, such as IPv6 destination remote triggered blackholing with the 6PE model - Part I and IPv6 destination remote triggered blackholing with the 6PE model - Part II, we have briefly covered Remotely Triggered Blackhole (RTBH) options for source and destination addresses, originally defined under [RFC3882] and [RFC5635]. I included there different references to basic RTBH techniques, mostly using a local beacon route pointing to a discard next hop, that would be used for recursive resolution to mitigate the attack (either for destination or source addresses).

 

However, RTBH mechanisms still lacked in more granular mitigation options on a per flow basis and had side-effects on attacked hosts, as all traffic is nuked upon recursive resolution to this discard next hop.

 

BGP Flow Specification has evolved through different draft versions until becoming [RFC5575], and provides further benefits, such as:

 

  • Centralized SDN orchestration to program distributed filters at the network perimeter
  • Efficient point-to-multipoint distribution of consistent control plane information to define crafted firewall filters (or access lists, depending on the vendor parlance)
  • Utilization of BGP as SDN protocol (well-known protocol, reusing mesh topologies, maybe just adding an extra family to existing sessions), with support to all standard BGP attributes, mechanisms and interaction with routing policies
  • Inherent inter-domain distribution capabilities, eventually using existing external BGP sessions
  • Tight and coherent integration with unicast routing

In fact, all these DDoS thwarting technologies are not exclusive but actually complimentary. Depending on the attack magnitude, on distribution across sources and if flows are properly identified, an operator can combine destination-RTBH, source-RTBH and BGP Flow Specification (FlowSpec) mechanisms to neutralize the attack effects in the network.

 

With this post and its follow-up, I am just covering a simple BGP FlowSpec use case with basic Junos OS and PyEz tidbits in our usual Junosphere topology with an SDN-based approach. Junosphere topology and scripts are attached to each post for your perusal.

 

By no means it can be taken as any kind of professional service or solution development to mitigate DDoS attacks. I just intend to provide a very simplistic gem to illustrate that a bunch of Python lines suffice to deploy an SDN-based BGP FlowSpec mechanism and can become baseline for a proper solution, provided that DDoS is a trending topic in our industry.

 

If you have any thoughts, feedback, learned lessons or improvement proposals, please be our guest and drop your comments!

 

 

About BGP Flow Specification

 

After several draft versions and lengthy discussions, BGP Flow Specification reached final [RFC5575] status with consensus and support across several vendors.

 

Our configuration guide provides an overview under http://www.juniper.net/techpubs/en_US/junos14.2/topics/example/routing-bgp-flow-specification-routes.htm..., but I also found that David Roy has provided an excellent summary for BGP Flow Specification options and Junos OS configuration knobs and results at his article under http://www.junosandme.net/article-junos-and-bgp-flowspec-119193150.html. Other useful references are https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks/, https://njetwork.wordpress.com/2013/04/30/mitigating-ddos-attacks-with-bgp-flow-specification/ and http://www.slideshare.net/jgrahamc/cloud-flarejgc-secure2013andvirusbulletin2013.

 

My colleague Justin Ryburn recently delivered anoter DDoS Mitigation review presentation at NANOG63 (also cross-refered from our http://forums.juniper.net/t5/Security-Now/DDoS-Mitigation-using-BGP-Flowspec/ba-p/268609 J-net blog), that summarizes rationale and tidbits for BGP FlowSpec, including multi-vendor references (not only Junos OS) and a very interesting final survey.

 

All these references and [RFC5575] provide a comprehensive overview of BGP Flow Specification details and techniques and I will follow up from there.

 

Flow route validation options

 

Default [RFC5575] validation rules are defined in Section 6 and state that the receiving BGP-enabled device validates a flow route if it passes the following criteria:

 

   a) The originator of the flow specification matches the originator of
      the best-match unicast route for the destination prefix embedded
      in the flow specification.

   b) There are no more specific unicast routes, when compared with the
      flow destination prefix, that have been received from a different
      neighboring AS than the best-match unicast route, which has been
      determined in step a).

 

However, some implementations are based on a centralized route distribution point (such as route reflector, as in these articles) and it may be required to bypass these requirements to effectively enable a flow route at the perimeter devices.

 

Junos OS offers the no-validate configuration option to attach such bypass policies to the corresponding BGP family, using standard routing policy engine resources where, for instance, communities can be mainly used to identify flow routes and accept them from a given BGP peer. This option yields the substrate for a route reflector (RR) based deployment where the flow route originator attribute value will mostly never match the unicast NLRI originator for the flow route embedded destination prefix.

 

Flow route filter term order

 

[RFC5575] also defines a certain predefined order of rules or terms to be chosen out of flow routes and to be compiled into the same common filter.

 

In Junos OS, multiple flow routes in a routing instance are represented as multiple ordered terms in an internal and orthogonal input Forwarding Table Filter (FTF). As per standard definition, the term evaluation order is independent of the order of the flow route configuration, or the flow route receiving order from a BGP neighbor, but rather computed internally based on the prefix bits.

 

Previous [RFC5575] draft versions (up to revision 5) defined another term ordering algorithm and for that reason, Junos OS suppors two term-order types: legacy (default, to be backwards compatible) and standard (considered a best practice, compliant with the post-version 5 algorithm definition and focused on a logical most specific route matching). Unless required otherwise, it is recommended here to change the Junos OS default and enable term-order standard flow route generation.

 

How does this feature get activated in Junos OS?

 

Implementing BGP Flow Specification requires interaction with a new routing table per instance and a new BGP family. To keep it short, I am going to focus in these articles just on the main inetflow.0 table and BGP AFI/SAFI 1/133 for the default instance, but the same concepts could be similarly applied inside VRF instances and BGP AFI/SAFI 1/134 for attack mitigation inside MPLS L3 VPNs.

 

  • Inet flow BGP family

It is first required to adequately tune the BGP family activation to enable peerings for flow route dissemination in the corresponding mesh:

[edit protocols bgp]
group flow {
    [...]
    local-address [...];
    family inet {
        flow;
    }
    neighbor [...];
}

 

  • BGP flow route identification via communities

BGP flow route actions are in point of fact encoded via extended communities, as explained in [RFC5575] Section 7 and registered as IANA extended communities.

 

Apart from that, I would encourage route tagging with simple communities (even more than one), for service identification purposes and because default validation bypass policies shall be based on them.

 

Such communities can be added in the local flow route definition with Junos OS or alternatively, with BGP Flow Specification export policies redistributing locally defined flow routes, for example:

 

[edit policy-options policy-statement tag-flow-routes]
term local-flow-routes {
    from rib inetflow.0;
    then {
        community add 65000:667;
        accept;
    }
}

 

  • BGP flow route evaluation order

As previously outlined, the filter term order when evaluating BGP flow routes has changed during the evolution of this standard.

 

In order to be fully compliant with the latest [RFC5575] Section 5 process definition (now adopted as best practice), it would be required to change the Junos OS default legacy behavior:

 

[edit routing-options flow]              
term-order standard;

 

  • BGP flow route validation policy adjustment

Default latest [RFC5575] Section 6 validation options may not fit the scenario too, as described above.

 

Specific no-validate policies can be defined and attached at the client side. We may use a specific simple community for such service identification:

 

[edit policy-options policy-statement allow-tagged-flow-routes]
  term flow-routes-from-R6 {
      from {
          protocol bgp;
          community 65000:667;
      }
      then accept;
  }

 

[edit protocols bgp group iBGP-inet-flow]
   type internal;
   family inet {
     flow {
        no-validate allow-tagged-flow-routes;
      }
    }
[…]

These basic steps allow BGP enablement as SDN protocol for flow route dissemination and may set a network-wide baseline for a proper solution. It is just now a question of defining the flow routes and rely on this BGP infrastructure.

 

  • Local flow route definition

Junos OS allows a local static definition for a flow route:

 

[edit routing-options flow]              
route foo {
    match {
        destination 198.51.100.1/32;
        protocol udp;
        destination-port 123;
    }
    then discard;
}
[...]

At the moment it is defined, it appears in the corresponding local inetflow.0 RIB:

 

root@R6> show route table inetflow.0

inetflow.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

198.51.100.1,*,proto=17,dstport=123/term:1            
                   *[Flow/5] 00:02:10
                      Fictitious

Based on a BGP export policy, such as the one illustrated above, this eligible active flow route is picked up and redistributed to peers, where it will be implemented as a filter term.

 

This local definition can be enforced in a scrubbing center, via an external system,... or we can use a sufficiently meshed and externally orchestrated central iBGP peer (such as a route reflector) as in the next use case with Junosphere.

 

Examples with Junosphere and Junos PyEz: Programming BGP FlowSpec


Considering feedback and survey results from Justin Ryburn's DDoS Mitigation talk at NANOG63 (see video at http://forums.juniper.net/t5/Security-Now/DDoS-Mitigation-using-BGP-Flowspec/ba-p/268609), it seems as if many operators are reluctant to deploy inter-AS BGP Flow Specification services.

Despite the inherent BGP route propagation nature beyond a single AS, enabling external BGP peers or customers populating BGP Flow Specification routes up- or downstream is perceived as a risky operation.

This seems to be mainly due to the lack of control options at the network perimeter and the possibility that an external entity could advertise a wrongly configured, incorrectly defined or even intentionally misbehaving flow route that would derive into a widely distributed filter causing major harm.

Even though Junos OS supports source-address-filter as a policy match option to provide further granular control to external BGP Flow Specification peerings, this is a very understandable concern, especially at the scale of hundreds or thousands of flow routes.

A preferred deployment option seems to be based on a centralized SDN controller view within an AS (or across ASs handled by the same organization) and a centralized scrubbing center or mitigation orchestrator. This proposition is going to inspire this simple use case and we are going to depict this with our usual Junosphere topology:

flow-spec-route-advertisement.jpg

For this extremely simplistic representation of concepts, let's consider our usual #TheRoutingChurn topological division in 2 Autonomous Systems: AS65000 including R1 under attack (198.51.100.1) and R10 and R11 with redundant external BGP peerings to AS65001 where the attack is being launched from in a distributed fashion.

Traditional inet unicast peering in this example ensures end-to-end connectivity between the 198.51.100.0/24 range from AS65000 and 203.0.113.0/24 range from AS65001.

 

root@R10> show route protocol bgp table inet.0 

inet.0: 39 destinations, 40 routes (39 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

198.51.100.0/24    *[BGP/170] 2w4d 05:13:18, localpref 100, from 192.168.255.6
                      AS path: I, validation-state: unverified
                    > to 192.168.13.1 via ge-0/0/0.0, Push 300560
                    [BGP/170] 2w4d 05:13:18, localpref 100, from 192.168.255.12
                      AS path: I, validation-state: unverified
                    > to 192.168.13.1 via ge-0/0/0.0, Push 300560
203.0.113.0/25     *[BGP/170] 2w5d 01:45:38, localpref 100
                      AS path: 65001 I, validation-state: unverified
                    > to 192.168.15.2 via ge-0/0/2.0
203.0.113.128/25   *[BGP/170] 2w5d 01:45:38, localpref 100
                      AS path: 65001 I, validation-state: unverified
                    > to 192.168.15.2 via ge-0/0/2.0

 

And to simply simulate distributed NTP packets from different sources, R1 (198.51.100.1) will be set as NTP server for all AS65001 systems:

 

root@R24> show ntp associations 
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*198.51.100.1    172.17.28.5      2 -  167  256  377    9.922   -3.323   0.62

The same fundament applies to DNS, setting R1 (198.51.100.1) as main DNS server across AS65001 in this Junosphere topology for widely spread DNS packets from several sources.

In this scenario, R6 as internal AS65000 route reflector and R10 and R11 as AS65000 border routers (ASBRs) and route reflector clients will be enabled not only with the inet unicast, but also with the inet flow BGP family:

 

[edit protocols bgp]
group flow {
    [...]
    local-address [...];
    family inet {
        flow;
    }
    neighbor [...];
}

And they will follow all previously described baseline configuration principles too: using a simple community for flow route identification purposes from R6, adjusting term order evaluation and bypassing default [RFC5575] Section 6 validation rules.

Last but not least, systems where Junos PyEz interaction is needed (R6 first, R10 and R11 later in the next post), require Netconf services to be enabled, in this case over SSH:

 

root@R6> show configuration system services 
ftp;
ssh;
telnet;
netconf {
    ssh;
}

This setup may suffice to illustrate this concept from a functional point of view.

BGP Flow Specification as SDN application

 

At this stage, R6 has become the SDN controller for BGP Flow Specification rules inside AS65000. Flow routes can get statically provisioned in R6 and BGP is the adequate distribution vehicle for them across the Autonomous System, as SDN protocol of choice.

 

Now, instead of locally programming flow routes in R6 via traditional Junos OS CLI, this simple use case aims at leveraging Junos PyEz for a Junos OS agnostic orchestrator. This orchestrator will use embedded Netconf RPCs in Junos PyEz to program flow routes in R6 (and monitor their deployment in the follow-up post). BGP will do the rest:

 

flow-spec-route-advertisement-II.jpg

 

To sum up, this example will be therefore based on these general action items: 

 

  • Local flow route creation, removal and activation in R6 via Junos PyEz from orchestrator
  • Automatic flow route activation and withdrawal with BGP

 

Creating BGP Flow Specification routes with Junos PyEz

 

As a general Junos PyEz resource, I found our current http://techwiki.juniper.net/Automation_Scripting/Junos_OS_PyEZ wiki to be a very useful and practical site.

 

The following example will be using main principles and topics described in that wiki, together with interaction with YAML Ain't Markup Language to incorporate variable values by means of a Jinja2 template for Python, in the following fashion:

 

  • Invoke a single Python file at the orchestrator to create flow routes based on a Jinja2 template and a YAML file that includes specific flow route details and values
  • Change YAML file variable values and invoke script again to create new flow routes

 

python-set-flow-route.jpg

and this can be simply achieved by invoking the Python script from the orchestrator (flow route printed to standard output here as well for the sake of argument):

 

user scripts $ python set-flow-routes.py
# Flow-route configure via vRR
routing-options {
    flow {
        route foo {              
            match {                     
                destination 198.51.100.1/32;
                destination-port 123;
                protocol udp;               
            }                           
            then {
                discard;
        }
        }                               
    }                                   
}

 

Basic Python script using PyEz to connect, lock, load, commit and unlock configuration

 

The basic Junos PyEz guidelines to be used to connect, lock, load, commit and unlock the configuration are explained under:

 

 

# Open netconf connection with RR
dev = Device(host='r6', user='juniper', password='Clouds')
dev.open()
# Bind and lock configuration and load it
dev.bind(cfg=Config)
dev.cfg.lock()
dev.cfg.load(template_path='set-flow-route.j2', template_vars=data, format='text', merge=True)
# Commit and unlock
dev.cfg.commit()
dev.cfg.unlock()
# Close netconf connection
dev.close()

Note that for simplicity purposes, I am not handling exceptions or tuning an RPC connectivity timeout in this example. But for any real-life deployment, it is highly recommended, among others, to adjust connection timeout values and capture any possible exceptions, from connection to commit errors, using the well-known try and except Python clauses and leveraging our specific Junos PyEz exceptions, as explained under https://techwiki.juniper.net/Automation_Scripting/010_Getting_Started_and_Reference/Junos_PyEZ/Excep...

 

Also, when instantiating the Device class in this example, I am including host='r6' and assuming then local name resolution. If you test this in Junosphere, you should either address this locally as well, or replace the value with the management IPv4 address that the system assigns to the corresponding Virtual Machine. This applies to all scripts in these both articles.

 

Jinja2 template to create the flow route


Jinja2 is a Python templating engine that can be used in this case to create the Junos OS configuration skeleton that would be fed with variable values.

 

By creating comprehensive and reusable configuration templates with Jinja2, we can effectively separate the values from the scheme and provide a more programmatic and reusable configuration approach for flow routes. This provides further scaling and avoids the need for specifically different configuration snippets, just reuse the same micro-framework with different values for each route!

 

Jinja2 has become a de facto standard for template rendering with Python. Jeremy Schulman has published several Jinja2 tutorials explaining diverse aspects from this template language. From Jinja2 - 101 - Intro to Jinja2 template building  to Jinja2 - 105 - Dealing with Data, there are multiple examples and use cases (go through all other Vimeo sessions in between or check under his Packet Pushers Jinja2 podcast) to quickly understand how Jinja2 works and how it can be quickly and easily leveraged to build up a Junos OS-like config.

 

As good summary for the Python, Junos PyEz, Jinja2 and YAML interaction, this Junos PyEZ - YAML, Jinja2, Template building, configuration deployment, Oh My! tutorial provides a 15 minute wrap-up to understand how each piece fits in the puzzle. As you may see there, Jinja2 templates can be generally constructed in Junos OS like format (in traditional curly braces in this example, as in the article attached script).

 

In our case, the definition for a given flow route may include a parameter or not. Not all flow routes may need to specify a destination or a source address etc. Our Jinja2 template should be prepared for that eventual condition and consider at least an option for a given variable to exist or not. This can be achieved in Jinja2, just by using {%- if ... is defined and ... != None %} conditional directives:

 

# Jinja2 flow route template
routing-options {
    flow {
        route {{flow_route_name}} {              
            match {                     
                destination {{destination_address_and_mask}};
                {%- if destination_port is defined and destination_port !=None %}
                destination-port {{ destination_port }};
                {%- endif %}
                {%- if dscp is defined and dscp !=None %}
                dscp {{dscp}};
                {%- endif %}
                [...]              
            }                           
            then {
                {%- if accept is defined and accept !=None %}
                {{accept}};
                {%- endif %}
                [..]
        }
        }                               
    }                                   
}

These variable existence checks are explained by Jeremy Schulman in detail at Jinja2 - 104 - Using "if / then / else" conditional directives.

 

YAML file with the new flow route variable values

 

Being a human friendly data serialization standard, the YAML Ain't Markup Language can be used to provide a given value to each variable used in the Jinja2 template in this provisioning case. YAML provides the option to express structured data in a human-readable format (friendlier than XML, for instance).

 

Because we have split the scheme (Jinja2 template) from the variable values (inside the YAML file), it would be just needed to feed this YAML file with different settings any time the Python script is invoked, and Junos PyEz would take care to render these data in every execution whenever loading the configuration for a new flow route:

 

---
# $Id$
# YAML file covering all possible variables

# Name of flow route
flow_route_name: foo
# Destination prefix and mask in format: A.B.C.D/Z
destination_address_and_mask: 198.51.100.1/32
# DSCP in decimal format
dscp:
# Destination port as alias or in decimal format
destination_port: 53
[...]              

Note that not all these variables need to be assigned with concrete details, our Jinja2 template conditional directives will just take care of enforcing the configuration whenever a real value exists.

 

And just with these 3 basic and simple building blocks, we can quickly build up a flow route creation scheme. But once the attack is over, how can we possibly delete the specific flow route?

 

Deleting BGP Flow Specification routes with Junos PyEz

 

Deleting a route can be achieved in a parallel fashion, based on the same fundaments:

 

  • Invoke a single Python file at the orchestrator to delete flow routes based on Jinja2 template and YAML file that includes the specific flow route name at R6
  • Change YAML file variable values and invoke script again to withdraw existing flow routes

 

python-delete-flow-route.jpg

 

 

and just by invoking a Python script from the orchestrator (route deletion snippet printed to standard output here to be clear as well):

 

user scripts $ python delete-flow-routes.py
routing-options {
    flow {
        delete:
        route foo;                 
    }                                   
}

 

The same device connectivity scheme can be reused here with Junos PyEz, it just becomes a question of loading a configuration patch that deletes an existing route instead of creating it.

As mentioned before, I am not capturing exceptions here or handling connectivity timeouts, but this should be considered in a more formal concept proposal. Among others, a relevant check here would be to try-except that the route to be deleted indeed exists beforehand in the route reflector configuration. Both creation and deletion can be combined in the same script too as another value proposition, just changing an argument to distinguish between them, for instance.

 

Jinja2 template to delete the flow route


Jinja2 can also be used here to create a basic template to identify the flow route to be deleted. This use case assumes that this removal is based on existing knowledge for the local flow route name at R6, so that this name is used as ultimate identifier for the route to be deleted (and withdrawn from the network, as a consequence):

 

routing-options {
    flow {
        delete:
        route {{flow_route_name}};   
    }                                   
}

Note that the delete keyword applied at the flow route top level determines the corresponding action to be applied in the configuration patch to be loaded. It is kept very simple just with the route name.

 

YAML file with the existing flow route name

 

For that reason, the YAML file for route removal becomes as simple as just including a variable for the local flow route name on R6 (where route advertisement is controlled):

 

---
# $Id$
# YAML file covering flow route name to delete

# Name of flow route
flow_route_name: foo

As an operator, you can use self-explanatory flow route names for identification purposes or add another recursion or interaction to pinpoint the correct route name based on other data, such as source or destination addresses.

 

Conclusions

 

In this first article, we have provided references, datapoints and descriptions for BGP Flow Specification [RFC5575] concepts, and we have seen how it can be leveraged as SDN WAN protocol, even providing transit services across networks, with a wide range of attributes and routing policy options, just using another family.

 

First, my intention has been to start depicting that just with a bunch of lines of code using Junos PyEz, eligible to be improved and complemented for a proper solution, and with existing Netconf services and a baseline BGP configuration, an SDN environment can be easily created, offering a provisioning API to a DDoS mitigation orchestrator just by filling the YAML file variables with values for every flow route to create or delete.

 

In the follow-up post, I will cover basic actions to monitor flow route existence and derived firewall filter term installation, so as to provide a surveillance tool-box to ascertain whereas a DDoS attack is still ongoing or has already finished, and end up with some final recommendations and conclusions.

 

In the meantime, if you have any questions, related experiences or comments, please drop them here or via #TheRoutingChurn!

 

Apr 22, 2015

Great post Gonzalo! Thank you!

May 7, 2015

Good stuff!

May 13, 2015
Juniper Employee

Great write up Gonzalo. This is gives an excellent view including one of a solution using Junos automation.

Aug 13, 2018
Xiomara

Wonderful article, Gonzalo! As a DevOps Consulting specialist at Clickittech, I can't stress how important it is to think of DevOps as a culture as opposed to a position. DevOps must be implemented across all levels of your organization instead of being relegated to a single position or department.

Feedback