Archive
Juniper Employee , Juniper Employee Juniper Employee
Archive
IPv6 destination remote triggered blackholing with the 6PE model - Part I
Apr 12, 2013

Some days after the largest publicly announced DDoS attack on the history of Internet so far, the recent Spamhaus vs Cyberbunker war has provided me the excuse to illustrate an effect when interacting the 6PE model with native IPv6 external peering: next-hop self is done per default when changing the context.

 

How are these concepts related? After these massive attacks, some folks have started to seriously evaluate again prevention mechanisms and wonder how DDoS mitigation could be applied to IPv6. The least sofisticated albeit most powerful mitigation technique with routing resources is the so-called destination Remote Triggered Blackhole (RTBH) originally defined under [RFC3882],

 

If you are to deploy this technique or source RTBH ([RFC5635]) for IPv6 with a 6PE model and Junos OS, you should be aware of the stage where the corresponding next hop needs to be overwritten. And this is because there is an inherent next-hop self rule when changing the lookup from a labeled to a non-labeled route, aligned with the context change in Junos OS.

 

I am using a sample Junosphere topology to illustrate this effect and provide some insight as for where and how this policy should be applied with a destination RTBH case. The same concept applies to source RTBH as well.

 

About destination Remote Triggered Blackholes

 

Remote Triggered Blackhole (RTBH) has been a widely used mitigation concept in service provider environments over the last years. Since the early 2000s, where first massive DDoS attacks started to occur frequently, destination-based RTBH techniques have been developed in different fashions and with multiple options by many colleagues in the field. After some real-life ocurrences in different ISPs, the original multi-vendor destination-based technique was documented some time later under [RFC3882], and this concept has evolved later with diverse ramifications and options.

 

It is not the intention to go through all RTBH tidbits in this post, there are multiple documented examples out there, but to sum up, all RTBH design variants are based on 3 major buidling blocks:

 

  • Local implementation of a discard route or prefix at the network perimeter (with per-incident, per-router-type or per-customer-type flavors)
  • BGP Policy implementation to map the corresponding next hop of affected routes to this discard route, so as to nuke offending traffic
  • More specific route distribution for affected destinations (next-hop lookup) or attacking sources (uRPF checks), depending on the mitigation technique, to distribute such blackholing rule across network(s)

 

Junos OS offers here multiple resources for Remote Triggered Blackhole (RTBH). I found that the NANOG32 preso: Options for Blackhole and Discard Routing was actually a very good old summary to recap all feasible issues and options (accounting, different communities or next hops etc.) with IPv4 destination-based RTBH under Junos OS.

 

The traditional destination-based RTBH concept has evolved in multiple fashions:

  • To offer RTBH as a service across different Autonomous Systems (check NANOG30 preso Customer-Triggered Real-Time Blackholes)
  • To also include source-based RTBH with a particular loose uRPF check implementation flavor (check [RFC5635])
  • To combine both source- and destination-based with BGP-based a-la-SDN flow dissemination techniques (check [RFC5575] and earlier FlowSpec drafts)
  • To directly provide DPI or traffic-cleaning products that may intercept traffic and nuke offending packets
  • etc.

I personally think this is also a very good application for Software Defined Networking (SDN) for the Service Provider Edge to implement on the fly mitigation rules at the network perimeter (For further details about SDN, please check Decoding SDN).

 

However, all these techniques are not exclusive but complimentary at different stages, and destination-based RTBH, despite its main drawback of neutralizing both ilegitimate and legitimate traffic towards a destination under attack, remains a quite popular network-wide vendor-agnostic mitigation artifact for massive DDoS attacks.

 

Destination-based RTBH is very appealing in the case of propagating rules beyond the own Autonomous System, because its routing-only implementation requires simple and automatic coordination between networks and allows service providers offering it as a service to their customers with very simple routing resources and low cost. This is the scenario replicated in the attached Junosphere topology.

 

A use case for next-hop rewrite policies in the 6PE to native IPv6 interaction: multi-AS IPv6 destination RTBH in the 6PE model

 

The same destination-based RTBH methodology can be extrapolated to an IPv6 environment, there are no specific IPv4 bits involved. Many folks have already implemented dual-stack RTBH (check RIPE65 INEX's Remotely Triggered Black Hole Tutorial) and RFC6666 already reserves an official IPv6 discard prefix for this specific mitigation purpose. Tools for IPv6 RTBH are available!

 

A common IPv6 destination-based RTBH scenario across Autonomous Systems may be based on the 6PE model inside a given AS but native IPv6 BGP peering among them. In Junos OS parlance, this requires changing from plain inet6 unicast routing context at the network edge to inet6 labeled unicast towards/from the core. Changing routing context for the adequate lookup requires in this case either label allocation in one direction (IPv6 explicit-null in this 6PE case) or label pop in the opposite direction, and a next-hop change.

 

Similarly to vpn route types, originated labeled routes use next-hop self per default. When changing from a non-labeled to a labeled milieu, an inherent next-hop self takes place at this edge to adjust the lookup context. Therefore, such an implicit next-hop rewrite becomes a consideration, when another explicit next-hop rewrite is envisioned, as in the multi-AS destination RTBH case.

 

Examples with Junosphere: IPv6 destination RTBH for 6PE

 

Let's analyze what happens when replicating a dual-AS IPv6 destination-based RTBH scenario using Junosphere, based on our well-known topology (Junosphere topology for this post can be found attached). The setup considers the 6PE model in each AS and native IPv6 peering between respective ASBRs:

 

6PE-rtbh-topology.jpg

For simplicity purposes, native IPv6 peering between ASBRs will occur over IPv4-mapped IPv6 addresses for the same outbound interfaces, i.e., as explained under http://www.juniper.net/techpubs/en_US/junos/topics/example/bgp-ipv6.html. For instance, these would be the relevant snippets for an arbitrary ASBR like R10:

 

juniper@R10> show configuration protocols bgp

log-updown;

vpn-apply-export;

[...]

group eBGP-inet {

    type external;

    local-address 192.168.15.1;

    family inet {

        unicast;

    }

    family inet-vpn {

        unicast;

    }

    family inet6 {

        unicast;

    }

    export export-own-space;

    peer-as 65001;

    neighbor 192.168.15.2;

}

 

juniper@R10> show configuration interfaces ge-0/0/3

description R10<-->R13;

unit 0 {

    family inet {

        address 192.168.15.1/30;

    }

    family iso;

    family inet6 {

        address ::ffff:192.168.15.1/126;

    }

    family mpls;

}

 

Note that there is a single eBGP group and peering session, and the inet6 physical address in the external interface corresponds to the IPv4-mapped IPv6 address with the respectively mapped netmask.

 

Automatic next-hop self at the Edge

 

As described above, note that in this edge environment, there is an automatic next-hop rewrite when propagating routes from labeled to non-labeled NLRIs and vice versa. Checking a simple iBGP group config without explicit next-hop self policy:

 

juniper@R10> show configuration protocols bgp

log-updown;

vpn-apply-export;

group iBGP-inet {

    type internal;

    local-address 192.168.255.10;

    family inet {

        unicast;

    }

    family inet-vpn {

        unicast;

    }

    family inet6 {

        labeled-unicast {

            explicit-null;

        }

    }

    export own-loopback-labeled;

    neighbor 192.168.255.6;

    neighbor 192.168.255.12;

}

[...]

 

And externally received IPv6 routes over the eBGP session...

 

juniper@R10> show route receive-protocol bgp 192.168.15.2 extensive

[...]

inet6.0: 17 destinations, 26 routes (17 active, 0 holddown, 0 hidden)

* 2001:db8:6500:1::/64 (2 entries, 1 announced)

     Accepted

     Nexthop: ::ffff:192.168.15.2

     AS path: 65001 I

     Communities: 65001:100

[...]

 

get their next-hop rewritten to self when populating it internally with an explicit-null label in the 6PE group:

 

juniper@R10> show route advertising-protocol bgp 192.168.255.6 extensive

 

inet6.0: 17 destinations, 26 routes (17 active, 0 holddown, 0 hidden)

[...]

 

* 2001:db8:6500:1::/64 (2 entries, 1 announced)

BGP group iBGP-inet type Internal

     Route Label: 2

     Nexthop: Self

     Flags: Nexthop Change

     Localpref: 100

     AS path: [65000] 65001 I

     Communities: 65001:100

 

End-to-end connectivity over the 6PE clouds and native IPv6 interface

 

To illustrate the topic, let's choose a representative end-to-end connectivity use case between remote routers R1 and R24, with respective bidirectional route advertisement:

6PE-rtbh-e2e-connectivity.jpg

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Connectivity is granted between R1 and R24 and their representative source adresses:

 

 

juniper@R1> ping 2001:db8:6500:1::1 source 2001:db8:6500::1 

PING6(56=40+8+8 bytes) 2001:db8:6500::1 --> 2001:db8:6500:1::1

16 bytes from 2001:db8:6500:1::1, icmp_seq=0 hlim=62 time=65.154 ms

16 bytes from 2001:db8:6500:1::1, icmp_seq=1 hlim=62 time=84.386 ms

16 bytes from 2001:db8:6500:1::1, icmp_seq=2 hlim=62 time=72.390 ms

16 bytes from 2001:db8:6500:1::1, icmp_seq=3 hlim=62 time=72.372 ms

16 bytes from 2001:db8:6500:1::1, icmp_seq=4 hlim=62 time=61.953 ms

^C

--- 2001:db8:6500:1::1 ping6 statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max/std-dev = 61.953/71.251/84.386/7.729 ms

 

juniper@R1> traceroute 2001:db8:6500:1::1 source 2001:db8:6500::1

traceroute6 to 2001:db8:6500:1::1 (2001:db8:6500:1::1) from 2001:db8:6500::1, 64 hops max, 12 byte packets

1  2001:db8:0:ffff:192:168:255:3 (2001:db8:0:ffff:192:168:255:3)  38.832 ms  42.045 ms  39.924 ms

     MPLS Label=300288 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=1 S=1

2  2001:db8:0:ffff:192:168:255:7 (2001:db8:0:ffff:192:168:255:7)  28.820 ms  29.945 ms  29.981 ms

     MPLS Label=300512 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=2 S=1

3  2001:db8:0:ffff:192:168:255:10 (2001:db8:0:ffff:192:168:255:10)  29.969 ms  30.030 ms  30.018 ms

4  2001:db8:0:ffff:192:168:255:20 (2001:db8:0:ffff:192:168:255:20)  65.953 ms  60.073 ms  65.991 ms

     MPLS Label=301056 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=2 S=1

5  2001:db8:0:ffff:192:168:255:23 (2001:db8:0:ffff:192:168:255:23)  73.466 ms  78.029 ms  71.986 ms

     MPLS Label=300064 CoS=0 TTL=1 S=0

     MPLS Label=2 CoS=0 TTL=3 S=1

6  2001:db8:6500:1::1 (2001:db8:6500:1::1)  65.977 ms  81.957 ms  72.015 ms

 

 

 

Preparing destination-based RTBH policies and RFC6666 discard route

 

Taking into account this end-to-end path, let's configure both preliminary major steps to get the network ready for IPv6 destination-based RTBH:

 

  • Discard route for the RFC6666 discard prefix:

 

juniper@R10> show configuration routing-options

rib inet6.0 {

    static {

        route 100::1/128 {

            discard;

            no-readvertise;

        }

    }

}

[...]

 

 

  • BGP policies to rewrite the next hop and map it to the discard prefix (Note that community, tag, local-preference and diverse other options have been chosen arbitrarily and may change at will)

 

[edit protocols bgp group eBGP-inet]

+    import RTBH-Next-Hop-rewrite;

[edit policy-options]

+   policy-statement RTBH-Next-Hop-rewrite {

+       term RTBH-from-community {

+           from {

+               protocol bgp;

+               community 65001:666;

+           }

+           then {

+               tag 666;

+               local-preference 666;

+               next-hop 100::1;

+           }

+       }

+   }

[edit policy-options]

+   community 65001:666 members 65001:666;

 

 

 

  • Last but not least, the corresponding eBGP session needs to be able to accept and eventually rewrite the next hop with an import policy before importing routes in the RIB. Traditionally, [RFC4271] implies that for a certain eBGP peer, which is single hop away,  BGP should advertise a next hop which shares a common subnet with it. In most cases, the next hop advertised is the address of the interface used for eBGP peering.  In Junos OS, if the next hop received from the one-hop eBGP peer is recognized as not sharing this common subnet, then the route gets hidden and becomes non-eligible per default. There are some options to overcome this situation:

 

    • Multihop eBGP sessions

 

In situations where a different next hop is needed, like this use case, a traditional approach is to configure this session as multihop.This is enforced not because of multiple hops between the peers, but because it causes BGP to relax this check on the next hop, allowing it to accept remote next hops on these routes.

 

This classically supported option is explained under the NANOG32 preso: Options for Blackhole and Discard Routing. There is no need to explicitly set other BGP peering addresses (that is, loopback interface addresses, as you may expect with traditional multihop peering), but this is simply an indication to be able to decouple next-hop settings from the eBGP peering interface addresses and rewrite the next hop at ingress:

 

[edit]

juniper@R10# set protocols bgp group eBGP-inet multihop ttl 3

 

(if not done before, the BGP session will reset when committing this command for obvious reasons)

 

This can be complemented with the no-nexthop-change extension on the peer that sends these routes, depending on how the export policy is conformed. In our Junosphere case, it is enough to activate this session as multihop and craft the policies correspondingly to keep next-hop self and just enforce a different next hop for the discard prefix on the peer that sends the route.

 

 

Another option could be to use the accept-remote-next-hop in the eBGP session configuration, as explained at http://www.juniper.net/techpubs/en_US/junos12.2/topics/example/bgp-accept-remote-nexthop.html. This option was exactly created to be able to accept non-local next hops in the incoming BGP NLRIs over the external sessions, without needing to redefine the complete session as multihop:

 

[edit]

juniper@R11# set protocols bgp group eBGP-inet accept-remote-nexthop

 

Both options are equally valid in this use case, with minimal output changes and differences but same final goal, so I have considered to configure each one of them in each eBGP session from the Junosphere topology to illustrate both.

 

 

At this stage, our policies are ready to get in action! An upcoming post will provide an attack mitigation example over this topology...

Apr 14, 2013

 

Hi Gonzalo,

 

Nice write up! I have check rfc regarding rtbh. such as rfc3882 and rfc6666. ALL of those rfc point that define a static route.why need it ? why not just set next-hop to discard. what is difference?

 

policy-options {
    policy-statement blackhole-by-route {
        term specific-routes {
            from {
                route-filter 10.10.10.1/32 exact;
                route-filter 10.20.20.2/32 exact;
                route-filter 10.30.30.3/32 exact;
                route-filter 10.40.40.4/32 exact;
            }
            then {
                next-hop 192.0.2.101;  ----> change to "next-hop discard"
            }
        }
    }
}
Apr 15, 2013
Juniper Employee

Hi ryantang,

 

many thanks for reading the article and providing feedback!

 

IMHO the point is that you need to distribute such blackholing advertisement throughout the network(s). This needs to be widely identified as an RTBH advertisement.

 

If you directly set a 'discard' next hop, how would this be distributed to the BGP peers? Which next hop would be set in the BGP NLRI? If you try to configure it for a BGP export policy in recent Junos OS releases, the policy verification code will trigger a commit warning:

 

[edit protocols bgp group foo]
  'export'
    next-hop discard not supported by policy blackhole-by-route user

 

because it cannot code the NEXT_HOP attribute (check [RFC4271], Sect 5.1.3) as discard but rather as an IP address per definition. So it may end up setting next-hop self, depending on other policies and your policy chain:

 

user@router> show route advertising-protocol bgp 192.168.255.21 192.0.2.1/32 extensive    

inet.0: 1793 destinations, 3487 routes (1792 active, 0 holddown, 148 hidden)
* 192.0.2.1/32 (1 entry, 1 announced)
 BGP group foo type Internal
     Nexthop: Self
     Localpref: 100
     AS path: [65000] I

 

Provided you have to include an IP-address-like field in the well-known mandatory NEXT_HOP attribute, why not directly including an agreed discard prefix? Even though you tag and mark the route correspondingly for blackholing (communities), a simple route recursive resolution for this discard prefix can make the trick (and can be probably easily implemented across different vendors and networks).

 

As I understand it, the idea behind selecting a so-called discard prefix is that everywhere (or at least at the network perimeter), there is a recursive resolution for the discard prefix (BGP route next hop) to a real discard next hop.

 

[BGP route]

   |--------------->[Next hop: discard prefix]------------> discard prefix local static route with next hop to discard

 

This recursive resolution for the discard prefix (used as NLRI NEXT_HOP attribute) would directly implement blackholing by itself. You can always tag the route with blackholing communities, but this is a direct and straightforward blackholing implementation.

 

Hope I could shed some light on this. I found that [RFC5635], Sect 3.2 and http://www.nanog.org/meetings/nanog32/presentations/soricelli.pdf provided me clarification in this area.

 

Would this make sense? Smiley Happy

 

Thanks,

 

Gonzalo

 

Apr 16, 2013

Hi Gonzalo,

 

Really appreciated for your reply. I mean apply this  blackhole-by-route policy under import .no export. Let's take this topology for example . we want protect C1 from DDoS. we creat  blackhole-by-route which match C1 route on R2 and R3,then apply it under BGP import .

 

 

If we set next-hop to discard . we also can block DDoS traffic from P.  it's same result with IP-address-like next-hop.

 

one issue is set  the NEXT_HOP attribute as discard,ending up setting next-hop self. so when R4 receive C1 's route , protocol next-hop is R2 .not R1.

 

 

C1----------R1-------R2----------R3-------R4-------C2

                                  |                  |  

                                 P                 P

 

(in this topology ,R2 R3 are RR, R1 R4 are client)

Apr 18, 2013
Juniper Employee

Hi ryantang,

 

OK, that is different Smiley Happy and feasible!

We could discuss about the tradeoff to define such a policy in all your peers' ibgp group import chain vs setting local discard routes (provided that discard prefix next-hop mapping is done at the network perimeter). This is debatable because at the end of the day you may need to configure all devices and may end up doing both for enforcement purposes.

I believe there could be some arguments that justify the existence of recursive resolution:

 

[1] Historical: the whole destination-based RTBH concept was discussed since the early 2000s, much before [RFC3882] was published (September 2004). Junos OS started to support directly setting discard next hop with an import bgp policy around 7.0+ timeframe. Beforehand, recursive resolution was the only way to achieve this.

[2] Multivendor support: I am not sure if all networking devices are able to support direct next-hop rewrite to discard/null with a bgp inbound policy/map.

[3] Attack accounting and tracing: setting different real test-net addresses as next hops grants attack traffic analysis. Check for instance latest slides http://www.nanog.org/meetings/nanog32/presentations/soricelli.pdf and [RFC5635], Sect 3.2 (Step 1) as examples. Setting real and per-incident/per-router-type/etc. addresses as next hops would allow you to discern different attack types and to quantify attack volumes by jflowing/accounting traffic towards these test-net addresses. An alternative with a direct discard route could be to use Destination Class Usage (DCU) with per-incident communities, but in that case your tracking options are not so wide and this can be always combined with the former.

[4] Deviation to DPI and cleaning systems: I have worked in the past in some designs where the discard prefix was not locally mapped to a discard interface, but rather to another locally defined resource (like a tunnel or a VRF towards a cleaning system). This is aligned with the use case exposed in [RFC3882], Sect 3. At a certain stage, you may want to change this local mapping or even use different advertisement types for the attack to set one or the other next hop, resulting into complete blackholing (tracking attack volumes with mentioned tools) or traffic deviation to a cleaning device.

From all these reasons, I tend to think that [3] is probably the most relevant one.

However, as usual, it surely depends on your scenario and constraints.

Thoughts? Agree/Disagree? Smiley Happy

Thanks,

Gonzalo

Apr 21, 2013

thx Gonzalo. very clear and reasonable.

Feedback