Routing

We run a multivendor mpls network spread across around 8 DC's. Currently I'm redesigning our "internet breakout" and I'm puzzled about some odd behavior I'm seeing (or misunderstanding the output).

I can simulate our problem in a very simple topology with only Juniper equipment:

Customer CPE/Router <-> QFX5100 <-> MX204

- Our IGP is OSPF, we run MPLS with LDP and BGP on top
- MPLS traffic engineering is on BGP, we configured explicit-null's
- Routing options are configured with indirect-next-hop, we use per-prefix-labels

We put the "connected" interface interco's on the MX and QFX in BGP to satisfy the route lookups with the right next-hop (OSPF is only responsible for backbone interfaces + loopbacks)
On the MX204 we also announce an aggregate by redistributing a static 0/0 towards the network. This problem happens on the default and as well on upstream interface ranges on the MX.

Problem:
When sending traffic out the customer CPE towards something behind the MX we get a loop:

X.X.X.2 = intero between CPE and QFX
Y.Y.Y.1 = IP behind an interface on the MX (something connected, but not the MX itself)
Z.Z.Z.1 = IP of the MX on the link between the MX/QFX

thomas@503-r51b09-2> traceroute source X.X.X.2 Y.Y.Y.1 ttl 5 
traceroute to Y.Y.Y.1 (Y.Y.Y.1) from X.X.X.2, 5 hops max, 40 byte packets
 1  X.X.X.1 (X.X.X.1)  23.820 ms  13.688 ms  23.873 ms
 2  Z.Z.Z.1 (Z.Z.Z.1)  10.475 ms  4.051 ms  1.992 ms
 3  Z.Z.Z.1 (Z.Z.Z.1)  2.674 ms  4.898 ms  3.866 ms
 4  Z.Z.Z.1 (Z.Z.Z.1)  3.412 ms  4.424 ms  1.500 ms
 5  Z.Z.Z.1 (Z.Z.Z.1)  2.679 ms  2.483 ms  2.566 ms

If we do a traceroute towards Y.Y.Y.2, the IP of the MX of that interco, it works fine.

Output's:

1) first the qfx does do a lookup on 0/0 in inet.0 and after that inet.3 should be checked to reach our MX-LOOPBACK:

root@qfx> show route table inet.0 0.0.0.0 

inet.0: 95 destinations, 167 routes (83 active, 0 holddown, 78 hidden)e
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[BGP/170] 1d 11:10:16, localpref 100, from MX-LOOPBACK
                      AS path: I, validation-state: unverified
                    >  to Z.Z.Z.1 via et-0/0/10.0, Push 0


root@qfx> show route table inet.3 MX-LOOPBACK

inet.3: 9 destinations, 12 routes (9 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

MX-LOOPBACK/32  *[LDP/9] 1w0d 11:32:50, metric 1
                    >  to Z.Z.Z.1 via et-0/0/10.0, Push 0

I guess pushing 0 to the packet isn't optimal as this router is also the last hop before our MX (they are back to back).

2)

The packet arrives at the MX and a lookup should be done on the mpls table:

root@mx> show route table mpls.0 label 0   

mpls.0: 28 destinations, 28 routes (28 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0                  *[MPLS/0] 6w2d 19:46:38, metric 1
                       to table inet.0
0(S=0)             *[MPLS/0] 6w2d 19:46:38, metric 1
                       to table mpls.0

So label 0 makes the MX look into inet.0 that one has all the right entries. 

So in theory, if you look at the outputs this should work, but it is not? Looking at the traceroute it looks like we are stuck on the MX itself, traffic is not bouncing back towards other interfaces.

It also seems to work for the MX IP's on the connected interfaces but not for the peer's.

a) I do get that the ingress PE is also the PHP, is this the cause of the problem? Shouldn't the fowarding logic be smart enough to handle this and just send an IP packet towards the MX?

b) I guess moving towards RSVP and doing tunnel-services & UHP on the MX would solve this but I do not fully understand why the current config isnt working. How are all you solving these kind of problems?

Thanks,

Thomas

Hello,

There is not enough information to analyze Your problem.

Please post complete sanitized configurations, topology diagram and traffic flows.

Also please post following printouts from QFX and MX:

show route extensive Y.Y.Y.1 | no-more
show route forwarding-table destination Y.Y.Y.1 | no-more

---

@thomaswoidt wrote:

- MPLS traffic engineering is on BGP, we configured explicit-null's

<skip>

I guess pushing 0 to the packet isn't optimal as this router is also the last hop before our MX (they are back to back).

<skip>

Shouldn't the fowarding logic be smart enough to handle this and just send an IP packet towards the MX?

 

This is expected if You configure explicit-null. And no, forwarding logic is doing what You instructed it to do.

HTH

Thx

Alex

.... wrong, see below.

I'll work on full configurations.

Hello,

---

@thomaswoidt wrote:

Y.Y.Y.1 = IP behind an interface on the MX (something connected, but not the MX itself)

and

---

@thomaswoidt wrote:

root@mx> show route Y.Y.Y.1 extensive table inet.0 | no-more

inet.0: 802801 destinations, 3192207 routes (802801 active, 0 holddown, 0 hidden)
Y.Y.Y.1/32 (1 entry, 1 announced)
        State: <FlashAll>
        *Local  Preference: 0
                Next hop type: Local, Next hop index: 0

---

so, Y.Y.Y.1 - which one is this?  Local IP on MX, or link IP on another box directly connected to MX?

HTH

Thx

Alex

Small mistake on the Y.Y.Y.1 (bad thing of not having just a simple lab atm, I also modified the QFX config here).

These are the right outputs (so the IP is behind the MX on a connected interface, I'll try to remove the others):

root@mx> show route Y.Y.Y.1 extensive table inet.0 | no-more                                         

inet.0: 802732 destinations, 3189419 routes (802732 active, 0 holddown, 0 hidden)
Y.Y.Y.0/30 (1 entry, 1 announced)
        State: <FlashAll>
TSI:
Page 0 idx 1, (group CORE type Internal) Type 1 val 0xe753e28 (adv_entry)
   Advertised metrics:
     Flags: Nexthop Change
     Nexthop: Self
     Localpref: 100
     AS path: [65000] I
     Communities: 65000:2
    Advertise: 0000001f
Path Y.Y.Y.0
Vector len 4.  Val: 0 1
        *Direct Preference: 0
                Next hop type: Interface, Next hop index: 0
                Address: 0x5221b7c
                Next-hop reference count: 1
                Next hop: via xe-0/1/0.109, selected
                State: <Active Int>
                Local AS: 65000 
                Age: 5d 2:01:10 
                Validation State: unverified 
                Task: IF
                Announcement bits (4): 2-LDP 7-BGP_RT_Background 8-Resolve tree 6 9-Resolve tree 8 
                AS path: I 



root@qfx> show route Y.Y.Y.1 extensive table inet.0 

inet.0: 95 destinations, 167 routes (83 active, 0 holddown, 78 hidden)
Y.Y.Y.0/30 (1 entry, 1 announced)
TSI:
KRT in-kernel Y.Y.Y.0/30 -> {indirect(131093)}
Page 0 idx 1, (group BORDER type Internal) Type 1 val 0xeb4ecfc (adv_entry)
   Advertised metrics:
     Nexthop: MX-LOOPBACK
     Localpref: 100
     AS path: [65000] I
     Communities: 65000:2
     Cluster ID: CLUSTER-ID
     Originator ID: MX-LOOPBACK
    Advertise: 00000003
Path Y.Y.Y.0
from MX-LOOPBACK
Vector len 4.  Val: 1 2 3
        *BGP    Preference: 170/-101
                Next hop type: Indirect, Next hop index: 0
                Address: 0xc664220
                Next-hop reference count: 14
                Source: MX-LOOPBACK
                Next hop type: Router, Next hop index: 2087
                Next hop: Z.Z.Z.1 via et-0/0/10.0, selected
                Label operation: Push 0
                Label TTL action: prop-ttl
                Load balance label: Label 0: None; 
                Label element ptr: 0xbcc4458
                Label parent element ptr: 0x0
                Label element references: 4
                Label element child references: 3
                Label element lsp id: 0
                Session Id: 0x0
                Protocol next hop: MX-LOOPBACK
                Indirect next hop: 0xce1dc04 131093 INH Session ID: 0x0
                State: <Active Int Ext>
                Local AS: 65000 Peer AS: 65000
                Age: 2d 8:36:27         Metric2: 1 
                Validation State: unverified 
                Task: BGP_65000.MX-LOOPBACK
                Announcement bits (4): 0-KRT 6-BGP_RT_Background 7-Resolve tree 4 8-Resolve tree 5 
                AS path: I 
                Communities: 65000:2
                Accepted
                Localpref: 100
                Router ID: MX-LOOPBACK
                Indirect next hops: 1
                        Protocol next hop: MX-LOOPBACK Metric: 1
                        Indirect next hop: 0xce1dc04 131093 INH Session ID: 0x0
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: Z.Z.Z.1 via et-0/0/10.0
                                Session Id: 0x0
                                MX-LOOPBACK/32 Originating RIB: inet.3
                                  Metric: 1     Node path count: 1
                                  Forwarding nexthops: 1
                                        Nexthop: Z.Z.Z.1 via et-0/0/10.0
                                        Session Id: 0

And the forwarding tables:

root@mx> show route forwarding-table destination Y.Y.Y.1 table default | no-more 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
Y.Y.Y.1/32 dest     1 0:24:dc:ce:3e:74   ucst      828 241368 xe-0/1/0.109


root@qfx> show route forwarding-table destination Y.Y.Y.1 table default | no-more 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
Y.Y.Y.0/30 user     0                    indr   131093     7
                              Z.Z.Z.1   Push 0     2087     3 et-0/0/10.0

This the config for the QFX. The relevant pieces towards the MX's is called BORDER. I tried to only keep relevant pieces of config (we also do EVPN/VXLAN over loopback/MPLS and L3VPN's here).

The BORDER peering group has 3 MX's, spread around some DC's. The problem only exists when the local connected MX is used. Connecting the CPE to a QFX without a local MX it all works fine (you'll see a swap 0 for the ingress label instead of push 0).

Filtering is strict as we have full tables on MX and only limited routes on QFX (aggregation network).

root@qfx> show configuration               
## Last commit: 2020-06-05 06:31:12 CEST by root
## Image name: jinstall-host-qfx-5-19.4R1.10-signed.tgz

version 20191212.201431_builder.r1074901;
system {
    host-name qfx;
    root-authentication {
        encrypted-password ****; ## SECRET-DATA
    }
    services {
        ssh {
            root-login allow;
        }
    }
    time-zone Europe/Brussels;
    ntp {
        server ****;
        server ****;
    }
}
chassis {
    aggregated-devices {
        ethernet {
            device-count 30;
        }
    }
    fpc 0 {
        pic 0 {
            ....
            port 16 {
                channel-speed 10g;
            }
            ....
        }
    }
}
security {
    authentication-key-chains {
        key-chain upstream-bfd {                   
            key 0 {
                secret ****; ## SECRET-DATA
                start-time "1970-1-1.01:00:01 +0100";
            }
        }
    }
    ipsec {
        security-association core {
            mode transport;
            manual {
                direction bidirectional {
                    protocol ah;
                    spi 256;
                    authentication {
                        algorithm hmac-sha1-96;
                        key ascii-text ****; ## SECRET-DATA
                    }
                }
            }
        }
    }
}
interfaces {
    ....
    et-0/0/10 {
        description "Link to MX et-0/0/2";
        mtu 9216;
        unit 0 {
            family inet {
                address Z.Z.Z.2/30;
            }
            family inet6 {
                address ..../127;
            }
            family mpls;
        }
    }
    ....
    xe-0/0/16:0 {
        description "Link to CPE";
        unit 0 {
            family inet {
                address X.X.X.1/30;
            }
            family inet6 {
                address ..../127;
            }
        }
    }
    ....
    lo0 {
        unit 0 {
            family inet {
                filter {
                    input protect-me;
                }
                address QFX-LOOPBACK/32 {
                    primary;
                }
            }
            family inet6 {
                address ..../128 {
                    primary;
                }
            }
        }
    }
}

policy-options {
    prefix-list interco_list {
        X.X.X.0/30;
    }
    prefix-list interco6_list {
        ..../127;
    }

    ....

    policy-statement export_border {
        term customers {
            from {
                protocol bgp;
                community customers;
            }
            then accept;
        }
        term loopbacks {
            from {
                protocol bgp;
                community customers_loopbacks;
            }
            then accept;
        }
        term underlay {
            from {
                protocol bgp;
                community [ interco statics ];
            }
            then accept;
        }
        term connected {
            from {
                protocol direct;
                prefix-list interco_list;
            }
            then {
                community set interco;
                next-hop self;
                accept;
            }                           
        }
        term static {
            from {
                protocol static;
                tag 10;
            }
            then {
                community set statics;
                accept;
            }
        }
        then reject;
    }

    policy-statement import_border {
        term underlay {                 
            from {
                protocol bgp;
                community [ interco statics ];
            }
            then accept;
        }
        term aggregate {
            from {
                protocol bgp;
                community aggregates;
            }
            then accept;
        }
        then reject;
    }

    ....

    policy-statement load-balance {
        then {
            load-balance per-packet;
        }
    }

    ....

    community aggregates members 65000:5;
    community all members *:*;

    community customers members 65000:50000;
    community customers_blackhole members 65000:5666;
    community customers_loopbacks members 65000:59999;

    community interco members 65000:2;
    community rtbh members 65000:666;
    community statics members 65000:10;
}
firewall {
    family inet {
        filter protect-me {
...
        }
    }
....
}


routing-instances {
....
}

routing-options {
    forwarding-table {
        export load-balance;
        ecmp-fast-reroute;
        indirect-next-hop;
    }
    router-id QFX-LOOPBACK;
    autonomous-system 65000;
}


protocols {

    ospf {
        area 0.0.0.0 {
            interface lo0.0 {
                passive;
            }
            interface et-0/0/10.0 {
                interface-type p2p;
                authentication {
                    simple-password ****; ## SECRET-DATA
                }
                bfd-liveness-detection {
                    minimum-interval 1500;
                    multiplier 3;
                }
            }
        }
        reference-bandwidth 1000g;
    }

    rsvp {
        interface lo0.0;
        interface et-0/0/10.0 {
            authentication-key ****; ## SECRET-DATA
        }
    }

    bgp {
        family inet {
            labeled-unicast {
                aggregate-label {
                    community aggregates;
                }
                per-prefix-label;
            }
        }
        family inet6 {
            labeled-unicast {
                aggregate-label {
                    community aggregates;
                }
            }
        }

        group BORDER {
            local-address QFX-LOOPBACK;
            hold-time 30;
            import import_border;
            family inet {
                any;
            }
            family inet-vpn {
                any;
            }
            authentication-key ****; ## SECRET-DATA
            export export_border;
            cluster CLUSTER-ID;
            peer-as 65000;
            neighbor MX-1;
            neighbor MX-2;
            neighbor MX-3;
        }
        group BORDER6 {
            local-address QFX-LOOPBACK;
            hold-time 30;
            import import6_border;
            family inet6 {
                any;
            }
            authentication-key ****; ## SECRET-DATA
            export export6_border;
            cluster CLUSTER-ID;
            peer-as 65000;
            neighbor MX-1;
            neighbor MX-2;
            neighbor MX-3;
        }

....

        log-updown;
        graceful-restart;
    }

    ldp {
        deaggregate;
        explicit-null;
        transport-address router-id;
        interface et-0/0/10.0;
        interface lo0.0;
        family {
            inet;
            inet6;
        }
        transport-preference ipv4;
    }

    mpls {
        traffic-engineering {
            bgp;
        }
        explicit-null;
        icmp-tunneling;
        no-decrement-ttl;
        no-cspf;
        ipv6-tunneling;

        interface lo0.0;
        interface et-0/0/10.0;

    }

....

    ospf3 {
        area 0.0.0.0 {
            interface lo0.0 {
                passive;
            }

            interface et-0/0/10.0 {
                interface-type p2p;
                ipsec-sa core;
                bfd-liveness-detection {
                    minimum-interval 1500;
                    multiplier 3;
                }
            }

        }
        reference-bandwidth 1000g;
    }


}

{master:0}
root@qfx> 

root@mx> show configuration 
## Last commit: 2020-06-05 06:19:04 CEST by root
version 20191212.201431_builder.r1074901;
system {
    host-name mx;
    root-authentication {
        encrypted-password ****; ## SECRET-DATA
    }
    services {
        ssh {
            root-login allow;
            sftp-server;
        }
    }
    time-zone Europe/Brussels;
    ntp {
        server ****;
        server ****;
    }
}
chassis {
    fpc 0 {
        pic 0 {
            port 2 {
                speed 40g;
            }
        }
        pic 1 {
            port 0 {
                speed 10g;
            }
        }
    }
    alarm {
        management-ethernet {
            link-down ignore;
        }
    }
    network-services enhanced-ip;
}
security {                              
    authentication-key-chains {         
        key-chain upstream-bfd {        
            key 0 {                     
                secret ****; ## SECRET-DATA
                start-time "1970-1-1.01:00:01 +0100";
            }                           
        }                               
    }                                   
    ipsec {                             
        security-association core {     
            mode transport;             
            manual {                    
                direction bidirectional {
                    protocol ah;        
                    spi 256;            
                    authentication {    
                        algorithm hmac-sha1-96;
                        key ascii-text ****; ## SECRET-DATA
                    }                   
                }                       
            }                           
        }                               
    }                                   
}    
interfaces {                            
    et-0/0/2 {                          
        description "Link to QFX et-0/0/10";
        mtu 9216;                       
        unit 0 {                        
            family inet {               
                address Z.Z.Z.1/30;
            }                           
            family inet6 {              
                address ****/127;
            }                           
            family mpls;                
        }                               
    }                        
....
    xe-0/1/0 {                          
        flexible-vlan-tagging;          
        mtu 9216;         
        unit 109 {                      
            vlan-id 109;                
            family inet {               
                mtu 1500;               
                address Y.Y.Y.2/30;
            }                           
            family inet6 {              
                mtu 1500;               
                address ****/126;
            }                           
        }   
....
    lo0 {
        unit 0 {
            family inet {
                address MX-LOOPBACK/32 {
                    primary;
                    preferred;
                }
            }
            family inet6 {
                address ****/128 {
                    primary;
                }
            }
        }
    }
}
policy-options {   
    prefix-list interco_list {                       
         Y.Y.Y.0/30;             
    }                                   
    prefix-list interco6_list {         
....          
    }         

    policy-statement export6_core {     
        term aggregate {                
            from {                      
                protocol static;        
                tag 5;                  
            }                           
            then {                      
                community set aggregates;
                next-hop self;          
                accept;                 
            }                           
        }                               
        then reject;                    
    }

    policy-statement export_core {
        term connected {
            from {
                protocol direct;
                prefix-list interco_list;
            }
            then {
                community set interco;
                next-hop self;
                accept;
            }
        }
        term aggregate {
            from {
                protocol static;
                tag 5;
            }
            then {
                community set aggregates;
                next-hop self;
                accept;
            }
        }
        term static {
            from {
                protocol static;
                tag 10;
            }
            then {
                community set statics;
                accept;
            }                           
        }                               
        term customers {                
            from {                      
                protocol bgp;           
                community customers;    
            }                           
            then accept;                
        }                               
        then reject;                    
    }  

    policy-statement import6_core {     
        term customers {                
            from {                      
                protocol bgp;           
                community customers;    
            }                           
            then accept;                
        }                               
        term loopbacks {                
            from {                      
                protocol bgp;           
                community customers_loopbacks;
            }                           
            then accept;                
        }                               
        term connected {                
            from {                      
                protocol bgp;           
                community interco;      
            }                           
            then accept;                
        }                               
        term static {                   
            from {                      
                protocol bgp;           
                community statics;      
            }                           
            then accept;                
        }                               
        then reject;                    
    }                              

    policy-statement import_core {      
        term customers {                
            from {                      
                protocol bgp;           
                community customers;    
            }                           
            then accept;                
        }                               
        term loopbacks {                
            from {                      
                protocol bgp;           
                community customers_loopbacks;
            }                           
            then accept;                
        }                               
        term connected {                
            from {                      
                protocol bgp;           
                community interco;      
            }                           
            then accept;                
        }                               
        term static {                   
            from {                      
                protocol bgp;           
                community statics;      
            }                           
            then accept;                
        }                               
        then reject;                    
    }                            

    community aggregates members 65000:5;
    community all members *:*;          
    community customers members 65000:50000;
    community customers_blackhole members 65000:5666;
    community customers_loopbacks members 65000:59999;
    community interco members 65000:2;  
    community rtbh members 65000:666;   
    community statics members 65000:10; 
    as-path private 64512-65535;   

     
}                                       

firewall {                              
    family inet {                       
        filter protect-me { 
....
        }                               
    }                                   
}  



protocols {
    ospf {
        area 0.0.0.0 {
            interface lo0.0 {
                passive;
            }
            interface et-0/0/2.0 {
                interface-type p2p;
                authentication {
                    simple-password ****; ## SECRET-DATA
                }
                bfd-liveness-detection {
                    minimum-interval 1500;
                    multiplier 3;
                }
            }
        }                               
        reference-bandwidth 1000g;      
    }   
     
    rsvp {                              
        interface lo0.0;                
        interface et-0/0/2.0 {          
            authentication-key ****; ## SECRET-DATA
        }    

    bgp {                               
        family inet {                   
            labeled-unicast {           
                aggregate-label {       
                    community aggregates;
                }                       
                per-prefix-label;       
            }                           
        }                               
        family inet6 {                  
            labeled-unicast {           
                aggregate-label {       
                    community aggregates;
                }                       
            }                           
        }   
                         
        group CORE {                    
            local-address MX-LOOPBACK;
            hold-time 30;               
            import import_core;         
            family inet {               
                any;                    
            }                           
            family inet-vpn {           
                any;                    
            }                           
            family inet6-vpn {          
                any;                    
            }                           
            authentication-key ****; ## SECRET-DATA
            export export_core;         
            cluster MX-CLUSTERID-1; 
            peer-as 65000;              
            neighbor QFX-LOOPBACK;
            ....
        }   
                            
        group CORE6 {                   
            local-address MX-LOOPBACK;
            hold-time 30;               
            import import6_core;        
            family inet6 {              
                any;                    
            }                           
            authentication-key ****; ## SECRET-DATA
            export export6_core;        
            cluster MX-CLUSTERID-1;     
            peer-as 65000;              
            neighbor QFX-LOOPBACK;
            .... 
        }
                               
        log-updown;                     
        graceful-restart;               
    } 

 
    ldp {                               
        deaggregate;                    
        explicit-null;                  
        transport-address router-id;    
        interface et-0/0/2.0;           

        interface lo0.0;                
        family {                        
            inet;                       
            inet6;                      
        }                               
        transport-preference ipv4;      
    }                                   
    mpls {                              
        traffic-engineering {           
            bgp;                        
        }                               
        explicit-null;                  
        icmp-tunneling;                 

        no-decrement-ttl;               
        no-cspf;                        
        ipv6-tunneling;                 
        interface lo0.0;                
        interface et-0/0/2.0;           

    }      

    ospf3 {                             
        area 0.0.0.0 {                  
            interface lo0.0 {           
                passive;                
            }                           
            interface et-0/0/2.0 {      
                interface-type p2p;     
                ipsec-sa core;          
                bfd-liveness-detection {
                    minimum-interval 1500;
                    multiplier 3;       
                }                       
            }                           
      
        }                               
        reference-bandwidth 1000g;      
    }                                   
}                      
             

Hello,

Thanks for providing the configurations.

First findings:

1/ on the forward path, the traffic does not take Your 0/0 route on QFX, it takes a specific Y.Y.Y.0/30 route

2/ I see no issues with encapsulation or forward routing on either QFX or MX

Let's examine the return path.

Please provide the following printouts from QFX and MX:

show route extensive X.X.X.2 | no-more
show route forwarding-table destination X.X.X.2 | no-more

Also, do You have NAT configured on MX (inline or regular MS-DPC|MS-MPC NAT)?

HTH

Thx
Alex

1/ True, in this case. As I'm trying to simulate the most simple scenario where this issue occurs .

Example with the default route looks like this:

root@qfx> show route forwarding-table destination 8.8.4.4 table default | no-more 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    indr   131093     7
                              40:de:ad:cb:20:27 Push 0     2087     3 et-0/0/10.0
default            perm     0                    rjct       51     1



root@mx> show route forwarding-table destination 8.8.4.4 table default | no-more    
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
8.8.4.0/24         user     0 SOME-TRANSIT-PROVIDER     ucst      869 425690 xe-0/1/0.101

2/ I've been looking at this for some days and cant figure out what I am missing, we do plenty of other stuff on mpls/l3vpn/evpn/vxlan/... that is just working fine.

This label is pushed at the DC next to it and just works fine:

root@qfx-other> show route table inet.0 0.0.0.0 

inet.0: 91 destinations, 119 routes (66 active, 0 holddown, 49 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[BGP/170] 4d 14:02:00, localpref 100, from MX-LOOPBACK
                      AS path: I, validation-state: unverified
                    >  to P2P-QFX via xe-0/0/12:3.0, Push 53


root@qfx> show route table mpls.0 label 53  

mpls.0: 41 destinations, 41 routes (41 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

53                 *[LDP/9] 1w1d 11:36:31, metric 1
                    >  to Z.Z.Z.1 via et-0/0/10.0, Swap 0
53(S=0)            *[LDP/9] 1w1d 11:36:31, metric 1
                    >  to Z.Z.Z.1 via et-0/0/10.0, Pop      

Return traffic is currently going over this qfx-other (and is working fine). The CPE is connected to both, On the CPE I'm currently pushing all traffic out the other DC and have more specifics for the test-ip's towards the qfx on the "broken route".

The route back (I checked and the problem also occurs if I do it with the source of the interco interface):

root@mx> show route forwarding-table destination X.X.X.2 table default 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
 X.X.X.2/30  user     0                    indr  1048578     3
                              Z.Z.Z.2   Push 0      607     3 et-0/0/2.0


root@qfx> show route forwarding-table destination X.X.X.2 table default 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
X.X.X.2/32  dest     1 3c:8a:b0:d3:cc:e7  ucst     2141     2 xe-0/0/16:0.0

It all looks perfectly fine for me :(. If you do traceroutes on the qfx or the mx with as source the same interfaces (from the RE), they also work.

I went through a couple of scenarios again:

1) QFX and QFX-OTHER are back to back like the QFX-MX. if we do the same thing here (communicate over connected interco's) the issue is not present, so pushing and pop'ing 0 on QFX is working fine.

2) If the traceroute is started on something behind an interface on the MX it does work fine too, so packets arriving on the MX get label 0 pushed and this is understood by the QFX (and pop'ed before going to the cpe)

3) If we start traceroute on the CPE it's broken:

- Pushing 0 here needs to be validated (but we might conclude it works based on the QFX <-> QFX-OTHER test)

- Lookup on MX for ingress traffic with label 0 needs more investigation 

Hello,

---

@thomaswoidt wrote:

 

3) If we start traceroute on the CPE it's broken:

- Pushing 0 here needs to be validated (but we might conclude it works based on the QFX <-> QFX-OTHER test)

- Lookup on MX for ingress traffic with label 0 needs more investigation 

---

Please check if You have any missing prefixes/labels NOT installed in MX HW

show krt queue | no-more

HTH

Thx

Alex

I was doing some hw filtering today to confirm more things. If I understand the flexible-match filter correctly this should take all label's 0 from mpls and will look further for the source-addresss of our CPE's (ipv4 header).

root@mx> show configuration firewall family mpls | display set 
set firewall family mpls filter in-count interface-specific
set firewall family mpls filter in-count term null from ip-version ipv4 protocol icmp
set firewall family mpls filter in-count term null from ip-version ipv4 source-address X.X.X.2/32
set firewall family mpls filter in-count term null from flexible-match-range match-start layer-3
set firewall family mpls filter in-count term null from flexible-match-range byte-offset 0
set firewall family mpls filter in-count term null from flexible-match-range bit-length 20
set firewall family mpls filter in-count term null from flexible-match-range range 0
set firewall family mpls filter in-count term null then count nullpkt
set firewall family mpls filter in-count term null then accept
set firewall family mpls filter in-count term accept then accept

root@mx> 

Output of our filter matches the amount of pings send from the CPE:

root@mx> show firewall |match et-0/0/2                                               

Filter: in-count-et-0/0/2.0-i                                  
nullpkt-et-0/0/2.0-i                                    0                    0


thomas@503-r51b09-2> ping source X.X.X.2 Y.Y.Y.2 count 1 wait 1    
PING Y.Y.Y.2 (Y.Y.Y.2): 56 data bytes
36 bytes from Z.Z.Z.1: Time to live exceeded
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 a47b   0 0000  01  01 ed90 X.X.X.2  Y.Y.Y.2 


root@mx> show firewall |match et-0/0/2    

Filter: in-count-et-0/0/2.0-i                                  
nullpkt-et-0/0/2.0-i                                   88                    1


thomas@503-r51b09-2> ping source X.X.X.2 Y.Y.Y.1 count 1 wait 1    
PING Y.Y.Y.1 (Y.Y.Y.1): 56 data bytes
64 bytes from Y.Y.Y.1: icmp_seq=0 ttl=63 time=1.448 ms


root@mx> show firewall |match et-0/0/2                                               

Filter: in-count-et-0/0/2.0-i                                  
nullpkt-et-0/0/2.0-i                                  176                    2

Output:

root@mx> show krt queue | no-more 
Routing table add queue: 0 queued
Interface add/delete/change queue: 0 queued
Top-priority deletion queue: 0 queued
Top-priority change queue: 0 queued
Top-priority add queue: 0 queued
high priority V4oV6 tcnh delete queue: 0 queued
high prioriy anchor gencfg delete queue: 0 queued
High-priority multicast add/change: 0 queued
Indirect next hop top priority add/change: 0 queued
Indirect next hop add/change: 0 queued
high prioriy anchor gencfg add-change queue: 0 queued
MPLS add queue: 0 queued
Indirect next hop delete: 0 queued
High-priority deletion queue: 0 queued
MPLS change queue: 0 queued
High-priority change queue: 0 queued
High-priority add queue: 0 queued
Normal-priority indirect next hop queue: 0 queued
Normal-priority deletion queue: 0 queued
Normal-priority composite next hop deletion queue: 0 queued
Low prioriy Statistics-id-group deletion queue: 0 queued
Normal-priority change queue: 0 queued
Normal-priority add queue: 0 queued
Least-priority delete queue: 0 queued
Least-priority change queue: 0 queued
Least-priority add queue: 0 queued
Normal-priority pfe table nexthop queue: 0 queued
EVPN gencfg queue: 0 queued
Normal-priority gmp queue: 0 queued
Routing table delete queue: 0 queued
Low priority route retry queue: 0 queued

root@mx> 

Hello,

Ok cool so You know how to use the JUNOS flex-offset filter 

Could You please check if this packet has BOS bit set?

Also, if would be good to mirror the whole packet to see if there isn't any artefact introduced by CPE.

HTH

Thx

Alex

So what I did was create a poor-man's tap by adding a sample action to the firewall filter and use the inline ipfix to send it to some tcpdump host:

root@mx# show | compare 
[edit chassis fpc 0]
-    sampling-instance flowlogs;
[edit services]
-   flow-monitoring {
-       version-ipfix {
-           template mplsflows {
-               flow-active-timeout 15;
-               flow-inactive-timeout 15;
-               mpls-template;
-           }
-       }
-   }
[edit]
-  forwarding-options {
-      sampling {
-          instance {
-              flowlogs {
-                  input {
-                      rate 1;
-                  }
-                  family mpls {
-                      output {
-                          flow-server **** {
-                              port 9011;
-                              version-ipfix {
-                                  template {
-                                      mplsflows;
-                                  }
-                              }
-                          }
-                          inline-jflow {
-                              source-address MX-LOOPBACK;
-                              flow-export-rate 200;
-                          }
-                      }
-                  }
-              }
-          }
-      }
-  }
[edit firewall family mpls filter in-count term null then]
-       sample;

[edit]
root@mx# 

Wireshark can decode the frames (you also need to wait till it receives the template). Screenshot is attached. I'll do the same exercise on the working path to validate the output:

Just as small update:

- it was impossible to find the packet from the other cpe (connected to qfx-other) when using mpls firewall filters. After adding it as plain family inet filter do get hits.

root@mx# show firewall | display set | match count 
set firewall family inet filter ip-count interface-specific
set firewall family inet filter ip-count term null from destination-address */32
set firewall family inet filter ip-count term null then count inetpkt
set firewall family inet filter ip-count term null then accept
set firewall family inet filter ip-count term accept then accept
...
set firewall family mpls filter in-count interface-specific
set firewall family mpls filter in-count term null from ip-version ipv4 protocol icmp
set firewall family mpls filter in-count term null from ip-version ipv4 destination-address */32
set firewall family mpls filter in-count term null from flexible-match-range match-start layer-3
set firewall family mpls filter in-count term null from flexible-match-range byte-offset 0
set firewall family mpls filter in-count term null from flexible-match-range bit-length 20
set firewall family mpls filter in-count term null from flexible-match-range range 0
set firewall family mpls filter in-count term null then count nullpkt
set firewall family mpls filter in-count term null then accept
set firewall family mpls filter in-count term accept then accept

Output after 1 ping from each CPE:

root@mx# run show firewall 

Filter: __default_bpdu_filter__                                

Filter: protect-me                                             

Filter: in-count-et-0/0/2.0-i                                  
Counters:
Name                                                Bytes              Packets
nullpkt-et-0/0/2.0-i                                   88                    1

Filter: ip-count-et-0/0/2.0-i                                  
Counters:
Name                                                Bytes              Packets
inetpkt-et-0/0/2.0-i                                   84                    1

- it seems that the QFX connected to the MX is actually doing 2 different things:

1. for the direct connected cpe it is pushing 0 to the packet (expl null option), this is to be expected

2. for the other cpe, the packet is arriving at qfx-other, gets label 53, is being transmitted to qfx, here the output show a 53-swap-0 but in reality seems the mpls label is pop'ed and we just get an ip packet at the MX

So it looks like we might have 2 independent issues:

- QFX is doing some labeling wrong towards MX (not pushing the 0 for label swap's)

- The MX works fine if it gets a normal IP packet, but has issues with label 0 (even if here we should get a normal lookup in inet.0)

Besides the debugging, following fix actually works like expected (UHP):

QFX:

root@qfx> show configuration protocols mpls label-switched-path qfx_to_mx 
to MX-LOOPBACK;
ultimate-hop-popping;

root@qfx> show rsvp session name qfx_to_mx extensive | match "Label out"
  Resv style: 1 FF, Label in: -, Label out: 80

On the MX:

root@mx> show route label 80 table mpls.0       

mpls.0: 32 destinations, 32 routes (32 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

80                 *[RSVP/0] 00:01:54
                       to table inet.0, label-switched-path qfx_to_mx
80(S=0)            *[RSVP/0] 00:01:54
                       to table mpls.0

Doesn't explain the weirdness but is an acceptable workaround. I think the MX is even doing multiple label lookups at ingress so it's not using tunnel-services for recirculation (even those can do high capacity on the mx204 platform).