Junos OS

I have two MX routers with intentical version 17.3R3.10

First MX is border router and have the following config:

show interfaces xe-1/0/0 
flexible-vlan-tagging;
mtu 9216;
unit 0 {
    vlan-id 506;
    family inet {
        mtu 9000;
        address 10.0.0.18/30;
    }
}

show interfaces gr-1/3/0
unit 0 {
    tunnel {
        source 10.0.0.18;
        destination 10.0.0.17;
    }
    family bridge {
        interface-mode trunk;
        vlan-id-list 3009;
    }
}

VLAN 3009 is bridged with irb with public IP address

Router 2 config is:

show interfaces xe-0/0/3                                            
flexible-vlan-tagging;
mtu 9216;
unit 0 {
    vlan-id 506;
    family inet {
        mtu 9000;
        address 10.0.0.17/30;
    }
}

show interfaces gr-0/0/0 
unit 0 {
    tunnel {
        source 10.0.0.17;
        destination 10.0.0.18;
    }
    family bridge {
        interface-mode trunk;
        vlan-id-list 3009;
    }
}

VLAN3009 is bridged with second interface:

show bridge-domains
test {
    vlan-id 3009;
}
show interfaces xe-0/0/1    
unit 0 {
    family bridge {
        interface-mode trunk;
        vlan-id-list 3009;
    }
}

On which have connected Linux server with configured public IP and gateway irb interface on first router.

Between routers jumbo frames is passing, but from the Linux server (default mtu 1500) to the gateway via gre tunnel largest passing packet is 1468 bytes, which create issues with tcp-mss. Already tried allow-fragmentation and do-not-fragment tunnel options, but nothing changes.

In both routers gre tunnel mtu is marked as unlimited:

run show interfaces gr-1/3/0      
Physical interface: gr-1/3/0, Enabled, Physical link is Up
  Interface index: 275, SNMP ifIndex: 851
  Type: GRE, Link-level type: GRE, MTU: Unlimited, Speed: 10000mbps
  Device flags   : Present Running
  Interface flags: Point-To-Point SNMP-Traps
  Input rate     : 0 bps (0 pps)
  Output rate    : 21616 bps (58 pps)

  Logical interface gr-1/3/0.0 (Index 568) (SNMP ifIndex 901)
    Flags: Up Point-To-Point SNMP-Traps 0x20024000 IP-Header 10.0.0.17:10.0.0.18:47:df:64:0000000000000000 Encapsulation: L2-over-GRE
    Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off
    Gre keepalives configured: Off, Gre keepalives adjacency state: down
    Input packets : 28868
    Output packets: 3953521
    Protocol bridge, MTU: Unlimited
      Flags: Trunk-Mode

run show interfaces gr-0/0/0   
Physical interface: gr-0/0/0, Enabled, Physical link is Up
  Interface index: 140, SNMP ifIndex: 566
  Type: GRE, Link-level type: GRE, MTU: Unlimited, Speed: 10000mbps
  Device flags   : Present Running
  Interface flags: Point-To-Point SNMP-Traps
  Input rate     : 17752 bps (45 pps)
  Output rate    : 0 bps (0 pps)

  Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 621)
    Flags: Up Point-To-Point SNMP-Traps 0x0 IP-Header 10.0.0.18:10.0.0.17:47:df:64:0000000000000000 Encapsulation: L2-over-GRE
    Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off
    Gre keepalives configured: Off, Gre keepalives adjacency state: down
    Input packets : 7362259
    Output packets: 27122
    Protocol bridge, MTU: Unlimited
      Flags: Trunk-Mode

I would appreciate any ideas

What is the MTU of  interfaces xe-0/0/1 ?

Please verify the MTU of physical interface in the path.

You can also ping with do-no-fragment knob and packet size to figure out the maximum possible MTU for the path. 

 ping 8.8.8.8 do-not-fragment size 1400 < keep increasing the size until packet drops

If you have some tcp-mss value in mind for GRE interface you can try to configure that

TEST# show interfaces gr-4/0/0
unit 0 {
family inet {
mtu 1400;
tcp-mss 600;
}
}

Usually if ethernet header is 1500 the TCP-MSS value is 1460 (1500 - IP Header and TCP Header)

Mostly, it should be some configuration issue. Please check that and let me know if you have further query.

PS: Please accpet my response as solution if it answers your query, kuods are appreciated too!

Thanks
Vishal

---

@vlsingh wrote:

What is the MTU of  interfaces xe-0/0/1 ?

MTU is 1500, but this interface is connected to the Linux server and not related with the gre tunnel.

---

@vlsingh wrote:

You can also ping with do-no-fragment knob and packet size to figure out the maximum possible MTU for the path. 

 

---

Between both routers jumbo frames is passing without issues, via tunnel max MTU is 1468 bytes.

---

@vlsingh wrote:

If you have some tcp-mss value in mind for GRE interface you can try to configure that

 

TEST# show interfaces gr-4/0/0
unit 0 {
family inet {
mtu 1400;
tcp-mss 600;
}
}

---

We are talking about family bridge (Layer 2 over GRE). With family inet and tcp-mss tune there is no issues.

Do you see "MTU errors" increasing on any interface involve in the topology when you send your traffic?

show interfaces ge-0/0/0 extensive | match "MTU errors"

Or do you see "Output MTU" counter increasing on the box when sending that traffic?

show pfe statistics traffic

Thanks Vishal

No MTU errors on both sides:

# run show interfaces xe-1/0/0 extensive | match "MTU errors"
Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0

# run show interfaces xe-0/0/3 extensive |match "MTU errors"
Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0

On second router where is connected the Linux server output mtu counter is increasing:

Packet Forwarding Engine hardware discard statistics:

    Timeout                    :                    0

    Truncated key              :                    0   

    Bits to test               :                    0        

    Data error                 :                    0

    TCP header length error    :                    0

    Stack underflow            :                    0

    Stack overflow             :                    0

    Normal discard             :               735218

    Extended discard           :                    0

    Invalid interface          :                    0

    Info cell drops            :                    0

    Fabric drops               :                    0

Packet Forwarding Engine Input IPv4 Header Checksum Error and Output MTU Error statistics:

    Input Checksum             :                    0

    Output MTU                 :                 1177

Hello,

Is that "Output MTU" drop increasing when you are sending the interested traffic?

As mentioned in the following blog:

         http://blog.ip.fi/2014/02/junos-l3-incompletes-what-and-why.html

We can check for pfe drops corresponding to "Output MTU" drop.

Can you please share following output?

start shell pfe network fpc0

show jnh 0 exception terse

exit

Thanks

Vishal

---

@vlsingh wrote:

Is that "Output MTU" drop increasing when you are sending the interested traffic?

 Yes, on each sended packet.

---

@vlsingh wrote:

Can you please share following output?

start shell pfe network fpc0

show jnh 0 exception terse

exit

From MX960:

# show jnh 0 exception terse

Reason                             Type           Packets      Bytes
==================================================================


PFE State Invalid
----------------------
unknown iif                        DISC(  1)        47216    5683339


Packet Exceptions
----------------------
ttl expired                        PUNT(  1)       131205    7460684


Bridging
----------------------
mlp pkt                            PUNT( 11)            1         29


Firewall
----------------------
firewall discard                   DISC( 67)        29239    1659966


Routing
----------------------
hold route                         DISC( 70)     53206547 2347192094
resolve route                      PUNT( 33)     11873244  535881749
control pkt punt via nh            PUNT( 34)         1182      49644
host route                         PUNT( 32)     21807739  930704110
reject route                       PUNT( 40)      3523413  197335715

From MX80:

# show jnh 0 exception terse

Reason                             Type           Packets      Bytes
==================================================================


Error Parcels
----------------------
from WAN                           M2L ERROR         6812


PFE State Invalid
----------------------
sw error                           DISC( 64)           42       2248
unknown family                     DISC( 73)         1076      83912
iif down                           DISC( 87)           11        638
unknown iif                        DISC(  1)      2463941  134519193
invalid L2 token                   DISC( 86)            1         46


Packet Exceptions
----------------------
mtu exceeded                       DISC( 21)        12086   18663387
IP options                         PUNT(  2)            2         64
my-mac check failed                DISC( 28)           12       1020


Bridging
----------------------
bridge ucast split horizon         DISC( 26)            1         90
mlp pkt                            PUNT( 11)         5361     155469
input trunk vlan lookup failed     DISC( 91)        75787    3509085


Routing
----------------------
hold route                         DISC( 70)            2        324
resolve route                      PUNT( 33)            7        748
control pkt punt via nh            PUNT( 34)     54472896 1525280380
host route                         PUNT( 32)       942262   58209572
reject route                       PUNT( 40)            2        188

"unknown iif" is increasing on each packet.

Hi Lyubomir,

I think from MX80 this output is interesting, does "MTU exceeded" increase with each packet that you are sending?

# show jnh 0 exception terse

mtu exceeded DISC( 21) 12086 18663387

You can debug this drop counter (21) to confirm if its your GRE traffic

start shell pfe network tfeb0

debug jnh exceptions 21 discard

debug jnh exceptions-trace

show jnh exceptions-trace

undebug jnh exceptions 21 discard

refer: http://blog.ip.fi/2014/02/junos-l3-incompletes-what-and-why.html

If you can see your packet in "show jnh exceptions-trace" output on MX80. That means packet is getting dropped on MX80 and you need to verify all interface MTU on MX80 again.

Thanks

Vishal

Hello,

MTU on all interfaces is > 9000 bytes. Here is the debug result when I am pinging from the server via gre tunnel MX960 router with 1470 bytes:

mtu exceeded - its increasing on each packet

# show jnh exceptions-trace
[Jan 23 20:30:01.891] [241572] jnh_exception_packet_trace:1800 ###############
[Jan 23 20:30:01.891] [241573] jnh_exception_packet_trace:1805 [iif:334,code/info:149 D(mtu exceeded)/0x5ea,score:(0x1),ptype:2/0,orig_ptype:2,offset:60,orig_offset:60,len:1558,l2iif:328,oif:337,BD 3,l2-off=180388627013,token=2791800 ]
[Jan 23 20:30:01.891] [241574] jnh_exception_packet_trace:1829 0x00: 20 01 95 00 5e a0 01 4e 2a 3c 06 16 80 01 48 20 
[Jan 23 20:30:01.891] [241575] jnh_exception_packet_trace:1829 0x10: 3c 00 03 40 02 00 00 02 45 00 00 00 00 15 10 08 
[Jan 23 20:30:01.891] [241576] jnh_exception_packet_trace:1829 0x20: 81 f4 89 15 f3 28 c0 da fa 28 a5 81 00 01 fa 08 
[Jan 23 20:30:01.891] [241577] jnh_exception_packet_trace:1829 0x30: 00 45 00 06 04 6d 00 00 00 3f 2f f5 0b b9 cd d0 
[Jan 23 20:30:01.891] [241578] jnh_exception_packet_trace:1829 0x40: 12 b9 cd d0 11 00 00 65 58 00 25 90 a2 80 ab 28 
[Jan 23 20:30:01.892] [241579] jnh_exception_packet_trace:1829 0x50: c0 da fa 2f f0 81 00 0b c1 08 00 45 00 05 da 6c 
[Jan 23 20:30:01.892] [241580] jnh_exception_packet_trace:1829 0x60: ff 40 00 40 01 98 20 5b 84 3c 01 5b 84 3c fa 00 
[Jan 23 20:30:01.892] [241581] jnh_exception_packet_trace:1829 0x70: 00 6b 66 0c 21 00 55 10 f5 29 5e 00 00 00 00 bc

iif:334 is the gre tunnel - Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 621)

show interfaces gr-0/0/0   

Physical interface: gr-0/0/0, Enabled, Physical link is Up

  Interface index: 140, SNMP ifIndex: 566

  Description: S3C-TEST

  Type: GRE, Link-level type: GRE, MTU: Unlimited, Speed: 10000mbps

  Device flags   : Present Running

  Interface flags: Point-To-Point SNMP-Traps

  Input rate     : 43096 bps (69 pps)

  Output rate    : 23664 bps (6 pps)

  Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 621)

    Flags: Up Point-To-Point SNMP-Traps Reassemble-Pkts 0x0 IP-Header xx.xx.xx.xx:xx.xx.xx.xx:47::64:0000000000000000 Encapsulation: L2-over-GRE

    Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off

    Gre keepalives configured: Off, Gre keepalives adjacency state: down

    Input packets : 131830202

    Output packets: 2781297

    Protocol bridge, MTU: Unlimited

      Flags: Trunk-Mode

Hi Lyubomir,

## Here is the ICMP reply packets which is geting dropped:

Frame 1 (97 bytes on wire, 97 bytes captured)
Ethernet II, Src: 28:c0:da:fa:28:a5 (28:c0:da:fa:28:a5), Dst: 08:81:f4:89:15:f3 (08:81:f4:89:15:f3)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 506
Internet Protocol, Src: 185.205.208.18 (185.205.208.18), Dst: 185.205.208.17 (185.205.208.17)
Generic Routing Encapsulation (Transparent Ethernet bridging)
Ethernet II, Src: 28:c0:da:fa:2f:f0 (28:c0:da:fa:2f:f0), Dst: SuperMic_a2:80:ab (00:25:90:a2:80:ab)
802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3009
Internet Protocol, Src: 91.132.60.1 (91.132.60.1), Dst: 91.132.60.250 (91.132.60.250)
Internet Control Message Protocol
Checksum: 0x6b66 [incorrect, should be 0xfd35]
0000 10 f5 29 5e 00 00 00 00 bc ..)

## Incoming interface is 334 ( gr-0/0/0.0 ), outgoing interface is (337), packet length is 1558 and its getting dropped due to MTU issue.  Please check the MTU for interface index 337 and 328. Please check MTU for interface in that bridge domain.

jnh_exception_packet_trace:1805 [iif:334,code/info:149 D(mtu exceeded)/0x5ea,score:(0x1),ptype:2/0,orig_ptype:2,offset:60,orig_offset:60,len:1558,l2iif:328,oif:337,BD 3,l2-off=180388627013,token=2791800 ]

## If there is no MTU misconfiguration you can open a JTAC case.

PS: Please accpet my response as solution if it answers your query, kudos are appreciated too!

Thanks
Vishal

Hello,

---

@Lyubomir wrote:

Already tried allow-fragmentation and do-not-fragment tunnel options, but nothing changes.

 

---

You need to use 2 knobs together: "allow-fragmentation" and "reassemble-packets" 

https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/reassemble-packets-edit-interfaces.html

One more stupid question - do You use DPC/I-CHIP cards for GRE tunneling, by any chance?

ICHIP cards do not support inline GRE reassembly, only Trio cards do.

HTH

Thx

Alex

---

@aarseniev wrote:

Hello,

 

---

@Lyubomir wrote:

Already tried allow-fragmentation and do-not-fragment tunnel options, but nothing changes.

 

---

You need to use 2 knobs together: "allow-fragmentation" and "reassemble-packets" 

https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/reassemble-packets-edit-interfaces.html

---

Just added in both ends "allow-fragmentation" and "reassemble-packets", but still max MTU which pass tunnel is 1468.

---

@aarseniev wrote:

One more stupid question - do You use DPC/I-CHIP cards for GRE tunneling, by any chance?

ICHIP cards do not support inline GRE reassembly, only Trio cards do.

---

Router 1 is with MPC 3D 16x 10GE

Router 2 is MX80 with integrated 4 x 10GE