Ethernet Switching
Highlighted
Ethernet Switching

QFX5100 packet drop

‎05-18-2020 03:21 AM

Hi, Experts,

 

I found the QFX5100 interfaces xe-0/0/ has packet drop when connecting to a WAN link  ( oppsite partner = Cisco device, for our case).

But there is no packet drop when connecting to a LAN Link ( opposite partner = Cisco device,. also ).

 

Interfaces (MTU size is default)  in QFX5100 and Cisco device have no any error, like CRC, input/out, RUNT/Giant, unknown packet drop, unknown protocol drop... 

 

No QoS configuration in our devices ( Juniper and Cisco ).

 

Any advice for investigation of this problem (packet drop), thx so much for your kind help.

 

 

 

13 REPLIES 13
Highlighted
Ethernet Switching

Betreff: QFX5100 packet drop

‎05-18-2020 03:27 AM

Hello Ben Ben,

 

"packet drop" is quite general. Please post an output of "show interfaces xe-0/0/X extensive", so that we can see more details.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution".
If you think that my answer was helpful, please spend some Kudos.
Highlighted
Ethernet Switching

Re: QFX5100 packet drop

‎05-18-2020 06:55 AM

Hello Ben,

 

Need more details of the impact, but I can share a couple KBs with examples of packet drops on QFX5100s.

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB35334&actp=RSS

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB30744&actp=METADATA

 

About the links, I recommend packet captures on both scenarios and compare the packets to see what is different to isolate the issue.

 

Randall,

 

Highlighted
Ethernet Switching

Betreff: QFX5100 packet drop

[ Edited ]
‎05-20-2020 04:04 AM

Hi, Guys,

From Cisco switch pinged to QFX5100:

 

C38SwInTT#ping 10.88.8.2 source 10.88.8.3 size 1500 r 1000
Type escape sequence to abort.
Sending 1000, 1500-byte ICMP Echos to 10.88.8.2, timeout is 2 seconds:
Packet sent with a source address of 10.88.8.3
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!...!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!
Success rate is 98 percent (987/1000), round-trip min/avg/max = 1/11/70 ms

 

 

No CRC error, input/out errors ...no any error found int QFX5100 interfaces and Cisco switch;

but these devices are connected through a WAN link/private line. 

 

 

Below is the QFX5100 interfaces status ( Output errors are not increasing 😞

 

QFX5100> show interfaces xe-0/0/37 extensive
Physical interface: xe-0/0/37, Enabled, Physical link is Up
Interface index: 766, SNMP ifIndex: 570, Generation: 259
Description: TO_001DGD_1G
Link-level type: Ethernet, MTU: 1514, MRU: 0, Link-mode: Auto, Speed: 1000 Mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Disabled,
Remote fault: Offline, Media type: Copper
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Link flags : None
CoS queues : 12 supported, 12 maximum usable queues
Hold-times : Up 0 ms, Down 0 ms
Current address: e4:fc:82:6c:57:68, Hardware address: e4:fc:82:6c:57:68
Last flapped : 2020-05-12 14:57:15 HKT (6d 08:37 ago)
Statistics last cleared: 2020-05-14 22:28:42 HKT (4d 01:05 ago)
Traffic statistics:
Input bytes : 1599668834797 34872280 bps
Output bytes : 1838308564215 41129296 bps
Input packets: 11155925208 29156 pps
Output packets: 10141971911 26492 pps
IPv6 transit statistics:
Input bytes : 0
Output bytes : 0
Input packets: 0
Output packets: 0
Input errors:
Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Bucket drops: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
Output errors:
Carrier transitions: 0, Errors: 0, Drops: 28, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0, Bucket drops: 0
Egress queues: 12 supported, 5 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 0 10141952867 28
3 0 0 0
4 0 2 0
7 0 12598 0
8 0 52789 0
Queue number: Mapped forwarding classes
0 best-effort
3 fcoe
4 no-loss
7 network-control
8 mcast
Active alarms : None
Active defects : None
MAC statistics: Receive Transmit
Total octets 1599668834797 1838308564215
Total packets 11155925208 10141971911
Unicast packets 11154927756 10141900604
Broadcast packets 108553 58148
Multicast packets 888899 13159
CRC/Align errors 0 0
FIFO errors 0 0
MAC control frames 0 0
MAC pause frames 0 0
Oversized frames 0
Jabber frames 0
Fragment frames 0
VLAN tagged frames 11155856318
Code violations 0
MAC Priority Flow Control Statistics:
Priority : 0 0 0
Priority : 1 0 0
Priority : 2 0 0
Priority : 3 0 0
Priority : 4 0 0
Priority : 5 0 0
Priority : 6 0 0
Priority : 7 0 0
Filter statistics:
Input packet count 0
Input packet rejects 0
Input DA rejects 0
Input SA rejects 0
Output packet count 0
Output packet pad count 0
Output packet error count 0
CAM destination filters: 1, CAM source filters: 0
Autonegotiation information:
Negotiation status: Complete
Link partner:
Link mode: Full-duplex, Flow control: None, Remote fault: OK, Link partner Speed: 1000 Mbps
Local resolution:
Flow control: None, Flow control tx: Disabled, Flow control rx: Disabled, Remote fault: Link OK
Packet Forwarding Engine configuration:
Destination slot: 0 (0x00)
CoS information:
Direction : Output
CoS transmit queue Bandwidth Buffer Priority Limit
% bps % usec
0 best-effort 5 50000000 5 0 low none
3 fcoe 35 350000000 35 0 low none
4 no-loss 35 350000000 35 0 low none
7 network-control 5 50000000 5 0 low none
8 mcast 20 200000000 20 0 low none
Interface transmit statistics: Disabled

Logical interface xe-0/0/37.0 (Index 711) (SNMP ifIndex 815) (Generation 377)
Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
Traffic statistics:
Input bytes : 15772708
Output bytes : 20578584
Input packets: 24698
Output packets: 31082
Local statistics:
Input bytes : 15772708
Output bytes : 20578584
Input packets: 24698
Output packets: 31082
Transit statistics:
Input bytes : 0 0 bps
Output bytes : 0 0 bps
Input packets: 0 0 pps
Output packets: 0 0 pps
Protocol eth-switch, MTU: 1514, Generation: 378, Route table: 6
Flags: Trunk-Mode

{master:0}

Highlighted
Ethernet Switching

Betreff: QFX5100 packet drop

‎05-20-2020 04:41 AM

Hi Ben, 

From the outputs, we see drops in the output errors, this seems to me like L1 issue. 
Please check the transceiver and clean the port and reseat the optics. Clear the counters and check the status after performing all the steps. Framing errors are mostly issue at physical layer.

Output errors:
Carrier transitions: 0, Errors: 0, Drops: 28, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0, Bucket drops: 0

Errors—Sum of the outgoing frame aborts and FCS errors.
Link: https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/show-interfaces-g...

 

Please mark "Accept as solution" if this answers your query.  Kudos are appreciated too! 

 

Regards, 

Sharat Ainapur

Highlighted
Ethernet Switching

Betreff: QFX5100 packet drop

[ Edited ]
‎05-20-2020 05:02 AM

Hi,

It seems not this issue, this counter is not increasing.

 

But will check this.

 

Thanks so much for your kind help

Highlighted
Ethernet Switching

Betreff: QFX5100 packet drop

‎05-20-2020 05:03 AM

Hi Ben

 

If it is point to point link between the qfx5100 and cisco and you are trying to ping an interface on qfx5100 (which is a host bound icmp packet) then you may see 2% drop because of the policing of the icmp packet towards the qfx5100 host path of the RE.

 

However if the qfx5100 is acting as a transit for the packet then you would not see any drops.

 

Hope this explains the behavior.

Highlighted
Ethernet Switching

Betreff: QFX5100 packet drop

‎05-20-2020 05:06 AM

Hi Ben,

 

Can you share this output?

 

show interfaces queue xe-0/0/37

 

Regards,

 

Randall

Highlighted
Ethernet Switching

Re: QFX5100 packet drop

‎05-20-2020 09:47 AM

Hello Ben Ben 


From the interface xe-0/0/37 extensive you point out I have notice the following :

On the queue # 0 that is the best effort queue it is the only one where we are having drops.

Queue counters: Queued packets Transmitted packets Dropped packets
0 0 10141952867 28
3 0 0 0
4 0 2 0
7 0 12598 0
8 0 52789 0
Queue number: Mapped forwarding classes
0 best-effort
3 fcoe
4 no-loss
7 network-control
8 mcast

 

This best effort queue is using the default values for the transmition of the traffic and it is 5%


CoS transmit queue Bandwidth Buffer Priority Limit
% bps % usec
0 best-effort 5 50000000 5 0 low none

Base on this could you tell me if the drops are happening on the switch right now or if they intermitent or during specifc time frames on the day ?

There could be multiple reasons for the Drops to happen for example High CPU, interface flaps etc, however as they are quite low and only on this queue i would point out you have this case the traffic is exedding the 5%  available by default for the Best effort by default. 

Also please share the command show interfaces queue xe-0/0/37 in order to debug even more as for drops detail information is needed in order to get the proper Diagnostic. 

 

 

Highlighted
Ethernet Switching

Betreff: QFX5100 packet drop

[ Edited ]
‎05-20-2020 08:16 PM

Hi, Randero.

Here is the output of the command:

QFX5100> show interfaces queue xe-0/0/37
Physical interface: xe-0/0/37, Enabled, Physical link is Up
Interface index: 766, SNMP ifIndex: 570
Description: TO_001DGD_1G
Forwarding classes: 16 supported, 5 in use
Egress queues: 12 supported, 5 in use
Queue: 0, Forwarding classes: best-effort
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 15066280087 28437 pps
Bytes : 2778694077821 42009072 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 28 0 pps
Total-dropped bytes : 26660 0 bps
Queue: 3, Forwarding classes: fcoe
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 0 0 pps
Bytes : 0 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 4, Forwarding classes: no-loss
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 2 0 pps
Bytes : 136 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 7, Forwarding classes: network-control
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 18748 0 pps
Bytes : 7761672 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps
Queue: 8, Forwarding classes: mcast
Queued:
Packets : 0 0 pps
Bytes : 0 0 bps
Transmitted:
Packets : 77108 0 pps
Bytes : 4998047 0 bps
Tail-dropped packets : Not Available
RL-dropped packets : 0 0 pps
RL-dropped bytes : 0 0 bps
Total-dropped packets: 0 0 pps
Total-dropped bytes : 0 0 bps

 

 

------------------------------------------------------------------------------------------------------------

Hi, guys,

 

I do not think this is problem of the QoS, CoS..due to the normal traffic is not high ( most often the BW utilization is less than 20% of the link capacity ).

 

 

 

 

Highlighted
Ethernet Switching

Re: QFX5100 packet drop

‎05-20-2020 09:00 PM

Hi, guys,

 

When I replaced the QFX5100 with a Cisco Switch/Router ( like Cisco 2960x) , there is no packet drop...so it is strange.

 

 

 

 

 

 

 

Highlighted
Ethernet Switching

Re: QFX5100 packet drop

‎05-21-2020 05:15 AM

@Ben Ben - could you please send your QoS config on the QFX5100.  'show configuration class-of-service' output into a word type doc.

 

I see you have 5 forwarding-classes and 5 queues configured - sort of wondering why, especially something like fcoe, which I see has no packets being rcv/sent but could be holding queue space.

 

I am not at all sure, if this indeed an area of concern for you, but maybe worth looking at.

 

At the same time, do you have a JTAC case open for this, and what are they saying?

Highlighted
Ethernet Switching

Re: QFX5100 packet drop

[ Edited ]
‎05-21-2020 07:58 PM

Hi, RCCPGM,

 

The QoS/CoS of all interfaces are default setting.

 

 

 

Highlighted
Ethernet Switching

Re: QFX5100 packet drop

[ Edited ]
‎05-21-2020 11:37 PM

Hi Ben,

 

Greetings!
I have been through the post and see that you are seeing a packet loss of ~2% when you are pinging with 1000 packets from the cisco side. This is not seen when you replace the Juniper device with the cisco switch. This doesn't look like a software bug, nor a problem with the link. Instead, the drops occurred on the device might be because of the default rate-limiting of ICMP packets on Junos platforms.

 

Juniper is receiving all of the ICMP request packets and at the same time generating ICMP replies to send back to Cisco. However, during this process, it is hitting the default rate limit of ICMP within the kernel, which is on the routing-engine. The default ICMP rate limit on the system can be checked using the start shell command. If the 1000 ping requests sent by the Cisco side come in in less than 3 seconds, there is a chance that some ICMP requests will be discarded by the Juniper device based on the rate setting as below. This is a mechanism to protect the CPU from being affected by an ICMP flood attack.

 

# sysctl -a | grep "inet.icmp"
net.inet.icmp.maskrepl: 0
net.inet.icmp.bucketsize: 5
net.inet.icmp.tokenrate: 1000
net.inet.icmp.drop_redirect: 0
net.inet.icmp.log_redirect: 0
net.inet.icmp.traceroute_l3vpn: 0
net.inet.icmp.bmcastecho: 1

 

Notice in the output above that the ICMP rate is set to 1000; this is the default system value.

 

However to verify if the packets are being dropped due to rate limiting, you can use the command as below.

> show system statistics icmp

 

However ICMP is not the best way to test the delay or capacity on the link. One of the best ways to test the delay or capacity is to attach a host or traffic generator to both sides. Then traffic can be sent through, or transit to, the devices. By taking a transit path through the router, there is not any default rate limiting that would limit the traffic.

 


Regards,
Vishaal


Accept as Solution = cool ! (Help fellow community members with similar query be redirected here instead of them reposting again)
Accept as Solution+Kudo = You are a Star !
Feedback