Ethernet Switching
Highlighted
Ethernet Switching

qfx-5200 COS (QOS) drops packets

‎03-31-2020 03:26 PM

Hello

Can one of you QOS engineers "un-dumb" me ?

we have a QFX-5200 with HCI cluster (all 100g) and 80G channel uplinks to the "old" core (in migration from cisco N9K to the qfx)

we recently moved the Firewall connection this QFX, there a EX4600 in the middle

so Old core 80g > QFX (with hci direct) > 10g link to ex4600 > 10g link to FW

 

this 10g link is running at about 1gig all the time during peak hours. so about 10%

the defualt OQS on the QFX was dropping 300-500pps

I put QOS on it and no matter what I adjust it still drops packets.

 

since it is traffic to the Inet firewall most of it is BE

set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VOIP-EF loss-priority low code-points cs5
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VOIP-EF loss-priority low code-points ef
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class CONTROL-CS6-CS7 loss-priority low code-points cs6
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class CONTROL-CS6-CS7 loss-priority low code-points cs7
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points cs3
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points cs4
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points af31
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points af32
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points af33
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points af41
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points af42
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class VIDEO-AF41 loss-priority low code-points af43
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points cs1
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points cs2
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points af11
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points af12
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points af13
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points af21
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points af22
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class DATA-AF21 loss-priority low code-points af23
set groups cos-COS class-of-service classifiers dscp cos-dscp-classifier forwarding-class Best-Effort-0 loss-priority high code-points be
set groups cos-COS class-of-service drop-profiles Best-effort interpolate fill-level 98
set groups cos-COS class-of-service drop-profiles Best-effort interpolate fill-level 100
set groups cos-COS class-of-service drop-profiles Best-effort interpolate drop-probability 0
set groups cos-COS class-of-service drop-profiles Best-effort interpolate drop-probability 100
set groups cos-COS class-of-service forwarding-classes class VOIP-EF queue-num 5
set groups cos-COS class-of-service forwarding-classes class VIDEO-AF41 queue-num 2
set groups cos-COS class-of-service forwarding-classes class CONTROL-CS6-CS7 queue-num 7
set groups cos-COS class-of-service forwarding-classes class DATA-AF21 queue-num 1
set groups cos-COS class-of-service forwarding-classes class Best-Effort-0 queue-num 0
set groups cos-COS class-of-service interfaces xe-*/*/* scheduler-map cos-sched-map
set groups cos-COS class-of-service interfaces xe-*/*/* unit 0 classifiers dscp cos-dscp-classifier
set groups cos-COS class-of-service interfaces ae255 scheduler-map cos-sched-map
set groups cos-COS class-of-service interfaces ae255 unit 0 classifiers dscp cos-dscp-classifier
set groups cos-COS class-of-service interfaces et-*/*/* unit 0 classifiers dscp cos-dscp-classifier
set groups cos-COS class-of-service rewrite-rules dscp Branch-Rewrite forwarding-class VOIP-EF loss-priority low code-point ef
set groups cos-COS class-of-service rewrite-rules dscp Branch-Rewrite forwarding-class VIDEO-AF41 loss-priority low code-point af41
set groups cos-COS class-of-service rewrite-rules dscp Branch-Rewrite forwarding-class CONTROL-CS6-CS7 loss-priority low code-point cs7
set groups cos-COS class-of-service rewrite-rules dscp Branch-Rewrite forwarding-class DATA-AF21 loss-priority low code-point af21
set groups cos-COS class-of-service rewrite-rules dscp Branch-Rewrite forwarding-class Best-Effort-0 loss-priority low code-point 000000
set groups cos-COS class-of-service scheduler-maps cos-sched-map forwarding-class VOIP-EF scheduler cos-VOIP-scheduler
set groups cos-COS class-of-service scheduler-maps cos-sched-map forwarding-class VIDEO-AF41 scheduler cos-video-scheduler
set groups cos-COS class-of-service scheduler-maps cos-sched-map forwarding-class CONTROL-CS6-CS7 scheduler cos-control-scheduler
set groups cos-COS class-of-service scheduler-maps cos-sched-map forwarding-class DATA-AF21 scheduler cos-data-scheduler
set groups cos-COS class-of-service scheduler-maps cos-sched-map forwarding-class Best-Effort-0 scheduler cos-best-effort
set groups cos-COS class-of-service schedulers cos-VOIP-scheduler transmit-rate percent 1
set groups cos-COS class-of-service schedulers cos-VOIP-scheduler buffer-size percent 1
set groups cos-COS class-of-service schedulers cos-VOIP-scheduler priority low
set groups cos-COS class-of-service schedulers cos-video-scheduler transmit-rate percent 2
set groups cos-COS class-of-service schedulers cos-video-scheduler buffer-size percent 2
set groups cos-COS class-of-service schedulers cos-video-scheduler priority low
set groups cos-COS class-of-service schedulers cos-control-scheduler transmit-rate percent 1
set groups cos-COS class-of-service schedulers cos-control-scheduler buffer-size percent 2
set groups cos-COS class-of-service schedulers cos-control-scheduler priority low
set groups cos-COS class-of-service schedulers cos-data-scheduler transmit-rate percent 1
set groups cos-COS class-of-service schedulers cos-data-scheduler buffer-size percent 1
set groups cos-COS class-of-service schedulers cos-data-scheduler priority low
set groups cos-COS class-of-service schedulers cos-best-effort transmit-rate percent 95
set groups cos-COS class-of-service schedulers cos-best-effort buffer-size remainder
set groups cos-COS class-of-service schedulers cos-best-effort priority low

and what im seeing

 Egress queues: 10 supported, 8 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                        109550315            109355079               195236
    1                           316138               316138                    0
    2                           399357               399357                    0
    3                                0                    0                    0
    4                                0                    0                    0
    5                          1038004              1038004                    0
    7                              833                  833                    0
    8                                1                    1                    0
  Queue number:         Mapped forwarding classes
    0                   Best-Effort-0
    1                   DATA-AF21
    2                   VIDEO-AF41
    3                   fcoe
    4                   no-loss
    5                   VOIP-EF
    7                   CONTROL-CS6-CS7
    8                   mcast           

 CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 Best-Effort-0          95     9500000000     r              0      low    none
    1 DATA-AF21               1      100000000     1              0      low    none
    2 VIDEO-AF41              2      200000000     2              0      low    none
    5 VOIP-EF                 1      100000000     1              0      low    none
    7 CONTROL-CS6-CS7         1      100000000     2              0      low    none
Physical interface: xe-0/0/26:2, Enabled, Physical link is Up
  Interface index: 653, SNMP ifIndex: 554
Forwarding classes: 12 supported, 8 in use
Egress queues: 10 supported, 8 in use
Queue: 0, Forwarding classes: Best-Effort-0
  Queued:
    Packets              :             117233136                 79998 pps
    Bytes                :          169432766982             936969056 bps
  Transmitted:
    Packets              :             117022062                 79746 pps
    Bytes                :          169111887825             933907152 bps
    Tail-dropped packets : Not Available  
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                211074                   252 pps
    Total-dropped bytes  :             320879157               3061904 bps
Queue: 1, Forwarding classes: DATA-AF21
  Queued:
    Packets              :                339222                   201 pps
    Bytes                :             213323209               1471928 bps
  Transmitted:
    Packets              :                339222                   201 pps
    Bytes                :             213323209               1471928 bps
    Tail-dropped packets : Not Available  
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps
Queue: 2, Forwarding classes: VIDEO-AF41
  Queued:
    Packets              :                430761                   300 pps
    Bytes                :             262409561               1547872 bps
  Transmitted:
    Packets              :                430761                   300 pps
    Bytes                :             262409561               1547872 bps
    Tail-dropped packets : Not Available  
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps

so i know the ques look funny but i'm trying everything here. 99% does seem to be um BE

I CAN add another link or make it 40G, but that just moves the issues to the EX4600

im hoping Im just QOS DUMB and am forgetting something or there is something i need to do on the QFX that normal EX do not need

PS: I know the drop profile is not applied, in the past having a drop profile would help in this case it has not helped.

 

Thank you in advance for you time

3 REPLIES 3
Highlighted
Ethernet Switching

Re: qfx-5200 COS (QOS) drops packets

‎04-01-2020 07:08 AM

#1 I assume you are using a QFX5120-32C, yes?  There are differences, beyond just interface speeds between 48Y and 32C, so always best to call out full model #.

 

#2 These are egress drops.  From you description, you have a 100GE pipe (or maybe multiple 10GE pipes) that eventually go down to a 10GE pipe to your FW.  I fully expect that no matter what your QoS/CoS settings are that drops will still occur, as the QFX5120 buffers will always be over-run.

 

I think maybe you need to see what exact traffic is being sent to and then out of this 10GE interface to see if all of this is valid. For a potential example, there could be some BUM or Multicast traffic coming to and out this interface, which is not really needed.  Cleaning this up might help, or putting this traffic in BE and real traffic to FW in some other queue might help.

 

Additionally I am not sure if Flow-Control (pause frames) might help and actually work on the Ingress 100GE interfaces.  I am not even sure if 100GE support Flow-Control/pause frames.

 

Just my 2 cents worth.  Good luck.

Highlighted
Ethernet Switching

Re: qfx-5200 COS (QOS) drops packets

[ Edited ]
‎04-01-2020 08:24 AM

1. no, QFX5200-32C-32c (10/25/50/40/100g)

 

2. the server are on 100G MOST of there traffic traffic does not hit the 10G, the traffic that is going out is any INET/DMZ traffic to the firewalls, that is and 80G link off our old core (we moved the firewalls to the QFX as part 1 of the migration) 

 

would it be better to only have 2 ques at this location maybe? or is it just way to much? 

Highlighted
Ethernet Switching

Re: qfx-5200 COS (QOS) drops packets

‎04-01-2020 09:32 AM

I would think 2 or more queues could not hurt, but I still think with potential 80GE in and 10GE out, you'll still see queue over-runs, which equals dropped packets.

Feedback