Ethernet Switching
Highlighted
Ethernet Switching

QFX5100 interface drops

‎03-21-2018 09:23 AM

Hello,

 

I have a pair of QFX5100's configured as MC-LAG peers. I have an MC-LAG going up to an MX5 router - ge-0/0/41 on both QFX5100-A and QFX5100-B connect to MX5.

 

The problem is I'm seeing drops in my best-effort queue even though I'm pushing less than 500Mbps through the interface. I'm using iperf to generate UDP traffic. When I push more than 100Mbps, I'm seeing thousands of drops on my ge-0/0/41 interfaces (the mc-ae interface is ae8). Here is my QoS configuration:

admin@QFX5100-B# show class-of-service
classifiers {
    dscp CLASSIFIER-DSCP {
        forwarding-class FC-BE {
            loss-priority low code-points [ 000000 000001 000010 000011 000100 000101 000110 000111 001000 001001 001010 001011 001100 001101 001110 001111 010000 010001 010010 010100 010011 010101 010110 010111 011000 011100 011001 011010 011011 011101 011110 011111 100000 100001 100011 100100 100101 100110 100111 101000 101001 101010 101011 101100 101101 101111 110001 110010 110011 110100 110101 110110 110111 111000 111001 111010 111011 111100 111101 111110 111111 ];
        }
        forwarding-class FC-AF {
            loss-priority low code-points 100010;
        }
        forwarding-class FC-NC {
            loss-priority low code-points 110000;
        }
        forwarding-class FC-EF {
            loss-priority low code-points 101110;
        }
    }
}
drop-profiles {
    DP-STANDARD {
        interpolate {
            fill-level [ 95 99 ];
            drop-probability [ 0 50 ];
        }
    }
}
shared-buffer {
    ingress {
        percent 100;
        buffer-partition lossless {
            percent 5;
        }
        buffer-partition lossless-headroom {
            percent 0;
        }
        buffer-partition lossy {
            percent 95;
        }
    }
    egress {
        percent 100;
        buffer-partition lossless {
            percent 5;
        }
        buffer-partition lossy {
            percent 75;
        }
        buffer-partition multicast {
            percent 20;
        }
    }
}
forwarding-classes {
    class FC-BE queue-num 0;
    class FC-AF queue-num 1;
    class FC-NC queue-num 2;
    class FC-EF queue-num 3;
}
traffic-control-profiles {
    TC-PROFILE-STANDARD {
        scheduler-map SCHEDMAP-STANDARD;
    }
    TC-PROFILE-STRICT {
        scheduler-map SCHEDMAP-STRICT;
    }
}
forwarding-class-sets {
    FC-SET-STANDARD {
        class FC-BE;
        class FC-AF;
        class FC-NC;
    }
    FC-SET-STRICT {
        class FC-EF;
    }
}
interfaces {
    ge-* {
        forwarding-class-set {
            FC-SET-STANDARD {
                output-traffic-control-profile TC-PROFILE-STANDARD;
            }
            FC-SET-STRICT {
                output-traffic-control-profile TC-PROFILE-STRICT;
            }
        }
        unit 0 {
            classifiers {
                dscp CLASSIFIER-DSCP;
            }
        }
    }
    xe-* {
        forwarding-class-set {
            FC-SET-STANDARD {
                output-traffic-control-profile TC-PROFILE-STANDARD;
            }
            FC-SET-STRICT {
                output-traffic-control-profile TC-PROFILE-STRICT;
            }
        }
        unit 0 {
            classifiers {
                dscp CLASSIFIER-DSCP;
            }
        }
    }
    ae* {
        forwarding-class-set {
            FC-SET-STANDARD {
                output-traffic-control-profile TC-PROFILE-STANDARD;
            }
            FC-SET-STRICT {
                output-traffic-control-profile TC-PROFILE-STRICT;
            }
        }
        unit 0 {
            classifiers {
                dscp CLASSIFIER-DSCP;
            }
        }
    }
}
scheduler-maps {
    SCHEDMAP-STANDARD {
        forwarding-class FC-BE scheduler SCHED-BE;
        forwarding-class FC-AF scheduler SCHED-AF;
        forwarding-class FC-NC scheduler SCHED-NC;
    }
    SCHEDMAP-STRICT {
        forwarding-class FC-EF scheduler SCHED-EF;
    }
}
schedulers {
    SCHED-BE {
        transmit-rate percent 80;
        buffer-size {
            remainder;
        }
        priority low;
        drop-profile-map loss-priority low protocol any drop-profile DP-STANDARD;
    }
    SCHED-AF {
        transmit-rate percent 5;
        buffer-size percent 5;
        priority low;
        drop-profile-map loss-priority low protocol any drop-profile DP-STANDARD;
    }
    SCHED-NC {
        transmit-rate percent 5;
        buffer-size percent 5;
        priority low;
        drop-profile-map loss-priority low protocol any drop-profile DP-STANDARD;
    }
    SCHED-EF {
        shaping-rate percent 10;
        buffer-size percent 0;
        priority strict-high;
    }
}

Here is the interface stats:

admin@QFX5100-B# run show interfaces ge-0/0/41 extensive
Physical interface: ge-0/0/41, Enabled, Physical link is Up
  Interface index: 670, SNMP ifIndex: 651, Generation: 161
  Description: L3 Link to MX5-B - NeutralPath
  Link-level type: Ethernet, MTU: 1514, MRU: 0, Speed: Auto, Duplex: Auto, BPDU Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled,
  Auto-negotiation: Enabled, Remote fault: Online, Media type: Copper
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 28:a2:4b:42:33:d8, Hardware address: 28:a2:4b:42:33:0c
  Last flapped   : 2018-02-14 22:47:30 CST (4w6d 11:44 ago)
  Statistics last cleared: 2018-03-21 09:35:39 CDT (01:55:53 ago)
  Traffic statistics:
   Input  bytes  :           1584906758                14984 bps
   Output bytes  :            943837004               101200 bps
   Input  packets:              1214425                   19 pps
   Output packets:              1471527                   96 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Bucket drops: 0, Policed discards: 0,
    L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 12360, Collisions: 0, Aged packets: 0, FIFO errors: 0,
    HS link CRC errors: 0, MTU errors: 0, Resource errors: 0, Bucket drops: 0
  Egress queues: 12 supported, 7 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0              1307498                12360
    1                                0                    0                    0
    2                                0                   14                    0
    3                                0                   39                    0
    4                                0                    0                    0
    7                                0               160280                    0
    8                                0                    0                    0
  Queue number:         Mapped forwarding classes
    0                   FC-BE
    1                   FC-AF
    2                   FC-NC
    3                   FC-EF
    4                   no-loss
    7                   network-control
    8                   mcast
  Active alarms  : None
  Active defects : None
  MAC statistics:                      Receive         Transmit
    Total octets                    1584906758        943837004
    Total packets                      1214425          1471527
    Unicast packets                    1207472          1460998
    Broadcast packets                        0               23
    Multicast packets                     6953            10506
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                 1207472
    Code violations                          0
  MAC Priority Flow Control Statistics:
    Priority :  0                             0                0
    Priority :  1                             0                0
    Priority :  2                             0                0
    Priority :  3                             0                0
    Priority :  4                             0                0
    Priority :  5                             0                0
    Priority :  6                             0                0
    Priority :  7                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 1, CAM source filters: 0
  Autonegotiation information:
    Negotiation status: Complete
    Link partner:
        Link mode: Full-duplex, Flow control: None, Remote fault: OK, Link partner Speed: 1000 Mbps
    Local resolution:
        Flow control: None, Flow control tx: None, Flow control rx: None, Remote fault: Link OK
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  Interface transmit statistics: Disabled

  Logical interface ge-0/0/41.0 (Index 552) (SNMP ifIndex 662) (HW Token 2147483649) (Generation 184)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Traffic statistics:
     Input  bytes  :                    0
     Output bytes  :               389618
     Input  packets:                    0
     Output packets:                 2861
    Local statistics:
     Input  bytes  :                    0
     Output bytes  :               389618
     Input  packets:                    0
     Output packets:                 2861
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol aenet, AE bundle: ae8.0, Generation: 208, Route table: 0

And here are some QoS output commnads:

admin@QFX5100-B# run show interfaces queue ge-0/0/41
Physical interface: ge-0/0/41, Enabled, Physical link is Up
  Interface index: 670, SNMP ifIndex: 651
  Description: L3 Link to DAL-MX5-B - NeutralPath
Forwarding classes: 16 supported, 7 in use
Egress queues: 12 supported, 7 in use
Queue: 0, Forwarding classes: FC-BE
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :               1317060                    36 pps
    Bytes                :             933203718                 45760 bps
    Tail-dropped packets : Not Available
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                 12360                     0 pps
    Total-dropped bytes  :              17389225                     0 bps
Queue: 1, Forwarding classes: FC-AF
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
    Tail-dropped packets : Not Available
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps
Queue: 2, Forwarding classes: FC-NC
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :                    14                     0 pps
    Bytes                :                  1372                     0 bps
    Tail-dropped packets : Not Available
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps
Queue: 3, Forwarding classes: FC-EF
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :                    39                     0 pps
    Bytes                :                  3822                     0 bps
    Tail-dropped packets : Not Available
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps
Queue: 4, Forwarding classes: no-loss
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
    Tail-dropped packets : Not Available
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps
Queue: 7, Forwarding classes: network-control
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :                162251                    22 pps
    Bytes                :              12489518                 13936 bps
    Tail-dropped packets : Not Available
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps
Queue: 8, Forwarding classes: mcast
  Queued:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
  Transmitted:
    Packets              :                     0                     0 pps
    Bytes                :                     0                     0 bps
    Tail-dropped packets : Not Available
    RL-dropped packets   :                     0                     0 pps
    RL-dropped bytes     :                     0                     0 bps
    Total-dropped packets:                     0                     0 pps
    Total-dropped bytes  :                     0                     0 bps
admin@QFX5100-B# run show class-of-service interface ge-0/0/41
Physical interface: ge-0/0/41, Index: 670
Queues supported: 12, Queues in use: 7
  Congestion-notification: Disabled
  Forwarding class set: FC-SET-STANDARD, Index: 52944
    Output traffic control profile: TC-PROFILE-STANDARD, Index: 36518
  Forwarding class set: FC-SET-STRICT, Index: 43469
    Output traffic control profile: TC-PROFILE-STRICT, Index: 46173

  Logical interface: ge-0/0/41.0, Index: 552
Object                  Name                   Type                    Index
Classifier              CLASSIFIER-DSCP        dscp                    49277
Classifier              ieee8021p-default      ieee8021p                  11
admin@QFX5100-B# run show class-of-service shared-buffer
Ingress:
  Total Buffer     :  12480.00 KB
  Dedicated Buffer :  2912.81 KB
  Shared Buffer    :  9567.19 KB
    Lossless          :  478.36 KB
    Lossless Headroom :  0.00 KB
    Lossy             :  9088.83 KB

  Lossless Headroom Utilization:
  Node Device         Total          Used                  Free
  0                   0.00 KB        0.00 KB               0.00 KB

Egress:
  Total Buffer     :  12480.00 KB
  Dedicated Buffer :  3744.00 KB
  Shared Buffer    :  8736.00 KB
    Lossless          :  436.80 KB
    Multicast         :  1747.20 KB
    Lossy             :  6552.00 KB

Finally, the interface config:

admin@QFX5100-B# show interfaces ae8
description "L3 Link to DAL-MX5-B - NeutralPath";
aggregated-ether-options {
    lacp {
        active;
        system-id 00:00:00:00:00:08;
        admin-key 1;
    }
    mc-ae {
        mc-ae-id 8;
        chassis-id 1;
        mode active-active;
        status-control standby;
        init-delay-time 420;
    }
}
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members [ VLAN-100 VLAN-200 VLAN-300 VLAN-400 ];
        }
    }
}

{master:0}[edit]
admin@QFX5100-B# show interfaces ge-0/0/41
description "L3 Link to DAL-MX5-B - NeutralPath";
ether-options {
    802.3ad ae8;
}

 

In my opinion, there should be NO reason for the QFX to drop anything on this interface. Like I said, I'm pushing between 200Mbps and 400Mbps on a 1Gbps interface. Why are there drops?

 

I'm running 14.1X53-D40.8. Any help would be appreciated. Thanks!

Feedback