Ethernet Switching
Highlighted
Ethernet Switching

QFX5100 interface is always down

‎07-02-2020 02:00 AM

We are using a QFX5100 switch and multiple servers are connected to it. Each server has two connections to the switch the first connection is working fine, the second connection does not the switch shows that the link state is always down. In fact, traffic arrives at the physical interface of the switch, and the link negotiation was complete.

 

The logical interface is always down:

 

uniscon@DCUA02-SW01> show interfaces xe-0/0/20 terse 
# interface connected to a server X (eth1) Interface Admin Link Proto Local Remote xe-0/0/20 up up xe-0/0/20.0 up up eth-switch uniscon@DCUA02-SW01> show interfaces xe-0/0/19 terse
# interface connected to a server X (eth0) Interface Admin Link Proto Local Remote xe-0/0/19 up down xe-0/0/19.0 up down eth-switch

 

here are the non-working interface details:

 

show interfaces xe-0/0/19 extensive    
Physical interface: xe-0/0/19, Enabled, Physical link is Up
  Interface index: 672, SNMP ifIndex: 533, Generation: 163
  Description: eth0 on appsrv5
  Link-level type: Ethernet, MTU: 9188, LAN-PHY mode, Link-mode: Auto, Speed: 10 Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Enabled, Remote fault: Offline, Media type: Copper
  Device flags   : Present Running
  Interface flags: Link-Layer-Down SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 44:f4:77:b5:4a:96, Hardware address: 44:f4:77:b5:4a:96
  Last flapped   : 2020-07-01 16:35:39 CEST (17:59:55 ago)
  Statistics last cleared: 2020-07-01 14:26:51 CEST (20:08:43 ago)
  Traffic statistics:
   Input  bytes  :              1005534                    0 bps
   Output bytes  :                    0                    0 bps
   Input  packets:                 2959                    0 pps
   Output packets:                    0                    0 pps
   IPv6 transit statistics:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Bucket drops: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 16, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0, Bucket drops: 0
  Egress queues: 12 supported, 5 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    3                                0                    0                    0
    4                                0                    0                    0
    7                                0                    0                    0
    8                                0                    0                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    3                   fcoe
    4                   no-loss
    7                   network-control
    8                   mcast
  Active alarms  : None
  Active defects : None
  PCS statistics                      Seconds
    Bit errors                             0
    Errored blocks                         0
  Ethernet FEC statistics              Errors
    FEC Corrected Errors                    0
    FEC Uncorrected Errors                  0
    FEC Corrected Errors Rate               0
    FEC Uncorrected Errors Rate             0
  MAC statistics:                      Receive         Transmit
    Total octets                       1005534                0
    Total packets                         2959                0
    Unicast packets                          0                0
    Broadcast packets                     2890                0
    Multicast packets                       69                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
  MAC Priority Flow Control Statistics:
    Priority :  0                             0                0
    Priority :  1                             0                0
    Priority :  2                             0                0
    Priority :  3                             0                0
    Priority :  4                             0                0
    Priority :  5                             0                0
    Priority :  6                             0                0
    Priority :  7                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 1, CAM source filters: 0
  Autonegotiation information:
    Negotiation status: Complete
    Link partner:                       
        Link mode: Full-duplex, Flow control: Symmetric/Asymmetric, Remote fault: OK, Link partner Speed: 10 Gbps
    Local resolution:
        Flow control: None, Flow control tx: Disabled, Flow control rx: Disabled, Remote fault: Link OK
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort             5      500000000     5              0      low    none
    3 fcoe                   35     3500000000    35              0      low    none
    4 no-loss                35     3500000000    35              0      low    none
    7 network-control         5      500000000     5              0      low    none
    8 mcast                  20     2000000000    20              0      low    none
  Interface transmit statistics: Disabled

  Logical interface xe-0/0/19.0 (Index 579) (SNMP ifIndex 575) (Generation 194)
    Flags: Device-Down SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Traffic statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Local statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol eth-switch, MTU: 9188, Generation: 217, Route table: 5
      Input Filters: storage-dst

For comparison sake, here is the working interface that is connected to the server:

show interfaces xe-0/0/20 extensive    
Physical interface: xe-0/0/20, Enabled, Physical link is Up
  Interface index: 673, SNMP ifIndex: 534, Generation: 164
  Description: eth1 on appsrv5
  Link-level type: Ethernet, MTU: 9188, LAN-PHY mode, Link-mode: Auto, Speed: 10 Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Disabled, Remote fault: Offline, Media type: Copper
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 12 supported, 12 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 44:f4:77:b5:4a:97, Hardware address: 44:f4:77:b5:4a:97
  Last flapped   : 2020-07-01 11:50:55 CEST (22:44:58 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :            142019669                    0 bps
   Output bytes  :           4069747174                 4584 bps
   Input  packets:              1569096                    0 pps
   Output packets:              5075888                    1 pps
   IPv6 transit statistics:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Bucket drops: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 189, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0, Bucket drops: 0
  Egress queues: 12 supported, 5 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                          2108741              2108741                    0
    3                                0                    0                    0
    4                                0                    0                    0
    7                                0                    0                    0
    8                           262441               262441                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    3                   fcoe
    4                   no-loss
    7                   network-control
    8                   mcast
  Active alarms  : None
  Active defects : None
  PCS statistics                      Seconds
    Bit errors                             0
    Errored blocks                         0
  Ethernet FEC statistics              Errors
    FEC Corrected Errors                    0
    FEC Uncorrected Errors                  0
    FEC Corrected Errors Rate               0
    FEC Uncorrected Errors Rate             0
  MAC statistics:                      Receive         Transmit
    Total octets                     142019669       4069726772
    Total packets                      1569096          5075849
    Unicast packets                    1521520          2461348
    Broadcast packets                    47370          2110717
    Multicast packets                      206           503784
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
  MAC Priority Flow Control Statistics:
    Priority :  0                             0                0
    Priority :  1                             0                0
    Priority :  2                             0                0
    Priority :  3                             0                0
    Priority :  4                             0                0
    Priority :  5                             0                0
    Priority :  6                             0                0
    Priority :  7                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 1, CAM source filters: 0
  Autonegotiation information:
    Negotiation status: Complete
    Link partner:                       
        Link mode: Full-duplex, Flow control: Symmetric/Asymmetric, Remote fault: OK, Link partner Speed: 10 Gbps
    Local resolution:
        Flow control: None, Flow control tx: Disabled, Flow control rx: Disabled, Remote fault: Link OK
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort             5      500000000     5              0      low    none
    3 fcoe                   35     3500000000    35              0      low    none
    4 no-loss                35     3500000000    35              0      low    none
    7 network-control         5      500000000     5              0      low    none
    8 mcast                  20     2000000000    20              0      low    none
  Interface transmit statistics: Disabled

  Logical interface xe-0/0/20.0 (Index 580) (SNMP ifIndex 576) (Generation 165)
    Flags: Up SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Traffic statistics:
     Input  bytes  :             25166180
     Output bytes  :                19934
     Input  packets:                42806
     Output packets:                   39
    Local statistics:
     Input  bytes  :             25166180
     Output bytes  :                19934
     Input  packets:                42806
     Output packets:                   39
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol eth-switch, MTU: 9188, Generation: 186, Route table: 5

Any help is appreciated 

11 REPLIES 11
Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 02:16 AM

Hi yassine_fadh1,

 

The below indicates that interface xe-0/0/19 is up but the link is down. Indicating the link is not up. It could be an issue on transit or an issue with the remote end connected to this specific interface. Please ensure both Admin and Link and UP.

uniscon@DCUA02-SW01> show interfaces xe-0/0/19 terse    
# interface connected to a server X (eth0) Interface Admin Link Proto Local Remote xe-0/0/19 up down xe-0/0/19.0 up down eth-switch

 

Also, you can see this alarm in the interface description for xe-0/0/19

 

show interfaces xe-0/0/19 extensive    
Physical interface: xe-0/0/19, Enabled, Physical link is Up
  Interface index: 672, SNMP ifIndex: 533, Generation: 163
  Description: eth0 on appsrv5
  Link-level type: Ethernet, MTU: 9188, LAN-PHY mode, Link-mode: Auto, Speed: 10 Gbps, BPDU Error: None, Loop Detect PDU Error: None, Ethernet-Switching Error: None,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Enabled, Remote fault: Offline, Media type: Copper
  Device flags   : Present Running
  Interface flags: Link-Layer-Down SNMP-Traps Internal: 0x4000

Please ensure the link is up and then everything should work fine.

 

Hope this helps 🙂

 

Please mark "Accepted Solution" if this helps you solve your query. Kudos are always appreciated!

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

[ Edited ]
‎07-02-2020 02:28 AM

Thank you for your reply, but this issue is on all servers :

# server N (eth1)
xe-0/0/9                up    down
xe-0/0/9.0              up    down eth-switch
# server N (eth0) xe-0/0/10 up up xe-0/0/10.0 up up eth-switch
# server N+1 (eth1) xe-0/0/11 up down xe-0/0/11.0 up down eth-switch
# server N+1 (eth0) xe-0/0/12 up up xe-0/0/12.0 up up eth-switch
# server N+2 (eth1) xe-0/0/13 up down xe-0/0/13.0 up down eth-switch
# server N+2 (eth0) xe-0/0/14 up up xe-0/0/14.0 up up eth-switch
# server N+3 (eth1) xe-0/0/15 up down xe-0/0/15.0 up down eth-switch
# server N+3 (eth0) xe-0/0/16 up up xe-0/0/16.0 up up eth-switch
xe-0/0/17 up down xe-0/0/17.0 up down eth-switch xe-0/0/18 up up xe-0/0/18.0 up up eth-switch xe-0/0/19 up down xe-0/0/19.0 up down eth-switch xe-0/0/20 up up xe-0/0/20.0 up up eth-switch

 

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 02:42 AM

Hi yassine_fadh1, 

 

Greetings, 

I see that there is an input filter in the output you shared for xe-0/0/19 whereas this isn't seen on xe-0/0/20.

I believe the input filter is blocking this traffic, could you confirm that for us ? Also, please check if there are any filter applied to this interface and the others as well and confirm if the same are applied/not on the other interface in the pair.

show interfaces xe-0/0/19 extensive ..
...
Input Filters: storage-dst

 

Hope this helps. Smiley Happy

 

Please mark "Accept as solution" if this answers your query. 

Kudos are appreciated too! 

  

Regards, 

Sharat Ainapur

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 03:03 AM

Thank you for your reply,

I did that before but it did not help here is output after removing the filter

uniscon@DCUA02-SW01# run show configuration interfaces xe-0/0/19                                            
description "eth0 on appsrv5";
mtu 9188;
ether-options {
    auto-negotiation;
}
unit 0 {
    family ethernet-switching {
        interface-mode access;
        vlan {
            members Storage;
        }
    }
}

# commit confirmed will be rolled back in 9 minutes
{master:0}[edit]
uniscon@DCUA02-SW01# run show interfaces xe-0/0/19 terse            
Interface               Admin Link Proto    Local                 Remote
xe-0/0/19               up    down
xe-0/0/19.0             up    down eth-switch

# commit confirmed will be rolled back in 9 minutes

 

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 03:23 AM

This looks like a physical issue.

If this is a duplex optic try flipping the two fibers inserted on one side reversing the transmit and receive connections.  I've seen issues where the pair gets flipping in the patch panel process across facilities or some cables are designed to do such a flip.

 

Also have a look at the optic details to see if there is a light or power level issue.

show interfaces diagnostics optics 

show interfaces diagnostics optics xe-0/0/19 

 

Steve Puluka BSEET - Juniper Ambassador
IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
http://puluka.com/home
Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 03:48 AM

Hi

 

If for each server you have 2 links towards the switch and one is up and another is down then i would suggest to check the following:

1. Compare the cable/optics of the working link with the non working link (for same server).

2. Check the auto neg settings on the qfx5100 and the state on the server.

3. Also check if there is any static bundling of the 2 links on the server side etc

 

Hope this helps

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 06:46 AM

A piece of information to add here,

 

From the server, I try to run  

dhclient eno1 
# eno1 is the interface linked on the server linked to that switch port

I cleared the statistics on the switch side for that interface

clear interfaces statistics xe-0/0/17

and after few seconds packets are arriving on that interface:

Screenshot from 2020-07-02 15-42-39.png

 

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 07:18 AM

Hi yassine_fadh1, 

 

Yes, the input packets were seen previously as well, and another difference I can think of is the auto-negotation in the configuration you shared.  Could you try disabling the auto-negotiation on the xe-0/0/19 interface on which this is not working and confirm ?  Command: 

 

# delete interfaces xe-0/0/19 ether-options auto-negotiation
# commit and-quit

 

 

 

show interfaces xe-0/0/19 extensive    
Physical interface: xe-0/0/19, Enabled, Physical link is Up
..  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Enabled, Remote fault: Offline, Media type: Copper

show interfaces xe-0/0/20 extensive    
Physical interface: xe-0/0/20, Enabled, Physical link is Up
..  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled, Auto-negotiation: Disabled, Remote fault: Offline, Media type: Copper


Hope this helps. Smiley Happy

 

Please mark "Accept as solution" if this answers your query / resolves the issue. 

Kudos are appreciated too! 

  

Regards, 

Sharat Ainapur

 

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 07:38 AM

yeah the default config is auto-negotiation disabled just for the sake of test I enabled it for a while. Below is the confirmation of disabling the auto-negotiation:

mtu 1500;
ether-options {
    no-auto-negotiation;
}
unit 0 {
    family ethernet-switching {
        interface-mode access;
        vlan {
            members Storage;
        }
        filter {
            input storage-dst;
        }
    }
}

but still not working Smiley Sad

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

‎07-02-2020 07:50 AM

And if you unplug a server's 'up' interface does its 'down' interface come up? It sounds like simple active/passive server-side interface bonding.

Highlighted
Ethernet Switching

Re: QFX5100 interface is always down

[ Edited ]
‎07-02-2020 10:35 AM

Thank-you @

 

Feedback