Junos OS

Had an FPC card stay online but all the ports are down.

Two of the FPC cards we used are all down even with rebooting the re's and re-seating.

show chassis fpcshows it up

                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
  0  Online            38     10          0       10     11     11    2048       49         13
  1  Empty           
  2  Online            36     14          2       13     14     14    2048       46         14
  3  Online            32     11          0       10     10     10    2048       33         13
  4  Empty           
  5  Empty           

Item             Version  Part number  Serial number     Description
Chassis                                JN1261B2DAFB      MX480
Midplane         REV 40   750-047862   ACRH0849          Enhanced MX480 Midplane
FPM Board        REV 04   760-059208   CAGG8322          Front Panel Display
PEM 0            Rev 01   740-063046   QCS1622U03X       PS 1.4-2.52kW; 90-264V AC in
PEM 1            Rev 01   740-063046   QCS1622U0BA       PS 1.4-2.52kW; 90-264V AC in
PEM 2            Rev 01   740-063046   QCS1622U01C       PS 1.4-2.52kW; 90-264V AC in
PEM 3            Rev 01   740-063046   QCS1622U054       PS 1.4-2.52kW; 90-264V AC in
Routing Engine 0 REV 03   740-051822   9013135351        RE-S-1800x4
Routing Engine 1 REV 03   740-051822   9013133255        RE-S-1800x4
CB 0             REV 07   750-062572   CAGC4744          Enhanced MX SCB 2
CB 1             REV 07   750-062572   CAGK2061          Enhanced MX SCB 2
FPC 0            REV 38   750-028467   ABCT9877          MPC 3D 16x 10GE
  CPU            REV 12   711-029089   ABCT9796          AMPC PMB
  PIC 0                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-011783   Q0507150584       SFP+-10G-LR
    Xcvr 2       REV 01   740-021308   Q0608020012       SFP+-10G-SR
  PIC 1                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-021309   Q0607141517       SFP+-10G-LR
    Xcvr 1       REV 01   740-021309   Q0508170437       SFP+-10G-LR
    Xcvr 2       REV 01   740-021308   Q0608020011       SFP+-10G-SR
  PIC 2                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0                NON-JNPR     STST85S81900210   DUAL-SFP+-SR/SFP-SX
    Xcvr 2                NON-JNPR     FS41010Q0052      SFP+-10G-LR
    Xcvr 3                NON-JNPR     FS41010Q0058      SFP+-10G-ER
  PIC 3                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-031980   D8830821027       SFP+-10G-SR
    Xcvr 1                NON-JNPR     STST85S81900217   DUAL-SFP+-SR/SFP-SX
    Xcvr 2       REV 01   740-021308   FS50210Q0176      SFP+-10G-SR
    Xcvr 3       REV 01   740-021309   Q07B1600006       SFP+-10G-LR
FPC 2            REV 09   750-062863   CAHT5466          MPC4E 3D 2CGE+8XGE
  CPU            REV 03   711-062860   CAHK6412          HMPC PMB 2G
  PIC 0                   BUILTIN      BUILTIN           4x10GE SFPP
    Xcvr 0       REV 01   740-030128   Q0611050384       SFP+-10G-ER
    Xcvr 1       '        NON-JNPR     FS41010Q0059      SFP+-10G-ER
    Xcvr 2                NON-JNPR     Q0611050383       SFP+-10G-ER
    Xcvr 3                NON-JNPR     STST85S81900218   DUAL-SFP+-SR/SFP-SX
  PIC 1                   BUILTIN      BUILTIN           1X100GE CFP
    Xcvr 0                NON-JNPR     J17K68938         CFP2-100G-LR4-D
  PIC 2                   BUILTIN      BUILTIN           4x10GE SFPP
    Xcvr 0       +        NON-JNPR     FS41010Q0048      SFP+-10G-LR
    Xcvr 1                NON-JNPR     FS41010Q0049      SFP+-10G-LR
    Xcvr 2                NON-JNPR     FS41010Q0046      SFP+-10G-LR
    Xcvr 3                NON-JNPR     STST85S81900220   DUAL-SFP+-SR/SFP-SX
  PIC 3                   BUILTIN      BUILTIN           1X100GE CFP
FPC 3            REV 38   750-028467   ABCR5714          MPC 3D 16x 10GE
  CPU            REV 12   711-029089   ABCL3301          AMPC PMB
  PIC 0                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0       REV 01   740-011613   FS40310Q0089      SFP+-10G-LR
    Xcvr 1                NON-JNPR     STST85S81900219   DUAL-SFP+-SR/SFP-SX
    Xcvr 2                NON-JNPR     STST85S81900216   DUAL-SFP+-SR/SFP-SX
    Xcvr 3       REV 01   740-021309   Q0508170439       SFP+-10G-LR
  PIC 1                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0                NON-JNPR     FS41010Q0051      SFP+-10G-LR
    Xcvr 1       REV 01   740-021308   Q0608020030       SFP+-10G-SR
    Xcvr 2       REV 01   740-021308   Q0608020023       SFP+-10G-SR
    Xcvr 3       A        NON-JNPR     FS41010Q0060      SFP+-10G-ER
  PIC 2                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0                NON-JNPR     STST85S81900209   DUAL-SFP+-SR/SFP-SX
    Xcvr 1       REV 01   740-021309   FS50127Q1427      SFP+-10G-LR
    Xcvr 2       REV 01   740-021308   Q0608020022       SFP+-10G-SR
    Xcvr 3       REV 01   740-021308   Q0608020029       SFP+-10G-SR
  PIC 3                   BUILTIN      BUILTIN           4x 10GE(LAN) SFP+
    Xcvr 0                NON-JNPR     STST85S81900208   DUAL-SFP+-SR/SFP-SX
    Xcvr 1       REV 01   740-021309   Q0601040067       SFP+-10G-LR
    Xcvr 2       REV 01   740-021308   Q0608020021       SFP+-10G-SR
    Xcvr 3       REV 01   740-021309   Q0508170441       SFP+-10G-LR
Fan Tray                                                 Enhanced Left Fan Tray

Doing show interface on any fpc0/fpc3 although fpc are up the ports are non functional an example light wave

Physical interface: xe-0/0/2
    Laser bias current                        :  10.804 mA
    Laser output power                        :  0.4050 mW / -3.93 dBm
    Module temperature                        :  32 degrees C / 90 degrees F
    Module voltage                            :  3.2990 V
    Receiver signal average optical power     :  0.4706 mW / -3.27 dBm
    Laser bias current high alarm             :  Off
    Laser bias current low alarm              :  Off
    Laser bias current high warning           :  Off
    Laser bias current low warning            :  Off
    Laser output power high alarm             :  Off
    Laser output power low alarm              :  Off
    Laser output power high warning           :  Off
    Laser output power low warning            :  Off
    Module temperature high alarm             :  Off
    Module temperature low alarm              :  Off
    Module temperature high warning           :  Off
    Module temperature low warning            :  Off
    Module voltage high alarm                 :  Off
    Module voltage low alarm                  :  Off
    Module voltage high warning               :  Off
    Module voltage low warning                :  Off
    Laser rx power high alarm                 :  Off
    Laser rx power low alarm                  :  Off
    Laser rx power high warning               :  Off
    Laser rx power low warning                :  Off
    Laser bias current high alarm threshold   :  100.000 mA
    Laser bias current low alarm threshold    :  0.000 mA
    Laser bias current high warning threshold :  90.000 mA
    Laser bias current low warning threshold  :  0.100 mA
    Laser output power high alarm threshold   :  1.0000 mW / 0.00 dBm
    Laser output power low alarm threshold    :  0.2510 mW / -6.00 dBm
    Laser output power high warning threshold :  0.7940 mW / -1.00 dBm
    Laser output power low warning threshold  :  0.3160 mW / -5.00 dBm
    Module temperature high alarm threshold   :  90 degrees C / 194 degrees F
    Module temperature low alarm threshold    :  -5 degrees C / 23 degrees F
    Module temperature high warning threshold :  85 degrees C / 185 degrees F
    Module temperature low warning threshold  :  0 degrees C / 32 degrees F
    Module voltage high alarm threshold       :  3.800 V
    Module voltage low alarm threshold        :  2.700 V
    Module voltage high warning threshold     :  3.700 V
    Module voltage low warning threshold      :  2.800 V
    Laser rx power high alarm threshold       :  1.0000 mW / 0.00 dBm
    Laser rx power low alarm threshold        :  0.0501 mW / -13.00 dBm
    Laser rx power high warning threshold     :  0.7943 mW / -1.00 dBm
    Laser rx power low warning threshold      :  0.0631 mW / -12.00 dBm

{master}

Light looks good but no link when moving the ports to fpc2 everything is functional

Module of card having issues - MPC 3D 16x 10GE

Hi

Could you check if anyone by chance have hard disabled the ports on those FPC(s)?

Get the following from CLI:

show configuration chassis fpc0

show configuration chassis fpc 3

show chassis fpc pic-status 0

show chassis fpc pic-status 3

show interfaces terse 

show chassis alarms

show configuration chassis fpc 0  
pic 0 {
    tunnel-services;
}
pic 1 {
    tunnel-services;
}
pic 2 {
    tunnel-services;
}
pic 3 {
    tunnel-services;
}
sampling-instance s1;

show configuration chassis fpc 1    
pic 0 {
    tunnel-services;
}
pic 1 {
    tunnel-services;
}
pic 2 {
    tunnel-services;
}
pic 3 {
    tunnel-services;
}
port-mirror-instance redirect-instance;
sampling-instance s1;

show chassis fpc pic-status 0
Slot 0   Online       MPC 3D 16x 10GE
  PIC 0  Online       4x 10GE(LAN) SFP+
  PIC 1  Online       4x 10GE(LAN) SFP+
  PIC 2  Online       4x 10GE(LAN) SFP+
  PIC 3  Online       4x 10GE(LAN) SFP+

show chassis fpc pic-status 3    
Slot 3   Online       MPC 3D 16x 10GE
  PIC 0  Online       4x 10GE(LAN) SFP+
  PIC 1  Online       4x 10GE(LAN) SFP+
  PIC 2  Online       4x 10GE(LAN) SFP+
  PIC 3  Online       4x 10GE(LAN) SFP+

3 alarms currently active
Alarm time               Class  Description
2018-10-18 03:21:15 PDT  Major  Host 1 fxp0 : Ethernet Link Down
2018-10-18 03:21:15 PDT  Major  Host 0 fxp0 : Ethernet Link Down
2018-10-18 03:20:43 PDT  Minor  Backup RE Active

show interfaces terse  --> this shows that the ports are up just no link light. It was working fine out of no where one shutdown i rebooted fpc0 and it never came back up in terms of ports but fpc came up fine.

You previously had mention fpc0 and fpc3 - why do you list the output of fpc1?

Also, FPC is UP and PIC(s) associcated to that FPC (0) are UP as well. 

I don't link its local issue but perhaps could be remote end issue.

Have you tried to change/replace XFP & the swap cable?

Try the following:

-> Move the affected FPC (0) to another empty SLOT.

-> When FPC and PIC are online, check the interface status.

-> what is the physical led link status on the FPC& on the Port?

Could you also get the following from the present FPC slot and post moving the FPC slot:

show interfaces xe-x/x/x extensive   >>> get this o/p from multiple port of the affected FPC

Hi Karand

Thanks for help a bit confusion on fpc1/3 that was actually moved from fpc1 to fpc3 which.

The physical LED lights all show down when we move it to another port such as fpc2 any of the sfp++ comes back online.

Physical interface: xe-0/1/1, Enabled, Physical link is Down
  Interface index: 511, SNMP ifIndex: 547, Generation: 514
  Link-level type: Ethernet, MTU: 9192, MRU: 9200, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Disabled
  Pad to minimum frame size: Disabled
  Device flags   : Present Running Down
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: 84:c1:c1:1b:57:c3, Hardware address: 84:c1:c1:1b:50:53
  Last flapped   : 2018-10-18 13:47:50 PDT (09:53:56 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :                    0                    0 bps
   Output bytes  :                    0                    0 bps
   Input  packets:                    0                    0 pps
   Output packets:                    0                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                                0                    0                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    1                   expedited-forwarding
    2                   assured-forwarding
    3                   network-control
  Active alarms  : LINK
  Active defects : LINK
  PCS statistics                      Seconds
    Bit errors                             0
    Errored blocks                         0
  MAC statistics:                      Receive         Transmit
    Total octets                             0                0
    Total packets                            0                0
    Unicast packets                          0                0
    Broadcast packets                        0                0
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
    Total errors                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 0, CAM source filters: 0
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95     9500000000    95              0      low    none
    3 network-control         5      500000000     5              0      low    none
  Preclassifier statistics:
    Traffic Class        Received Packets   Transmitted  Packets      Dropped Packets
    real-time                           0                      0                    0
    network-control                     0                      0                    0
    best-effort                         0                      0                    0
  Interface transmit statistics: Disabled

  Logical interface xe-0/1/1.705 (Index 428) (SNMP ifIndex 762) (Generation 902)
    Flags: Device-Down SNMP-Traps 0x4000 VLAN-Tag [ 0x8100.705 ]  Encapsulation: ENET2
    Traffic statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Local statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol aenet, AE bundle: ae3.705, Generation: 1142, Route table: 0

  Logical interface xe-0/1/1.805 (Index 429) (SNMP ifIndex 761) (Generation 903)
    Flags: Device-Down SNMP-Traps 0x4000 VLAN-Tag [ 0x8100.805 ]  Encapsulation: ENET2
    Traffic statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Local statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol aenet, AE bundle: ae3.805, Generation: 1143, Route table: 0

  Logical interface xe-0/1/1.32767 (Index 430) (SNMP ifIndex 725) (Generation 904)
    Flags: Device-Down SNMP-Traps 0x4004000 VLAN-Tag [ 0x0000.0 ]  Encapsulation: ENET2
    Traffic statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Local statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol aenet, AE bundle: ae3.32767, Generation: 1144, Route table: 0

show interfaces xe-0/3/0 extensive   
Physical interface: xe-0/3/0, Enabled, Physical link is Down
  Interface index: 518, SNMP ifIndex: 554, Generation: 521
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Enabled
  Pad to minimum frame size: Disabled
  Device flags   : Present Running Down
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: 84:c1:c1:1b:50:f6, Hardware address: 84:c1:c1:1b:50:f6
  Last flapped   : 2018-10-18 13:47:50 PDT (09:54:32 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :                    0                    0 bps
   Output bytes  :                    0                    0 bps
   Input  packets:                    0                    0 pps
   Output packets:                    0                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                                0                    0                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    1                   expedited-forwarding
    2                   assured-forwarding
    3                   network-control
  Active alarms  : LINK
  Active defects : LINK
  PCS statistics                      Seconds
    Bit errors                             0
    Errored blocks                         0
  MAC statistics:                      Receive         Transmit
    Total octets                             0                0
    Total packets                            0                0
    Unicast packets                          0                0
    Broadcast packets                        0                0
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
    Total errors                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 0, CAM source filters: 0
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95     9500000000    95              0      low    none
    3 network-control         5      500000000     5              0      low    none
  Preclassifier statistics:
    Traffic Class        Received Packets   Transmitted  Packets      Dropped Packets
    real-time                           0                      0                    0
    network-control                     0                      0                    0
    best-effort                         0                      0                    0
  Interface transmit statistics: Disabled

  Logical interface xe-0/3/0.0 (Index 477) (SNMP ifIndex 756) (Generation 945)
    Flags: Device-Down SNMP-Traps 0x4004000 Encapsulation: ENET2
    Traffic statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Local statistics:
     Input  bytes  :                    0
     Output bytes  :                    0
     Input  packets:                    0
     Output packets:                    0
    Transit statistics:
     Input  bytes  :                    0                    0 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                    0                    0 pps
     Output packets:                    0                    0 pps
    Protocol inet, MTU: 1500, Generation: 1187, Route table: 0
      Flags: Sendbcast-pkt-to-re
      Addresses, Flags: Dest-route-down Is-Preferred Is-Primary
        Destination: 10.50.50.0/30, Local: 10.50.50.2, Broadcast: 10.50.50.3, Generation: 805
    Protocol multiservice, MTU: Unlimited, Generation: 1188, Route table: 0
      Policer: Input: __default_arp_policer__

{master}

show interfaces xe-3/0/0 extensive     
Physical interface: xe-3/0/0, Enabled, Physical link is Down
  Interface index: 445, SNMP ifIndex: 1129, Generation: 448
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: None, Source filtering: Disabled,
  Flow control: Enabled
  Pad to minimum frame size: Disabled
  Device flags   : Present Running Down
  Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: 84:c1:c1:1b:53:de, Hardware address: 84:c1:c1:1b:53:de
  Last flapped   : 2018-10-18 04:33:20 PDT (19:10:01 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :                    0                    0 bps
   Output bytes  :                    0                    0 bps
   Input  packets:                    0                    0 pps
   Output packets:                    0                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                                0                    0                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    1                   expedited-forwarding
    2                   assured-forwarding
    3                   network-control
  Active alarms  : LINK
  Active defects : LINK
  PCS statistics                      Seconds
    Bit errors                             0
    Errored blocks                         0
  MAC statistics:                      Receive         Transmit
    Total octets                             0                0
    Total packets                            0                0
    Unicast packets                          0                0
    Broadcast packets                        0                0
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
    Total errors                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 0, CAM source filters: 0
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95     9500000000    95              0      low    none
    3 network-control         5      500000000     5              0      low    none
  Preclassifier statistics:
    Traffic Class        Received Packets   Transmitted  Packets      Dropped Packets
    real-time                           0                      0                    0
    network-control                     0                      0                    0
    best-effort                         0                      0                    0
  Interface transmit statistics: Disabled

Hi,

Your port is physically enabled but the link is down. You also have active alarms such as LINK alarms

Active alarms  : LINK
Active defects : LINK

Link—Interface has lost its link state, which usually means that the cable is unplugged, the far-end system has been turned off, or the PIC is malfunctioning.

To verify the if your PIC/Local H/w interface is fine or malfunctioning, run diagnose test as follows:

For instance:

run show interfaces xe-3/0/1 extensive
Physical interface: xe-3/0/1, Enabled, Physical link is Down    <<<< Link Down

<snip>

re0# set interfaces xe-3/0/1 gigether-options loopback  

re0# set interfaces lo0 unit 0 family inet address x.x.x.x/32

re0# commit

re0# run show interfaces xe-3/0/1 extensive                       
Physical interface: xe-3/0/1, Enabled, Physical link is Up    <<<< loops port link to verify that local port is fine.

Active Alarm should go off at the point. Check and let me know the results.

Karan

Thank you for your help here binding local seem to work

 run show interfaces xe-3/0/1 extensive
Physical interface: xe-3/0/1, Enabled, Physical link is Up
  Interface index: 446, SNMP ifIndex: 1130, Generation: 449
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, LAN-PHY mode, Speed: 10Gbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Local, Source filtering: Disabled,
  Flow control: Enabled
  Pad to minimum frame size: Disabled
  Device flags   : Present Running Loop-Detected
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Schedulers     : 0
  Hold-times     : Up 0 ms, Down 0 ms
  Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
  Current address: 84:c1:c1:1b:53:df, Hardware address: 84:c1:c1:1b:53:df
  Last flapped   : 2018-10-19 01:07:29 PDT (00:00:49 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :                    0                    0 bps
   Output bytes  :                    0                    0 bps
   Input  packets:                    0                    0 pps
   Output packets:                    0                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0                                0                    0                    0
    1                                0                    0                    0
    2                                0                    0                    0
    3                                0                    0                    0
  Queue number:         Mapped forwarding classes
    0                   best-effort
    1                   expedited-forwarding
    2                   assured-forwarding
    3                   network-control
  Active alarms  : LINK
  Active defects : None
  PCS statistics                      Seconds
    Bit errors                             0
    Errored blocks                         0
  MAC statistics:                      Receive         Transmit
    Total octets                             0                0
    Total packets                            0                0
    Unicast packets                          0                0
    Broadcast packets                        0                0
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
    Total errors                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                     0
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 0, CAM source filters: 0
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95     9500000000    95              0      low    none
    3 network-control         5      500000000     5              0      low    none
  Preclassifier statistics:
    Traffic Class        Received Packets   Transmitted  Packets      Dropped Packets
    real-time                           0                      0                    0
    network-control                     0                      0                    0
    best-effort                         0                      0                    0
  Interface transmit statistics: Disabled

However when moving fiber to the fpc2 ( which is working ) these come up fine. This issue is extremely odd

OK, Thanks for testing out that.

Now the delele diag knobs:

#delete interfaces xe-3/0/1 gigether-options loopback  

#commit

Connect cable back-2-back. For instance

xe-3/0/0 <---> xe-3/0/1

If the port doesn't up, change the XFP/SFP form factor & cable once ( could be that XFP/SFP not support)

Check and let me know goes its goes.

Hi -

We'll need to test this out tomorrow as this is in a dc and will require onsite changes.

The odd part was fpc0/fpc1( now fpc3) were all in production.

fpc1 all 16 ports just shut off - rebooted nothing happen

so we rebooted route-engine 0/1

Then we rebooted fpc0 and then thats when fpc0 just completely shut off as well. We did receive an alarm on cb 0 clock sync problem likely a scbe2 card is bad so we'll be pulling something out.

That does sound to be H/w condition though run all the possible test required. 

If possible, ask the on-site technicain to move this FPC to another MX chassis  & check once as well.

Also, would suggest to log a case with JTAC to help narrow down & isolate if it happens to be h/w condition.