Switching

last person joined: yesterday 

Ask questions and share experiences about EX and QFX portfolios and all switching solutions across your data center, campus, and branch locations.
  • 1.  EX3300 Getting periodic disconnects of VC via SFP

    Posted 04-19-2020 13:25

    Hi!

     

    I have  a problem with my 2xEX3300 switches. I would like to get some suggestions on what to look at, or what could possibly be wrong. I don't have any other hardware (SFP:s) to test with. They are running in a VPC (One master and one slave).

    The second switch is disconnected roughly once every second hour (seems to be more frequent when there is more data going between the switches).

     

    They are connected with 2xSFP-10GE-T RJ45, and I'm using a roughly 25 meter cat6A cable.

     

    What I have ruled out:

    - Cable is faulty (I have measure it with sveral different ethernet testers, the cable is fine) , I guess it could still be faulty but 
    it seems not that likely.

    - Temperature, I have run both switches when ambient temperature was really low, didn't see any different patterns on the disconnects, they also report they have helthy temperaures, although it somewhat high for one of the switches.

     

    What could the problem be, and how likely is it?

    - Faulty cable?

    - Faulty SFP?

    - Faulty switch?

    - Config?

     

    When it's disconnecting the ports on the slave switch are turned off and then restarted, which causes my POE-devices to reboot.

    I have this in the logs: (You can see it's removed, and at 20:16:38 inserted).

     

     

    Apr 19 20:16:30  FPC 1 removed
    Apr 19 20:16:30 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX3300 24-Port POE+ @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Apr 19 20:16:30 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 1 offline: Removal
    Apr 19 20:16:30  fpc_down slot 1 reason Removal cargs 0xa90bc0
    Apr 19 20:16:30  pic detach portinfo, pic 0 fpc 1
    Apr 19 20:16:30  pic detach portinfo, pic 1 fpc 1
    Apr 19 20:16:30  fpc_disconnect_generic: fpc 1 state Empty cargs 0xa90bc0 clean_shutdown 0, offline_reason=Removal
    Apr 19 20:16:30  -- FPC 1, last request 132, state Empty
    Apr 19 20:16:30  fpc_disconnect_generic - FPC 1 was removed!
    Apr 19 20:16:30 CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 1
    Apr 19 20:16:30 CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(1)
    Apr 19 20:16:30  ifdev_detach: skipping ifd vcp-255/1/3
    Apr 19 20:16:30  ifd ge-1/0/0 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/1 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/2 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/3 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/4 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/5 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/6 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/7 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/8 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/9 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/10 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/11 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/12 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/13 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/14 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/15 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/16 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/17 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/18 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/19 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/20 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/21 marked as gone
    Apr 19 20:16:31  ifd ge-1/0/22 marked as gone
    Apr 19 20:16:32  ifd ge-1/0/23 marked as gone
    Apr 19 20:16:32  fpc_offline_now - slot 1, reason: Removal, error Chassis connection dropped transition state 1
    Apr 19 20:16:32 CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperDisabled (entPhysicalIndex 17, entStateAdmin 3, entStateAlarm 0)
    Apr 19 20:16:32  fpc_offline_now - slot 1, is_resync_ready cleared
    Apr 19 20:16:32  fpc_offline_now - Setup NMI timer on fpc 1 for 180 sec
    Apr 19 20:16:32  mic_get_mic_slot: clp1: fpc_slot=1, pic_slot=0, i2c=0xf052
     
    Apr 19 20:16:32  mic_get_mic_slot: clp1: fpc_slot=1, pic_slot=1, i2c=0xf0c5
     
    Apr 19 20:16:32  hwdb: entry for fpc 2541 at slot 1 deleted
     
    Apr 19 20:16:32 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX3300 24-Port POE+ @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Apr 19 20:16:32 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX3300 24-Port POE+ @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Apr 19 20:16:32  send: clear all FPC JAVA 1 alarms
    Apr 19 20:16:32  op 3 for ifd bme0
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  function returned 2
    Apr 19 20:16:32  PIC (fpc 1 pic 0) message operation: change. ifd count 3, flags 0x2 in mesg
    Apr 19 20:16:32  PIC (fpc 1 pic 0) message operation: change. ifd count 1, flags 0x2 in mesg
    Apr 19 20:16:32  PIC (fpc 1 pic 0) message operation: change. ifd count 0, flags 0 in mesg
    Apr 19 20:16:32  Time to clean up PIC FPC 1, PIC 0
    Apr 19 20:16:32  PIC (fpc 1 pic 0) message operation: delete. ifd count 0, flags 0 in mesg
    Apr 19 20:16:32  pic_handle_message_idl: PIC fpc 1 pic 0 got deleted
    Apr 19 20:16:34  op 1 for ifd bme0
    Apr 19 20:16:35  ipc pipe 0xa9f100 created
    Apr 19 20:16:35  ch_signal_proc: Sent signal 1 to tnp.sntpd, pid=1330
    Apr 19 20:16:36  fpc 1 ready, pipe 0x0xa9f100
    Apr 19 20:16:36  ifdev_detach_check: skipping ifd vcp-255/1/3 (what=0)
    Apr 19 20:16:36  fpc 1 clean, bringing online
    Apr 19 20:16:36  send_chassisd_capabs: send chassisd capabilities
    Apr 19 20:16:36  send: fpc 1 online cmd
    Apr 19 20:16:37  FPC 1 online ack
    Apr 19 20:16:37  fpc_announce_online_generic: fpc 1
    Apr 19 20:16:37  fpc_online_now - slot 1 - Online
    Apr 19 20:16:37 CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperEnabled (entPhysicalIndex 17, entStateAdmin 4, entStateAlarm 6)
    Apr 19 20:16:37  ch_java_sfpplus_send_config_to_cm: sending info fpc 1 pic 1 mode 10
    Apr 19 20:16:37  send: fpc 0 Disk Mon stats
    Apr 19 20:16:37  send: fpc 1 Disk Mon stats
    Apr 19 20:16:37  send: fpc 0 Disk Mon stats
    Apr 19 20:16:37  send: fpc 1 Disk Mon stats
    Apr 19 20:16:37  send: fpc 0 Disk Mon stats
    Apr 19 20:16:37  send: fpc 1 Disk Mon stats
    Apr 19 20:16:37  send: fpc 0 Disk Mon stats
    Apr 19 20:16:37  send: fpc 1 Disk Mon stats
    Apr 19 20:16:37 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX3300 24-Port POE+ @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Apr 19 20:16:37  fpc 1 creates device
     
    Apr 19 20:16:37  fpc_create_devs: No fpc based dev is required for fpc slot 1, skip fpc dev creation
     
    Apr 19 20:16:38  pic online req, pic 0 type 61522, fpc 1
    Apr 19 20:16:38  fpc slot 1 pic_present 0x0 => 0x1
    Apr 19 20:16:38  fpc_send_pic_online_ack: fpc 1 pic 0 pic_type 0xf052 msg_len 72 tlv_len 0
    Apr 19 20:16:38  fpc_send_pic_online_ack: fpc fru slot 1
    Apr 19 20:16:38  pic_get_ingress_shaping_overhead: 1/0 eso val = 0
    Apr 19 20:16:38  pic_get_egress_shaping_overhead: 1/0 eso val = 0
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC:  @ 1/0/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 319470272, jnxFruLastPowerOn 319470892)
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperEnabled (entPhysicalIndex 50, entStateAdmin 4, entStateAlarm 0)
    Apr 19 20:16:38  send: fpc 1 pic 0 online ack
    Apr 19 20:16:38  pic online req, pic 1 type 61637, fpc 1
    Apr 19 20:16:38  fpc slot 1 pic_present 0x1 => 0x3
    Apr 19 20:16:38  fpc_send_pic_online_ack: fpc 1 pic 1 pic_type 0xf0c5 msg_len 72 tlv_len 0
    Apr 19 20:16:38  fpc_send_pic_online_ack: fpc fru slot 1
    Apr 19 20:16:38  pic_get_ingress_shaping_overhead: 1/1 eso val = 0
    Apr 19 20:16:38  pic_get_egress_shaping_overhead: 1/1 eso val = 0
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 2, jnxFruL2Index 2, jnxFruL3Index 0, jnxFruName PIC: 4x GE/XE SFP+ @ 1/1/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 319470272, jnxFruLastPowerOn 319470894)
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperEnabled (entPhysicalIndex 51, entStateAdmin 4, entStateAlarm 0)
    Apr 19 20:16:38  send: fpc 1 pic 1 online ack
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 2, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName Power Supply: Power Supply 0 @ 1/0/*, jnxFruType 7, jnxFruSlot 1)
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Online (jnxFruContentsIndex 2, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName Power Supply: Power Supply 0 @ 1/0/*, jnxFruType 7, jnxFruSlot 1)
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Ok (jnxFruContentsIndex 2, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName Power Supply: Power Supply 0 @ 1/0/*, jnxFruType 7, jnxFruSlot 1)
    Apr 19 20:16:38  fpc 1: power budget received, old = 0 new = 405
    Apr 19 20:16:38  pic attach pic 0, flags 0x40, portcount 24, fpc 1
    Apr 19 20:16:38 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 8, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName PIC:  @ 1/0/*, jnxFruType 11, jnxFruSlot 1)
    Apr 19 20:16:38  pic_set_online: i2c 0xf052 pic 0 fpc 1 state 3 in_issu 0
    Apr 19 20:16:38   pic_type=61522 pic_slot=0 fpc_slot=1 pic_i2c_id=61522
     
    Apr 19 20:16:38  hwdb: entry for pic 61522 at slot 0 in fpc 1 inserted
    Apr 19 20:16:38  ifdev_detach_check: skipping ifd vcp-255/1/3 (what=1)
    Apr 19 20:16:38  FPC 1 PIC 0, attaching clean
    Apr 19 20:16:38  System config mode msg to CM is sent
    Apr 19 20:16:38  fpc 1: new port map received pic slot 0
    Apr 19 20:16:38  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 0
    Apr 19 20:16:38  capability: ge-1/0/0, cap flag = 48135
    Apr 19 20:16:38  looking for interface: ge-1/0/0
    Apr 19 20:16:38  ifd_tree: added ifd ge-1/0/0
    Apr 19 20:16:38  PoE capability: ge-1/0/0, cap flag = 2
    Apr 19 20:16:38  ifdev_create entered ge-1/0/0
    Apr 19 20:16:38  ge-1/0/0: large delay buffer cleared
    Apr 19 20:16:38  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/0 Not SyncE capable
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/0 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:38  PoE capability: ge-1/0/0, cap flag = 2
    Apr 19 20:16:38  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 1
    Apr 19 20:16:38  capability: ge-1/0/1, cap flag = 48135
    Apr 19 20:16:38  looking for interface: ge-1/0/1
    Apr 19 20:16:38  ifd_tree: added ifd ge-1/0/1
    Apr 19 20:16:38  PoE capability: ge-1/0/1, cap flag = 2
    Apr 19 20:16:38  ifdev_create entered ge-1/0/1
    Apr 19 20:16:38  ge-1/0/1: large delay buffer cleared
    Apr 19 20:16:38  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/1 Not SyncE capable
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/1 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:38  PoE capability: ge-1/0/1, cap flag = 2
    Apr 19 20:16:38  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 2
    Apr 19 20:16:38  capability: ge-1/0/2, cap flag = 48135
    Apr 19 20:16:38  looking for interface: ge-1/0/2
    Apr 19 20:16:38  ifd_tree: added ifd ge-1/0/2
    Apr 19 20:16:38  PoE capability: ge-1/0/2, cap flag = 2
    Apr 19 20:16:38  ifdev_create entered ge-1/0/2
    Apr 19 20:16:38  ge-1/0/2: large delay buffer cleared
    Apr 19 20:16:38  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/2 Not SyncE capable
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/2 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:38  PoE capability: ge-1/0/2, cap flag = 2
    Apr 19 20:16:38  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 3
    Apr 19 20:16:38  capability: ge-1/0/3, cap flag = 48135
    Apr 19 20:16:38  looking for interface: ge-1/0/3
    Apr 19 20:16:38  ifd_tree: added ifd ge-1/0/3
    Apr 19 20:16:38  PoE capability: ge-1/0/3, cap flag = 2
    Apr 19 20:16:38  ifdev_create entered ge-1/0/3
    Apr 19 20:16:38  ge-1/0/3: large delay buffer cleared
    Apr 19 20:16:38  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/3 Not SyncE capable
     
    Apr 19 20:16:38  pic_update_ifdev_tlvs: ge-1/0/3 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/3, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 4
    Apr 19 20:16:39  capability: ge-1/0/4, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/4
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/4
    Apr 19 20:16:39  PoE capability: ge-1/0/4, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/4
    Apr 19 20:16:39  ge-1/0/4: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/4 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/4 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/4, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 5
    Apr 19 20:16:39  capability: ge-1/0/5, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/5
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/5
    Apr 19 20:16:39  PoE capability: ge-1/0/5, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/5
    Apr 19 20:16:39  ge-1/0/5: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/5 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/5 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/5, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 6
    Apr 19 20:16:39  capability: ge-1/0/6, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/6
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/6
    Apr 19 20:16:39  PoE capability: ge-1/0/6, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/6
    Apr 19 20:16:39  ge-1/0/6: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/6 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/6 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/6, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 7
    Apr 19 20:16:39  capability: ge-1/0/7, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/7
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/7
    Apr 19 20:16:39  PoE capability: ge-1/0/7, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/7
    Apr 19 20:16:39  ge-1/0/7: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/7 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/7 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/7, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 8
    Apr 19 20:16:39  capability: ge-1/0/8, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/8
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/8
    Apr 19 20:16:39  PoE capability: ge-1/0/8, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/8
    Apr 19 20:16:39  ge-1/0/8: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/8 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/8 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/8, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 9
    Apr 19 20:16:39  capability: ge-1/0/9, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/9
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/9
    Apr 19 20:16:39  PoE capability: ge-1/0/9, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/9
    Apr 19 20:16:39  ge-1/0/9: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/9 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/9 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/9, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 10
    Apr 19 20:16:39  capability: ge-1/0/10, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/10
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/10
    Apr 19 20:16:39  PoE capability: ge-1/0/10, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/10
    Apr 19 20:16:39  ge-1/0/10: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/10 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/10 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/10, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 11
    Apr 19 20:16:39  capability: ge-1/0/11, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/11
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/11
    Apr 19 20:16:39  PoE capability: ge-1/0/11, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/11
    Apr 19 20:16:39  ge-1/0/11: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/11 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/11 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/11, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 12
    Apr 19 20:16:39  capability: ge-1/0/12, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/12
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/12
    Apr 19 20:16:39  PoE capability: ge-1/0/12, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/12
    Apr 19 20:16:39  ge-1/0/12: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/12 Not SyncE capable
                                            
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/12 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/12, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 13
    Apr 19 20:16:39  capability: ge-1/0/13, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/13
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/13
    Apr 19 20:16:39  PoE capability: ge-1/0/13, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/13
    Apr 19 20:16:39  ge-1/0/13: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/13 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/13 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/13, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 14
    Apr 19 20:16:39  capability: ge-1/0/14, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/14
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/14
    Apr 19 20:16:39  PoE capability: ge-1/0/14, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/14
    Apr 19 20:16:39  ge-1/0/14: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/14 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/14 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:39  PoE capability: ge-1/0/14, cap flag = 2
    Apr 19 20:16:39  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 15
    Apr 19 20:16:39  capability: ge-1/0/15, cap flag = 48135
    Apr 19 20:16:39  looking for interface: ge-1/0/15
    Apr 19 20:16:39  ifd_tree: added ifd ge-1/0/15
    Apr 19 20:16:39  PoE capability: ge-1/0/15, cap flag = 2
    Apr 19 20:16:39  ifdev_create entered ge-1/0/15
    Apr 19 20:16:39  ge-1/0/15: large delay buffer cleared
    Apr 19 20:16:39  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/15 Not SyncE capable
     
    Apr 19 20:16:39  pic_update_ifdev_tlvs: ge-1/0/15 IFM type is Ether, SyncE TLV is added
    Apr 19 20:16:40  PoE capability: ge-1/0/15, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 16
    Apr 19 20:16:40  capability: ge-1/0/16, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/16
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/16
    Apr 19 20:16:40  PoE capability: ge-1/0/16, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/16
    Apr 19 20:16:40  ge-1/0/16: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/16 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/16 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/16, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 17
    Apr 19 20:16:40  capability: ge-1/0/17, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/17
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/17
    Apr 19 20:16:40  PoE capability: ge-1/0/17, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/17
    Apr 19 20:16:40  ge-1/0/17: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/17 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/17 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/17, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 18
    Apr 19 20:16:40  capability: ge-1/0/18, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/18
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/18
    Apr 19 20:16:40  PoE capability: ge-1/0/18, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/18
    Apr 19 20:16:40  ge-1/0/18: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/18 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/18 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/18, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 19
    Apr 19 20:16:40  capability: ge-1/0/19, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/19
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/19
    Apr 19 20:16:40  PoE capability: ge-1/0/19, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/19
    Apr 19 20:16:40  ge-1/0/19: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/19 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/19 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/19, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 20
    Apr 19 20:16:40  capability: ge-1/0/20, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/20
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/20
    Apr 19 20:16:40  PoE capability: ge-1/0/20, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/20
    Apr 19 20:16:40  ge-1/0/20: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/20 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/20 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/20, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 21
    Apr 19 20:16:40  capability: ge-1/0/21, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/21
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/21
    Apr 19 20:16:40  PoE capability: ge-1/0/21, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/21
    Apr 19 20:16:40  ge-1/0/21: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/21 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/21 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/21, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 22
    Apr 19 20:16:40  capability: ge-1/0/22, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/22
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/22
    Apr 19 20:16:40  PoE capability: ge-1/0/22, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/22
    Apr 19 20:16:40  ge-1/0/22: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/22 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/22 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/22, cap flag = 2
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf052, hw qs 8 supported qs 8, flags 0x0, pic port 23
    Apr 19 20:16:40  capability: ge-1/0/23, cap flag = 48135
    Apr 19 20:16:40  looking for interface: ge-1/0/23
    Apr 19 20:16:40  ifd_tree: added ifd ge-1/0/23
    Apr 19 20:16:40  PoE capability: ge-1/0/23, cap flag = 2
    Apr 19 20:16:40  ifdev_create entered ge-1/0/23
    Apr 19 20:16:40  ge-1/0/23: large delay buffer cleared
    Apr 19 20:16:40  fpc_is_synce_cap: FPC i2c: 0x9ed, PIC type: 0xf052
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/23 Not SyncE capable
     
    Apr 19 20:16:40  pic_update_ifdev_tlvs: ge-1/0/23 IFM type is Ether, SyncE TLV is added
     
    Apr 19 20:16:40  PoE capability: ge-1/0/23, cap flag = 2
    Apr 19 20:16:40  ge-1/0/0: notify CM 1
    Apr 19 20:16:40  ge-1/0/1: notify CM 1
    Apr 19 20:16:40  ge-1/0/2: notify CM 1
    Apr 19 20:16:40  ge-1/0/3: notify CM 1
    Apr 19 20:16:40  ge-1/0/4: notify CM 1
    Apr 19 20:16:40  ge-1/0/5: notify CM 1
    Apr 19 20:16:40  ge-1/0/6: notify CM 1
    Apr 19 20:16:40  ge-1/0/7: notify CM 1  
    Apr 19 20:16:40  ge-1/0/8: notify CM 1
    Apr 19 20:16:40  ge-1/0/9: notify CM 1
    Apr 19 20:16:40  ge-1/0/10: notify CM 1
    Apr 19 20:16:40  ge-1/0/11: notify CM 1
    Apr 19 20:16:40  ge-1/0/12: notify CM 1
    Apr 19 20:16:40  ge-1/0/13: notify CM 1
    Apr 19 20:16:40  ge-1/0/14: notify CM 1
    Apr 19 20:16:40  ge-1/0/15: notify CM 1
    Apr 19 20:16:40  ge-1/0/16: notify CM 1
    Apr 19 20:16:40  ge-1/0/17: notify CM 1
    Apr 19 20:16:40  ge-1/0/18: notify CM 1
    Apr 19 20:16:40  ge-1/0/19: notify CM 1
    Apr 19 20:16:40  ge-1/0/20: notify CM 1
    Apr 19 20:16:40  ge-1/0/21: notify CM 1
    Apr 19 20:16:40  ge-1/0/22: notify CM 1
    Apr 19 20:16:40  ge-1/0/23: notify CM 1
    Apr 19 20:16:40 CHASSISD_SNMP_TRAP6: SNMP trap generated: Fan/Blower Removed (jnxContentsContainerIndex 4, jnxContentsL1Index 2, jnxContentsL2Index 1, jnxContentsL3Index 1, jnxContentsDescr FAN: Fan 1 @ 1/0/0, jnxOperatingState/Temp 1)
    Apr 19 20:16:40  PIC (fpc 1 pic 0) message operation: add. ifd count 24, flags 0x3 in mesg
    Apr 19 20:16:40  op 1 for ifd ge-1/0/0
    Apr 19 20:16:40  pic attach pic 1, flags 0x40, portcount 4, fpc 1
    Apr 19 20:16:40 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 8, jnxFruL1Index 2, jnxFruL2Index 2, jnxFruL3Index 0, jnxFruName PIC: 4x GE/XE SFP+ @ 1/1/*, jnxFruType 11, jnxFruSlot 1)
    Apr 19 20:16:40  pic_set_online: i2c 0xf0c5 pic 1 fpc 1 state 3 in_issu 0
    Apr 19 20:16:40   pic_type=61637 pic_slot=1 fpc_slot=1 pic_i2c_id=61637
     
    Apr 19 20:16:40  hwdb: entry for pic 61637 at slot 1 in fpc 1 inserted
    Apr 19 20:16:40  ifdev_detach_check: skipping ifd vcp-255/1/3 (what=1)
    Apr 19 20:16:40  FPC 1 PIC 1, attaching clean
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 0, nchannels:0
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf0c5, hw qs 8 supported qs 8, flags 0x0, pic port 0
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 1, nchannels:0
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf0c5, hw qs 8 supported qs 8, flags 0x0, pic port 1
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 2, nchannels:0
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf0c5, hw qs 8 supported qs 8, flags 0x0, pic port 2
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 3, nchannels:0
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf0c5, hw qs 8 supported qs 8, flags 0x0, pic port 3
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 0, nchannels:0
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf0c5, hw qs 8 supported qs 8, flags 0x0, pic port 0
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 1, nchannels:0
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf0c5, hw qs 8 supported qs 8, flags 0x0, pic port 1
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 2, nchannels:0
    Apr 19 20:16:40  create_pic_entry: pic i2c 0xf0c5, hw qs 8 supported qs 8, flags 0x0, pic port 2
    Apr 19 20:16:40  Creating pic entry, baseport 0, nports 4, port 3, nchannels:0
    Apr 19 20:16:40  System config mode msg to CM is sent
    Apr 19 20:16:40  op 1 for ifd ge-1/0/1
    Apr 19 20:16:40  op 1 for ifd ge-1/0/2
    Apr 19 20:16:40  fpc 1: new port map received pic slot 0
    Apr 19 20:16:40  PoE capability: ge-1/0/0, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/1, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/2, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/3, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/4, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/5, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/6, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/7, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/8, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/9, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/10, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/11, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/12, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/13, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/14, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/15, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/16, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/17, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/18, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/19, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/20, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/21, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/22, cap flag = 2
    Apr 19 20:16:40  PoE capability: ge-1/0/23, cap flag = 2
    Apr 19 20:16:40  op 1 for ifd ge-1/0/3
    Apr 19 20:16:40  fpc 1: new port map received pic slot 1
    Apr 19 20:16:40  PoE capability: xe-1/1/0, cap flag = 0
    Apr 19 20:16:40  PoE capability: xe-1/1/1, cap flag = 0
    Apr 19 20:16:40  PoE capability: xe-1/1/2, cap flag = 0
    Apr 19 20:16:40  PoE capability: xe-1/1/3, cap flag = 0
    Apr 19 20:16:40  PoE capability: ge-1/1/0, cap flag = 0
    Apr 19 20:16:40  PoE capability: ge-1/1/1, cap flag = 0
    Apr 19 20:16:40  PoE capability: ge-1/1/2, cap flag = 0
    Apr 19 20:16:40  op 1 for ifd ge-1/0/4
    Apr 19 20:16:40  op 1 for ifd ge-1/0/12
    Apr 19 20:16:40  op 1 for ifd ge-1/0/13
    Apr 19 20:16:40  op 1 for ifd ge-1/0/14
    Apr 19 20:16:40  op 1 for ifd ge-1/0/15
    Apr 19 20:16:40  op 1 for ifd ge-1/0/17
    Apr 19 20:16:43  ch_java_mcontrol_ipc_handler/1744: rcvd MCONTROL_MSG_RE_INFO replying back: master: 1 state ISSU_STATE_IDLE/0 chassis->ch_issu_state: ISSU_STATE_IDLE/0
    Apr 19 20:16:43  ch_java_mcontrol_force_send_re_info/1084 : sending MCONTROL_MSG_RE_INFO: master: 1  state ISSU_STATE_IDLE/0 old_issu_state: ISSU_STATE_IDLE/0
    Apr 19 20:16:43  ch_java_mcontrol_force_send_re_info/1113 : NOT in ISSU window MCONTROL_MSG_RE_INFO: master: 1 state ISSU_STATE_IDLE/0
    Apr 19 20:16:43  got lldp req poe info msg from lldpd intf:ge-1/0/0.0
    Apr 19 20:16:43  got lldp req poe info msg from lldpd intf:ge-1/0/0.0
    Apr 19 20:16:43  got lldp req poe info msg from lldpd intf:ge-1/0/1.0
    Apr 19 20:16:43  got lldp req poe info msg from lldpd intf:ge-1/0/1.0
    Apr 19 20:16:43  got lldp req poe info msg from lldpd intf:ge-1/0/4.0
    Apr 19 20:16:43  got lldp req poe info msg from lldpd intf:ge-1/0/4.0
    Apr 19 20:16:48 CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Ok (jnxFruContentsIndex 4, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FAN: FAN @ 1/*/*, jnxFruType 13, jnxFruSlot 1)
    Apr 19 20:16:48 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 4, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FAN: FAN @ 1/*/*, jnxFruType 13, jnxFruSlot 1)
    Apr 19 20:16:48  pic_copy_port_info:Got SFP Rev=REV 01, Pno=740-031980, Sno=F1940008156
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/14.0
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/14.0
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/13.0
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/13.0
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/17.0
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/17.0
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/15.0
    Apr 19 20:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/15.0
    Apr 19 20:16:52  got lldp req poe info msg from lldpd intf:ge-1/0/12.0
    Apr 19 20:16:52  got lldp req poe info msg from lldpd intf:ge-1/0/12.0
    Apr 19 20:17:55 CHASSISD_SNMP_TRAP6: SNMP trap generated: Fan/Blower OK (jnxContentsContainerIndex 4, jnxContentsL1Index 2, jnxContentsL2Index 1, jnxContentsL3Index 1, jnxContentsDescr FAN: Fan 1 @ 1/0/0, jnxOperatingState/Temp 2)
    
    {master:0}

     

     

    Are there any other logs i could look at?

     

     

    oot@juniper1> show chassis environment 
    Class Item                           Status     Measurement
    Power FPC 0 Power Supply 0           OK        
          FPC 1 Power Supply 0           OK        
    Temp  FPC 0 CPU                      OK         64 degrees C / 147 degrees F
          FPC 0 EX-PFE1                  OK         69 degrees C / 156 degrees F
          FPC 0 GEPHY1                   OK         34 degrees C / 93 degrees F
          FPC 0 Fan Exhaust              OK         56 degrees C / 132 degrees F
          FPC 0 SFP+ PHY                 OK         48 degrees C / 118 degrees F
          FPC 0 Local Sensor             OK         49 degrees C / 120 degrees F
          FPC 1 CPU                      OK         63 degrees C / 145 degrees F
          FPC 1 EX-PFE1                  OK         68 degrees C / 154 degrees F
          FPC 1 GEPHY1                   OK         35 degrees C / 95 degrees F
          FPC 1 Fan Exhaust              OK         55 degrees C / 131 degrees F
          FPC 1 SFP+ PHY                 OK         49 degrees C / 120 degrees F
          FPC 1 Local Sensor             OK         49 degrees C / 120 degrees F
    Fans  FPC 0 Fan 1                    OK         Spinning at normal speed
          FPC 0 Fan 2                    OK         Spinning at normal speed
          FPC 1 Fan 1                    OK         Spinning at normal speed
          FPC 1 Fan 2                    OK         Spinning at normal speed
    
    oot@juniper1:RE:0% cli
    {master:0}
    root@juniper1> show chassis fpc pic-status 
    Slot 0   Online       EX3300 24-Port POE+
      PIC 0  Online       24x 10/100/1000 Base-T
      PIC 1  Online       4x GE/XE SFP+
    Slot 1   Online       EX3300 24-Port POE+
      PIC 0  Online       24x 10/100/1000 Base-T
      PIC 1  Online       4x GE/XE SFP+
    
    {master:0}
    
    root@juniper1> show chassis fpc 
                         Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
    Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
      0  Online                    5          0                           1024                  47
      1  Online                    4          0                           1024                  42
      2  Empty           
      3  Empty           
      4  Empty           
      5  Empty           
      6  Empty           
      7  Empty           
      8  Empty           
      9  Empty           
    
    
    root@juniper1> show chassis hardware 
    Hardware inventory:
    Item             Version  Part number  Serial number     Description
    Chassis                                GE0214442561      Virtual Chassis
    Routing Engine 0 REV 15   750-034300   GE0214442561      EX3300 24-Port POE+
    Routing Engine 1 REV 15   750-034300   GE0214088662      EX3300 24-Port POE+
    FPC 0            REV 15   750-034300   GE0214442561      EX3300 24-Port POE+
      CPU                     BUILTIN      BUILTIN           FPC CPU
      PIC 0                   BUILTIN      BUILTIN           24x 10/100/1000 Base-T
      PIC 1          REV 15   750-034300   GE0214442561      4x GE/XE SFP+
        Xcvr 3       REV 01   740-031980   F1940008155       SFP+-10G-SR
      Power Supply 0                                         PS 550W AC
      Fan Tray                                               Fan Tray
    FPC 1            REV 15   750-034300   GE0214088662      EX3300 24-Port POE+
      CPU                     BUILTIN      BUILTIN           FPC CPU
      PIC 0                   BUILTIN      BUILTIN           24x 10/100/1000 Base-T
      PIC 1          REV 15   750-034300   GE0214088662      4x GE/XE SFP+
        Xcvr 3       REV 01   740-031980   F1940008156       SFP+-10G-SR
      Power Supply 0                                         PS 550W AC
      Fan Tray                                               Fan Tray
    

    Thanks in advance,

    J

     



  • 2.  RE: EX3300 Getting periodic disconnects of VC via SFP

     
    Posted 04-19-2020 22:21

    Hi josha,

     

    Firstly, as this is a virtual-chassis, the title should be "VC" not "VPC", as the latter is misleading and makes us think of  MC-LAG in Juniper world :).  Anyways your CLI output is giving it away...

     

    Here's what you can check:

    1) Please clarify the connectivity using "show virtual-chassis".

    2) Are the two physical interfaces/links actually flapping prior to the disconnection?

    3) What's the distance between the two EXs?

    4) Please check the logs prior to "FPC 1 is removed" that gives a clue what led to this.

    5) Hope you have "no-split-detection" in the virtual-chassis config to protect against disruption.

     

    Hope this helps.

    Regards,
    -r.

    --------------------------------------------------

    If this solves your problem, please mark this post as "Accepted Solution."
    Kudos are always appreciated :).



  • 3.  RE: EX3300 Getting periodic disconnects of VC via SFP

    Posted 04-20-2020 01:48

    Hi mriyaz!

     

    Thank you for  the response.

     

    so 

     

    1)

     

    root@juniper1> show virtual-chassis 
    
    Virtual Chassis ID: c074.ff9f.5522
    Virtual Chassis Mode: Enabled
                                                    Mstr           Mixed Route Neighbor List
    Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
    0 (FPC 0)  Prsnt    GE0214442561 ex3300-24p     128   Master*     NA  VC   1  vcp-255/1/3
    1 (FPC 1)  Prsnt    GE0214088662 ex3300-24p     128   Backup      NA  VC   0  vcp-255/1/3
    
    Member ID for next new member: 2 (FPC 2)
    
    {master:0}
    

    2) I don't see any flapps:

    Physical interface: vcp-255/1/3, Enabled, Physical link is Up
      Interface index: 68, SNMP ifIndex: 107
      Link-level type: Virtual-Chassis-Interface, MTU: 1514, LAN-PHY mode, Speed: 10000mbps, Media type: Fiber
      Device flags   : Present Running
      Link flags     : None
      CoS queues     : 0 supported, 0 maximum usable queues
      Current address: Unspecified, Hardware address: Unspecified
      Last flapped   : Never
        Input packets : 244698
        Output packets: 244809
      Interface transmit statistics: Disabled
    
      Logical interface vcp-255/1/3.32768 (Index 3) (SNMP ifIndex 108)
        Flags: Encapsulation: Unspecified
        Input packets : 244698
        Output packets: 244809
    
    

    3) I Believe it's about 25 meters of cable CAT6A

    4) Last message before it says removed is 4 hours earlier:

    Apr 19 16:16:46  got lldp req poe info msg from lldpd intf:ge-1/0/0.0
    Apr 19 16:16:46  got lldp req poe info msg from lldpd intf:ge-1/0/0.0
    Apr 19 16:16:46  got lldp req poe info msg from lldpd intf:ge-1/0/4.0
    Apr 19 16:16:46  got lldp req poe info msg from lldpd intf:ge-1/0/4.0
    Apr 19 16:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/13.0
    Apr 19 16:16:51  got lldp req poe info msg from lldpd intf:ge-1/0/13.0
    Apr 19 16:16:52  got lldp req poe info msg from lldpd intf:ge-1/0/14.0
    Apr 19 16:16:52  got lldp req poe info msg from lldpd intf:ge-1/0/14.0
    Apr 19 16:16:52  got lldp req poe info msg from lldpd intf:ge-1/0/12.0
    Apr 19 16:16:52  got lldp req poe info msg from lldpd intf:ge-1/0/12.0
    Apr 19 16:16:53  got lldp req poe info msg from lldpd intf:ge-1/0/15.0
    Apr 19 16:16:53  got lldp req poe info msg from lldpd intf:ge-1/0/15.0
    Apr 19 16:16:53  got lldp req poe info msg from lldpd intf:ge-1/0/17.0
    Apr 19 16:16:53  got lldp req poe info msg from lldpd intf:ge-1/0/17.0
    Apr 19 16:17:53 CHASSISD_SNMP_TRAP6: SNMP trap generated: Fan/Blower OK (jnxContentsContainerIndex 4, jnxContentsL1Index 2, jnxContentsL2Index 1, jnxContentsL3Index 1, jnxContentsDescr FAN: Fan 1 @ 1/0/0, jnxOperatingState/Temp 2)
    Apr 19 20:16:30  FPC 1 removed

    5) Yes I did that configuration early on.

     

    Thanks for explaing VC vs VPC, I don't have all the abbrivations etc correct, it's good to learn.



  • 4.  RE: EX3300 Getting periodic disconnects of VC via SFP
    Best Answer

    Posted 04-27-2020 00:18

    So a follow up. Turns out to be two bad cat6a patch cables causing this issue.

    Running a network testing tool on them didn't show any problems, but when replaced I got it working without disconnects.

     

    Regards, J