SRX Services Gateway
SRX Services Gateway

SRX CLUSTER SECONDARY port not showing

‎12-27-2013 01:03 PM

Dear all,

 

SRX  configured in cluster are working fine after upgradation to new SRX junos 

12.1x-44D25.5 .But after 3-4 days observation suddenly i can see my secondary nodes interfaces are not showing ,But my cluster status are fine. Could anyone tell me or assist me on this. URGENT

JMD
3 REPLIES 3
SRX Services Gateway

Re: SRX CLUSTER SECONDARY port not showing

‎12-27-2013 10:41 PM

As i see on the SRX, FPC is offline in the secondary node. Anyone quick responses on it.

 

how chassis fpc pic-status
node0:
--------------------------------------------------------------------------
Slot 0 Online FPC
PIC 0 Online 8x GE Base PIC

node1:
--------------------------------------------------------------------------
Slot 0 Offline FPC

================================================

 

 

some logs:

 

show log chassisd | last 200
Dec 27 19:13:36 LCC: ifdev_get_mgmt_if_hwaddr:Got fxp0 hardware address len 6 as 40:b4:f0:83:89:86
Dec 27 19:13:36 CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU insertion (jnxFruContentsIndex 9, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName node1 USB Hub, jnxFruType 6, jnxFruSlot 1)
Dec 27 19:13:36 LCC: hwdb: entry for re 2389 at slot 0 inserted
Dec 27 19:13:36 LCC: SRX220 Chassis fan 0 0 added
Dec 27 19:13:36 LCC: hwdb: entry for fan at slot 0 inserted
Dec 27 19:13:36 LCC: setting host 0 master led
Dec 27 19:13:36 LCC: send: clear all chassis class alarms
Dec 27 19:13:36 LCC: FPC 0 power is on
Dec 27 19:13:36 LCC: ch_ipc_reconnect reconnect proceed
Dec 27 19:13:36 LCC: ch_ipc_reconnect_wait for 30 seconds
Dec 27 19:13:36 LCC: ipc pipe 0xc14380 created
Dec 27 19:13:36 CHASSISD_IPC_UNEXPECTED_RECV: Received unexpected message from craftd: type = 4, subtype = 44

Dec 27 19:13:36 LCC: ch_jsrxnle_fru_from_msg: resolved for subtype 263, index=0
Dec 27 19:13:36 LCC: fwdd 0 ready, pipe 0x0xc14380
Dec 27 19:13:36 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting max-tcp-mss_value 0
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting debugmode off
---(more 100%)--- Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting soft-restart on
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting pfeman-reconnect on
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting source-route off
Dec 27 19:13:36 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting multicast-resolve-rate 66
Dec 27 19:13:36 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting multicast-mismatch-rate 50
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting ipv4-key-hash-L3 on
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting mpls-key-hash-2label off
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting multiservice-key-hash off
Dec 27 19:13:36 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting 7 0
Dec 27 19:13:36 LCC: set_kern_pplb_hash_seed net.rnh_pplb_hashseed from sysctlbyname: value 0
Dec 27 19:13:36 LCC: fpc_gen_wait_mask 0x0
Dec 27 19:13:36 LCC: .. power sequencer started ..
Dec 27 19:13:36 LCC: ch_fru_power_sequencer FPC 0 step 0
Dec 27 19:13:36 LCC: FPC 0 power is on
---(more 100%)--- Dec 27 19:13:36 LCC: ... power sequencer finished ...
Dec 27 19:13:36 LCC: fwdd_jsrxnle_pic_attach_retry: retry sending FWDD pic attach message for the failed FPCs
Dec 27 19:13:36 CHASSISD_IPC_UNEXPECTED_RECV: Received unexpected message from craftd: type = 4, subtype = 44

Dec 27 19:13:36 LCC: Got a FPC ready from fwdd fpc 0. Reconnect flag: True
Dec 27 19:13:36 LCC: Port presence on PIC 0

Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FPC 0 setting coredump on
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FPC 0 setting soft-restart on
Dec 27 19:13:36 LCC: fru_set_boolean: send: set_boolean_cmd FPC 0 setting pfeman-reconnect on
Dec 27 19:13:36 LCC: fpc_online_now - slot 0 - Online
Dec 27 19:13:36 LCC: hwdb: entry for fpc 1897 at slot 0 inserted
Dec 27 19:13:36 LCC: fpc_online_now - slot 0 - Online
Dec 27 19:13:36 LCC: Sending FPC 0 ready message to SCC
Dec 27 19:13:36 LCC: pic attach pic 0, flags 0x0, portcount 8, fpc 0
Dec 27 19:13:36 LCC: pic_set_online: i2c 0x0 pic 0 fpc 0 state 1 in_issu 0
Dec 27 19:13:36 LCC: pic_type=1641 pic_slot=0 fpc_slot=0 pic_i2c_id=1641

---(more 100%)--- Dec 27 19:13:36 LCC: fpc slot 0 pic_present 0x0 => 0x1
Dec 27 19:13:36 LCC: hwdb: entry for pic 1641 at slot 0 in fpc 0 inserted
Dec 27 19:13:36 LCC: not in vc mode
Dec 27 19:13:36 LCC: Forwarding pic attach to FWDD fpc 0, pic 0
Dec 27 19:13:36 LCC: Got a pic attach ack from fwdd fpc 0pic 0
Dec 27 19:13:36 LCC: FWDD pic attach ack recd fpc 0, pic 0
Dec 27 19:13:49 LCC: rcv: ch_ipc_dispatch() null ipc read for args 0xc11280 pipe 0xc14380, fru FWDD 0 errno 0
Dec 27 19:13:49 LCC: -- FWDD 0, last request 0, state Online
Dec 27 19:13:49 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 0 offline: Error
Dec 27 19:13:49 LCC: fpc_down slot 0 reason Error cargs 0xc11280
Dec 27 19:13:49 LCC: fpc_srxsme_disconnect slot is 0

Dec 27 19:13:49 LCC: fpc_offline_now - slot 0, reason: Error, error OK transition state 1
Dec 27 19:13:49 LCC: Power off FPC 0
Dec 27 19:13:49 CHASSISD_IPC_WRITE_ERR_NULL_ARGS: FRU has no connection arguments fru_send_msg FWDD
Dec 27 19:13:49 LCC: send: fwdd, fpc 0 powered off
Dec 27 19:13:49 LCC: hwdb: entry for fpc 1897 at slot 0 deleted

Dec 27 19:13:50 LCC: ipc pipe 0xc14380 created
Dec 27 19:13:54 LCC: ch_jsrxnle_fru_from_msg: resolved for subtype 263, index=0
---(more 100%)--- Dec 27 19:13:54 LCC: fwdd 0 ready, pipe 0x0xc14380
Dec 27 19:13:54 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting max-tcp-mss_value 0
Dec 27 19:13:54 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting debugmode off
Dec 27 19:13:54 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting soft-restart on
Dec 27 19:13:54 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting pfeman-reconnect on
Dec 27 19:13:54 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting source-route off
Dec 27 19:13:54 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting multicast-resolve-rate 66
Dec 27 19:13:54 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting multicast-mismatch-rate 50
Dec 27 19:13:54 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting ipv4-key-hash-L3 on
Dec 27 19:13:54 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting mpls-key-hash-2label off
Dec 27 19:13:54 LCC: fru_set_boolean: send: set_boolean_cmd FWDD 0 setting multiservice-key-hash off
Dec 27 19:13:54 LCC: fru_set_integer: send: set_integer_cmd FWDD 0 setting 7 0
---(more 100%)--- Dec 27 19:13:54 LCC: set_kern_pplb_hash_seed net.rnh_pplb_hashseed from sysctlbyname: value 0
Dec 27 19:13:54 LCC: fwdd ready received (msg-reconnect no, reconnect-in-progress yes)
Dec 27 19:14:06 LCC: ch_ipc_reconnect_expired: timer expired
Dec 27 19:14:06 LCC: ch_ipc_reconnect_expired: reconnect window closed
Dec 27 19:14:06 LCC: ch_jsrxnle_reconnect_expired
Dec 27 19:14:06 LCC: reconnect expired: power off fpc 1
Dec 27 19:14:06 LCC: Power off FPC 1
Dec 27 19:14:06 LCC: reconnect expired: power off fpc 2
Dec 27 19:14:06 LCC: Power off FPC 2
Dec 27 19:14:06 LCC: ch_lcc_send_lcc_online_ack: Sending LCC ONLINE ACK to SCC
Dec 27 19:14:06 LCC: fpc_gen_wait_mask 0x0
Dec 27 19:14:06 LCC: .. power sequencer started ..
Dec 27 19:14:06 LCC: ch_fru_power_sequencer FPC 0 step 0
Dec 27 19:14:06 LCC: FPC 0 power is off
Dec 27 19:14:06 LCC: Power on FPC 0
Dec 27 19:14:06 LCC: send: fwdd, fpc 0 powered on
Dec 27 19:14:06 LCC: setup_power_on_timeout FPC 0
Dec 27 19:14:06 LCC: ... power sequencer finished ...
Dec 27 19:14:06 CHASSISD_RECONNECT_SUCCESSFUL: Successfully reconnected on soft restart
---(more 100%)--- Dec 27 19:19:06 CHASSISD_FRU_UNRESPONSIVE_RETRY: Attempt 1 to power on FPC 0 timed out; restarted it
Dec 27 19:19:06 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 0 offline: Restarting unresponsive board
Dec 27 19:19:06 LCC: fpc_down slot 0 reason Restarting unresponsive board cargs 0x0
Dec 27 19:19:06 LCC: fpc_offline_now - slot 0, reason: Restarting unresponsive board, error OK transition state 1
Dec 27 19:19:06 LCC: Power off FPC 0
Dec 27 19:19:06 LCC: send: fwdd, fpc 0 powered off
Dec 27 19:19:06 LCC: fpc_gen_wait_mask 0x0
Dec 27 19:19:06 LCC: updated fpc_wait_mask 0x0
Dec 27 19:19:22 LCC: fru_power_on_state_timer: FPC 0 step 0
Dec 27 19:19:22 LCC: Power on FPC 0
Dec 27 19:19:22 LCC: send: fwdd, fpc 0 powered on
Dec 27 19:19:22 LCC: setup_power_on_timeout FPC 0
Dec 27 19:24:22 CHASSISD_FRU_UNRESPONSIVE_RETRY: Attempt 2 to power on FPC 0 timed out; restarted it
Dec 27 19:24:22 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 0 offline: Restarting unresponsive board
Dec 27 19:24:22 LCC: fpc_down slot 0 reason Restarting unresponsive board cargs 0x0
---(more 100%)--- Dec 27 19:24:22 LCC: fpc_offline_now - slot 0, reason: Restarting unresponsive board, error OK transition state 1
Dec 27 19:24:22 LCC: Power off FPC 0
Dec 27 19:24:22 LCC: send: fwdd, fpc 0 powered off
Dec 27 19:24:22 LCC: fpc_gen_wait_mask 0x0
Dec 27 19:24:22 LCC: updated fpc_wait_mask 0x0
Dec 27 19:24:38 LCC: fru_power_on_state_timer: FPC 0 step 0
Dec 27 19:24:38 LCC: Power on FPC 0
Dec 27 19:24:38 LCC: send: fwdd, fpc 0 powered on
Dec 27 19:24:38 LCC: setup_power_on_timeout FPC 0
Dec 27 19:29:38 CHASSISD_FRU_UNRESPONSIVE_RETRY: Attempt 3 to power on FPC 0 timed out; restarted it
Dec 27 19:29:38 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 0 offline: Restarting unresponsive board
Dec 27 19:29:38 LCC: fpc_down slot 0 reason Restarting unresponsive board cargs 0x0
Dec 27 19:29:38 LCC: fpc_offline_now - slot 0, reason: Restarting unresponsive board, error OK transition state 1
Dec 27 19:29:38 LCC: Power off FPC 0
Dec 27 19:29:38 LCC: send: fwdd, fpc 0 powered off
Dec 27 19:29:38 LCC: fpc_gen_wait_mask 0x0
Dec 27 19:29:38 LCC: updated fpc_wait_mask 0x0
Dec 27 19:29:55 LCC: fru_power_on_state_timer: FPC 0 step 0
---(more 31%)--- Dec 27 19:29:55 LCC: Power on FPC 0
Dec 27 19:29:55 LCC: send: fwdd, fpc 0 powered on
Dec 27 19:29:55 LCC: setup_power_on_timeout FPC 0
Dec 27 19:34:55 CHASSISD_FRU_UNRESPONSIVE: Error for FPC 0: unresponsive to attempts to start it; left it offline
Dec 27 19:34:55 CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 0 offline: Error
Dec 27 19:34:55 LCC: fpc_down slot 0 reason Error cargs 0x0
Dec 27 19:34:55 LCC: fpc_offline_now - slot 0, reason: Error, error Unresponsive transition state 1
Dec 27 19:34:55 LCC: Power off FPC 0
Dec 27 19:34:55 LCC: send: fwdd, fpc 0 powered off
Dec 27 19:34:55 LCC: send: red alarm set, device FPC 0, reason FPC 0 Hard errors
Dec 27 19:34:55 CHASSISD_SNMP_TRAP7: SNMP trap generated: Fru Failed (jnxFruContentsIndex 7, jnxFruL1Index 4, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName node1 FPC: FPC @ 0/*/*, jnxFruType 3, jnxFruSlot 3)
Dec 27 19:34:55 LCC: fpc_gen_wait_mask 0x0
Dec 27 19:34:55 LCC: updated fpc_wait_mask 0x0
Dec 27 19:34:55 CHASSISD_IPC_UNEXPECTED_RECV: Received unexpected message from craftd: type = 4, subtype = 43

 

 

===========================================================

 

if i see the logs and alarms its showing hard error, could anyone please tell me what to do?  What may be the reason for it.

JMD
SRX Services Gateway

Re: SRX CLUSTER SECONDARY port not showing

‎12-28-2013 11:29 AM

Hi

 

If I were you I would contact JTAC with all these messages.

 

Now if I happened to not have a support contract... FPC0 on this SRX

is offline so box seems to be not working. It is a backup node of the cluster, right?

Then I would reboot it (only this node) to see if the problem is fixed or when it repeats.

I don't expect the reboot to affect primary node however this is not guaranteed

in such a situation.

 

If the issue started after sw upgrade then it is likely a bug, but this is not 100%.

Could be a hw fault as well.  Again ideally you need JTAC help in such a situation.

 

Best Regards,
PK

Juniper Ambassador, Juniper Networks Certified Instructor,
JNCIE-SEC #98, JNCIE-ENT #393, JNCIE-SP #2253
Twitter: @JuniperTrain
GitHub: https://github.com/pklimai
[Juniper Authorized Education & Support in Russia]
SRX Services Gateway

Re: SRX CLUSTER SECONDARY port not showing

‎01-02-2014 05:41 AM

Hardware issue...confirmed from JTAC. Thanks for your reply anyws already i tried those methods.

JMD