SRX Services Gateway
Highlighted
SRX Services Gateway

SRX1400 - Urgent Request

‎01-13-2016 02:17 AM

Hi Experts,

 

Basically, I have an issue with our cluster SRX1400 in which intermittently the CPU goes to 100% and SRX stop responding ...  please see the error as below

 

PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 1 PIC 0 CPU utilization exceeds threshold, current value=93

 

I suspect there was issue with standby peer which is currently shutdown and cluster is running on single device 

 

I am also seeing the error below continously 

 

node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@2198:(errno=1000) create nsp tunnel failed 1
Jan 13 10:05:14 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3069:(errno=1000) tunnel session add(gr-0/0/0) failed
Jan 13 10:05:24 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@2198:(errno=1000) create nsp tunnel failed 1
Jan 13 10:05:24  node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3069:(errno=1000) tunnel session add(gr-0/0/0) failed
Jan 13 10:05:34 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@2198:(errno=1000) create nsp tunnel failed 1
Jan 13 10:05:34 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3069:(errno=1000) tunnel session add(gr-0/0/0) failed
Jan 13 10:05:44 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@2198:(errno=1000) create nsp tunnel failed 1
Jan 13 10:05:44  node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3069:(errno=1000) tunnel session add(gr-0/0/0) failed
Jan 13 10:05:54  node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@2198:(errno=1000) create nsp tunnel failed 1
Jan 13 10:05:54 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3069:(errno=1000) tunnel session add(gr-0/0/0) failed

 

Can you please sugguest on urgent basis 

 

Thanks 

 

FAZ

3 REPLIES 3
Highlighted
SRX Services Gateway

Re: SRX1400 - Urgent Request

‎01-13-2016 02:51 AM
Hi,

Could you attach the following :

show verison
show security monitoring fpc 1
show log chassisd
Regards,
A'bed AL-R.
[JNCSP-SEC JNCDA JNCIS-ENT Ingenious Champion|Sec]
https://srxtech.wordpress.com
Highlighted
SRX Services Gateway

Re: SRX1400 - Urgent Request

‎01-13-2016 02:56 AM

Hi thank you for the reply please see below output 

 

show log chassisd | last 100

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 3
Jan 12 17:43:29 SCC: Creating pic entry, baseport 0, nports 7, port 4

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 4
Jan 12 17:43:29 SCC: Creating pic entry, baseport 0, nports 7, port 5

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 5
Jan 12 17:43:29 SCC: Creating pic entry, baseport 0, nports 7, port 6

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 6
Jan 12 17:43:29 SCC: Creating pic entry, baseport 7, nports 3, port 7

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 7
Jan 12 17:43:29 SCC: Creating pic entry, baseport 7, nports 3, port 8

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 8
Jan 12 17:43:29 SCC: Creating pic entry, baseport 7, nports 3, port 9

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 9
Jan 12 17:43:29 SCC: Creating pic entry, baseport 10, nports 2, port 10

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 10
Jan 12 17:43:29 SCC: Created pic for ge-0/0/10

Jan 12 17:43:29 SCC: Creating pic entry, baseport 10, nports 2, port 11

Jan 12 17:43:29 SCC: create_pic_entry: pic i2c 0xa08, hw qs 8 supported qs 4, flags 0x1, pic port 11
Jan 12 17:43:29 SCC: Created pic for ge-0/0/11

Jan 12 17:43:29 CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for ge-0/0/10
Jan 12 17:43:29 SCC: ifdev_create entered ge-0/0/10
Jan 12 17:43:29 SCC: ha_update_ifdev_info entered ge-0/0/10 I2CID a08 PORT 10 is HA Cntl port
Jan 12 17:43:29 CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for ge-0/0/11
Jan 12 17:43:29 SCC: ifdev_create entered ge-0/0/11
Jan 12 17:43:29 SCC: ha_update_ifdev_info entered ge-0/0/11 I2CID a08 PORT 11 is HA Cntl port
Jan 12 17:43:29 LCC: ignoring PIC message on LCC
Jan 12 17:43:38 LCC: pic_copy_port_info:Got cable_type for FPC 0 PIC 0 port 6 cable num=0, str=
Jan 12 17:43:38 LCC: pic_copy_port_info:Got cable_type for FPC 0 PIC 0 port 7 cable num=59, str=10GBASE SR
Jan 12 17:43:38 LCC: pic_copy_port_info:Got SFP Rev=REV 01, Pno=740-021308, Sno=AKJ0SSR
Jan 12 17:43:38 LCC: pic_copy_port_info:Got cable_type for FPC 0 PIC 0 port 8 cable num=59, str=10GBASE SR
Jan 12 17:43:38 LCC: pic_copy_port_info:Got SFP Rev=REV 01, Pno=740-021308, Sno=14T511100977
Jan 12 17:43:38 LCC: pic_copy_port_info:Got cable_type for FPC 0 PIC 0 port 9 cable num=59, str=10GBASE SR
Jan 12 17:43:38 LCC: pic_copy_port_info:Got SFP Rev=REV 01, Pno=740-021308, Sno=AKJ0SJE
Jan 12 17:43:38 LCC: pic_copy_port_info:Got cable_type for FPC 0 PIC 0 port 10 cable num=0, str=
Jan 12 17:43:38 LCC: pic_copy_port_info:Got cable_type for FPC 0 PIC 0 port 11 cable num=0, str=
Jan 12 17:45:38 LCC: ipc pipe 0x206d600 created
Jan 12 17:45:38 LCC: ch_signal_proc: Sent signal 1 to tnp.sntpd, pid=1207
Jan 12 17:45:39 LCC: pic online req, pic 0 type 747, fpc 1
Jan 12 17:45:39 LCC: fpc_recv_pic_online_req() pic=0x1e5a000 fpc=0x1e4e000 (CH_MAX_FPCS=4, CH_MAX_PICS=4)
Jan 12 17:45:39 LCC: From LCC 0 send pic online req for fpc 1 pic 0 to SCC
Jan 12 17:45:39 SCC: pic_online_req for fpc 1, pic 0 lcc_slot 0 in lcc_recv_pic_online_req
Jan 12 17:45:39 SCC: lcc_send_pic_online_ack: On Switch-chassis: fpc 1 pic 0 pic_type 0x2eb msg_len 20 tlv_len 0
Jan 12 17:45:39 SCC: From SCC send: fru 34013184 lcc_slot 0 online ack to LCC
Jan 12 17:45:39 SCC: From Switch-Chassis send: fpc 1 pic 0 online ack to LCC
Jan 12 17:45:39 LCC: At LCC 0 recv pic online ack for fpc 1 pic 0 from SCC, length 28
Jan 12 17:45:39 LCC: fpc_send_pic_online_ack: fpc 1 pic 0 pic_type 0x2eb msg_len 64 tlv_len 0
Jan 12 17:45:39 LCC: fpc_send_pic_online_ack: fpc fru slot 1
Jan 12 17:45:39 LCC: pic_get_egress_shaping_overhead: 1/0 eso val = 0
Jan 12 17:45:39 CHASSISD_SNMP_TRAP10: SNMP trap generated: FRU power on (jnxFruContentsIndex 8, jnxFruL1Index 2, jnxFruL2Index 1, jnxFruL3Index 0, jnxFruName node0 PIC: @ 1/0/*, jnxFruType 11, jnxFruSlot 1, jnxFruOfflineReason 2, jnxFruLastPowerOff 0, jnxFruLastPowerOn 27993)
Jan 12 17:45:39 LCC: send: fpc 1 pic 0 online ack
Jan 12 17:45:39 LCC: fpc_a40_set_cp_state: state = 1
Jan 12 17:45:39 LCC: setting CP state to Up
Jan 12 17:45:39 LCC: pic attach pic 0, flags 0x100, portcount 10, fpc 1
Jan 12 17:45:39 LCC: pic_set_online: i2c 0x2eb pic 0 fpc 1 state 3 in_issu 0
Jan 12 17:45:39 LCC: pic_type=747 pic_slot=0 fpc_slot=1 pic_i2c_id=747

Jan 12 17:45:39 LCC: hwdb: entry for pic 747 at slot 0 in fpc 1 inserted
Jan 12 17:45:39 LCC: FPC 1 PIC 0, attaching clean
Jan 12 17:45:39 LCC: not in vc mode
Jan 12 17:45:39 SCC: lcc_recv_pic_attach: pic attach pic 0, flags 0x100, portcount 10, fpc 1
Jan 12 17:45:39 SCC: pic_set_online: i2c 0x2eb pic 0 fpc 1 state 1 in_issu 0
Jan 12 17:45:39 SCC: pic_type=747 pic_slot=0 fpc_slot=1 pic_i2c_id=747

Jan 12 17:45:39 SCC: fpc slot 1 pic_present 0x0 => 0x1
Jan 12 17:45:39 SCC: FPC 1 PIC 0, attaching clean
Jan 12 17:45:39 SCC: Creating pic entry, baseport 0, nports 1, port 0

Jan 12 17:45:39 SCC: create_pic_entry: pic i2c 0x2eb, hw qs 8 supported qs 8, flags 0x0, pic port 0
Jan 12 17:45:39 SCC: Created pic for mt-1/0/0

Jan 12 17:45:39 CHASSISD_IFDEV_CREATE_NOTICE: create_pics: created interface device for mt-1/0/0
Jan 12 17:45:39 SCC: ifdev_create entered mt-1/0/0
Jan 12 17:45:39 SCC: mt-1/0/0: large delay buffer cleared
Jan 12 17:45:39 SCC: lcc_a40_update_ch_info_on_pic_attach: On SCC: recvd PIC attach for fpc 1 pic 0, flags 0x100, portcount 10, belonging to ch_id 0
Jan 12 17:45:39 SCC: update_ch_info_llf_mode: ch_id:0 pic_slot:2 = 5

Jan 12 17:45:39 SCC: ch_info_pic_state_blob_set: ch_id 0, addflag 0x0 key 0x6 num_spu 1 num_npc 1 app_mode 0x0 ioc_npc_map 0x3 ch_info_version 0x1
Jan 12 17:45:39 SCC: PIC[0][0] => ioc PIC Up
Jan 12 17:45:39 SCC: PIC[1][0] => cp-flow combo PIC Up
Jan 12 17:45:39 SCC: PIC[3][0] => npc PIC Up
Jan 12 17:45:39 SCC: ch_info_update: UPDATE ch 0, fpc_slot 1, pic_slot 0, state 0x5 SUCCEEDED
Jan 12 17:45:39 SCC: ch_info_update: fpc_slot 1, pic_slot 0, ioc_npc_map 0x0
Jan 12 17:45:39 SCC: Chassis 0 all service PICs is up
Jan 12 17:45:39 SCC: PIC (fpc 1 pic 0) message operation: add. ifd count 1, flags 0x3 in mesg
Jan 12 17:45:39 LCC: ignoring PIC message on LCC
Jan 13 09:33:20 LCC: ENTER SHOW CRAFT
Jan 13 09:42:07 LCC: ENTER SHOW CRAFT
Jan 13 09:50:33 LCC: ENTER SHOW CRAFT

 

-------------------------------------------------

show security monitoring fpc 1
node0:
--------------------------------------------------------------------------
FPC 1
PIC 0
CPU utilization : 6 %
Memory utilization : 66 %
Current flow session : 9641
Current flow session IPv4: 9641
Current flow session IPv6: 0
Max flow session : 1048576
Current CP session : 9872
Current CP session IPv4: 9872
Current CP session IPv6: 0
Max CP session : 1048576
Total Session Creation Per Second (for last 96 seconds on average): 200
IPv4 Session Creation Per Second (for last 96 seconds on average): 200
IPv6 Session Creation Per Second (for last 96 seconds on average): 0

 

------------------------------------------

 

show version
node0:
--------------------------------------------------------------------------
Hostname: XXXXFW-1
Model: srx1400
JUNOS Software Release [12.1X44-D10.4]

 

Thanks again 

 

FAZ

 

Highlighted
SRX Services Gateway

Re: SRX1400 - Urgent Request

‎01-13-2016 03:17 AM

Hello ,

 

 

1)  PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 1 PIC 0 CPU utilization exceeds threshold, current value=93

 

> This error is basically due to DataPlane CPU going high due to traffic ,  As this is FPC1 PIC0 its seens to be the CP-Session going high . So this have to be troubelshooted realtime . There are 2 possibilities , first is that the CP session craeted during the same have exceeded the thrushold value . So you need to monitor the CP session count if you see this message again  ( show security flow cp-session summary ) . Second could be due to high amount of traffic getting hit during the same . So you need to monitor the interfaces to see if there is any sudden outburst of traffic .

Please open a JTAc ticket next time if you get high CPU , to troubleshoot this realtime.

 

2)  Jan 13 10:05:14 node0.fpc1.pic0 IFP error> ../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3069Smiley Saderrno=1000) tunnel session add(gr-0/0/0) failed

 

This error is when you need the static route if gr-0/0/0 and egress interface are in two different routing-instances. So you need to point the static route of the gr tunnel to the table that have the external interface .

 

Reference :

 

PR 982043 [Confidential] - Odd syslog error with IFP errors
PR 1146306 [Confidential] - 15.1X49-D40 : Citadel : Stress : continuous Err log observed IFP error> ../../../../../../../../src/pfe/usp/control/applications/interface/ifp.c@3176:(errno=1000) tunnel session add(ppe0) failed


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Feedback