SRX Services Gateway
SRX Services Gateway

Anyone here has facing IOC II CPU utlization 100%

3 weeks ago

Hi all,

 

Currently i have 4 pair SRX5800 chassis cluster and all the cluster facing issue on FPC11 (IOC II - that all the interface traffic) CPU 100% High Utilization. I already open JTAC case and JTAC said need to reseat the FPC but when i request PR no then they said no PR number. So it difficult to get RFC to execute activity reseat FPC. So i ask in this forum incase have someone facing issue same as me.

 

 

{primary:node0}
root@SRX5800> show chassis fpc
node0:
--------------------------------------------------------------------------
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer
0 Online 30 2 0 2 2 2 1024 5 26
1 Online 26 20 0 20 20 20 1024 5 26
2 Online 26 2 0 2 2 2 1024 5 26
3 Online 26 2 0 2 2 2 1024 5 26
4 Empty
5 Empty
6 Empty
7 Empty
8 Empty
9 Empty
10 Online 28 10 0 9 9 9 2048 15 14
11 Online 28 100 0 99 99 99 2048 16 27

node1:
--------------------------------------------------------------------------
Temp CPU Utilization (%) CPU Utilization (%) Memory Utilization (%)
Slot State (C) Total Interrupt 1min 5min 15min DRAM (MB) Heap Buffer
0 Online 31 2 0 2 2 2 1024 5 26
1 Online 27 21 0 20 20 20 1024 5 26
2 Online 26 2 0 2 2 2 1024 5 26
3 Online 26 2 0 2 2 2 1024 5 26
4 Empty
5 Empty
6 Empty
7 Empty
8 Empty
9 Empty
10 Online 30 9 0 9 10 10 2048 15 14
11 Online 30 100 0 99 99 99 2048 16 27

 

 

Thanks

 

4 REPLIES 4
SRX Services Gateway

Re: Anyone here has facing IOC II CPU utlization 100%

2 weeks ago

Hi all,

 

Any one can show the step how to login into IOC II in SRX5800?

 

{primary:node0}
test@SRX5800-Cluster> start shell user root
root@SRX5800-Cluster1% node0.fpc11
node0.fpc11: Command not found.

 

Thanks and appreciate any help

 

SRX Services Gateway

Re: Anyone here has facing IOC II CPU utlization 100%

2 weeks ago

try %vty node0.fpc11

 

However, not recommended to go into IOCs without JTAc supervision or suggestion.


Shailesh
[KUDOS PLEASE! If you think I earned it!
If this solution worked for you please flag my post as an "Accepted Solution" so others can benefit..]
SRX Services Gateway

Re: Anyone here has facing IOC II CPU utlization 100%

2 weeks ago

Hi ssn,

 

If i execute command below as per requested by JTAC, is it will have any service impact in your knowledge?

 

# show sched            <<<<<   Please check the PID of the Top Thread

# show threads cpu

# show threads verbose

# show threads XXX      <<<<<   Please collect it for Top Thread

 

 

Thanks and appreciate your feedback

 

SRX Services Gateway

Re: Anyone here has facing IOC II CPU utlization 100%

2 weeks ago

Hi all,

 

Next level JTAC already give some analysis as per below. Anyone here has visiblity to check internal PR whether it hit any PR number?

 

=== IOC11, node0.fpc11> show sched

 

Total system uptime 529+17:43:27, (45769106030 ms), 42367285954 thread dispatches

CPU load is 100% (1 second), 100% (5 second)

            100% (10 second), 100% (1 minute)

Total interrupt time 111290642 (ms)

 

CPU       Name      Time(ms)

  2%       Idle      1117522898

97%    Threads      44540593884

  0%        ISR      111290642

  0%    Level 1      111238331

  0%    Level 2      52311

 

Top Thread:

   pid      = 15

   name     = mac_db

   time     = 39713325140 ms

   cpu      = 86%

 

=== IOC11, node0.fpc11> show threads

 

PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu

--- -- -------   ---------------------  ---------  ---------------------

15 M  ready     mac_db                2200/8200   0/1/39713325141 ms 86%

 

As I stated above, the “mac_db” process is consuming more CPU load on IOC2 on both the cluster nodes. Since, the CPU load is constant on the nodes I suspect the process was stuck.