Switching

Hi,

I hope ti get your help here. We have 2 EX3300 built in virtual chassis and runing ther version [12.3R3.4]. Our nagios shows us a high cpu usage and i do  not why!?!

From the shell i execute "top" command for 30 min and get a maximum of 48% usage

% top
last pid: 37688;  load averages:  0.45,  0.53,  0.47                                                                                                                                       up 23+02:41:30  11:12:38
48 processes:  2 running, 46 sleeping
CPU states: 42.4% user,  0.0% nice, 48.3% system,  0.6% interrupt,  8.7% idle
Mem: 290M Active, 94M Inact, 78M Wired, 134M Cache, 86M Buf, 389M Free
Swap:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 1257 root        1 114    0 23440K 11124K RUN     28.0H 19.63% mib2d
 1256 root        1 114    0 18600K 14972K select 917:57 10.25% snmpd
 1237 root        2  44  -52   140M 24008K select  35.5H  7.32% sfid
 1239 root        1   8    0   135M 15900K nanslp  45.4H  3.52% pfem
 1249 root        1  96    0 35372K 14136K select  92:59  2.54% chassisd
 1236 root        1   4    0   127M 10560K kqread 765:09  0.00% chassism
... ALL other processes consumes 0.00%

From the CLI "show system processes summary" " an usage of sometimes up to 99%

Stack1> show system processes summary
last pid: 37689;  load averages:  0.26,  0.44,  0.44  up 23+02:43:11    11:14:19
125 processes: 3 running, 105 sleeping, 17 waiting

Mem: 288M Active, 93M Inact, 78M Wired, 134M Cache, 86M Buf, 391M Free
Swap:


  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   11 root        1 171   52     0K  3548K RUN    391.1H 79.74% idle

And "show chassis fpc"

Stack1> show chassis fpc

                     Temp  CPU Utilization (%)   Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer
  0  Online                   97          0       1024                  39
  1  Online                   33          1       1024                  33
  2  Empty
  3  Empty
  4  Empty
  5  Empty
  6  Empty
  7  Empty
  8  Empty
  9  Empty

And "

Stack1> show chassis routing-engine | <piped>
  Slot 0:
    CPU utilization:
      User                      59 percent
      Background                 0 percent
      Kernel                    36 percent
      Interrupt                  0 percent
      Idle                       4 percent
    Load averages:                 1 minute   5 minute  15 minute
                                       0.21       0.39       0.42

  Slot 1:
    CPU utilization:
      User                       8 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      90 percent
    Load averages:                 1 minute   5 minute  15 minute
                                       0.18       0.24       0.20

can anyone tell me some cli cmds to more investigate this issue and find out which process is most using cpu? or is theire any bugs in this version?

Thank you

 I am curious to follow this discussion. We have some 2200s that also seem to "run hot" CPU-wise. In our case, the sfid process is always the busiest process for reasons I can't yet fathom. Our EX4200s and EX3200s running virtually an identical configuration seem idle by comparion. In fact, the configuration for these 2200s was brought over from our standard 3200 config and they aren't being used in any different manner or topology. We just bought them based on cost for a few new closets in a new building. 

 By comparison, CLI interactions on the 2200s are always slow like they are under load. 

 The main difference, aside from hardware, is 12.3R3.4 on the 2200s vs. a 10.4 release on most of the 3200s and 4200s (have had little compelling need to upgrade beyond what has been stable for our needs in recent years).

Thoughts?

 - Aaron

Can you disable nagios monitoring and manually check the CPU usage for a while?

I see that SNMPD/MIB2D is taking a lot of your CPU resources:

 1257 root        1 114    0 23440K 11124K RUN     28.0H 19.63% mib2d
 1256 root        1 114    0 18600K 14972K select 917:57 10.25% snmpd

Well at the time I copy/past the output of "top" shell command were snmp and mib2d using more cpu, but both processes are not requiring permanently cpu

I will deactivate snmp at first und observe how the cpu usage behaves

I'll provide a update asap

Hi all,

I've observed the cpu usage these last days through a perl script. I was unable to redirect the output of the shell to the StdIn to the script but I've got the output from cli.

When the script runs and executed following cmds "show system processes summary", "show chassis routing-engine | match "Slot|state|emperature|utilization|User|Background|Kernel|Interrupt|Idle"" and "show chassis fpc | match "Temp|State|Online"", the cpu went high by cli on the fpc but still low on the RE.

==== show system processes summary ====
last pid: 47870;  load averages:  0.29,  0.17,  0.16  up 29+06:48:54    16:20:02
128 processes: 4 running, 107 sleeping, 17 waiting

Mem: 296M Active, 94M Inact, 78M Wired, 134M Cache, 86M Buf, 383M Free
Swap:


  PID USERNAME  THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   11 root        1 171   52     0K  3548K RUN    489.7H 83.06% idle
47868 admin       1 132    0 20276K 13512K RUN      0:00 14.00% cli

==== show chassis routing-engine | match "Slot|state|emperature|utilization|User|Background|Kernel|Interrupt|Idle" ====
  Slot 0:
    Current state                  Master
    Temperature                 45 degrees C / 113 degrees F
    CPU temperature             45 degrees C / 113 degrees F
    Memory utilization          39 percent
    CPU utilization:
      User                       6 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      92 percent
  Slot 1:
    Current state                  Backup
    Temperature                 42 degrees C / 107 degrees F
    CPU temperature             42 degrees C / 107 degrees F
    Memory utilization          33 percent
    CPU utilization:
      User                       3 percent
      Background                 0 percent
      Kernel                     1 percent
      Interrupt                  0 percent
      Idle                      96 percent
==== show chassis fpc | match "Temp|State|Online" ====
                     Temp  CPU Utilization (%)   Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer
  0  Online                  100          0       1024                  40
  1  Online                    7          0       1024                  33

 Any suggestions?

Thk you

Are you seeing this very often? The CPU usage not matching between the two commands output?

I did quickly a test and I see something similar:

root@EX4200-VC:RE:0% cat output
Wed Apr  9 11:22:29 CEST 2014
Routing Engine status:
  Slot 0:
    Current state                  Master
    Temperature                 37 degrees C / 98 degrees F
    CPU temperature             37 degrees C / 98 degrees F
    DRAM                      1024
    Memory utilization          52 percent
    CPU utilization:
      User                       8 percent
      Background                 0 percent
      Kernel                    10 percent
      Interrupt                  0 percent
      Idle                      81 percent
    Model                          EX4200-24F
    Serial ID                      BR0208360644
    Start time                     2013-12-18 05:07:46 CET
    Uptime                         47 days, 13 hours, 26 minutes, 43 seconds
    Last reboot reason             0x2:watchdog
    Load averages:                 1 minute   5 minute  15 minute
                                       0.18       0.08       0.02
Routing Engine status:
  Slot 1:
    Current state                  Backup
    Temperature                 39 degrees C / 102 degrees F
    CPU temperature             39 degrees C / 102 degrees F
    DRAM                      1024
    Memory utilization          45 percent
    CPU utilization:
      User                       2 percent
      Background                 0 percent
      Kernel                     1 percent
      Interrupt                  0 percent
      Idle                      97 percent
    Model                          EX4200-24T, 8 POE
    Serial ID                      BM0209412253
    Start time                     2013-12-18 05:06:10 CET
    Uptime                         26 days, 19 hours, 11 minutes, 9 seconds
    Last reboot reason             0x2:watchdog
    Load averages:                 1 minute   5 minute  15 minute
                                       0.18       0.08       0.02
                     Temp  CPU Utilization (%)   Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer
  0  Online                    3          0       1024                  52
  1  Online                    3          0       1024                  45
  2  Empty           
  3  Empty           
  4  Empty           
  5  Empty           
  6  Empty           
  7  Empty           
  8  Empty           
  9  Empty           
root@EX4200-VC:RE:0%

yes i do!! the script ran every 5 min. please see attached file

Attachment(s)

Stack-CPU-from-cli.txt   2.04 MB 1 version

Please contact Juniper support and open a case to start to troubleshoot this further.

I experienced the same behavior on my 3300 in VC.

When a SNMP query runs mib2d and snmpd take up to 40-45% of the CPU threatening the device stability.

Is that possible that a snmpwalk o something like that could have a so high impact on the switch?

Is thare any way to limit this problem?

Thank You

Regards

Anyone were able to fix this? I am also having problem with my Ex switches. which is reaching 90% CPU when there is snmpbulkwalk is done.

Please help.

Hi opsadmin,

Bulk SNMP walk is bound to increase CPU on this EX.  And if we poll for a lot of data in a short time, it could spike "mgd" or CLI process as too.  It's beter to only probe for critical alarms/events like interface downs, chassis/system alarms etc., work out a lesser aggressive polling interval (if you poll every 5mins, try 10mins for example) and limit to a lesser no. of commands per poll rather than bulk.  Please check this out: 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB30713&cat=EX3300&actp=LIST

The best to do is stay on the recommended Junos to avoid any others, but I believe for CPU utilization spike, that's something to expect during a bulk SNMP walk.  If limited polling is done for what's critical, we must be alright:

https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476&actp=METADATA#ex_series

While you can look for the owner of this discussion for confirmation, please let us know if this helps.

@RealMatrix, it'll be helpful for the community if you mark a resolution for the issue you've mentioned on this discussion.  Thank you in advance for your time. 

Hope this helps.

Regards,
-r.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution."
Kudos are always appreciated :).