SRX

Hello!

Model: srx100h2
JUNOS Software Release [12.1X44-D20.3]

I am getting this message in logs:

May  5 08:46:29  tco PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=96
May  5 08:46:56  tco PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=100
May  5 08:46:59  tco PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99

 show chassis routing-engine 

Routing Engine status:
    Temperature                 49 degrees C / 120 degrees F
    Total memory              2048 MB Max  1024 MB used ( 50 percent)
      Control plane memory    1104 MB Max   486 MB used ( 44 percent)
      Data plane memory        944 MB Max   538 MB used ( 57 percent)
    CPU utilization:
      User                      28 percent
      Background                 0 percent
      Kernel                    62 percent
      Interrupt                  0 percent
      Idle                       9 percent
    Model                          RE-SRX100H2
    Serial ID                      BZ5013AF0799
    Start time                     2014-04-14 16:03:47 GMT+4
    Uptime                         20 days, 17 hours, 11 minutes, 28 seconds
    Last reboot reason             0x1:power cycle/failure
    Load averages:                 1 minute   5 minute  15 minute
                                       2.04       1.58       1.29

 show system processes extensive

last pid: 54611;  load averages:  0.87,  0.93,  0.70  up 20+16:48:00    08:51:17
136 processes: 16 running, 106 sleeping, 2 zombie, 12 waiting

Mem: 221M Active, 108M Inact, 1024M Wired, 157M Cache, 112M Buf, 463M Free
Swap:


  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
 1440 root        5  76    0   994M 59316K select 0 567.5H 104.44% flowd_octeon_hm
   22 root        1 171   52     0K    16K RUN    0 375.6H 27.78% idle: cpu0
 1098 root        1 100    0 12788K  4784K RUN    0 701:19 20.26% eventd
 1458 root        1  98    0 13572K  6356K select 0 590:25 18.75% rtlogd
   24 root        1 -20 -139     0K    16K WAIT   0 370:43  0.00% swi7: clock
27076 root        5  76    0 80568K 70704K select 0 273:44  0.00% named
 1462 root        1  76    0 12396K  5784K select 0 132:00  0.00% license-check

 show security monitoring fpc 0

FPC 0
  PIC 0
    CPU utilization          :   13 %
    Memory utilization       :   57 %
    Current flow session     :  546
    Current flow session IPv4:  546
    Current flow session IPv6:    0
    Max flow session         : 524288
Total Session Creation Per Second (for last 96 seconds on average):   67
IPv4  Session Creation Per Second (for last 96 seconds on average):   67
IPv6  Session Creation Per Second (for last 96 seconds on average):    0

 show task accounting

Task                       Started    User Time  System Time  Longest Run
Scheduler                     1181        0.184        0.200        0.000
LMP Client                     282        0.089        0.097        0.000
Memory                          23        0.001        0.001        0.000
hakr                             5        0.000        0.000        0.000
PIM I/O./var/run/ppmd_con      105        0.007        0.007        0.000
BFD I/O./var/run/bfdd_con      105        0.007        0.007        0.000
KRT                            425        0.024        0.025        0.000
Redirect                         8        0.000        0.000        0.000
MGMT_Listen./var/run/rpd_        8        0.003        0.002        0.000
SNMP Subagent./var/run/sn      196        0.046        0.051        0.000

 Should I be worried about this messages?

Hi Nomad,

     CPU can go high if you receive high transistian traffic; the below provided information eventd , rtlogd processes are utilizing some CPU resource. Do you have any policy logging enable (as below)

                then {
                    permit;
                    log {
                        session-init;

                        session-close;
                    }

If enabled, please disable those. And you can consider freeing up device resources by disabling unused processes as suggested in the KB (it will not reduce the CPU usage, but as part of good practice, you can do this)

http://kb.juniper.net/InfoCenter/index?page=content&id=KB28933

Thanks

SHKM

Thank you for your reply, there is no policy logging enabled.

I will try to disable system services later, but doesn't  it shows that we should consider buying higher lvl device?

oh I see....ok, I suggest you to please go through the datasheet performance numbers

http://www.juniper.net/us/en/local/pdf/datasheets/1000281-en.pdf  page number 7

and compare with the traffic level comming into firewall and decide on further hardware upgrading. I strongly recommend to consult with Juniper Sales/Account team for upgading the hardware.

Thanks,

SHKM

Everything within perfomance numbers.

Ok, is it possible for you to share rsi and varlogs?

Here they are.

Attachment(s)

log.zip   124 KB 1 version

rsi.txt   637 KB 1 version

Its clear Data-plane CPU goes high, not always or in a sequence; its random. Device interfaces are in Ethernet switching, lot of broadcast or any flood in traffic can cause high Data plane CPU. I suggest you to please gather incoming and outgoing packet bytes, pps (packets per sec) during normal working time and during high cpu time to identify further. We can use the below command to get bps/pps per interface.

monitor interface traffic

Thanks,

SHKM

I watched on the output of "monitor interface traffic" for some time and I didn't notice any difference in pps

But wher bps reaching about 22299336 bps cpu reaches high marks.

I wonder if bps stands for bytes per second or bits per second? According to http://www.juniper.net/techpubs/en_US/junos11.4/information-products/topic-collections/security/software-all/monitoring-and-troubleshooting/index.html?jd0e24088.html it's bytes. So traffic is to high for 100mbit interface.

bps is Bytes per second.

You should watch for delta value as described below

Example:

Interface    Link     Input bytes        (bps)      Output bytes        (bps)
 fe-0/0/0      Up       155221315        (472)            149856          (0)

You should watch for value mentioned in bold (not in sticked one) The Input bytes means it is the total number of bytes received on this interface since the interface come up. (bps) is the number of bytes received per second.

Thanks,

SHKM

Yup, I was watching correct line, thank you.

Oh I see...~22Mbps of traffic; is it for entire device or only on specific interface? I feel it would be better to open Jtac case; it needs extensive troubleshooting.

Thanks,

SHKM

It's for specific interface. I'll split traffic between several interfaces, hope it will help.

Thank you for your help!

All the packets get proceed by PFE (ir-respective of the interface it received) so It really doesn’t affect even we split the traffic between interfaces however, we can try it.

As for me it helped a bit. But to as much as I hoped 😞

Oh ok..there is some improvement doing so...Good to hear..!