SRX

Hello all,

I have an SRX220 running high CPU. Find out because off complaints that the traffic between two vlan's was considerd very slow.

Eventd takes a lot of CPU without being configured. But my best guess it is because the packet fragmentation. I'm not sure how to determine where this happens. So I changed the following settings;

reduced the st0 interface mtu tot 1420 from default

reduced the tcp-mms all-tcp to 1300 from default

reduced the ipsec-vpn mss to 1350 from default

But there is no major decrease in utilization. Any tips how to find out where the high cpu usage comes from?

Model: srx220h
JUNOS Software Release [12.1X46-D50.4]

Process

  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
 1420 root        5  76    0   516M 58864K select 0    ??? 109.72% flowd_octeon_hm
 1440 root        1 127    0 13836K  2988K RUN    0 2577.8 30.03% rtlogd
57306 root        1 127    0 13036K  5284K RUN    0 539:25 27.78% eventd
   22 root        1 171   52     0K    16K RUN    0 3360.0  6.01% idle: cpu0

forwarding 

root@srx220> show chassis forwarding
FWDD status:
  State                                 Online
  Microkernel CPU utilization        18 percent
  Real-time threads CPU utilization 100 percent
  Heap utilization                   53 percent
  Buffer utilization                  1 percent
  Uptime:                               515 days, 2 hours, 33 minutes, 55 seconds

Routing-Engine (this is during the weekend, within business hours idle wil flat out 0% the entire day).

root@srx220> show chassis routing-engine
Routing Engine status:
    Temperature                 51 degrees C / 123 degrees F
    Total memory              1024 MB Max   768 MB used ( 75 percent)
      Control plane memory     544 MB Max   511 MB used ( 94 percent)
      Data plane memory        480 MB Max   254 MB used ( 53 percent)
    CPU utilization:
      User                      31 percent
      Background                 0 percent
      Kernel                    60 percent
      Interrupt                  1 percent
      Idle                       9 percent
    Model                          RE-SRX220H
    Serial ID                      AAFK6429
    Start time                     2016-06-28 07:23:36 GMT+1
    Uptime                         515 days, 2 hours, 46 minutes, 39 seconds
    Last reboot reason             0x20:power-button soft power off
    Load averages:                 1 minute   5 minute  15 minute
                                       2.10       2.12       2.13

root@srx220> show security flow statistics
    Current sessions: 299
    Packets forwarded: 245878319729
    Packets dropped: 606183520
    Fragment packets: 18446744072410236784

VPN assosiations
 <131076 ESP:aes-cbc-128/sha1 - 48530/unlim - root 500 x.x.x.x    (to SSG5)
  >131076 ESP:aes-cbc-128/sha1 - 48530/unlim - root 500 x.x.x.x

  <131074 ESP:3des/sha1 - 3283/ unlim   -   root 500  x.x.x.x (to SRX220/proposal standard)
  >131074 ESP:3des/sha1 - 3283/ unlim   -   root 500  x.x.x.x

  <131073 ESP:3des/sha1 - 1683/ unlim   -   root 500   x.x.x.x (to SRX220/proposal standard)
  >131073 ESP:3des/sha1 - 1683/ unlim   -   root 500   x.x.x.x

Config items

root@srx220> show configuration security flow
tcp-mss {
    all-tcp {
        mss 1300;
    }
    ipsec-vpn {
        mss 1350;
  
VPN Interfaces
unit 2 {
    description gw_;
    family inet {
        mtu 1420;
    }
root@srx220> show configuration system syslog
archive size 100k files 3;
user * {
 any emergency;
}
file messages {
 any critical;
 authorization info;
}
file interactive-commands {
 interactive-commands error;
}
file kmd-logs {
 daemon info;
 match KMD;

Update:

Ok this is wierd. I though of rebooting/updating so I checked the partitions. Backup partition was empty so I sliced it with the primary snapshot to be sure. After that the syslog process is using 0% CPU.

Other values dropped also:

Real-time threads CPU utilization 1 percent 

processes:

1420 root 5 76 0 516M 58864K select 0 ??? 99.17% flowd_octeon_hm
22 root 1 171 52 0K 16K RUN 0 3360.5 81.20% idle: cpu0

Ping trough the vpn decreased from 100ms+ to a acceptable 8ms.

So I have to wait to business hours to check if the CPU will increase again.

Any thought about the fragmentation will be appreciated. 

The eventd process takes care of logging and traps.  Typically when this is running high on a branch device I have seen logging enabled on security policies that have just too many session events for the device to handle.

Take a look at your security policy "then log" statements and turn off those that you can afford to not have logged any more to reduce the load.

Hi Spuluka,

Thank you, I forgot to mention I already disabled all the policy logging yesterday and restarted the syslog daemon.

root@srx220> show security policies | match log

root@srx220>

Maybe syslogd restarted again because of the partition touch.

Not much else feeds eventd.

https://www.juniper.net/documentation/en_US/junos/topics/concept/junos-script-automation-event-notifications-and-policy-overview.html

So with eventd causing the cpu, there needs to be something generating lots of events. Check screens and local logs to see what is showing up.

Your uptime is also very high so maybe a system reboot would help.

If you are screening good amount traffic via flow module of SRX then also you may observe the high CPU.

if possible for you, you may selectively bypass the traffic across fw to reduce cpu util.

Follow the kb fyr..

https://kb.juniper.net/InfoCenter/index?page=content&id=KB26757

Fragmentation is on dataplane and I dont expect that to cause high control plane CPU.

The process eventd and rtlogd are high, these process mainly deal with logging and if reboot fix the issue most possible reason is the process were stuck.