SRX Services Gateway
Highlighted
SRX Services Gateway

SRX100H performance issue - PFE maxes out

‎08-24-2015 02:53 PM

Hi everyone,


I have perfomance issue on my SRX100H1. I have done extensive tests and from iperf tests I have found that if:

- source and destination is in the same zone and same subnet:  93Mb traffic download or upload, which is max for the FastEthernet interface, CPU in norm
- source and destination is in new testing zones, each on separate physical interface : about 17Mb throughput and PFE maxes out
- source and destination is in the same zone but different subnets: same as above : about 17Mb throughput and PFE maxes out

show security monitoring fpc 0    
FPC 0
  PIC 0
    CPU utilization          :  100 %
    Memory utilization       :   46 %
    Current flow session     :   58
    Current flow session IPv4:   58
    Current flow session IPv6:    0
    Max flow session         : 12288
Total Session Creation Per Second (for last 96 seconds on average):    8
IPv4  Session Creation Per Second (for last 96 seconds on average):    8
IPv6  Session Creation Per Second (for last 96 seconds on average):    0

I have removed most of the configuration including UTM, CoS so the only thing I can try is to factory default box and slowly add configuration and see what impact it has.
MSS is setup to 1300 (I have PPPoE), otherwise I could not browse websites, but the problem lies in ethernet and is not touching PPPoE.

I have tried various firmwares and upgraded to newest code. I do not have access to device at the moment, but I think it is newest 12.1X45, which is last for H1 platform.


I have also tried to completely disable TCP checks:

set security flow tcp-session no-syn-check
set security flow tcp-session no-sequence-check

But still the same.

I would welcome any ideas as I think I have tried all tests I could think about and started running in circles.

Thanks

6 REPLIES 6
Highlighted
SRX Services Gateway

Re: SRX100H performance issue - PFE maxes out

‎08-24-2015 03:53 PM

How did you run iperf?

 

- if that packet rate was a new flow in the SRX session table per packet then I would expect fairly low performance through an SRX100. On the other hand, 17Mb/s for a single connection such as a file transfer is pretty bad.

 

I'd be interested to know the bandwidth you get for a big file transfer over a single flow, such as with scp or ftp.

 

 

Highlighted
SRX Services Gateway

Re: SRX100H performance issue - PFE maxes out

‎08-24-2015 04:29 PM

Hi Kez,

 

Thanks for replay.

 

I have done multiple tests beteen different platforms (Windows7, OSX and Linux). The tests where mostly 10+ flows plus, but I have tested from 1 (I have noticed that if flows are under 3 the performance is rather poor) up to 100+ in multiple time frames, from 10sec to 2h.

 

In the beginning I thought that there is a problem with SRX configuration or implementation of PPPoE (there is other topic I have opened about it) as I was getting 7-17Mb download and 15-17Mb upload on PPPoE link, but more tests I have done showed it is something deeper. In the past I never noticed the problem as I use (or I should say used) SRX only as firewall  with one link in trust and one in untrust zone and I had ADSL2+ . Issue came up when I have upgraded link to FTTC 76/20Mb and ASA5505 can handle it without problems (max 8% CPU during tests).

 

Looks like PFD process is hammered. In the last tests I have done there was only permit any any in the same zone, but different subnets, so looks like something with routing process on the box.

Highlighted
SRX Services Gateway

Re: SRX100H performance issue - PFE maxes out

‎08-27-2015 06:45 AM

do you have traceoptions running or policy logging enabled?

 

can you post your config?

Highlighted
SRX Services Gateway

Re: SRX100H performance issue - PFE maxes out

‎08-27-2015 07:04 AM

Hi Cogenesis,

 

I have disabled most of it.

 

No counting or logging on policies. I have only syslog and basic traceoption as per below:

 

 

set security flow traceoptions file trace_thsoot
set security flow traceoptions file files 5
set security flow traceoptions file no-world-readable
set security flow traceoptions flag basic-datapath


set system syslog archive size 10m
set system syslog archive files 3
set system syslog archive world-readable
set system syslog user * any emergency
set system syslog host <SYSLOG_SRV_IP> any any
set system syslog host <SYSLOG_SRV_IP> ntp info
set system syslog host <SYSLOG_SRV_IP> security info
set system syslog host <SYSLOG_SRV_IP> kernel info
set system syslog host <SYSLOG_SRV_IP> firewall info
set system syslog host <SYSLOG_SRV_IP> pfe info
set system syslog host <SYSLOG_SRV_IP> change-log info
set system syslog host <SYSLOG_SRV_IP> source-address <SOURCE_IP>
set system syslog file messages any critical
set system syslog file messages authorization info
set system syslog file interactive-commands interactive-commands error
set system syslog file traffic-log any any
set system syslog file traffic-log match RT_FLOW_SESSION
set system syslog file traffic-deny any any
set system syslog file traffic-deny match RT_FLOW_SESSION_DENY
set system syslog file default-log-messages any any
set system syslog file default-log-messages match "(requested 'commit' operation)|(copying configuration to juniper.save)|(commit complete)|ifAdminStatus|(FRU power)|(FRU removal)|(FRU insertion)|(link UP)|transitioned|Transferred|transfer-file|(license add)|(license delete)|(package -X update)|(package -X delete) | GRES"
set system syslog file default-log-messages structured-data
set system syslog time-format year
set system syslog time-format millisecond
set system syslog source-address <SOURCE_IP> c 

 

Highlighted
SRX Services Gateway

Re: SRX100H performance issue - PFE maxes out

‎08-27-2015 02:30 PM

Hi

 

can you do the test again but this time run the following commands to see which thread is causing the high PFE cpu.  once you isolate the PID with high cpu (last column), run the second command show threads <pid>

 

and post your results

 

root@lab-srx> start shell pfe network fwdd 


BSD platform (OCTEON processor, 896MB memory, 16384KB flash)

FLOWD_OCTEON(lab-srx vty)# show threads


PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
  1 H  asleep    Maintenance           1848/73824  0/8/6573 ms  0%
  2 L  running   Idle                  4048/73824  0/16/111618322 ms  0%
  3 H  asleep    Timer Services        1968/73824  0/15/1385457 ms  0%
  5 L  asleep    Ukern Syslog           856/73824  0/0/0 ms  0%
  6 L  asleep    Sheaf Background      1824/73824  0/8/111614 ms  0%
  7 M  asleep    mac_db                 856/73824  0/0/0 ms  0%
  8 M  asleep    Docsis                1776/73824  0/8/763252 ms  0%
  9 M  asleep    ATMX                  2064/73824  0/15/1602503 ms  0%
 10 M  asleep    XDSL                  2216/73824  0/15/55507743 ms  0%
 11 M  asleep    DSX50ms               1984/73824  0/15/6746938 ms  0%
 12 M  asleep    DSXonesec             1976/73824  0/8/786469 ms  0%
 13 M  asleep    SFP                   1920/73824  0/8/1717015 ms  0%
 14 M  asleep    Ethernet              3584/73824  0/16/416912843 ms  95%
 15 M  asleep    RSMON syslog thread   1264/73824  0/8/6206 ms  0%
 16 L  asleep    Syslog                2128/73824  0/8/17149 ms  0%
 17 M  asleep    Fwdd Notif Recv       2640/73824  0/30/21786718 ms  0%
 18 M  asleep    Forwarding Thread     4240/73824  0/15/82735377 ms  0%
 19 M  asleep    Periodic             13328/73824  0/30/59615007 ms  0%



FLOWD_OCTEON(lab-srx vty)# show threads 14      
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
 14 M  asleep    Ethernet              3584/73824  0/16/416919335 ms  0%

Wakeups:
      Type  ID  Enabled  Pending   Context
 Semaphore  00      Yes       No  0x50e57d30
     Timer  00      Yes       No  0x50e57e80

Frame 00: sp = 0x511e37a0, pc = 0x08014d94
Frame 01: sp = 0x511e3818, pc = 0x08215310
Frame 02: sp = 0x511e3858, pc = 0x0802b960
Frame 03: sp = 0x511e3880, pc = 0x00012060

FLOWD_OCTEON(lab-srx vty)# 

Highlighted
SRX Services Gateway

Re: SRX100H performance issue - PFE maxes out

‎08-28-2015 03:16 AM

Thanks Cogenesis,

 

It will have to wait till next week when I will be able to test it, but will see if it can show the root cause of the issue.

Feedback