SRX Services Gateway
Showing results for 
Search instead for 
Do you mean 
Reply
rjs
Visitor
Posts: 3
Registered: ‎02-09-2011
0 Kudos
Accepted Solution

SRX Branch + NSM + Logging + High CPU

We are using 210B, 240h and 650 SRX's with NSM 10.2. We are running 10.3.R1.9 on the devices. Our issue is high CPU with and dropping traffic. We have been troubleshooting this issue for some time with JTAC. I'm trying to get a feel if anyone is running a similar environment with success. We are not doing IDP, ALG's, etc. These boxes are a basic firewall with logging and average 10% usage. 

Trusted Contributor
Posts: 330
Registered: ‎01-08-2010
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

SRX was designed to log using SYSLOG, so when you log useing NSM mode it logs to the local flash then during the rollup it converts the log to NSM format and sends it..

 

Two solutions:

 

1. Log directly to a syslog server instead

2. NSM 2011.1 appears to support SRX sd-syslog format.

Super Contributor
Posts: 222
Registered: ‎12-16-2008
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

Depends which CPU is high, the routing-engine or the flow CPU. I assume its the routing-engine CPU as that is what most of us suffer from.

 

The best solution is to upgrade NSM to 2011.1 and use syslog to send traffic to the NSM server. This might require a few changes to the NSM setup, but will greatly reduce the load on the SRX. We are switching all our high-traffic devices to this new method, and can confirm that it works with SRX running 10.3R2 & 10.2R3. Sadly, NSM doesn't seem to parse the logs from earlier versions.

Contributor
Posts: 200
Registered: ‎05-05-2010
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

Hi motd ,  do you mean that syslog server will receive traffic & send it to NSM , how to do that on NSM side ?

Super Contributor
Posts: 222
Registered: ‎12-16-2008
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

NSM can be configured to receive syslog messages directly. See page 765 of

http://www.juniper.net/techpubs/software/management/security-manager/nsm2011_1/nsm-admin-guide.pdf

 

Its one setting on NSM to listen to the messages, the second part is configuring the SRX to stream messages directly to an external syslog server (nsm in this case), without passing the messages to the routing engine.

Trusted Contributor
Posts: 70
Registered: ‎01-23-2008
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

[ Edited ]

"..with SRX running 10.3R2 & 10.2R3. Sadly, NSM doesn't seem to parse the logs from earlier versions"

 

Do you have this working with the the SRX branch? We are running 10.2R3 and when trying to switch to "security log mode stream" the entire configuration is accepted but in the logs following the commit we see the message " Stream has no meaning when system-event-mode is on" and no logs forwarded to Syslog. We have not tested with JunOS 10.4R3 yet, we're waiting on the NSM schema to be released first.

 

NSM will handle traffic logs from JunoS 10.x versions if you have configured the log mode to event and set the event rate to 1000 eps as recommended. NSM log categorization under event mode works well, but the standard STRM log categorization is basic. Hopefully once we can get stream working, both STRM & NSM logging will be categorized best.

 

Here's our configuration (a reboot does not resolve this).

 

    set security log mode stream
    set security log format sd-syslog
    set security log source-address <source address of dmi device>
    set security log stream nsm-dataplane-log category all

    set security log stream nsm-dataplane-log severity info

    set security log stream nsm-dataplane-log format sd-syslog
    set security log stream nsm-dataplane-log host <ip of nsm devSvr>
    set security log stream nsm-dataplane-log port 5140


The error in default-log-messages is

 

"UI_CONFIGURATION_ERROR: Process: rtlogd, path: [edit security log], statement: stream nsm-dataplane-log, Stream has no meaning when system-event-mode is on"

 

 

 

 

Trusted Contributor
Posts: 70
Registered: ‎01-23-2008
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

[ Edited ]

"Our issue is high CPU with and dropping traffic"

 

We haven't seen this in our enviroinments, but we are sticking on 10.2R3 and selectively choose NSM releases that work best for us. 2010.4q14 and above was a good release and 2011.1 is looking very good for JunOS device management at last. There is a PR that references high CPU with event mode logging to NSM and the recommendation is NSM2011.1 with the SRX in stream mode as others have stated here - if you can get this to work (let us know if you do).

 

rjs
Visitor
Posts: 3
Registered: ‎02-09-2011
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

UPDATE

 

Per SomeITGuy's post I pursued to look over the release notes for 2011.1. We've upgraded to this release. I can report this is working well for us. We are still running 10.3.R1.9 on the branch devices. We found what may be a bug, that you must set the severity to none in order for this to work.

 

rjs
Visitor
Posts: 3
Registered: ‎02-09-2011
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

[ Edited ]

markpr...

   

set security log stream nsm-dataplane-log severity info <==Change to none in NSM or omit from cli 

   

Trusted Contributor
Posts: 70
Registered: ‎01-23-2008
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

[ Edited ]

I tried removing severity and setting the log mode to "stream" again and this this did not help on the 210H I tested with. I also tried a reboot and then adding in the minimal set of commands for a new syslog destination. This commits with success, no errors, but checking default-log-messages I get the same error.

 

" UI_CONFIGURATION_ERROR: Process: rtlogd, path: [edit security log], statement: stream strm-stream-log, Stream has no meaning when system-event-mode is on"

 

[edit security log]
# show

mode stream;
format sd-syslog;
source-address <source ip address of the srx210h>;

 

stream strm-stream-log {
    format sd-syslog;
    host {
        <ip address of strm>;
    }
}

 

I thought this may be a problem with 10.2r3 so upgraded the same SRX210H to 10.4r3.4 but I still get the same error. Here's the syslog configuration, in event mode traffic logs are written into default-log-messages. When the configuration is in stream mode, the errors are written here but no traffic logs appear (or are forwarded to the syslog destination).

 

[edit system syslog]
# show

file default-log-messages {
    any any;
    match "!(.*dfwd_ev_client.*)";
    structured-data;
}

 

I believe the releases notes and PR that putting the SRX into stream mode will reduce the CPU and still support logging to an NSM2011.1 - if we can get the SRX branch into stream mode. I don't have an SRX HE available in this environment to test with.

 

 

 

 

Contributor
Posts: 200
Registered: ‎05-05-2010
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

[ Edited ]

Hi motd ,

 

And when turning NSM to a sysog server from page   765  as  you mentioned , i will configure its ip at SRx as external syslog srever , right ?

 

Also will i be able to view these logs from Log viewer Tab (  the same as if i was adding the SRX to NSM ) ?

 

Super Contributor
Posts: 222
Registered: ‎12-16-2008
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

Yes, only one change is required on the NSM server itself.

 

The other changes you are making is simply instructing the SRX to send its traffic logs to a syslog server instead of logging them locally to disk or flash.

 

The logs show up in the log viewer just like before

Trusted Contributor
Posts: 70
Registered: ‎01-23-2008
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

If anyone else is struggling with this, the configurations detailed here are good and we are successfully logging in stream mode to external syslog servers and to NSM2011.1 now - but there are a few caveats as of JunOS 10.4

 

1) " UI_CONFIGURATION_ERROR: Process: rtlogd, path: [edit security log], statement: stream strm-stream-log, Stream has no meaning when system-event-mode is on" - this is a bogus error and can be ignored. It appears whether the system is in log mode stream or log mode event.

 

2) default-log-messages - in log mode event, the traffic logs appear here by default (or more correctly the control plane logs appear here). When in log mode stream, the traffic logs are sent straight from the dataplane and traffic logs do not appear here. Assume this means the webui will not show the traffic logs either.

 

3) stream mode only supported outbound on a physical interface not over a VPN (st interfaces). If you have branch offices connected via VPN you will not be able forward traffic logs in stream mode. You have to use event mode if you want to collect those traffic logs. This was our big problem in testing out stream mode.

 

We noticed some small differences in the data contents in structured (stream) vs non-structured (event) but the STRM DSM and NSM does support both.

 

Highlighted
Trusted Contributor
Posts: 345
Registered: ‎06-11-2009
0 Kudos

Re: SRX Branch + NSM + Logging + High CPU

Hello,

I'm having issues with this.  We had our SRX set to log to NSM using event mode.  This was causing high CPU so i have made the changes suggested in this thread. 

 

We have our SRXs running in a HA cluster.  I have put the configuration under each node in the groups configuration section.  NSM has also been changed to allow syslog over UDP. 

 

Also - We are managing the SRX cluster via the FXP, and NSM is using the FXP of each node for management as well.

 

Here is the config:


 show configuration groups node0 security log
mode stream;
format sd-syslog;
source-address 10.150.7.18;
stream nsm-stream {
    category all;
    host {
        10.203.0.34;
        port 5140;


show configuration groups node0 system syslog

file default-log-messages {
    any any;
    structured-data;

 

This configuration is the same on node1.

In NSM all i see is logs from source 0.0.0.0 destintion 0.0.0.0.  Nothing else.

 

Any thoughts on this?

Setting it to stream mode has fixed the CPU issue, but now logs do not appear properly.