SRX Services Gateway
SRX Services Gateway

SRX-650 crashed to high CPU

04.15.10   |  
‎04-15-2010 07:03 AM

Hard reboot got it back online, we were seeing this message via console:

 

"kern.maxfiles limit exceeded by uid 2001, please see tuning(7)."

 

We do have a case open with Juniper, just curious if anyone else has ran into this?  We think NSM is the source of the problem, but aren't sure.

 

Thanks,

10 REPLIES
SRX Services Gateway

Re: SRX-650 crashed to high CPU

04.15.10   |  
‎04-15-2010 12:15 PM

which image are you using? are you using FWauth anywhere?

 

thanks

raheel 

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
SRX Services Gateway

Re: SRX-650 crashed to high CPU

04.15.10   |  
‎04-15-2010 12:30 PM

10.0R2.10

 

We are not using FWauth.

 

Thanks,

SRX Services Gateway

Re: SRX-650 crashed to high CPU

04.15.10   |  
‎04-15-2010 10:42 PM

Your box seems to have hit the maximum files open limit. What is the customer case id for this problem?

 

SRX Services Gateway

Re: SRX-650 crashed to high CPU

04.19.10   |  
‎04-19-2010 12:12 PM

2010-0414-0848

SRX Services Gateway

Re: SRX-650 crashed to high CPU

11.20.10   |  
‎11-20-2010 04:55 AM

Hi,

 

Could you please share how this issue was solved by JTAC. We faced the same issue yesterday on our srx650 cluster.

 

Thanks,

Abhilash

SRX Services Gateway

Re: SRX-650 crashed to high CPU

11.20.10   |  
‎11-20-2010 01:09 PM

We have seen instances where NSM was causing high CPU and maxproc issues. There are fixes in latest NSM 2010.3 version. I would recommend upgrading NSM to latest version and schema. Be sure to also run at the very least the latest maintenance release for your JUNOS version (i.e. for 10.0 it would be 10.0R4). Current JTAC recommended JUNOS release for SRX is 10.2R3.

 

-Richard

SRX Services Gateway

Re: SRX-650 crashed to high CPU

11.21.10   |  
‎11-21-2010 10:16 PM

Thanks Richard. This issue looks to be the one mentioned in http://kb.juniper.net/KB15068

 

The KB however doesn't point to the NSM version upgrade as solution.

 

Regards,

Abhilash

Highlighted
SRX Services Gateway

Re: SRX-650 crashed to high CPU

11.22.10   |  
‎11-22-2010 07:27 AM

There were a lot of issues with this case :smileyvery-happy

 

The main one:

 

"1.  Changed the config of the VPNs to dynamic hostname to an email address format.  The
VPN config was originally created by an NSM template, and it created the dynamic
hostname with a format of hostname.global.  The code had problems when trying to resolve
DNS name with this format.  Changing the hostname to email format is a workaround for
this issue.
2.  Downgraded JUNOS code to 10.0R3.1.

The plan moving forward is to go through the same steps on the production site this
weekend.

10.1R3 is going to be out soon, and the when NSM schema update supports this, then an
upgrade to that version of code will be planned.

Putting case into monitoring for now.  Also got the okay to bring this to a P2, since we
seem to have a workable solution on the DR site."

 

We are presently setting on [10.1R3.7], and it's been pretty stable.

 

NSM version:

Version: 2010.2
Build: LGB13z1n63

 

Our SE is telling us to hold off on 2010.3.

 

I would definately recomend going to a .3 release of SRX code.  Juniper code (netscreen or SRX) seems to stabilize by the R3 release. 

SRX Services Gateway

Re: SRX-650 crashed to high CPU

11.22.10   |  
‎11-22-2010 09:58 PM

Thanks a lot for this detailed reply. Its hard to conclude now the NSM caused this or a bug on the 10.0R3.10. If we create a case, JTAC would directly recommend to upgrade to the recommended version 10.2R3.10.

 

We are planning to upgarde our box to 10.2 in the coming weekend and will remove out the NSM.

 

Regards,

Abhilash 

SRX Services Gateway

Re: SRX-650 crashed to high CPU

11.29.10   |  
‎11-29-2010 01:25 AM

We meet the same issue. Thanks for all your information.

What was the conclusion of case 2010-0414-0848?