SRX Services Gateway
Showing results for 
Search instead for 
Do you mean 
Reply
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

vpn dropping

I have an SRX240 running 10.1.R1.8 with routed IPSec vpn to 2 SSG320's.

the tunnel stays up for a few days and then it starts to bounce every minute to 2 minutes until the srx is rebooted.

the SSGs are runnit 6.2r5 and 6.2r3 same issue on both.

 

SXR ---- internet ------- SSG#1 -------------- SSG#2

 

SRX = 10.1r1.8

ssg#1 6.2r3

ssg#2 6.2r5

Frank Dias
Distinguished Expert
Posts: 414
Registered: ‎06-18-2008
0 Kudos

Re: vpn dropping

please attach relevant logs,

 

/var/log/kmd

/var/log/messages

/var/log/jsrpd

 

 

thanks

raheel

 

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

here are the files requested

Frank Dias
Highlighted
Juniper Employee
Posts: 15
Registered: ‎01-08-2010
0 Kudos

Re: vpn dropping

nice logs.

 

Interface is flapping. from jsrpd logs.

 

Mar 1 18:28:07 Interface ge-0/0/2 is going up

Mar 1 18:30:46 Interface ge-0/0/2 is going down

Mar 1 18:30:50 Interface ge-0/0/2 is going up

Mar 1 18:37:06 Interface ge-0/0/2 is going down

Mar 1 18:37:10 Interface ge-0/0/2 is going up

Mar 1 18:43:58 Interface ge-0/0/2 is going down

Mar 1 18:44:02 Interface ge-0/0/2 is going up

 

 

Possible memory leak. from kmd logs.

 

Mar 1 03:27:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:33:30 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

 

 

 

Distinguished Expert
Posts: 414
Registered: ‎06-18-2008
0 Kudos

Re: vpn dropping

thanks for posting logs,

 

could you please run following commands and provide output.

 

1/ show security ike memory-usage

 

 

If you notice that any of the counters  are incrementing significantly over a period of time, then that could be the root cause of the problem.This command does not account for the memory allocated by libraries used by iked. You can also peroidically monitor the output of "top" on the shell. Please note that top is executed on the shell and NOT on cli.

 

2/ top -n (run it for multiple times, when encounter issue)

 

 

thanks

raheel

 

 

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

here is the output;

 

thanks

Frank Dias
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

hi drive567,

 

the ge-0/0/2 is a connection to a server, the ipsec is bound to ge-0/0/0

 

so you think there is a possible memory leak, what other logs can I provide to confirm this assumption?

 

 

Frank Dias
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

just happened again

Frank Dias
Distinguished Expert
Posts: 414
Registered: ‎06-18-2008
0 Kudos

Re: vpn dropping

[ Edited ]

Frank,

memory usage seems to be fine from the first log o/p, I couldn't able to see "top -n" output in your second log. could you please directly paste the output again?

 

also, please attach your configs.

 

thanks

raheel

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
Distinguished Expert
Posts: 414
Registered: ‎06-18-2008
0 Kudos

Re: vpn dropping

Looking at the top command, I see there is only very little memory available in kernel (1040k) generally it should be more than 100M. KMD occupied memory is reasonable. So it could be some other daemon leaking the memory.

o/p of top command is not complete. Some of the daemon is next page. Will it possible to get all pages of top? or try CLI cmd "
show system processes extensive "

 

thanks

raheel

 



% top -n^M
last pid:  7073;  load averages:  3.09,  3.10,  3.04  up 2+12:56:00    16:49:32^M
67 processes:  6 running, 61 sleeping^M
^M
Mem: 128M Active, 79M Inact, 685M Wired, 78M Cache, 112M Buf, 1040K Free^M
Swap: ^M
^M
^M
  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND^M
  998 root        5  76    0   457M 41296K select 0 187.8H 281.88% flowd_octeon_hm^M
  994 root        1  76    0  7408K  3300K select 0   8:52  0.00% ppmd^M
  983 root        1  76    0 17696K 11700K select 0   7:30  0.00% snmpd^M
  984 root        1  76    0 13132K  7508K select 0   6:35  0.00% mib2d^M
 1012 root        1  76    0 11576K  5132K select 0   4:02  0.00% utmd^M
 1009 root        1  76    0 10440K  4128K select 0   2:44  0.00% jdiameterd^M
 1018 root        1   4    0 11176K  5684K kqread 0   2:03  0.00% eswd^M
  986 root        1  76    0 13664K  4732K RUN    0   2:02  0.00% l2ald^M
  978 root        1  76    0 26512K 13960K select 0   1:49  0.00% chassisd^M
  988 root        1  76    0  8996K  3904K select 0   1:44  0.00% vrrpd^M
  975 root        1  76    0  2612K  1232K select 0   1:20  0.00% bslockd^M
 1016 root        1  76    0  3244K  1036K select 0   1:19  0.00% license-check^M
  979 root        1  76    0  5996K  2552K select 0   0:53  0.00% alarmd^M
 993 root        1  76    0 14680K  6268K select 0   0:47  0.00% kmd^M
1014 root        2  76    0  9804K  3616K select 0   0:40  0.00% wland^M
  985 root        1   4    0 36260K 18500K kqread 0   0:40  0.00% rpd^M
  977 root        1  76    0 23352K  8096K select 0   0:38  0.00% dcd^M
 1011 root        1  76    0  9044K  3580K select 0   0:29  0.00% rtlogd^M

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

Hi rachel.

 

I ran the commands you requested, but the unit is running fine at this time the last set of file I sent was when the unit was having issues and so far my resolve has been to reboot.

 

Frank Dias
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

config from srx attached with logs, unit is working fine at this time.

 

another observation it seems to happen at about 60 hrs uptime.

Frank Dias
Distinguished Expert
Posts: 414
Registered: ‎06-18-2008
0 Kudos

Re: vpn dropping

[ Edited ]

yes- reboot is not a good option.

 

please keep an eye on device, and if problem occurs again, collect all relevant logs (as mentioned earlier to collect)

 

thanks

raheel

 

 

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

rachel,

 

I was doing some checking and I noticed that the available memory has been dropping.

 

here is the latest output 1 day 10hrs uptime

 

Frank Dias
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

here are the stats from this mornig and by tonight I will neeed to reboot

Frank Dias
Super Contributor
Posts: 242
Registered: ‎11-06-2007
0 Kudos

Re: vpn dropping

What version of JunOS are you running?  There were some issues with memory leak in 9.6, which would have affected kmd.  This has been addressed in 10..1R1, but cannot remember if it is addressed in 10.0R2 or not.

Distinguished Expert
Posts: 414
Registered: ‎06-18-2008
0 Kudos

Re: vpn dropping

Frank,

whatelse are you running with IPSEC and other traffic details etc.

 

could you also please provide following details,

 

1/ show system license

2/ request support information

 

thanks

raheel

Follow me on Twitter @anwar_raheel

--
If this post was helpful, please mark this post as an "Accepted Solution".
Kudos are always appreciated!
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

we are using routed tunnels to communicate with DB's and web services along with local services. I do not think we are doing any thing diffiernet, we originally had this runnig using  an ssg320M and we did not see this issue.

 

I am also using the switch function, which would  be the newest thing for us..

 

this is a remote site  with two routed site-to-site tunnels. one is for support and suppport tools, the other is the main site the the remote communicates with DB's and send syslog, snmp and event.

Frank Dias
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

one more thing, you will see VRRP setup on the unit as we have a second unit for redundancy. we were seeing this same issue before we added the second unit.

Frank Dias
Contributor
Posts: 16
Registered: ‎06-10-2008
0 Kudos

Re: vpn dropping

does anyone know of an SNMP mib that I can use to monitor the available memory (free memory)?

Frank Dias