vpn dropping

[frank3427]

I have an SRX240 running 10.1.R1.8 with routed IPSec vpn to 2 SSG320's.

the tunnel stays up for a few days and then it starts to bounce every minute to 2 minutes until the srx is rebooted.

the SSGs are runnit 6.2r5 and 6.2r3 same issue on both.

SXR ---- internet ------- SSG#1 -------------- SSG#2

SRX = 10.1r1.8

ssg#1 6.2r3

ssg#2 6.2r5

[Raheel]

please attach relevant logs,

/var/log/kmd

/var/log/messages

/var/log/jsrpd

thanks

raheel

[frank3427]

here are the files requested

[drive567]

nice logs.

Interface is flapping. from jsrpd logs.

Mar 1 18:28:07 Interface ge-0/0/2 is going up

Mar 1 18:30:46 Interface ge-0/0/2 is going down

Mar 1 18:30:50 Interface ge-0/0/2 is going up

Mar 1 18:37:06 Interface ge-0/0/2 is going down

Mar 1 18:37:10 Interface ge-0/0/2 is going up

Mar 1 18:43:58 Interface ge-0/0/2 is going down

Mar 1 18:44:02 Interface ge-0/0/2 is going up

Possible memory leak. from kmd logs.

Mar 1 03:27:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:29:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:29 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:31:58 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

Mar 1 03:33:30 KMD_INTERNAL_ERROR: Error in adding ipsec SA pair , No buffer space available

[Raheel]

thanks for posting logs,

could you please run following commands and provide output.

1/ show security ike memory-usage

If you notice that any of the counters  are incrementing significantly over a period of time, then that could be the root cause of the problem.This command does not account for the memory allocated by libraries used by iked. You can also peroidically monitor the output of "top" on the shell. Please note that top is executed on the shell and NOT on cli.

2/ top -n (run it for multiple times, when encounter issue)

thanks

raheel

[frank3427]

here is the output;

thanks

[frank3427]

hi drive567,

the ge-0/0/2 is a connection to a server, the ipsec is bound to ge-0/0/0

so you think there is a possible memory leak, what other logs can I provide to confirm this assumption?

[frank3427]

just happened again

[Raheel]

Frank,

memory usage seems to be fine from the first log o/p, I couldn't able to see "top -n" output in your second log. could you please directly paste the output again?

also, please attach your configs.

thanks

raheel

[Raheel]

Looking at the top command, I see there is only very little memory available in kernel (1040k) generally it should be more than 100M. KMD occupied memory is reasonable. So it could be some other daemon leaking the memory.

o/p of top command is not complete. Some of the daemon is next page. Will it possible to get all pages of top? or try CLI cmd "show system processes extensive "

thanks

raheel

% top -n^M
last pid:  7073;  load averages:  3.09,  3.10,  3.04  up 2+12:56:00    16:49:32^M
67 processes:  6 running, 61 sleeping^M
^M
Mem: 128M Active, 79M Inact, 685M Wired, 78M Cache, 112M Buf, 1040K Free^M
Swap: ^M
^M
^M
  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND^M
  998 root        5  76    0   457M 41296K select 0 187.8H 281.88% flowd_octeon_hm^M
  994 root        1  76    0  7408K  3300K select 0   8:52  0.00% ppmd^M
  983 root        1  76    0 17696K 11700K select 0   7:30  0.00% snmpd^M
  984 root        1  76    0 13132K  7508K select 0   6:35  0.00% mib2d^M
 1012 root        1  76    0 11576K  5132K select 0   4:02  0.00% utmd^M
 1009 root        1  76    0 10440K  4128K select 0   2:44  0.00% jdiameterd^M
 1018 root        1   4    0 11176K  5684K kqread 0   2:03  0.00% eswd^M
  986 root        1  76    0 13664K  4732K RUN    0   2:02  0.00% l2ald^M
  978 root        1  76    0 26512K 13960K select 0   1:49  0.00% chassisd^M
  988 root        1  76    0  8996K  3904K select 0   1:44  0.00% vrrpd^M
  975 root        1  76    0  2612K  1232K select 0   1:20  0.00% bslockd^M
 1016 root        1  76    0  3244K  1036K select 0   1:19  0.00% license-check^M
  979 root        1  76    0  5996K  2552K select 0   0:53  0.00% alarmd^M
 993 root        1  76    0 14680K  6268K select 0   0:47  0.00% kmd^M
1014 root        2  76    0  9804K  3616K select 0   0:40  0.00% wland^M
  985 root        1   4    0 36260K 18500K kqread 0   0:40  0.00% rpd^M
  977 root        1  76    0 23352K  8096K select 0   0:38  0.00% dcd^M
 1011 root        1  76    0  9044K  3580K select 0   0:29  0.00% rtlogd^M