SRX Services Gateway
Highlighted
SRX Services Gateway

Dynamic VPN reconnect issue

‎06-12-2020 04:53 AM

We are using dynamic VPN with SRX320. Normally connections do work, but there happens often so that Pulse app shows that connection is taken, but actually no access works. This problem occurs for many of our users. Often the case is reconnecting in morning etc. but sometimes connection totally fails. Any ideas what can cause this?

Client side information:
All use with Win10. Pulse Secure, Firewall SRX connection type. We have tried Pulse 9.1r6, r5, r4, r3.1, r2, 9.0r4 and 5.3r3 versions and all of those have had issues, so I can say that the problem is not a wrong Pulse release etc.

On Juniper side 'show log kmd-logs' in failure cases like this:

IKE succeeds:

Jun 11 10:21:49 OUR-FW kmd[2001]: IKE negotiation successfully completed. IKE Version: 1, VPN: DYNAMIC-VPN Gateway: DYNAMIC-VPN, Local: 123.123.123.123/4500, Remote: 80.81.82.83/17000, Local IKE-ID: 123.123.123.123, Remote IKE-ID: user123dynvpn, VR-ID: 0, Role: Responder


Jun 11 10:21:58 OUR-FW kmd[2001]: KMD_VPN_UP_ALARM_USER: VPN DYNAMIC-VPN from 80.81.82.83 is up. Local-ip: 123.123.123.123, gateway name: DYNAMIC-VPN, vpn name: DYNAMIC-VPN, tunnel-id: 67110011, local tunnel-if: , remote tunnel-ip: Not-Available, Local IKE-ID: 123.123.123.123, Remote IKE-ID: user123dynvpn, AAA username: user123, VR id: 0, Traffic-selector: , Traffic-selector local ID: ipv4_subnet(any:0,[0..7]=10.0.0.0/8), Traffic-selector remote ID: ipv4(any:0,[0..3]=172.20.16.1), SA Type: Static

After <60 seconds later IPSec fails:
Jun 11 10:22:54 OUR-FW kmd[2001]: IPSec negotiation failed with error: Timed out. IKE Version: 1, VPN: DYNAMIC-VPN Gateway: DYNAMIC-VPN, Local: 123.123.123.123/4500, Remote: 80.81.82.83/17000, Local IKE-ID: 123.123.123.123, Remote IKE-ID: user123dynvpn, VR-ID: 0


After <10min later:
Jun 11 10:32:14 OUR-FW kmd[2001]: KMD_VPN_DOWN_ALARM_USER: VPN DYNAMIC-VPN from 80.81.82.83 is down. Local-ip: 123.123.123.123, gateway name: DYNAMIC-VPN, vpn name: DYNAMIC-VPN, tunnel-id: 67110011, local tunnel-if: , remote tunnel-ip: Not-Available, Local IKE-ID: 123.123.123.123, Remote IKE-ID: user123dynvpn, AAA username: user123, VR id: 0, Traffic-selector: , Traffic-selector local ID: ipv4_subnet(any:0,[0..7]=10.0.0.0/8), Traffic-selector remote ID: ipv4(any:0,[0..3]=172.20.16.1), SA Type: Static, Reason: IPSec SA delete payload received from peer, corresponding IPSec SAs cleared

Here are some snapshots of our configuration that are relevant:

IKE proposal:
lifetime-seconds 28800;

vpn DYN-VPN {
ike {
gateway DYN-VPN;
idle-time 72000; => Was not set ealier, but tried this too (to be removed?)
ipsec-policy DYN-VPN;
install-interval 3; => Was not set ealier, but tried this too (to be removed?)


gateway DYN-VPN {
ike-policy DYN-VPN;
dynamic {
hostname dynvpn;
connections-limit 25;
ike-user-type group-ike-id;
}
dead-peer-detection { => Was not set ealier, but tried this too (to be removed?)
optimized;
interval 10;
threshold 5;
}

Tunnel is configured as a split tunnel.


Should for example e.g. proxy-identies be configured also for Dynamic VPN? This is something we haven't yet tried as Juniper's dynamic VPN documentation show this (https://kb.juniper.net/InfoCenter/index?page=content&id=KB29364&actp=METADATA)

In 80% of cases (re)connection do work, but not always. We have already spent quite long time in solving this but the problem persists. Issue has been also reported to Juniper, but so far ticket has proceeded very slowly so all possible tips that you have are good.

19 REPLIES 19
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

[ Edited ]
‎06-12-2020 05:06 AM

Hello , 

 

Dynamic VPN is used as policy based VPN and there is no need to configure proxy IDs . We have something as remote protected resource which does the traffic handling part . 

 

Regarding your connection issue , it looks like the initial XAUTH itself is not working or coming to that stage . Have you tried clearing the token info for Dyn-VPN  from Shell and test it :

 

file delete /var/db/dynamic-vpn-ipsec/tokens-info


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-14-2020 06:59 AM

Thank you. Deleting token-info I haven't tried before. But seems that it did not help in this case. I am still getting error reproduced where kmd-logs have "IPSec negotiation failed with error: Timed out" trace.

 

Any other ideas what to try?

 

Should VPN's public IP be listed on remote-protected-resources list by the way? That we don't have there.

 

Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-14-2020 09:32 PM

Hello , 

 

VPN remote protected resorces are the IP/Subnet that is allowed for Dynamic VPN users to access ( like access list ) . If token information did not  fix the issue , can you try to restart web management to see if that fixes the issue . Also let us know the JUNOS running on the SRX . 

 

How many licenses are there for the Dynamic VPN users in your device ?  ( show system lisences )  

 

 

 


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-14-2020 09:38 PM

Hello , 

 

Also can you remove these 2 configurations and test :

 

idle-time 72000; => Was not set ealier, but tried this too (to be removed?)
install-interval 3; => Was not set ealier, but tried this too (to be removed?)

 

1) Removed above configs 
2) Delete the token info 
3) Restrat web-menagement  ( "restart web-management " from CLI) 

 

Then try again to connect .  


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 12:39 AM

Hi!
Thanks Sam. I removed idle-time and install-interval. Token info deleted and restarted web-management.
I the first tests I was still seeing the issues, but further testing continues.

I tried to reboot Juniper fully during last weekend too.


'show system license' shows licenses installed, but used count stays all the time 0. Is that how it should be for dynamic-vpn? There are 5+ users connected by looking via show 'security dynamic-vpn users'.


show system license

License usage:
Licenses Licenses Licenses Expiry
Feature name used installed needed
dynamic-vpn     0      50    0 permanent
logical-system   1      3      0 permanent
remote-access-ipsec-vpn-client   0    2     0 permanent

Licenses installed:
License identifier: JUNOS12345678
License version: 4
Valid for device: CW4567890
Customer ID: customer here
Features:
dynamic-vpn-50-clients - Dynamic VPN
permanent

...

connections-limit is set correctly to 50 (I wrote it earlier wrong to be 25).

We have Junos 18.4R3-S2 (was 15.1X49-D120.3 earlier and that had issues too).

 

*****

About the problem:
Problem is rather easy to get produced by disconnecting in Pulse and reconnecting very soon after.

 

What I have noticed that exiting Pulse, ipconfig /renew in admin command prompt and reconnecting in Pulse often restores the connection. Renew is updating IP by one digit. But disconnect+reconnect brings Pulse easily back into problematic state.

Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 12:58 AM

Hello , 

 

Thanks for the testing and the results . Can you share your configuration  ( especially the VPN and NAT part ) . Do we have any NAT configuration attached to the Interface where the Dynamic VPN connects ? Like static or destination NAT .


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 03:11 AM

I didn't notice anything special in NAT configuration that would impact.
NAT and VPN related configuration what I can share now attached as a text file.

 


There are two options that we haven't tried yet. These are not instructed by the default Dynamic VPN guide. Could these impact?

https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/security-...

1) config-checkEnable extra dynamic VPN configuration checking. If you include this statement in your configuration, it is automatically enabled. If the statement is not present in your configuration, the configuration check option is not enabled. This feature is supported on SRX300, SRX320, SRX340, SRX345, and SRX550HM devices.
2) force-upgradeForce upgrade the dynamic vpn.
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 03:28 AM

Hello , 

 

config-check is fine . But force upgrade will upgrade the pulse to the latest , which may not be supported as per Juniper . If you still face the issue . 

When you try to connect and fail , does it give any error on the pulse client , or just simply hungs and never connects ? 


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 06:09 AM

There are no errors on Pulse app. UI just shows green that connection gets up.

There are no lines in Pulse's detailed logs either that I see to be real errors. This is pretty much only error line that gets printed there. But I see the same line for succesful connection too...

00167,09 2020/06/15 14:14:07.750 2 SYSTEM PulseSecureService.exe vpnAccessMethod p5008 t5260 uiPluginRequests.cpp:28 - 'jamUIPlugin' DSAccessMonitorPlugin failed with error 3


On router side:

********
kmd-logs have either of these:
* IPSec negotiation failed with error: Timed out.
* IPSec negotiation failed with error: Aborted.

********

show security ike sa detail 90.90.90.90
IKE peer 90.90.90.90, Index 5147101, Gateway Name: DYNAMIC-VPN
Role: Responder, State: UP
Initiator cookie: abc, Responder cookie: def
Exchange type: Aggressive, Authentication method: Pre-shared-keys
Local: 123.123.123.123:4500, Remote: 90.90.90.90:60795
Lifetime: Expires in 86046 seconds
Reauth Lifetime: Disabled
IKE Fragmentation: Disabled, Size: 0
Remote Access Client Info: Unknown Client
Peer ike-id: user123dynvpn
AAA assigned IP: 172.10.100.148
Algorithms:
Authentication : hmac-sha1-96
Encryption : aes128-cbc
Pseudo random function: hmac-sha1
Diffie-Hellman group : DH-group-2
Traffic statistics:
Input bytes : 4139
Output bytes : 5844
Input packets: 35
Output packets: 38
Input fragmentated packets: 0
Output fragmentated packets: 0
IPSec security associations: 1 created, 0 deleted
Phase 2 negotiations in progress: 1

Negotiation type: Quick mode, Role: Responder, Message ID: 0
Local: 123.123.123.123:4500, Remote: 90.90.90.90:60795
Local identity: 123.123.123.123
Remote identity: user123dynvpn
Flags: IKE SA is created



show security ipsec sa detail index 67109999
ID: 67109999 Virtual-system: root, VPN Name: DYNAMIC-VPN
Local Gateway: 123.123.123.123, Remote Gateway: 90.90.90.90
Local Identity: ipv4_subnet(any:0,[0..7]=10.0.0.0/8)
Remote Identity: ipv4(any:0,[0..3]=172.10.100.148)
Version: IKEv1
DF-bit: clear, Copy-Outer-DSCP Disabled , Policy-name: DYNAMIC-VPN-policy
Port: 60795, Nego#: 363, Fail#: 0, Def-Del#: 0 Flag: 0x608c29
Multi-sa, Configured SAs# 1, Negotiated SAs#: 1
Tunnel events:
Mon Jun 15 2020 14:45:02 +0300: No response from peer. Negotiation failed (1 times)
Mon Jun 15 2020 14:44:06 +0300: IPSec SA negotiation successfully completed (1 times)
Mon Jun 15 2020 14:44:06 +0300: Tunnel is ready. Waiting for trigger event or peer to trigger negotiation (1 times)
Mon Jun 15 2020 14:43:56 +0300: IKE SA negotiation successfully completed (1 times)
Direction: inbound, SPI: 33a4b048, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 3308 seconds
Lifesize Remaining: 499963 kilobytes
Soft lifetime: Expires in 2715 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (128 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
Direction: outbound, SPI: dad77e08, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 3308 seconds
Lifesize Remaining: 499963 kilobytes
Soft lifetime: Expires in 2715 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (128 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 06:34 AM

Hello , 

 

I tested on SRX320 with 18.4  and it did not give any connection issues for more than 10 attempts using local authentication . Are you doing local or remote radius Auth ? for the users . If you are using local . Configure a new local user and test .  

 

If still you see the same issue , we need more detailed investigation with traces enabled . Probebaly a JTAC case will help .  


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 07:35 AM

Big thanks Sam for helping in this one. Unfortunately did not find yet a working solution for the problem.


local or remote radius Auth ? for the users . If you are using local . Configure a new local user and test .  
We did use Radius in the beginning but rolled back to local accounts because originally suspected that could solve the problem. We have ~40 accounts on the list and I belive that creating a new one will help...

 

If still you see the same issue , we need more detailed investigation with traces enabled . Probebaly a JTAC case will help .  

We have a case open too. It has been open 5 weeks now. Originally the problem was more visible as Pulse continued popping up password query and error description has changed somewhat. Tips that you have provided here have been much concrete so far.

 

 

 

Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 08:41 AM

Hello , 

 

I guess we have tried all possible tips as per my knowledge . The only thing left is to activate specific traces related to IKE , IPSEC and web management to understand the reason why the VPN client disconnects during the connected state . And also probebaly needed advanced log setting from Pulse client also . 

The error indicates a miss communication between the Pulse adapter and the SRX . But we can't figure out where its geting dropped until we have a detailed logs collected from both the ends . 

Also I would suggest to disable any monitoring  ( DPD or VPN-monitoring ) in the configuration to avoid any disconnection in between . 

We have seem multiple connectivity issues on Junos below 15.1X49 . But 18.4 is by far very stable for Pulse connection , not much reported issues . 

 

 


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-15-2020 08:46 AM

Hello , 

 

One more point is that , during your disconnection/error time , do you see the HTTPD process going high or move to "ucond"  state  ? 

 

> show system process extensive | match http


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-23-2020 01:33 PM

For the longest time, I have the same issue with Pulse and SRX Dynamic VPN. I can connect to the VPN but I can't access any resource.

 

What I found out is that the "Juniper Network Service" on the client PC interferes with the Pulse VPN.

aaa.png

 

Here's the link where this is described

https://kb.pulsesecure.net/pkb_mobile#article/l:en_US/kA1f1000000fz7VCAQ/s

 

 

What I do now is before I connect to the VPN, I disable the Juniper Network Service attached on my physical interface first using the Powershell command

 

Disable-NetAdapterBinding -Name * -DisplayName "Juniper Network Service"

 

Not an elegant solution but so far works

 

 

 

 

 

Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-23-2020 09:45 PM

Hello , 

 

Can this network adapter be removed somehow ?  Like removing and re-installing the network adapter driver will not help ? 


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-23-2020 11:33 PM

Our VPN stability issue root cause looks to be solved. At least so last couple days VPN has been more stable for our users.


This is ending up to be a client side issue.  In the end this got solved via our JTAC/Juniper support. The case just took 6 weeks to proceed enough, but thanks to Juniper support for this finding.

What I wrote in the beginning:
> We have tried Pulse 9.1r6, r5, r4, r3.1, r2, 9.0r4

There is a bigger issue in Pulse Secure application releases. Those (or many of those) are packaging wrong Juniper driver version or that the driver is not getting updated by the installer at all. Thus there comes an incompatiblity. 

The problematic file is C:\Windows\System32\drivers\jnprns.sys. Even though you would have 9.1.x Pulse, you might have 5.x Juniper driver. Easiest way to check this incompatility is to run this in command prompt:

powershell -command "(Get-Item -Path 'C:\Windows\System32\drivers\jnprns.sys').VersionInfo"
=> If you get 5.x.x, then you have it wrong. But should be:

ProductVersion FileVersion FileName
-------------- ----------- --------
9.1.0.0001 9.1.0.0001 bu... C:\Windows\System32\drivers\jnprns.sys


Seems that none of Pulse app versions downloaded from Pulse's web pages contained right driver version. Or well I didn't try to reinstall those again to check. But that was at least the case for 9.1r6. There was not mentioned anywhere in release notes what Juniper driver what version is using.

Right Pulse version to use is 9.1r2 that is downloaded from Juniper's pages. This release is dated 11-Oct-2019:
https://support.juniper.net/support/downloads/?p=pulse


There were at least two users who uninstalled newer Pulse properly, rebooted and installed this 9.1r2. But after installation Juniper driver file was not installed at all. There is also possibility that the driver file doesn't get upgraded properly. Below are the steps how to fix this driver incompatibility.

Update driver to be right one

  1. Open Command Prompt in Admin mode and run: sc stop jnprns
  2. Check do you have file jnprns.sys in this folder:

C:\Program Files (x86)\Common Files\Juniper Networks\JNPRNA\Drivers\jnprns\

Ctrl+c or right-click to copy the file.
(File can be under C:\Program Files\Common Files\Juniper Networks\JNPRNA\Drivers\jnprns\ )

  1. Via File Explorer, locate the jnprns.sys at C:\Windows\System32\drivers\ and rename it e.g. to jnprns.sys.old
  2. Paste the file to C:\Windows\System32\drivers\
  3. Check that you have there now file C:\Windows\System32\drivers\jnprns.sys
  4. Run command in admin cmd: sc stop jnprns

In the end run this command again to check that the file is finally right version:
powershell -command "(Get-Item -Path 'C:\Windows\System32\drivers\jnprns.sys').VersionInfo"

 

Juniper should work together with Pulse company to get Pulse Secure releases fixed so that all would contain right Juniper driver file. And would be nice to see that version information in release notes. Our company uses also pure Pulse Connect Secure VPN in another country. If we would need to upgrade Pulse Secure app e.g. because of security reasons, we can't do it, because we are now tight to older 9.1r2 release.

Even though this ended up to not to be an router side, BIG THANKS @joses for the help in investigating this.

Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-23-2020 11:45 PM

Hello , 

 

Thanks for posting the detailed update on the fix .  


Thanks,
Sam

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-26-2020 05:08 AM

Post script:

 

General information about these tracelines:

I still see in Juniper kmd logs these IPSec timeout lines. E.g. in yesterday's log 46 times (roughly per 20 users):
  IPSec negotiation failed with error: Timed out. IKE Version: 1, VPN: DYNAMIC-VPN Gateway:

-> So these lines are perhaps normal and do not mean problems (probably).

 

I still see in Juniper kmd logs these IKE aborted line. E.g. yesterday 6 times:
  IPSec negotiation failed with error: Aborted. IKE Version: 1, VPN: DYNAMIC-VPN Gateway:

 

General feedback to Juniper company:

1) I am still sometimes getting password re-query in Pulse. Probably when IKE rekeying happens (once a day if connection is kept on). Reconnecting doesn’t work by giving password but nowadays Pulse exit and start again brings connection back (-> not optimal, but much better than what is used to be). So there is still some issue in Juniper or Pulse left, but this is not as critical any more. This matches timely to the 'aborted' trace line.

 

2) 9.1r2 installer that is provided on Juniper pages contains the right driver file. But the file doesn’t get installed at all (at least for many users) but require copying that after the install. -> Please fix the installer.

 

3) Pulse company’s own 9.1rX releases. E.g. the latest 9.1r6 (and some before) do contain the old 5.x.x Juniper driver. There is no mention in Pulse release notes that driver would have changed. Feedback to Juniper company would be be collaborate with Pulse company to get the releases fixed. And preferably to get Pulse to include Juniper driver information into their release notes.

Highlighted
SRX Services Gateway

Re: Dynamic VPN reconnect issue

‎06-29-2020 01:20 PM

I uninstalled the latest from Pulse and installed the Juniper one. So far it's been working properly so that's a good thing.

Feedback