SRX

Hello All,

We're newcomers in the Pulse Secure/Juniper Community and technology as well, seeking assistance for a resolution of a problem summarized in the subject of this post.

Our desktops/laptops  are running Win10 ver 1803 (OS Build 17134.590).

The gateways are:

Model: srx300
Junos: 15.1X49-D70.3
JUNOS Software Release [15.1X49-D70.3]

We own two SRX300 gateways setup to connect 2 business locations. PuTTy ssh can connect to both gateways w/o a problem even if the problem explained below is active.    

The Pulse secure VPN client installed on the Win10 machines is v. 5.2.11 (1195). We didn't buy Pulse Secure VPN client. We downloaded a copy of it from Pulse Secure's site and we use it with a license that comes by default with the purchase of an SRX 300.

When we restart the SRX gateway  error 1453 goes away but it comes back about 12-15  hrs later.

The body of network connection error 1453 reads as follows:

Network errors can be caused by temporary conditions such as an invalid URL , a server not available, and so on. Please try the operation again. Restart your system and try the operation again. If the problem persists, contact your network administrator.

We select Firewall(SRX) in the Type of Connection on Pulse and we're very sure we are using the correct Server URL.

To alleviate error 1453 we restart the gateway of the location the VPN client is connecting to, an hr or so  before we open  for business and that keeps our remote users' connectivity going for half a day, sometimes more than a day but on avg it lasts 12-15 hrs as mentioned above.

There's plenty of disk apce on the appliance. That's how, at least, we interpret the results showing below:

user@location1> show system storage detail
Filesystem 1024-blocks Used Avail Capacity Mounted on
/dev/da0s1a 2528708 251256 2075156 11% /
devfs 1 1 0 100% /dev
/dev/md0 20012 11820 6592 64% /junos
/cf/packages 2528708 251256 2075156 11% /junos/cf/packages
devfs 1 1 0 100% /junos/cf/dev
/dev/md1 808018 808018 0 100% /junos
/cf 20012 11820 6592 64% /junos/cf
devfs 1 1 0 100% /junos/dev/
/cf/packages 2528708 251256 2075156 11% /junos/cf/packages1
procfs 4 4 0 100% /proc
/dev/bo0s3e 189552 80 174308 0% /config
/dev/bo0s3f 2218426 114380 1926572 6% /cf/var
/dev/md2 687956 20384 612536 3% /mfs
/cf/var/jail 2218426 114380 1926572 6% /jail/var
/cf/var/jails/rest-api 2218426 114380 1926572 6% /web-api/var
/cf/var/log 2218426 114380 1926572 6% /jail/var/log
devfs 1 1 0 100% /jail/dev
/dev/md3 1884 292 1442 17% /jail/mfs

We tried troubleshooting the problem at hand with Pulse Secure official tech support but the moment we talk about VPN out to a Juniper hardware product they immediately point us to Juniper tech support, hence reaching out to Juniper Community, just in case someone else has experienced something similar and has an advise/recommendation how to resolve it.

Any help will be much appreciated

Thanks

Stavros

Hi,

Can you try the same connection but form a PC that is running Windows 7? or Windows 10 but with a version prior 1803?

Some imcopatibility problems between Pulse software and Windows 10 were introduced after windows version 1803. This is one exmaple of it:

https://kb.juniper.net/InfoCenter/index?page=content&id=TSB17441&actp=METADATA&act=login

Also take a packet capture with Wireshark on your PC when the problem is happening, I would like to confirm if the PC is sending packets to the SRX when the problem happens and if the SRX is responding.

Thanks for replying.

I forgot to mention that one of our remote users is using Win7 SP1 and she experiences the same connection error when the other one who uses Win10  Build 1803 can not connect.  

Per your advise when the error re-appears

1.  I'll test a Win10 machine with Build < 1803

2.  capture packets with Wireshark

Thanks

Stavros

When the connection error came back I initiated Pulse Secure and while kept clicking 'Retry' to create more VPN traffic I  started collecting packets with Wireshark. After a few secs I stopped and filtered for the destimation IP. That's what shows in the jpg attached.

As far as I know black packets are problematic ones so I understand if the jpg is likely not enough to offer further advice what to troubleshoot next. If that's the case how can I share the pcapng file securely ?  I'm concerned sharing it in the open b/c of the destination public IP address.

Thanks

Stavros

I tried capturing packets again this morning, only this time I filtered all traffic captured for packets that have the public IP as source

or destination. Example of the syntax w/ a non-routable IP follows below.

ip.src== 192.168.0.1 && ip.dst == 192.168.0.5

All VPN traffic in this collection and on my previous reply is going through my Wless interface.

The pattern of packets between this attempt and the one before seems the same. Anything I need to change so I can provide more info to troubleshoot the connection error ? 

Thanks

Stavros

correction on filter on previous reply.

the filter I applied was  

ip.src== Public_IP_Address || ip.dst == Public_IP_Address

where Public_IP_Address is the same on both sides of the OR

Thanks

Stavros

Hi stavrosk

On both packet captures we can see that the PC is sending SYN messages in order to create  a TCP connection with the SRX, however it is not receiving any replies and start retransmiting the SYN messages (black packets). We need to find out if the SRX is indeed not responding or if any device in between is dropping those packets at the time of the issue.

1. We can tell that the SRX's configuration is fine because the PC can connect from time to time.

2. Create a filter on the SRX with a counter that will increase if it receives the packets from the PC on its external interface. This way we can monitor the counter increases during the outages:

set firewall family inet filter FILTER term COUNTER from source-address [Public_IP_of_PC]
set firewall family inet filter FILTER term COUNTER from destination-port 443
set firewall family inet filter FILTER term COUNTER then count DynVPN
set firewall family inet filter FILTER term COUNTER then accept
set firewall family inet filter FILTER term ALLOW-ELSE then accept

set interfaces [SRX_External_Interface] unit 0 family inet filter input FILTER

commit

Review the counter with the following operational command: > show firewall

3. You could also try taking a pcap on the external interface of the SRX during the time of the issue:

               https://kb.juniper.net/InfoCenter/index?page=content&id=KB11709

4. Confirm if during the time of the issue, hosts with different public IP addresses (connecting from different locations) experience the same issue. This will help us to isolate a problem with the SRX or a device sitting in front of the SRX.

Thanks for your recommendations @lpaniagua

In the end the problem I had was internal.

Juniper's JTAC team investigated the SRX300 Gateway, where Pulse Secure VPN client suppose to connect, while the VPN connectivity was failing and found out that it was caused by an over-utilization of its Routing Engine.

Next, we will show the Juniper commands the JTAC engineer ran on the SRX in config mode

user@SRXGateway# run show chassis routing-engine
Routing Engine status:
Temperature 45 degrees C / 113 degrees F
CPU temperature 59 degrees C / 138 degrees F
Total memory 4096 MB Max 983 MB used ( 24 percent)
Control plane memory 2624 MB Max 656 MB used ( 25 percent)
Data plane memory 1472 MB Max 309 MB used ( 21 percent)
5 sec CPU utilization:
User 50 percent
Background 0 percent
Kernel 45 percent
Interrupt 0 percent
Idle 5 percent <---- idle was down to 0% when we initally executed the command at the time the VPN client was failing to connect
Last reboot reason 0x200:normal shutdown
Load averages: 1 minute 5 minute 15 minute

user@SRXGateway# run show system processes extensive | except 0.00
last pid: 49064; load averages: 2.33, 1.67, 1.38 up 3+07:22:08 10:40:48
160 processes: 16 running, 130 sleeping, 2 zombie, 12 waiting

Mem: 304M Active, 144M Inact, 1574M Wired, 385M Cache, 112M Buf, 1572M Free
Swap:

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
........................

.......................
49023 nobody 3 84 0 11600K 5032K ucondt 0 0:01 1.46% httpd <---- was in the low 90% when the VPN client could not connect

Finally he executed

run show security flow session interface st0.21

and that returned a list of pair of communicating IPs, their Session IDs and the Policy Names that allow them.

The overwhelming majority of the records were indicating 3 specific IP addresses trying to aggressively connect to the SRX itself. All IPs were assigned on Hardware in the internal network and there was no config to get accessed externally, via DDNS for example.

Two of those IPs were given to IP power switches [to power up hardware when power from the grid restores after an outage] and the 3rd one on a server.

The IP power switches by design ping 5 different external preset targets every few seconds; if those become unreacheable then they would power-cycle one or both of their AC power outlets according to the user's configuration.

Unexpectedly, the SRX300 logs showed the traffic from the IP power switches wasn't icmp, as expected, but plain http, so the fact it was directed to the SRX raised suspicions of having been hacked, so I removed them from the network.

The server among other SW had 2 very resources intensive network mngt SW [SolarWinds and PRTG Network Monitor] that were hitting the Gateway to extract descriptive network info and stats for reporting purposes.

Both pieces of SW were completely removed [down to the Windows registry level] from the server and the machine was deeply scanned with a couple of AVs to assess/ensure its virus/spyware clean status.

The Juniper engineer also disabled http mngt access on the SRX

[edit system services web-management] <--- config setting he changed

and we agreed not to restart the SRX again but monitor Pulse connectivity daily. So far it has been 6 days and we haven't experienced any Pulse secure VPN outage hoping it'll stay that way.

However, Juniper highly recommends NOT to use Pulse Secure as a VPN client accessing their gateways, especially from Win10 machines [albeit, from personal experience Pulse Secure still works from Win7 and it's pretty stable and reliable].

Instead they propose to use NCP. Below is a KB article about how to set it up on the SRX and a PDF with NCP set up instructions for the Client:

https://kb.juniper.net/InfoCenter/index?page=content&id=KB32418&actp=RSS

https://kb.juniper.net/library/CUSTOMERSERVICE/BK17364/NCP%20Secure%20Entry%20Client%20Configuration%20v3.pdf

I hope the above info is helpful providing enough insight to help others in their troubleshooting efforts.

Thanks

Stavros

For those users who will make the decision to purchase an NCP client here is the generic URL of their site:    https://www.ncp-e.com/en/exclusive-remote-access-solution/vpn-client/#c12977

In our case we don't need a volume license so, in order to obtain a few of those one will need what it's called an "NCP Exclusive Entry Client for Windows" as it shows at

https://www.ncp-e.com/en/exclusive-remote-access-solution/vpn-client/exclusive-entry-client/

How-To-Buy site of NCP Exclusive Entry Client for Windows, MacOS and Android devices with lots of FAQs

https://www.ncp-e.com/en/exclusive-remote-access-solution/how-to-buy/

With your permission we'll reply back after we run the installation of an NCP client so others can benefit from our findings. Please bear with us as it'll be our first time to set it up.

Thanks

Stavros

As you have noticed the pulse client is no longer supported and is having issues connecting on the most recent windows system patching.

So for official and tested support you would need to purchase the NCP licenses for the clients.

A free open source alternative you could test, but also would not be officially supported would be Shrew Soft.

https://www.shrew.net/

Thanks for your input @spuluka.  I wasn't aware of Shrew Soft.

With lots of help from Juniper's JTAC team we managed to configure NCP client and make it talk VPN to SRX300.

Here's an excerpt of the SRX config with generic values specifically for the NCP client-Gateway VPN communication

ALL THE CREDIT FOR WRITING THE EXCERPT BELOW GOES TO THE JTAC TEAM and not me.

set security ike proposal NCP-PROP authentication-method pre-shared-keys
set security ike proposal NCP-PROP dh-group group5
set security ike proposal NCP-PROP authentication-algorithm sha1
set security ike proposal NCP-PROP encryption-algorithm aes-128-cbc
set security ike proposal NCP-PROP lifetime-seconds 86400

set security ike policy NCP-POL mode aggressive
set security ike policy NCP-POL proposals NCP-PROP
set security ike policy NCP-POL pre-shared-key ascii-text juniper123

set security ike gateway NCP-GW ike-policy NCP-POL
set security ike gateway NCP-GW dynamic user-at-hostname "user@juniper.net"
set security ike gateway NCP-GW dynamic connections-limit 2
set security ike gateway NCP-GW dynamic ike-user-type shared-ike-id
set security ike gateway NCP-GW external-interface ge-0/0/0.0
set security ike gateway NCP-GW aaa access-profile ncp-vpn-profile
set security ike gateway NCP-GW version v1-only

set security ipsec proposal NCP_IPSEC_PRO protocol esp
set security ipsec proposal NCP_IPSEC_PRO authentication-algorithm hmac-sha1-96
set security ipsec proposal NCP_IPSEC_PRO encryption-algorithm aes-128-cbc
set security ipsec proposal NCP_IPSEC_PRO lifetime-seconds 28800

set security ipsec policy NCP_IPSEC_POL proposals NCP_IPSEC_PRO
set security ipsec policy NCP_IPSEC_POL perfect-forward-secrecy keys group5

set security ipsec vpn NCP-IPSEC bind-interface st0.0
set security ipsec vpn NCP-IPSEC ike gateway NCP-GW
set security ipsec vpn NCP-IPSEC ike ipsec-policy NCP_IPSEC_POL
set security ipsec vpn NCP-IPSEC traffic-selector TS1 local-ip 0.0.0.0/0
set security ipsec vpn NCP-IPSEC traffic-selector TS1 remote-ip 0.0.0.0/0

set security zones security-zone trust interfaces st0.0 host-inbound-traffic system-services all
set security zones security-zone trust interfaces st0.0 host-inbound-traffic protocols all
set interfaces st0 unit 0 family inet

set access profile ncp-vpn-profile authentication-order password
set access profile ncp-vpn-profile client test firewall-user password test123
set access profile ncp-vpn-profile address-assignment pool NCP-pool

set access address-assignment pool NCP-pool family inet network 10.1.1.0/24
set access address-assignment pool NCP-pool family inet xauth-attributes primary-dns 8.8.8.8/32
set access firewall-authentication web-authentication default-profile ncp-vpn-profile

Also, do not forget to check the security policies, and confirm you may need  a policy from Tunnel to Internal resources. Ensure that traffic started by checking the encrypted packets flow is increasing executing

# run show security ipsec sa

in order to get the TUNNEL_ID and then run

# run show security ipsec statistics index TUNNEL_ID

to confirm the encypted packets traffic is increasing

It is our experience that the NCP VPN client is VERY robust and connects really fast.

Lastly, if your Juniper Gateways are SRX300s please make sure you buy the Exclusive Entry client for Windows from https://www.ncp-e.com/en/exclusive-remote-access-solution/vpn-client/#c12977

I hope this helps someone else in a similar technical deadlock.

Please feel free to close this thread.

Thanks

Stavros