SRX

Hi,

we have a weird issue with a SRX210 and a VPN towards another location (think it's a Cisco on the other side - they're not too generous with information on that end). There ARE 5 subnets they have access to on the other location, so there's 1 phase 1 with 5 phase 2's.

SRX's have issues with this so you have to use NHTB. Basically this comes down to sucking some fake (unused) addresses out of your thumb, attaching them to phase 2's and then using these addresses in the routing table. Many have commented on it already - no clue why it's still like this.

Anyways 4 of the tunnels always work fine for all users (besides the proxy addresses of the phase 2, and the NHTB address which is unique for every tunnel, everything is identical). The 5th tunnel is always up as well but it's causing head aches (in the config it's actually the 2nd tunnel, so 1, 3,4 and 5 always work fine).

At some point a user can't access 10.150.0.0/16 any more, which is the second tunnel. At this point in the SRX the tunnels are up, traffic from that subnet towards their monitoring is still flowing and colleagues might or might not be able to access the subnet.

There's no fancy routing per user - just 1 route table, nothing special.

At this point if a user shoots traffic towards *any one* of the 4 other tunnels, 10.150.0.0/16 will come alive again. For *that* user. If his colleague was having the same issue at the same time - it is *not* yet resolved for the colleague. Colleague has to do the same thing (shooting traffic to any other tunnel part of this VPN/location) and it will work again.

Did some flow debugging with a ping (ping doesn't respond - it never does host on the other end blocks it) with and without the issue, but the traces are identical minus some session numbers and the likes.

Since tunnels are up, routes are the same for all users, traffic is flowing to other stuff from the same tunnel whilst other machines have the issue, etc. it seriously looks like a bug in JunOS. Client updated to latest 11.4 release (already were on 11.4 quite a bit older) 3 days ago.

Any ideas? Some other stuff I can debug? Or perhaps a less weird set up (without NHTB - the proxy addresses pretty much should make it clear where the traffic is supposed to go...).

PS there's also a tunnel to another location - it has no issues. --- all external IP addresses, identifying names, passwords etc. were replaced with fakes in the config. Shouldn't make a difference for the general idea - better safe than sorry :). I did adhere to replacing internal with internal addresses and external with external ones and yes - 3 of the tunnels out of the 5 are for public space - this is not an error - they're just not for the ranges in the attached config :). Some subnet masks (or well net addresses) might not match but that should be it.

TIA

Attachment(s)

the-config.txt   18 KB 1 version

Hi Freaky,

When that specific tunnel(2) is not working , help me with the following:

1. show security ipsec sa index number.
2. show security ipsec statistics index number ( check if encrypt counter is increasing)
3. show security flow session source-prefix x.x.x.x destination-prefix y.y.y.y

also does clearing that particular ipsec sa resolves the issue?

Regards
rparthi
 

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated Too

Hi,

sorry for the response time. Had to wait for the issue to occur and the customer having enough time to let me troubleshoot.

Unfortunately clearing the tunnels doesn't seem to resolve a thing. The counters are also increasing, but it's not clear whether that's due to other traffic (there's traffic from monitoring probes reporting to the customer for example - these never have the issues with the tunnel failing probably because they don't have (extended) periods of time where there is no traffic).

Below is the output from the commands. IP addresses have been altered obviously. Hoping the other side will be available to troubleshoot at some point. As it's apparently not possible to sniff the actual tunnel interface I doubt it's possible to see if the traffic actually hits the tunnel at this end. Could sniff the external interface but I would see the encrypted tunnel packets then and it's quite hard to determine the original source of those.

Maybe there are some other useful commands to figure that out tho'. From below commands it appears the routes and the policies it hits are correct. Remains a weird issue.

Thanks for the re' :).

user@hostname> show security ipsec sa index 131076
ID: 131076 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_Range1
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=x1.x1.x1.x1/24)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2

Direction: inbound, SPI: 7b4b802d, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 892 seconds
Lifesize Remaining: 4603484 kilobytes
Soft lifetime: Expires in 288 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

Direction: outbound, SPI: 2a51c17a, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 892 seconds
Lifesize Remaining: 4603484 kilobytes
Soft lifetime: Expires in 288 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

user@hostname> show security ipsec sa index 131077
ID: 131077 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_Range2
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=x2.x2.x2.x2/23)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2

Direction: inbound, SPI: 6fae5d23, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 3505 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 2882 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

Direction: outbound, SPI: 405ba2fc, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 3505 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 2882 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

user@hostname> show security ipsec sa index 131078
ID: 131078 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_Range3
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=x3.x3.x3.x3/24)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2

Direction: inbound, SPI: ea2e5410, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1048 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 405 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

Direction: outbound, SPI: ebe172c5, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1048 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 405 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

user@hostname> show security ipsec sa index 131074
ID: 131074 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_VPC1
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=10.248.0.0/21)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2

Direction: inbound, SPI: 17dc1250, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 817 seconds
Lifesize Remaining: 4607885 kilobytes
Soft lifetime: Expires in 205 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

Direction: outbound, SPI: a47242e5, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 817 seconds
Lifesize Remaining: 4607885 kilobytes
Soft lifetime: Expires in 205 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

user@hostname> show security ipsec sa index 131075
ID: 131075 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_VPC2
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=10.150.0.0/16)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2

Direction: inbound, SPI: 2a988379, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1053 seconds
Lifesize Remaining: 4606301 kilobytes
Soft lifetime: Expires in 430 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

Direction: outbound, SPI: d1ca00d6, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1053 seconds
Lifesize Remaining: 4606301 kilobytes
Soft lifetime: Expires in 430 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64

user@hostname> show security ipsec statistics index 131075
ESP Statistics:
Encrypted bytes: 246044544
Decrypted bytes: 634637272
Encrypted packets: 1872674
Decrypted packets: 1863674
AH Statistics:
Input bytes: 0
Output bytes: 0
Input packets: 0
Output packets: 0
Errors:
AH authentication failures: 0, Replay errors: 0
ESP authentication failures: 0, ESP decryption failures: 0
Bad headers: 0, Bad trailers: 0

(waited 15 secs)

user@hostname> show security ipsec statistics index 131075
ESP Statistics:
Encrypted bytes: 246045688
Decrypted bytes: 634640096
Encrypted packets: 1872685
Decrypted packets: 1863686
AH Statistics:
Input bytes: 0
Output bytes: 0
Input packets: 0
Output packets: 0
Errors:
AH authentication failures: 0, Replay errors: 0
ESP authentication failures: 0, ESP decryption failures: 0
Bad headers: 0, Bad trailers: 0

counters have increased - error counters still 0

user@hostname> show security flow session source-prefix 172.31.0.13 destination-prefix 10.150.150.170
Session ID: 24121, Policy name: Trust_Location-With-Issues-VPC/12, Timeout: 18, Valid
In: 172.31.0.13/49318 --> 10.150.150.170/3389;tcp, If: ge-0/0/1.0, Pkts: 2, Bytes: 104
Out: 10.150.150.170/3389 --> 172.31.0.13/49318;tcp, If: st0.2, Pkts: 0, Bytes: 0

Hi

user@hostname> show security flow session source-prefix 172.31.0.13 destination-prefix 10.150.150.170
Session ID: 24121, Policy name: Trust_Location-With-Issues-VPC/12, Timeout: 18, Valid
In: 172.31.0.13/49318 --> 10.150.150.170/3389;tcp, If: ge-0/0/1.0, Pkts: 2, Bytes: 104
Out: 10.150.150.170/3389 --> 172.31.0.13/49318;tcp, If: st0.2, Pkts: 0, Bytes: 0 <<<<<<<<<<< PKTs=0

above output  shows that 2 packets have  been sent across the tunnel but there is no reply packet from remote peer.

Packets are going out through the vpn tunnel . it is not VPN monitoring packet. Monitoring is not enabled.

user@hostname> show security ipsec statistics index 131075
ESP Statistics:
Encrypted packets: 1872674
Decrypted packets: 1863674

user@hostname> show security ipsec statistics index 131075
ESP Statistics:
Encrypted packets: 1872685
Decrypted packets: 1863686

It confirms that it is an issue on the remote server and not SRX/

Regards
rparthi
 

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated Too

Could be routing on the remote VPN endpoint or most likely routing/default route on the devices behind the remote VPN endpoint.

Just wanted to say thanks for the re's. It's holiday season, customer hasn't called with the issue occuring again yet and when it happens both I and the remote side will need time for troubleshooting and it might take quite some times for all those to align nicely due to the holidays.

So thanks :).

Much faster than anticipated: it was the remote end.

The remote end does source NAT (very large corporate network - don't have much choice). This NAT/MIP was only triggered/configured for rules to the other 4 subnets over that tunnel. Once the source IP would be in the NAT table it would also be NAT'ed for the problematic range, which explains why it works after shooting traffic to one of the 4 other subnets.

Until it's in the table however... 🙂

Thanks a bunch.

Hi Freaky,

Thanks for the update.

Regards

rparthi