Hi,
sorry for the response time. Had to wait for the issue to occur and the customer having enough time to let me troubleshoot.
Unfortunately clearing the tunnels doesn't seem to resolve a thing. The counters are also increasing, but it's not clear whether that's due to other traffic (there's traffic from monitoring probes reporting to the customer for example - these never have the issues with the tunnel failing probably because they don't have (extended) periods of time where there is no traffic).
Below is the output from the commands. IP addresses have been altered obviously. Hoping the other side will be available to troubleshoot at some point. As it's apparently not possible to sniff the actual tunnel interface I doubt it's possible to see if the traffic actually hits the tunnel at this end. Could sniff the external interface but I would see the encrypted tunnel packets then and it's quite hard to determine the original source of those.
Maybe there are some other useful commands to figure that out tho'. From below commands it appears the routes and the policies it hits are correct. Remains a weird issue.
Thanks for the re' :).
user@hostname> show security ipsec sa index 131076
ID: 131076 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_Range1
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=x1.x1.x1.x1/24)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2
Direction: inbound, SPI: 7b4b802d, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 892 seconds
Lifesize Remaining: 4603484 kilobytes
Soft lifetime: Expires in 288 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
Direction: outbound, SPI: 2a51c17a, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 892 seconds
Lifesize Remaining: 4603484 kilobytes
Soft lifetime: Expires in 288 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
user@hostname> show security ipsec sa index 131077
ID: 131077 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_Range2
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=x2.x2.x2.x2/23)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2
Direction: inbound, SPI: 6fae5d23, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 3505 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 2882 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
Direction: outbound, SPI: 405ba2fc, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 3505 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 2882 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
user@hostname> show security ipsec sa index 131078
ID: 131078 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_Range3
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=x3.x3.x3.x3/24)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2
Direction: inbound, SPI: ea2e5410, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1048 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 405 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
Direction: outbound, SPI: ebe172c5, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1048 seconds
Lifesize Remaining: 4608000 kilobytes
Soft lifetime: Expires in 405 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
user@hostname> show security ipsec sa index 131074
ID: 131074 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_VPC1
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=10.248.0.0/21)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2
Direction: inbound, SPI: 17dc1250, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 817 seconds
Lifesize Remaining: 4607885 kilobytes
Soft lifetime: Expires in 205 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
Direction: outbound, SPI: a47242e5, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 817 seconds
Lifesize Remaining: 4607885 kilobytes
Soft lifetime: Expires in 205 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
user@hostname> show security ipsec sa index 131075
ID: 131075 Virtual-system: root, VPN Name: IPSEC_Location-With-Issues_VPN_VPC2
Local Gateway: 1.2.3.4, Remote Gateway: 5.6.7.8
Local Identity: ipv4_subnet(any:0,[0..7]=172.31.0.0/16)
Remote Identity: ipv4_subnet(any:0,[0..7]=10.150.0.0/16)
Version: IKEv1
DF-bit: clear
Bind-interface: st0.2
Direction: inbound, SPI: 2a988379, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1053 seconds
Lifesize Remaining: 4606301 kilobytes
Soft lifetime: Expires in 430 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
Direction: outbound, SPI: d1ca00d6, AUX-SPI: 0
, VPN Monitoring: -
Hard lifetime: Expires in 1053 seconds
Lifesize Remaining: 4606301 kilobytes
Soft lifetime: Expires in 430 seconds
Mode: Tunnel(0 0), Type: dynamic, State: installed
Protocol: ESP, Authentication: hmac-sha1-96, Encryption: aes-cbc (256 bits)
Anti-replay service: counter-based enabled, Replay window size: 64
user@hostname> show security ipsec statistics index 131075
ESP Statistics:
Encrypted bytes: 246044544
Decrypted bytes: 634637272
Encrypted packets: 1872674
Decrypted packets: 1863674
AH Statistics:
Input bytes: 0
Output bytes: 0
Input packets: 0
Output packets: 0
Errors:
AH authentication failures: 0, Replay errors: 0
ESP authentication failures: 0, ESP decryption failures: 0
Bad headers: 0, Bad trailers: 0
(waited 15 secs)
user@hostname> show security ipsec statistics index 131075
ESP Statistics:
Encrypted bytes: 246045688
Decrypted bytes: 634640096
Encrypted packets: 1872685
Decrypted packets: 1863686
AH Statistics:
Input bytes: 0
Output bytes: 0
Input packets: 0
Output packets: 0
Errors:
AH authentication failures: 0, Replay errors: 0
ESP authentication failures: 0, ESP decryption failures: 0
Bad headers: 0, Bad trailers: 0
counters have increased - error counters still 0
user@hostname> show security flow session source-prefix 172.31.0.13 destination-prefix 10.150.150.170
Session ID: 24121, Policy name: Trust_Location-With-Issues-VPC/12, Timeout: 18, Valid
In: 172.31.0.13/49318 --> 10.150.150.170/3389;tcp, If: ge-0/0/1.0, Pkts: 2, Bytes: 104
Out: 10.150.150.170/3389 --> 172.31.0.13/49318;tcp, If: st0.2, Pkts: 0, Bytes: 0