SRX Services Gateway
Highlighted
SRX Services Gateway

SRX210 & Azure VPN

‎07-18-2019 05:17 PM

Hi,

We have two VPN tunnels to Azure and we're experiencing an issue were traffic stops passing through the Azure side of the tunnel.

Both tunnels stay up, just not passing traffic.

I can see the traffic leaving from our side of the tunnel, just not working on Azure.

Azure support have no idea whats going on and want me to engage Juniper.

Does anyone have any ideas?

Thanks.

8 REPLIES 8
SRX Services Gateway

Re: SRX210 & Azure VPN

[ Edited ]
‎07-18-2019 06:08 PM

wyze,

 

How are you confirming that you are sending traffic to them?

 

You can run the following command in order to confirm if the SRX is encrypting data via that VPN:

 

user@host> show security ipsec statistics index [tunnel_index]
ESP Statistics:
  Encrypted bytes:              952
  Decrypted bytes:              588
  Encrypted packets:              7
  Decrypted packets:              7

 

The tunnel_index highlighted above can be found by confirming the index of the tunnel towards Azure running a "show security ipsec security-associations" command.

 

Also you can create a firewall-filter with a counter to tell if the SRX is sending ESP/encrypted packets towards Azure:

 

1. Create filter:

#set firewall family inet filter FILTER term ESP_COUNTER from destination-address [Azure_Public_IP]
#set firewall family inet filter FILTER term ESP_COUNTER from source-address [SRX_Public_IP]
#set firewall family inet filter FILTER term ESP_COUNTER from protocol esp
#set firewall family inet filter FILTER term ESP_COUNTER then count ESP_PACKETS-TO_AZURE
#set firewall family inet filter FILTER term ESP_COUNTER then accept
#set firewall family inet filter FILTER term ALLOW_ELSE then accept

2. Apply filter:

#set interfaces [SRX_External_Interface] unit 0 family inet filter output FILTER

3. Commit changes for only 5 minutes

#commit confirmed 5

4. Check the counter:

> show firewall

 

If the counter increases you have another proof that the SRX is sending encrypted traffic to Azure. If Azure tells that they are not receiving any encrypted packets, then you will have to engage your ISP in order to confirm why this traffic is being dropped/not reaching Azure.

 

If you see ESP packets being sent by the SRX, you could try rebooting the ISP modem in front of the SRX; sometimes they drop traffic unexpectedly.

 

Pura Vida from Costa Rica - Mark as Resolved if it applies.
Kudos are appreciated too!
SRX Services Gateway

Re: SRX210 & Azure VPN

‎07-18-2019 07:30 PM

Hi epanigua,

 

Thank you the reply. Will try what you have suggested.

 

When I'm told users cant access a virtual machine in Azure, I check both the SRX and Azure to see what the status of the tunnel is, and both sides say Up. I then ask them to initiate traffic and I run the below:

 

root@SRXFW> show security flow session destination-prefix 10.0.7.6
Session ID: 11712, Policy name: jump05-to-vpn-azure/149, Timeout: 18, Valid
In: 172.18.41.64/56193 --> 10.0.7.6/3389;tcp, If: vlan.1841, Pkts: 2, Bytes: 104
Out: 10.0.7.6/3389 --> 172.18.41.64/56193;tcp, If: st0.10, Pkts: 0, Bytes: 0
Total sessions: 1

 

Azure support say they can see the SRX not replying, but I cant see anything.

 

Can attach some logs if it will help.

 

Thanks.

 

SRX Services Gateway

Re: SRX210 & Azure VPN

[ Edited ]
‎07-19-2019 09:20 AM

wyze,

 

Can you elaborate on your topology so that I can better understand the output? Is it like this?:

 

172.18.41.64------SRX------------INTERNET----------Azure---------10.0.7.6

 

If so, then the output shows that the SRX is not seeing the reply from Azure VM over the VPN.

 

Besides the tests mentioned on my first comment, you can try a flow traceoptions to confirm if the SRX is processing the packets coming from Azure:

 

1.Configure the traces

#set security flow traceoptions file TRACE
#set security flow traceoptions flag basic-datapath
#set security flow traceoptions packet-filter TEST source-prefix [Azure_VM_IP]
#set security flow traceoptions packet-filter TEST destination-prefix [Internal_IP]

2. Commit

#commit

3. Send traffic from Azure side

4. Check the file

>show log TRACE

 

You can attach the outputs from the suggested commands as well.

 

Pura Vida from Costa Rica - Mark as Resolved if it applies.
Kudos are appreciated too!
SRX Services Gateway

Re: SRX210 & Azure VPN

[ Edited ]
‎07-21-2019 05:34 PM

Thank you epaniagua.

Will try your suggestions and let you know.

The tunnel traffic is only one way (LAN > Azure)

 

As this is a special case, the topology is:

 

172.18.41.64---------SRX--------Checkpoint firewall---------INTERNET---------Azure--------10.0.7.6.

 

The Checkpoint has the necessary policies to allow the tunnels to establish. We have two other tunnels from the SRX to two overseas offices and they are stable. Both sites are SRX as well.

 

We've had the Azure issue before the SRX was moved behind the Checkpoint.

SRX Services Gateway

Re: SRX210 & Azure VPN

‎07-21-2019 06:19 PM

So it just stopped passing traffic again on the Azure side. Both are saying that the tunnel is up.

 

I did a trace log when it was working and when it wasnt.

 

Attachments

SRX Services Gateway

Re: SRX210 & Azure VPN

[ Edited ]
a month ago

wyze,

 

Thanks for the topology and the files. Everything shows that the SRX is not receiving the traffic back.

 

The session shows no traffic being received:

 

root@SRXFW> show security flow session destination-prefix 10.0.7.6
Session ID: 11712, Policy name: jump05-to-vpn-azure/149, Timeout: 18, Valid
In: 172.18.41.64/56193 --> 10.0.7.6/3389;tcp, If: vlan.1841, Pkts: 2, Bytes: 104
Out: 10.0.7.6/3389 --> 172.18.41.64/56193;tcp, If: st0.10, Pkts: 0, Bytes: 0
Total sessions: 1

 

Also the flow traces show the same during the time of the issue (we see only traffic from the LAN):

 

Jul 23 10:44:57 10:44:57.705270:CID-0:RT:<172.18.41.64/54324->10.0.7.6/3389;6> matched filter TEST:
Jul 23 10:45:00 10:45:00.716327:CID-0:RT:<172.18.41.64/54324->10.0.7.6/3389;6> matched filter TEST:
Jul 23 10:45:06 10:45:06.730469:CID-0:RT:<172.18.41.64/54324->10.0.7.6/3389;6> matched filter TEST:

 

When the problem is not happening, we can see replies from the remote end:

 

Jul 23 10:07:16 10:07:16.425076:CID-0:RT:<172.18.41.64/54215->10.0.7.6/3389;6> matched filter TEST:
Jul 23 10:07:16 10:07:16.435822:CID-0:RT:<10.0.7.6/3389->172.18.41.64/54215;6> matched filter TEST:
Jul 23 10:07:16 10:07:16.436530:CID-0:RT:<172.18.41.64/54215->10.0.7.6/3389;6> matched filter TEST:
Jul 23 10:07:16 10:07:16.447383:CID-0:RT:<172.18.41.64/54215->10.0.7.6/3389;6> matched filter TEST:
Jul 23 10:07:16 10:07:16.460668:CID-0:RT:<10.0.7.6/3389->172.18.41.64/54215;6> matched filter TEST:
Jul 23 10:07:16 10:07:16.505918:CID-0:RT:<172.18.41.64/54215->10.0.7.6/3389;6> matched filter TEST:

 

Regarding the counters, I didnt had the same understanding of the issue that I have now so in order to have relevant information we will require to configure another counter that could tell us if we are receiving ESP packets from Azure during the time of the issue:

 

1. Create filter:

#set firewall family inet filter FILTER term TO_AZURE from destination-address [Azure_Public_IP]
#set firewall family inet filter FILTER term TO_AZURE from source-address [SRX_Public_IP]
#set firewall family inet filter FILTER term TO_AZURE from protocol esp
#set firewall family inet filter FILTER term TO_AZURE then count ESP_PACKETS-TO_AZURE
#set firewall family inet filter FILTER term TO_AZURE then accept
#set firewall family inet filter FILTER term FROM_AZURE from destination-address [SRX_Public_IP]
#set firewall family inet filter FILTER term FROM_AZURE from source-address [Azure_Public_IP]
#set firewall family inet filter FILTER term FROM_AZURE from protocol esp
#set firewall family inet filter FILTER term FROM_AZURE then count ESP_PACKETS-FROM_AZURE
#set firewall family inet filter FILTER term FROM_AZURE then accept
#set firewall family inet filter FILTER term ALLOW_ELSE then accept

2. Apply filter:

#set interfaces [SRX_External_Interface] unit 0 family inet filter output FILTER
#set interfaces [SRX_External_Interface] unit 0 family inet filter input FILTER

3. Commit changes for only 5 minutes

#commit confirmed 5

4. Check the counter:

> show firewall

 

As a last test, you could take a packet capture on the external interface during the problem so that you can demonstrate that the SRX is sending ESP traffic but no receiving anything back:

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB11709

 

 

Pura Vida from Costa Rica - Mark as Resolved if it applies.
Kudos are appreciated too!
SRX Services Gateway

Re: SRX210 & Azure VPN

4 weeks ago

Hi epaniagua,

 

Thank you again for your reply.

 

I applied the suggested filter and attached is the log for it.

 

Have also applied the packet capture on the external interface. Its log is also included.

 

I'm noticing that its looping?

Attachments

SRX Services Gateway

Re: SRX210 & Azure VPN

[ Edited ]
3 weeks ago

wyze,

 

If the outputs were taken during the time of the problem, then we can see that Azure is not replying with ESP packets or something is dropping their ESP packets. In order to investigate further Azure could take a pcap during the time of the problem to confirm if they are receiving our ESP packets and if they are sending ESP traffic back to us. This can be done at the checkpoint side as well.

 

# commit confirmed will be rolled back in 8 minutes
root@SRXFW> show firewall filter FILTER

Filter: FILTER
Counters:
Name                                                Bytes              Packets
ESP_PACKETS-FROM_AZURE                                  0                    0
ESP_PACKETS-TO_AZURE                                 3832                   26

 

Im having a hardtime understanding the pcacp file, as a text file, because I cant see further details of each of the packets shown, however we can see that only the SRX is sending ESP packets:

 

No.	Time		Source	Destination	Protocol	Length	Info
1	0			AZURE	SRXFW		ISAKMP		130	INFORMATIONAL MID=780 Initiator Request
2	0.000317	SRXFW	AZURE		ISAKMP		130	INFORMATIONAL MID=780 Responder Response
3	0.555779	SRXFW	AZURE		ESP			182	ESP (SPI=0x1ced495e)
4	1.55021		AZURE	SRXFW		ISAKMP		130	INFORMATIONAL MID=781 Initiator Request
5	1.550338	SRXFW	AZURE		ISAKMP		130	INFORMATIONAL MID=781 Responder Response
6	3.554959	AZURE	SRXFW		ISAKMP		130	INFORMATIONAL MID=782 Initiator Request
7	3.555114	SRXFW	AZURE		ISAKMP		130	INFORMATIONAL MID=782 Responder Response
8	5.555151	AZURE	SRXFW		ISAKMP		130	INFORMATIONAL MID=783 Initiator Request
9	5.555276	SRXFW	AZURE		ISAKMP		130	INFORMATIONAL MID=783 Responder Response
10	5.555365	SRXFW	AZURE		ESP			150	ESP (SPI=0x1ced495e)
11	7.549306	AZURE	SRXFW		ISAKMP		130	INFORMATIONAL MID=784 Initiator Request
12	7.549436	SRXFW	AZURE		ISAKMP		130	INFORMATIONAL MID=784 Responder Response
13	8.554029	SRXFW	AZURE		ESP			150	ESP (SPI=0x1ced495e)

 

We can also see traffic from Azure but we see ISAKMP traffic only. It says "Informational" maybe refering to the Informational Exchange mentioned in the following doc:

 

      https://www.shrew.net/static/help-2.0.x/files/%7BFD711C32-A0FE-40E2-B038-13F5FA1E72F3%7D.htm

 

I would like to confirm if maybe the SRX and the Azure FW are using different SPI values at the time of the issue and if this Informational messages are sent from Azure in order to deleted an old SA?

 

-How often is this issue happening?

-Does the lifetimes values match on both ends?

-Do you see any logs on the messages log file refering to Bad SPI? "> show log messages"

-Is messages log file configured properly? "# show system syslog "

-Can Azure confirm if by the time of the issue their VPN is using the same SPI that the SRX is using? 

 

 

Pura Vida from Costa Rica - Mark as Resolved if it applies.
Kudos are appreciated too!