Screen OS

Hi,

I have an interface with a /29, I have a webserver sat on my LAN which can see the internet just fine on the IP address of the interface.

I also have policy based NAT to allow ping, http, and https to the webserver with a different IP address on the same /29 subnet as the interface - this also works fine.

What doesn't seem to be working with this setup, is that nothing within the LAN subnet can access the external IP address of the webserver.

I can turn on logging on the the Trust to Untrust for anything coming from a specific machine on the LAN, and I try to go to the website via it's external IP address and I can see that request going through the firewall correctly.  Unfortunately it doesn't seem to pass back through the firewall and into the server to answer the request.

I've tried checking the firewall, and can't seem to figure out what zones might be involved in the request coming back in - from looking at the policy logs of the outgoing policy, it would appear to be an untrust intrazone policy which is already allowing traffic through successfully.  I also can't see it being a routing issue, as everything is within the same subnet and should therefore come in and out of the same interface without ever leaving the Juniper.

Hope I've explained this ok, if anyone could help it would be appreciated.

Thanks,

Here's a snippet of the relevant parts of my config file so you can see what is going on (note IP addresses aren't real):

set interface "ethernet0/8" zone "Untrust"
set interface "ethernet0/9" zone "Trust"
set interface ethernet0/8 ip 22.55.66.163/29
set interface ethernet0/8 route
set interface ethernet0/9 ip 10.61.20.254/24
set interface ethernet0/9 nat
set policy id 25 from "Untrust" to "Untrust"  "Any" "22.55.66.162/32" "HTTP" nat dst ip 10.61.20.20 permit
set policy id 25
set service "HTTPS"
set service "PING"
exit
set route 0.0.0.0/0 interface ethernet0/8 gateway 22.55.66.161

set arp nat-dst

Can you run a debug flow basic?

like this

set ff dst-ip <ipaddress server>

debug flow basic

access the server

undebug all

get db stream

I was thinking that may be the traffic is nated to another zone so you need a policy from trust to dmz or something. A debug should make that clear.

Hi,

Ok after scouring the log here's the packet I believe is the one I'm interested in:

****** 621230.0: <Trust/ethernet0/9> packet received [48]******
  ipid = 26518(6796), @1d5c5914
  packet passed sanity check.
  flow_decap_vector IPv4 process
  ethernet0/9:10.61.20.20/4594->22.55.66.162/80,6<Root>
  no session found
  flow_first_sanity_check: in <ethernet0/9>, out <N/A>
  chose interface ethernet0/9 as incoming nat if.
  flow_first_routing: in <ethernet0/9>, out <N/A>
  search route to (ethernet0/9, 10.61.20.20->22.55.66.162) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 28.route22.55.66.162->22.55.66.162, to ethernet0/8
  routed (x_dst_ip 22.55.66.162) from ethernet0/9 (ethernet0/9 in 0) to ethernet0/8
  policy search from zone 2-> zone 1
 policy_flow_search  policy search nat_crt from zone 2-> zone 1
  RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 22.55.66.162, port 80, proto 6)
  No SW RPC rule match, search HW rule
swrs_search_ip: policy matched id/idx/action = 1/0/0x1
  Permitted by policy 1
  dip id = 2, 10.61.20.20/4594->22.55.66.163/6517
  choose interface ethernet0/8 as outgoing phy if
  check nsrp pak fwd: in_tun=0xffffffff, VSD 0 for out ifp ethernet0/8
  vsd 0 is active
  no loop on ifp ethernet0/8.
  session application type 6, name HTTP, nas_id 0, timeout 300sec
  service lookup identified service 0.
  flow_first_final_check: in <ethernet0/9>, out <ethernet0/8>
  existing vector list 123-3df4aa4.
  Session (id:47555) created for first pak 123
  flow_first_install_session======>
  route to 22.55.66.162
  wait for arp rsp for 22.55.66.162
  ifp2 ethernet0/8, out_ifp ethernet0/8, flag 10000800, tunnel ffffffff, rc 0
  outgoing wing prepared, not ready
  handle cleartext reverse route
  search route to (ethernet0/8, 22.55.66.162->10.61.20.20) in vr trust-vr for vsd-0/flag-3000/ifp-ethernet0/9
  [ Dest] 5.route 10.61.20.20->10.61.20.20, to ethernet0/9
  route to 10.61.20.20
  arp entry found for 10.61.20.20
  ifp2 ethernet0/9, out_ifp ethernet0/9, flag 00800801, tunnel ffffffff, rc 1

Hi,

Instead of nat-dst on policy, can you test with a MIP?

MIP has loopback session processing functionality, not sure if NAT-dst also has it.

Thanks.

Hardeep

Hi

Oh yeah MIP works but we can't keep that, as we're using the same IP addressing for multiple internal servers and we can't use VIP either as we need it to respond to pings for monitoring purposes (although also in testing that also works internally).

Thanks

Hi,

Have you enabled the src-NAT in the policy? If the looping connection is not source-natted the response cannot be sent along the same way the request came in (asymmetric routing -> no handshaking).

If you run a basic flow debug you will see that two policy checks are performed for the looping connections. The second policy is an Untrust-to-Untrust one which is not logged! So, the only way to analyze the looping connections is a debug. The logging option logs only "the first half" of the event.

Hi,

Ok I've added the following rule:

set policy id 25 from "Untrust" to "Untrust"  "Any" "22.55.66.162" "HTTP" nat src dst ip 10.61.20.20 permit

and added a policy to go from untrust to trust as I don't believe it was getting that far in the first debug flow:

set policy id 26 from "Trust" to "Untrust"  "Any" "22.55.66.162" "HTTP" nat dst ip 10.61.20.20 permit

This results in the following output:

****** 660109.0: <Trust/ethernet0/9> packet received [48]******
  ipid = 12389(3065), @1d5f8914
  packet passed sanity check.
  flow_decap_vector IPv4 process
  ethernet0/9:10.61.20.20/1625->22.55.66.162/80,6<Root>
  no session found
  flow_first_sanity_check: in <ethernet0/9>, out <N/A>
  chose interface ethernet0/9 as incoming nat if.
  flow_first_routing: in <ethernet0/9>, out <N/A>
  search route to (ethernet0/9, 10.61.20.20->22.55.66.162) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 28.route 22.55.66.162->22.55.66.162, to ethernet0/8
  routed (x_dst_ip 22.55.66.162) from ethernet0/9 (ethernet0/9 in 0) to ethernet0/8 
  policy search from zone 2-> zone 1
 policy_flow_search  policy search nat_crt from zone 2-> zone 1
  RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 22.55.66.162, port 80, proto 6)
  No SW RPC rule match, search HW rule
swrs_search_ip: policy matched id/idx/action = 26/0/0x1
  Permitted by policy 26
  DST xlate: 22.55.66.162(80) to 10.61.20.20(80)
  search route to (ethernet0/9, 10.61.20.20->10.61.20.20) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 5.route 10.61.20.20->10.61.20.20, to ethernet0/9
  routed (10.61.20.20) from ethernet0/9 (ethernet0/9 in 0) to ethernet0/8 
  packet dropped, routed to different zone

 So now the packet is dropped because of routing to a different zone, which I don't understand because I've also just disabled "block intrazone traffic" on the untrust zone.

Hi,

If I understand it right, you have the server and other LAN in trust zone.
If yes, the policies should be as follows:
Trust to untrust, no natting done, just ensure that source nat happens.
Untrust to trust, nat-dst configured.
I dont think you need untrust to untrust policy for this.

Thanks.
Hardeep

Hi,

Unfortunately that doesn't work either.  Basically I have a server sitting on a LAN in the Trust zone along with some other servers too.

Our external IP address range is sitting in the untrust zone.

I need the untrust to untrust policy in order to allow access from the internet to the webserver via nat-dst, and this does work fine.

What I'm also trying to accomplish is to allow the servers inside the LAN on the Trust zone to access the other servers via the external IP addresses.  This is what isn't working.

I have just tried adding the trust to untrust and untrust to trust policies as suggested and unfortunately that did not work either.

Thanks.

Oh just to say about the source NAT that is happening by default because the trust interface is in NAT mode so everything gets NAT'd to the untrust interface IP address by default.

Hi,

Here is a working example:

192.168.1.1 and 1.1.1.2 are the private and public IPs of the server respectively.

Create a dip or a proxy-arp-entry for the dst policy-based NAT to work:

set interface ethernet0/0 dip 4 1.1.1.2 1.1.1.2   or  set interface ethernet0/0 proxy-arp-entry 1.1.1.2 1.1.1.2

Create the object and the policies:

set address "Untrust" "1.1.1.2" 1.1.1.2 255.255.255.255

set policy  from "Trust" to "Untrust"  "Any" "Any" "ANY" nat src permit log  (src-NAT to eth0/0 IP)

set policy  from "Untrust" to "Untrust"  "Any" "1.1.1.2" "ANY" nat dst ip 192.168.1.1 permit log

That's it.

I have incorrectly written that the second policy does not log. It was a slightly different situation in my previous tests. Both policies are logging.

Thanks, that got it working!! 🙂

I don't get why though... why do you need to go out via the same IP address as the incoming policy?  Surely the device is smart enough to know this is all on the same interface ie (the interface IP was 22.55.66.163/28 and the nat-dst policy was 22.55.66.162/32), let's just pass it through, but instead I was getting Arp errors - don't get it....

Would this have worked if the Juniper was on 6.3 with proxy arp instead of (what seems like a hack) set arp nat-dst?

Actually I just been playing around with it - it's purely the DIP that got it working, I've removed the src nat from trust to untrust and it still works!

Thinking about the entries you had, that would suggest proxy-arp would be the better solution to this...

Hi,

If Trust interface is in NAT mode the packets are automatically src-natted to the Untrust interface IP. I never use NAT mode and configure the src-nat in the policy. The packets in my example are also src-natted to the Untrust interface IP which is 1.1.1.1.

If you use a DIP or a proxy-arp entry makes no difference. The DIP makes the IP arp-responsive. I prefer to use DIP especially if the same IP should be used both for the src- and dst-NAT in the access policies.

With other words DIP is a proxy-arp entry plus ability to be used in the policy for a src-NAT.

Set arp nat-dst is not a good option:

"Because the <proxy-arp-entry> option allows better control of the device, the command set arp nat-dst is not recommended. For more information on <proxy-arp-entry>, see “set (Layer 3 Interfaces)” on page 364." (ScreenOS Reference Guide:IPv4 Command Descriptions). Besides, it can result in a perfomance degrade.

Hi,

Sorry to bring this up again, we were forced to do some network changes by our provider which was a good thing, it gave us a far better connection (redundancy wise) to the internet, IP addresses have now changed.  Whilst doing that change we also upgrade to the latest firmware on the Junipers (6.2r05 to 6.3r14) - mostly we wanted proxy arp.

This all went well or so I thought.

Unfortunately I can't loopback again from anything in the trusted zone to the untrust IP addresses on the interface.

We got the config working, and then I ran:

unset arp nat-dst

As this command is no longer recommended.  The policies are the same, we have a trust to unstrust with src nat to the external IP for one of our webservers, and I've also got an untrust to untrust for nat-dst (also tried untrust to trust and having a route in place).

I just don't understand, DIP or proxy-arp don't seem to work at all - I get the same debug output regardless of what I use...

Testing with both proxy-arp enabled for that external IP, or with a DIP (which it's currently setup as) doesn't work:

get int eth0/8 dip

id =   4:  ip range 5.5.184.184 ~ 5.5.184.184; (port-xlate)

Looking through the debug it looks like it can't resolve the ARP again.  However I have no idea why now.

Here's a debug flow basic:

****** 153316.0: <Trust/ethernet0/9> packet received [48]******
  ipid = 31681(7bc1), @1d6d0914
  packet passed sanity check.
  flow_decap_vector IPv4 process
  ethernet0/9:10.61.20.20/4340->5.5.184.184/80,6<Root>
  no session found
  flow_first_sanity_check: in <ethernet0/9>, out <N/A>
  chose interface ethernet0/9 as incoming nat if.
  flow_first_routing: in <ethernet0/9>, out <N/A>
  search route to (ethernet0/9, 10.61.20.20->5.5.184.184) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 1.route 5.5.184.184->5.5.184.184, to ethernet0/8
  routed (x_dst_ip 5.5.184.184) from ethernet0/9 (ethernet0/9 in 0) to ethernet0/8
  policy search from zone 2-> zone 1
 policy_flow_search  policy search nat_crt from zone 2-> zone 1
  RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 5.5.184.184, port 80, proto 6)
  No SW RPC rule match, search HW rule
swrs_search_ip: policy matched id/idx/action = 25/0/0x1
  Permitted by policy 25
  src-nat dip id = 4, 10.61.20.20/4340->5.5.184.184/1999
  choose interface ethernet0/8 as outgoing phy if
  check nsrp pak fwd: in_tun=0xffffffff, VSD 0 for out ifp ethernet0/8
  vsd 0 is active
  no loop on ifp ethernet0/8.
  session application type 6, name HTTP, nas_id 0, timeout 300sec
  service lookup identified service 0.
  flow_first_final_check: in <ethernet0/9>, out <ethernet0/8>
  existing vector list 123-40a47d4.
  Session (id:45815) created for first pak 123
  flow_first_install_session======>
  route to 5.5.184.184
  wait for arp rsp for 5.5.184.184
  ifp2 ethernet0/8, out_ifp ethernet0/8, flag 10000800, tunnel ffffffff, rc 0
  outgoing wing prepared, not ready
  handle cleartext reverse route
  search route to (ethernet0/8, 5.5.184.184->10.61.20.20) in vr trust-vr for vsd-0/flag-3000/ifp-ethernet0/9
  [ Dest] 3.route 10.61.20.20->10.61.20.20, to ethernet0/9
  route to 10.61.20.20
  arp entry found for 10.61.20.20
  ifp2 ethernet0/9, out_ifp ethernet0/9, flag 00800801, tunnel ffffffff, rc 1
****** 153319.0: <Trust/ethernet0/9> packet received [48]******
  ipid = 31741(7bfd), @1d6fe914
  packet passed sanity check.
  flow_decap_vector IPv4 process
  ethernet0/9:10.61.20.20/4340->5.5.184.184/80,6<Root>
  existing session found. sess token 3
  flow got session.
  flow session id 45815
  flow_main_body_vector in ifp ethernet0/9 out ifp N/A
  flow vector index 0x123, vector addr 0x40a47d4, orig vector 0x40a47d4
  vsd 0 is active
  tcp seq check.
  Got syn, 10.61.20.20(4340)->5.5.184.184(80), nspflag 0x801801, 0x10000800
  route to 5.5.184.184
  wait for arp rsp for 5.5.184.184
  ifp2 ethernet0/8, out_ifp ethernet0/8, flag 10000800, tunnel ffffffff, rc 0

 I just don't get why proxy-arp or DIP isn't responding to the internal ARP request.  It doesn't make sense at all.

Any further help would be very much appreciated.

Thanks in advance

Hi,

You should use another IP for the src-NAT on eth0/8. You cannot establish a connection if both src IP and dst IP are the same IP 5.5.184.184:

ethernet0/9:10.61.20.20/4340->5.5.184.184/80,6

src-nat dip id = 4, 10.61.20.20/4340->5.5.184.184/1999

The packet 5.5.184.184/1999 -> 5.5.184.184/80 does not make any sense.

Ok,

I told the policy to use src-nat (trust to untrust) but on the egress interface IP instead.

This still doesn't work, here's the new debug log:

****** 158182.0: <Trust/ethernet0/9> packet received [48]******
  ipid = 14956(3a6c), @1d660114
  packet passed sanity check.
  flow_decap_vector IPv4 process
  ethernet0/9:10.61.20.20/4526->5.5.184.184/80,6<Root>
  no session found
  flow_first_sanity_check: in <ethernet0/9>, out <N/A>
  chose interface ethernet0/9 as incoming nat if.
  flow_first_routing: in <ethernet0/9>, out <N/A>
  search route to (ethernet0/9, 10.61.20.20->5.5.184.184) in vr trust-vr for vsd-0/flag-0/ifp-null
  [ Dest] 1.route 5.5.184.184->5.5.184.184, to ethernet0/8
  routed (x_dst_ip 5.5.184.184) from ethernet0/9 (ethernet0/9 in 0) to ethernet0/8 
  policy search from zone 2-> zone 1
 policy_flow_search  policy search nat_crt from zone 2-> zone 1
  RPC Mapping Table search returned 0 matched service(s) for (vsys Root, ip 5.5.184.184, port 80, proto 6)
  No SW RPC rule match, search HW rule
swrs_search_ip: policy matched id/idx/action = 25/0/0x1
  Permitted by policy 25
  src-nat dip id = 2, 10.61.20.20/4526->5.5.184.180/16340
  choose interface ethernet0/8 as outgoing phy if
  check nsrp pak fwd: in_tun=0xffffffff, VSD 0 for out ifp ethernet0/8
  vsd 0 is active
  no loop on ifp ethernet0/8.
  session application type 6, name HTTP, nas_id 0, timeout 300sec
  service lookup identified service 0.
  flow_first_final_check: in <ethernet0/9>, out <ethernet0/8>
  existing vector list 123-40a47d4.
  Session (id:47379) created for first pak 123
  flow_first_install_session======>
  route to 5.5.184.184
  wait for arp rsp for 5.5.184.184
  ifp2 ethernet0/8, out_ifp ethernet0/8, flag 10000800, tunnel ffffffff, rc 0
  outgoing wing prepared, not ready
  handle cleartext reverse route
  search route to (ethernet0/8, 5.5.184.184->10.61.20.20) in vr trust-vr for vsd-0/flag-3000/ifp-ethernet0/9
  [ Dest] 3.route 10.61.20.20->10.61.20.20, to ethernet0/9
  route to 10.61.20.20
  arp entry found for 10.61.20.20
  ifp2 ethernet0/9, out_ifp ethernet0/9, flag 00800801, tunnel ffffffff, rc 1
****** 158185.0: <Trust/ethernet0/9> packet received [48]******
  ipid = 15042(3ac2), @1d6f1114
  packet passed sanity check.
  flow_decap_vector IPv4 process
  ethernet0/9:10.61.20.20/4526->5.5.184.184/80,6<Root>
  existing session found. sess token 3
  flow got session.
  flow session id 47379
  flow_main_body_vector in ifp ethernet0/9 out ifp N/A
  flow vector index 0x123, vector addr 0x40a47d4, orig vector 0x40a47d4
  vsd 0 is active
  tcp seq check.
  Got syn, 10.61.20.20(4526)->5.5.184.184(80), nspflag 0x801801, 0x10000800
  route to 5.5.184.184
  wait for arp rsp for 5.5.184.184
  ifp2 ethernet0/8, out_ifp ethernet0/8, flag 10000800, tunnel ffffffff, rc 0
****** 158191.0: <Trust/ethernet0/9> packet received [48]******
  ipid = 15123(3b13), @1d631914
  packet passed sanity check.
  flow_decap_vector IPv4 process
  ethernet0/9:10.61.20.20/4526->5.5.184.184/80,6<Root>
  existing session found. sess token 3
  flow got session.
  flow session id 47379
  flow_main_body_vector in ifp ethernet0/9 out ifp N/A
  flow vector index 0x123, vector addr 0x40a47d4, orig vector 0x40a47d4
  vsd 0 is active
  tcp seq check.
  Got syn, 10.61.20.20(4526)->5.5.184.184(80), nspflag 0x801801, 0x10000800
  route to 5.5.184.184
  wait for arp rsp for 5.5.184.184
  ifp2 ethernet0/8, out_ifp ethernet0/8, flag 10000800, tunnel ffffffff, rc 0

 It appears to still be doing exactly the same thing...

Hi,

Can 5.5.184.184 be arp-resolved from another host/device connected to Untrust network (eg from the CP router)?

... if you are going to contact your ISP for this ask them to clear arp first.

Hi,

The ISP has confirmed they are getting arp responses for that IP address on our interface, so from that side things are working - it's just internally.

I might give the firewall a reboot tonight, I'm dubious that my unset arp nat-dst didn't "unset" fully or something, I'll see how that goes, and reply in the morning 🙂

Thanks for your help so far, appreciate it...

Ok that was a long shot, and it didn't work either, rebooted and it still doesn't work

Hi,

Please run this debug again and capture additionally the packets with the source IP 5.5.184.180 and destination IP 5.5.184.184.

Hi,

Ok here is the filter:

get ff
Flow filter based on:
id:0 src ip 5.5.184.180 dst ip 5.5.184.184 dst port 80

Unfortunately that output of the debug flow is empty with that filter applied...

Hi,

If 5.5.184.184 is not used somewhere else I can only assume that this ScreenOS release has a bug.

What is the network mask on Untrust interface?

Hi,

I was starting to wonder if it's got a bug as well... Here's the netmask: 5.5.184.180/28 although it wouldn't let me set the proxy-arp or dip on this interface if the netmask was wrong...

Is there anyway to highlight this thread to Juniper?

Thanks

Hi,

I would downgrade to r13. My tests with this release were OK.

The firewall should not try to resolve an IP if it is mapped to its own MAC. It should send the packet to itself. But your debug does not capture any packets after the src-NAT. This is strange. Do you see any entry for 5.5.184.184 if you type get arp? There should not be such an entry. But this entry should be seen on other devices like ISP router and be mapped to the FW interface MAC.

Hi,

I'll give downgrading the firmware some thought...

For the get arp, I am getting the following entry:

5.5.184.184  000000000000       trust-vr/eth0/8    PND     0      2       2     0

Which as you say is odd, and it's also resolving it to nothing looking at that?

Thanks

Hi,

Just an update.

I didn't want to keep messing around with our production firewalls so we bought an SSG-20 to use in a lab environment to test this.

I've set it up to simulate our main system, and the problem still persists.  I've also downgraded our firmware as suggested to the r13 release and it's still doing it.  Source NAT makes no difference, and neither does whether I use proxy-arp or DIP.  It's simply not working.

I am now hoping as this is a new unit that I can raise a case with Juniper under warranty about it though.

Hi,

Create the host routes for the DIPs used for the looping connections:

set route DIP/32 interface ethx/y gateway ISP-router-IP preference 0

Here ethx/y is the interface facing the ISP. Preference 0 is required as the static routes have Preference 20 per default.

The packets will be sent to the router and returned back to the firewall.