SRX Services Gateway
Reply
Contributor
rebus
Posts: 56
Registered: ‎05-28-2009
0

MPLS Ethernet Handoff to SRX -- SOME services fail

[ Edited ]

I've been blind-sided by a problem, and am trying to understand what is happening and why.

 

At present we have a Sonicwall providing general internet connectivity to a single upstream ISP.  It's old and we bought a new SRX 220 to replace the Sonicwall.

 

Plus, to connect our home office with 6 branch offices, we have a vendor-managed MPLS cloud that connects all our LANs privately via RFC 1918 space.

 

The vendor's MPLS router sits "inside" our LAN and has its own connection into the vendor's cloud, completely separate from our internet connection.  From our perspective it's just a "Black Box" with an ethernet port addressed within our LAN as 10.1.1.50--- our gateway to the other branch office LANs.  Traffic hits that port and travels across the vendor's MPLS cloud to the branches.  (LAN to LAN only, no transit)

 

Our home office LAN is 10.1.1.0 /24.  Branches are their own /24 space at 10.1.2.0, 10.1.3.0, 10.1.4.0, etc.

 

Sonicwall has routes:

0.0.0.0/0  default points to our ISP's gateway.

10.1.2.0/24  is reached via  10.1.1.50 (....

10.1.3.0/24  is reached via  10.1.1.50 (.... the port of the MPLS router inside our LAN

10.1.4.0/24  is reached via  10.1.1.50 (....

And so on.

 

OK-- so far so good, and it's worked great for years.

 

PROBLEM:  I set up our new SRX 220 with EXACTLY the same routes.  When I removed the Sonicwall and replaced it with the SRX 220, **some** services in remote offices (reached via MPLS) started failing.  But not all services.

 

For example, a user in a branch at desktop 10.1.4.25 could ping our home office RDP server (across the MPLS) at 10.1.1.8 but could not connect via RDP.   Same user could ping our application server at 10.1.1.22 but the client component installed on his machine could not connect to the server.  However, he COULD connect to Windows shares back hosted at home office (\\share\directory) to copy and paste files back and forth between home office and his branch desktop.  Also applications like http to access the corporate intranet still worked OK. 

 

To summarize:  when a home office host wants to reach a branch (in a different RFC 1918 network) the host sends packets to the default gateway (our router) which in turn bounces it back out the same LAN interface to the MPLS port on our LAN at 10.1.1.50.   When a branch host reaches our LAN, it comes inbound through 10.1.1.50 and into our LAN.

 

But some services just cannot connect LAN to LAN.

 

I thought maybe there was an MTU issue.   So I set the MTU on the LAN interface of the SRX downward in small steps, all the way down to 1350.  But those services still failed.  (NOTE:  The MTU on the old Sonicwall worked fine when left at its default setting of 1500.  We did not have to manually set it lower.)  Also note, I can ping hosts across the MPLS up to 1472 bytes with the do-not-fragment bit set.  At 1473 bytes, I get the usual error, fragmentation required but DF set.

 

Can anyone think of what might be wrong, or what I can check next?  Does this sound like an MTU issue?  If you had this problem, what would be your next step? 

 

Any good advice will SINCERELY be appreciated.

Distinguished Expert
spuluka
Posts: 2,499
Registered: ‎03-30-2009
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

What I think you have is some asymetrical routing for you home office 10.1.1.0/24 network.  I had a similar problem when I replaced a Sonicwall a few years back.  The issue is that the sonicwall does not care if the routing is asymetrical but the Juniper products do care. 

 

The asymetrical route is a direct connection from the mpls router to the server.  But a return path from the server to the firewall then the router.

 

Your remote office traffic come in the mpls router.  This is on the 10.1.1.0/24 network so it does a direct arp and makes the inbound connection.

 

The server does not have a route directly to the mpls router for the remote sites.  So this forwards the reply to the firewall which then forwards it to the mpls router.  The firewall is seeing the connection for the first time and sets up a session in the opposite direction.  But replys never arrive because they deliver direct.  So the session times out and drops.

 

You have some choices.

 

Create a new network segment for the inbound mpls router on the firewall.  Then all traffic hits the firewall in both directions.  Connect the mpls router to the firewall on a new subnet and permit the traffic.

 

Create the direct route on your local servers that points to the mpls router for the remote networks.

 

Turn off the check for asymetrical routing on the SRX.

Steve Puluka BSEET
Juniper Ambassador
Senior Network Engineer - UPMC Pittsburgh, PA
JNCIA-ER JNCIA-EX JNCIS-SEC JNCIP-SEC
JNCIS-FWV JNCIS-SSL
MCP - Managing Server 2003 MCP - Windows XP Professional
MCTS Windows 7
http://puluka.com/home
Contributor
rebus
Posts: 56
Registered: ‎05-28-2009
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

[ Edited ]

If this is the case, do you know why SOME services work?  For example, HTTP and NetBIOS traffic work fine, but RDP and some client-server apps do not. 


Both the LAN and MPLS are in "trust" zone and no NAT is being performed, and I have the rule Trust to Trust is permit all services and all addresses.

 

Distinguished Expert
spuluka
Posts: 2,499
Registered: ‎03-30-2009
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

I don't know the command off the top of my head as this is not the recommended solution and I made changes to the routing to clear the asymetrical routes.

 

The reason it only affects some traffic is this is strictly a tcp issue.  The SRX does tcp syn sequence checking as part of the session process.  What happens during the handshake is that the system is only seeing one side of the conversation so is missing a portion of the three step handshake.

 

You need to turn off tcp syn check to clear this.  But this is a security issue for other traffic so it is not recommended.

 

There should be some kb articles on the topic.

Steve Puluka BSEET
Juniper Ambassador
Senior Network Engineer - UPMC Pittsburgh, PA
JNCIA-ER JNCIA-EX JNCIS-SEC JNCIP-SEC
JNCIS-FWV JNCIS-SSL
MCP - Managing Server 2003 MCP - Windows XP Professional
MCTS Windows 7
http://puluka.com/home
Distinguished Expert
spuluka
Posts: 2,499
Registered: ‎03-30-2009
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

i had some time to pull through the kb.  Here is the description of the issue from the troubleshooting kb16110

http://kb.juniper.net/InfoCenter/index?page=content&id=KB16110

 

Is the session asymmetric?
Stateful SYN checking is on by default, and if the firewall does not see the SYN packet, it will drop the subsequent packets and the session will not be established. This should be clear from the debug. In this situation you must either ensure that the SRX is in the initial packet path so that it sees the SYN packet (recommended!) or turn off SYN checking. If the SRX cannot see both directions of the flow then some features such as IDP may not be able to work at the full capacity. Therefore it is recommended to design your network so that asymmetric flow does not occur.

 

The preferred solution is to get the flow to be symetrical as I mention above.  Either add routes to the servers for the mpls router networks or bring the mpls router into a firewall interface to force symetrical flows.

 

The alternative solution would be to turn off syn checking globally and then apply it by policy to the normal flows.  This is outlined in kb21266.

 

http://kb.juniper.net/InfoCenter/index?page=content&id=KB21266

Steve Puluka BSEET
Juniper Ambassador
Senior Network Engineer - UPMC Pittsburgh, PA
JNCIA-ER JNCIA-EX JNCIS-SEC JNCIP-SEC
JNCIS-FWV JNCIS-SSL
MCP - Managing Server 2003 MCP - Windows XP Professional
MCTS Windows 7
http://puluka.com/home
Contributor
rebus
Posts: 56
Registered: ‎05-28-2009
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

[ Edited ]

I very much appreciate the extra work you've done.  In case anyone is unclear of our layout, I've attached a graphic (link is at the bottom of this post

 

Here is what I've applied since my original post:

 

root@router# show security flow   
tcp-session {
    no-syn-check;
    no-syn-check-in-tunnel;
}

 

root@router# show system internet-options
path-mtu-discovery;
ipip-path-mtu-discovery;

 

SOME of the people having problems yesterday can now connect to services.  However there are SOME people having problems with certain apps.  It is completely random.  One user can connect OK, while the person next  to them (same PC config, same LAN) cannot.

We ran into another (probably related) problem.  In another location we discovered that all HP Thin Client devices cannot make an RDP connection.   But laptops and desktops in the same office are OK.

 

Question-- if I do "no-syn-check" globally, will that be applied globally or must I still apply it per-policy before it becomes effective?  I'm not quite clear on that after reading the KB.

 

Distinguished Expert
spuluka
Posts: 2,499
Registered: ‎03-30-2009
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

When you do the global change it is applied globally.  Then the recommendation is to turn it ON for the other policies that are not asymetrical.

 

The kb also lists no-sequence-check as another global parameter to remove.

 

flow {
        tcp-session {                   
            no-syn-check;               
            no-sequence-check;          
        }                        

It really isn't that big a deal to add the routes to your servers.  Just pasting these into the dos prompt will add persistent routes from the server directly to your mpls router for those remote sites.  This then elimates the issue because the routing returns by the same path it enters.

 

route add 10.1.2.0 mask 255.255.255.0 10.1.1.50 -p
route add 10.1.3.0 mask 255.255.255.0 10.1.1.50 -p
route add 10.1.4.0 mask 255.255.255.0 10.1.1.50 -p

 

Steve Puluka BSEET
Juniper Ambassador
Senior Network Engineer - UPMC Pittsburgh, PA
JNCIA-ER JNCIA-EX JNCIS-SEC JNCIP-SEC
JNCIS-FWV JNCIS-SSL
MCP - Managing Server 2003 MCP - Windows XP Professional
MCTS Windows 7
http://puluka.com/home
Distinguished Expert
dfex
Posts: 705
Registered: ‎04-17-2008
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

The best fix would be to create a new subnet on the SRX (with a dedicated port and security zone), and have your provider re-address the "black box" to be in this subnet.  That way there would only be a single path into and out of your home subnet, and likewise with the WAN.  

 

It would mean the provider would need to change the routes in your network slightly (your home LAN would be one hop away from their NTU) but it would remove the need for removing strict-syn-checking and other security features.

 

As Steve has said though, adding static routes to your servers will also fix the issue, but is kinda hacky.

 

Ben

Ben Dale
JNCIP-ENT, JNCIS-SP, JNCIE-SEC #63
Juniper Ambassador
Follow me @labelswitcher
Contributor
rebus
Posts: 56
Registered: ‎05-28-2009
0

Re: MPLS Ethernet Handoff to SRX -- SOME services fail

I was really trying to avoid static routes on individual servers (we have a couple dozen of them) and keep everything centralized.  When there are too many Band Aid fixes, eventually the environment will change, things will break, and nobody will remember why.

 

But the path of least resistence was to script the static routes and apply them to all servers accessed from the branches.  Fortunately, the reason we bought more than a dozen SRX boxes was to create our own VPNs and get rid of MPLS.... so MPLS will be gone within a couple months.

 

Steve-- THANK YOU for all the help.  We ended up using the lowest tech solution, but you really went above and beyond the call and it's appreciated!!

Copyright© 1999-2013 Juniper Networks, Inc. All rights reserved.