SRX

Hi,

 

I'd like clear an SRX cluster issue about managemnet interface.

Can I use the same IP subnet as I use in one of the reth interfaces?

If I cannot that means that I need to use a router other than the SRX cultster?

 

Please help me with my example

 

fpx addresses are

node0 192.168.10.2

node1 192.168.10..3

 

I have a mgmt zone with interface reth1.0

reth1.0 ip is 192.168.10.1/24

 

Thanks,

Balázs

 

 

This has actually been covered a lot on these forums... but just to summarize, the fxp0 interface(s) are designed to be completely out-of-band from your production network -- meaning they need to connect to a separate router, and your management network can't intermingle with your actual production network.  In my opinion, that's just silly.  I would venture to say that a majority of people have their management workstations on production networks.

 

Many have posed their opinions on this design, myself included.  I think it was a completely shortsighted way to design a product's management interfaces for products that were positioned to be installed in remote offices that would not have the capability of an out-of-band network.  Many remote offices are connected with cable modems, DSL, metro ethernet, etc. and as such don't have the luxury of being able to carry an out-of-band network into those remote offices.

 

As the saying goes, "it is what it is," even if "it" is not a great design.  There are threads here about managing the SRX branch clusters using the virtual chassis mode rather than via the fxp0 interfaces.  It has some pros/cons, but it's worth looking into.  There are also some threads about working around this idiotic [... er, sorry did I say that out loud?] design by separating the fxp interfaces into isolated routing instances from your transit interfaces.  My opinion there is that if a product requires that kind of workaround to be useful, it was designed wrong from the start.

Jez.. I couldnt agree more. I've been building firewalls for some time and this design has me scratching my head. Why the FXP interfaces dont have a default gateway is beyond me. How much coding would it take to fix this ?

 

I'm at the point now where I can ping each fxp but cant ssh or browse to them.

I would just completely ignore the fact that there is a dedicated management interface fxp0. Just configure local interfaces or a reth and use that for in-band management.

Right.. We're just going to use out of band appliances for console access.

Using routing-instances with your out-of-band is not just a workaround, in some environments it's a necessity and a welcome option. For example if your FW is protecting any resources that need to access services also used by the cluster itself (e.g. common SNMP, Syslog servers etc.) and you still want to maintain function separation you just put your transit-traffic zones into a separate routing-instance and you're good to go. Now whether you believe you need a dedicated mgmt int in in the first place is another matter. You can lockdown and secure mgmt access through revenue ports if you wish but it's just a lot easier to manage from a dedicated interface sitting in your functional management zone to begin with. It's also often mandated by various standards for compliancy (as is layer-3+ separation of traffic between depts/roles within the company etc.). Generally the bigger the network the more likely you are to need that separation.

Setting up a new routing-instance is quite easy in most deployments (until you get into some funky routing operations which mgmt usually won't need anyway) and you have the ability to configure mgmt access pretty much anyway you want regardless. Use Fxp0...don't...it's up to you.

---

@Ahriakin wrote:

Using routing-instances with your out-of-band is not just a workaround, in some environments it's a necessity and a welcome option. For example if your FW is protecting any resources that need to access services also used by the cluster itself (e.g. common SNMP, Syslog servers etc.) and you still want to maintain function separation you just put your transit-traffic zones into a separate routing-instance and you're good to go. Now whether you believe you need a dedicated mgmt int in in the first place is another matter. You can lockdown and secure mgmt access through revenue ports if you wish but it's just a lot easier to manage from a dedicated interface sitting in your functional management zone to begin with. It's also often mandated by various standards for compliancy (as is layer-3+ separation of traffic between depts/roles within the company etc.). Generally the bigger the network the more likely you are to need that separation.

Setting up a new routing-instance is quite easy in most deployments (until you get into some funky routing operations which mgmt usually won't need anyway) and you have the ability to configure mgmt access pretty much anyway you want regardless. Use Fxp0...don't...it's up to you.

---

Agreed.

 

But. You forgot to mention something that seems rather important. If you make use of the dedicated management ports (aka fxp0) you would expect that the machine really uses that for everything that's related to the firewall's management. However, if you decide to use stream logging vs. normal logging (which is still recommended because normal logging bogs down your CPU) then the stream logs will go out one of the revenue interfaces (stream logs go out directly through the data plane, hence no way to push it to the control plane's fxp0). 

 

So. If you end up putting your revenue ports in a separate routing instance and want exclusive management through fxp0, your logs will still go out the revenue interfaces in the other routing instance. Which kind of defeats the purpose. Who wants their logs sent over production interfaces (unencrypted, I might add)? Not to mention the routing issues you might introduce with such a design (fpr which you need yet another workaround). 

 

Things like this make me feel that Juniper just doesn't think things through. Or it makes you wonder if the SRX architecture in all it's glory genius is really the right architecture.

I've tried to do this very thing for these very reasons (i.e. put fxp0 in a seprate routing instance so I can manage the firewalls AND the devices behind them from the sam location) and it doesn't work:

 

dgeist@hostname# show | compare
[edit]
+  routing-instances {
+      mgt {
+          instance-type virtual-router;
+          interface fxp0.0;
+          routing-options {
+              static {
+                  route 0.0.0.0/0 next-hop 10.174.77.129;
+              }
+          }
+      }
+  }

{primary:node0}[edit]
dgeist@hostname# commit check
[edit routing-instances mgt]
  'interface fxp0.0'
    RT Instance: Interface fxp0.0 not supported under routing-instances.
error: configuration check-out failed

I also tried ge-0/0/0.0 (which logically is the same as fxp0.0 in cluster mode) and it took it but did nothing since (I would guess) the route table has no IP associated with the interface named that. Is there a super-secret-squirrel way to implement this? BTW, I'm on  10.4R7.5 and a SRX240h cluster. 

 

The solution that some have suggested (managing the devices via the revenue ports) is fine, unless you actually want access to the individual chassis at the same time and don't want to re-write your entire monitoring infrastructure to deal with the way JunOS cluster works.....<deep breath for frustration>. Am I missing something or are these "features" really as short-sighted as they seem?

 

Dan

Dan, even if you put fxp0 in a separate routing instance you will not be able to manage devices BEHIND the firewall (if I understand your postig right, that's what you want to do). fxp0 will never route any packets through the firewall. You can only manage the firewall with it. 

The intent is exactly that, to only be able to manage the firewall chassis via fxp0, but to be able to do so from some of same locations as the devices BEHIND the firewalls (i.e. be able to ssh from a bastion host to the firewall fxp0 and also be able to ssh to a device logically behind the firewalls via policy on revenue ports).

 

This is trivial on ScreenOS devices. You simply put the revenue ports in untrust-vr and then have a revenue routing table and a trust-vr routing table for the mgt connections. Easy peasy.

 

I just have a hard time believing there's no way to do this on JunOS clusters... and am looking to everyone here to please proove me wrong 🙂

 

Dan

It's a "feature" not a bug.

 

It's a pretty massive oversight. Someone who owns 100 of these needs to complain and force Juniper to fix it.

 

 

The reason that your config did not work is that you tried putting fxp0 in a separate routig instance. That won't work. Instead, you have to put all other interfaces into a different routing instance and leave fxp0 in the default instance.

 

The key to understanding how fxp0 operates is to know that it is connected directly to the control plane, not the data plane. The result is that no packets will ever hit fxp0 unless those packet come in through that very interface. So if you want to talk to fxp0 and have to go through another interface first to reach it, you will fail. The packets arriving at "the other" interface come in through the data plane and there is no way for the data plane to forward the packets to the control plane (where fxp0 is connected). 

 

You can have routing for fxp0, but you can not have a default route for fxp0. So if your management station is that bastion host in a DMZ you have to route the IP of your management station to the bastion host's router. The problem here is that if your SRX has another route to that bastion host, your fxp0 route will not work (you can't have the same route going out through fxp0 and a revenue interface). That's why some people use a separate instance.

 

That's how I understand it. Someone please correct me if I am wrong.

 

In my experience it's just too much of a hassle and too many workarounds. So I tend to just forget about fxp0 and use a normal (data plane) interface for management. You don't have to re-write your whole monitoring infrastructure for this either. You can use virtual chassis mode for your cluster. If using SNMP, the virtual chassis will report for all the nodes in the cluster (so your monitoring will actually see both). And if you really need to log into the inactive node, you can always login to the active node and issue the "request routing-engine login node 1" command. This will connect you to the other node with full CLI access.

 

 

I've read that putting interfaces in virtual routers interferes with their ability to do IPSec termination. Would the following accomplish what we're talkign about:

 

# management
routing-options { 
    static {
        route 0.0.0.0/0 {
            next-hop 5.6.7.8;
       }

    }

}



# everything else

routing-instances {     

    revenue {
       instance-type virtual-router;
        interface [ st0.0 reth0.1 reth1.1 reth1.2 reth1.3 ....] ;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 1.2.3.4;

                route 10.0.0.0/8 next-hop st0.0;
            }
        }
    }
}


routing-options { 
    static {
        route 0.0.0.0/0 {
            next-hop 5.6.7.8;
       }

    }

}

 

Thanks.

Dan

It was true that some features could not be used when using routing instances, among them IPSec. But that limitation has been removed in one of the later Junos releases. Not sure which release exactly, should be easy to find out though.

 

Your config looks ok to me but let others have a look to. One thing to remember is that the secondary node does not have an active routing engine, so your default route will not work there. You will have to use the "backup-router" statement to deploy routes for fxp0 on secondary. Keep in mind that setting a default route is not supported for backup-router so you must explicitly route your mangement networks.

 

The backup router on group nodes is known, thanks.

 

I tried the config below, and the ipsec interface would not establish. Based on KB articles, the interface on which the IKE session is terminated has to be in inet.0. Unfortunately, that's also the interface where the default route for revenue traffic is, so it's a bit of a non-starter 😞

 

Dan

---

dan.geist@cox.com wrote:

The backup router on group nodes is known, thanks.

 

I tried the config below, and the ipsec interface would not establish. Based on KB articles, the interface on which the IKE session is terminated has to be in inet.0. Unfortunately, that's also the interface where the default route for revenue traffic is, so it's a bit of a non-starter 😞

 

Dan

---

Yep. As mentioned, this is fixed in a later release (I am afraid it's still in 10.4). 

 

Again, managing your firewall through a revenue interface would make things so much easier for you! 🙂

 

I am with cryptochrome on this one.  I usually try to tell me customers to pretend like fxp0 does not exist.  This usually gets greeted with more questions and complaints about loosing two ports but they get over it.

 

I have seen one setup where they had a TRUE out of band MGMT network and this "seemed" great.  We could manage the units truly out of band from the MGMT pc's.  All was great until we got into the business of sysloging and SNMP. 

 

On High End SRX you should stream your logs out the data plane, and a lot of companies must have their SNMP accessible from production networks for monitoring reasons.  So now device management is done one way and logging and SNMP is done another.  Overall its a mess in all cases I have seen.

 

Just ignore fxp and use an In-Band MGMT vlan or buy a console server.

 

 

My 2 cents

I understand what you guys are saying, but managing the platform via a revenue port is not desireable for a cluster configuration (and philisophically in general) as that's the port that you're most likey to by impacted if you have a critical event or attack that's impacting the platform's reachability. The syslog thing is yet another poor design choice. Your recommended resolutions are simply workarounds "enabling" the continuation of poor design desicions by the SRX/JunOS engineering team. 

 

What version of JunOS-sec includes the ability to terminate an IPSec connection in a virutal router?

 

Dan

I believe the recommended code 10.4R7 supports this now.  If not your looking at 11.2

---

dan.geist@cox.com wrote:

Your recommended resolutions are simply workarounds "enabling" the continuation of poor design desicions by the SRX/JunOS engineering team. 

 

---

You're absolutely right about that and no one said otherwise (well, except for your Juniper sales people probably). The situation sucks and when there is no real solution available, all that you're left with are workarounds. 

 

And you're right that when your box is under attack you may no longer have a chance to get into the machine through a revenue port. For that we use a console port. 

I got this back from support:

"There are at least 7 ER’s opened on this problematic issue. (26985, 26983, 25921, 27771, 29840, 29841, 32317). Thus this issue is definitely on the Product Line Managers (PLM) radar. Most of the above ER’s concern a request for  fxp0 interface to be placed in

a non-default routing instance. The branch SRX as a JUNOS device is specifically mentioned in many of the ER’s thus I don’t believe an additional ER is going to be appropriate in this case.

 

Please try to be patient as Juniper considers this issue and works towards an appropriate resolution."

 

 

And by the looks of it the final solution will involve VRs, just a simpler configuration whereby it is segregated to begin with (which I agree with).

I still don't get folks thinking this is an oversight, it's a normal routing limitation (the actual functions originally required, not the argument over whether to use fxp0 or not). You cannot have traffic concurrently egress from 2 different interfaces to the same host(s) with one routing table, on ANY IP unit. A forwarding table has to include the egress interface. The only way to have multiple forwarding tables is through VRs. And again VRs are very easy to work with on the SRX, changing your config to use them is really just a few minutes of work.

Remember to take at look in the KB, TN212: "Best Practices for SRX Series Cluster Management".

 

There are lots of good discussions around fxp0s, including the trick of a master-only address to send SNMP etc to the active master, and a routing trick to forward syslog streams out a "revenue" interface in the default routing instance.