Routing

Here is a basic a schematic of the network I am working on.  The switch is an EX3300 running 12.3R6.  Each firewall is connected to an ADSL internet connection.

I am attempting to accomplish the following:

• Device 1 at 172.17.6.90 should use 172.17.0.1 as its route to 8.8.8.8
• Device 2 at 172.17.6.91 should use 172.17.0.3 as its route to 8.8.8.8
• Device 3 at 172.17.99.99 should use 172.17.99.2 as its route to 8.8.8.8
• Devices 1 and 2 should use 172.17.7.240 as their route between the two private subnets
• Device 3 should use 172.17.99.2 as its route between the two private subnets

Here are the relevant portions of the configuration that I have created on the switch:

interfaces {
...
    vlan {
        unit 0 {
            family inet {
                dhcp {
                    vendor-id Juniper-ex3300-48t;
                }
            }
        }
        unit 10 {
            family inet {
                filter {
                    input {
                        first-fbf-filter;
                    }
                }
                address 172.17.7.240/21;
            }
        }
        unit 99 {
            family inet {
                filter {
                    input {
                        second-fbf-filter;
                    }
                }
                address 172.17.99.240/24;
            }
        }
    }
}

routing-options {
    interface-routes {
        rib-group inet fbf-group;
    }
    static {
        route 0.0.0.0/0 next-hop 172.17.0.1;
    }
    rib-groups {
        fbf-group {
            import-rib [ inet.0 ri-a.inet.0 ri-b.inet.0 ri-c.inet.0];
        }
    }
    router-id 172.17.7.240;
}

firewall {
    family inet {
        filter first-fbf-filter {
            term t1 {
                from {
                    source-address {
                        172.17.6.90/32;
                    }
                }
                then {
                    routing-instance ri-a;
                }
            }
            term t2 {
                from {
                    source-address {
                        172.17.6.91/32;
                    }
                }
                then {
                    routing-instance ri-b;
                }
            }
            term default {
                then accept;
            }
        }
        filter second-fbf-filter {
            term t1 {
                from {
                    source-address {
                        172.17.99.99/32;
                    }
                }
                then {
                    routing-instance ri-c;
                }
            }
            term default {
                then accept;
            }
        }
    }
}

routing-instances {
    ri-a {
        instance-type forwarding;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 172.17.0.1;
            }
        }
    }
    ri-b {
        instance-type forwarding;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 172.17.0.3;
            }
        }
    }
    ri-c {
        instance-type forwarding;
        routing-options {
            static {
                route 0.0.0.0/0 next-hop 172.17.99.22;
            }
        }
    }
}

With the above config, my 172.17.6.90 and 172.17.6.91 devices each still seem to use 172.17.0.1 as their gateway to get to 8.8.8.8 (as seen with tracert; they're windows machines).

I am new.  Is what I am trying to accomplish possible?  Am I going about it in the right way?  My thanks for your expertise.

Hi,

Can you show me the config for the interfaces the devices are connected to?

What are the default routes on the devices?

Cheers./C.

Definitely, thank you for your help.

Default routes as follows:

• Device 1: 172.17.7.240
• Device 2: 172.17.7.240
• Device 3: 172.17.99.240

Here is the config on those interfaces:

ge-0/0/1 {
    description "Device 1 Connected HERE";
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members [ vlan10 ]:
            }
        }
    }
}
ge-0/0/2 {
    description "Device 2 Connected HERE";
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members [ vlan10 ]:
            }
        }
    }
}
ge-0/0/3 {
    description "Device 3 Connected HERE";
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members [ vlan99 ]:
            }
        }
    }
}
ge-0/0/4 {
    description "Firewall A Connected HERE";
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members [ vlan10 ]:
            }
        }
    }
}
ge-0/0/5 {
    description "Firewall B Connection 1 Connected HERE";
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members [ vlan10 ]:
            }
        }
    }
}
ge-0/0/6 {
    description "Firewall B Connection 2 Connected HERE";
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members [ vlan99 ]:
            }
        }
    }
}
ge-0/0/7 {
    description "Firewall C Connected HERE";
    unit 0 {
        family ethernet-switching {
            port-mode access;
            vlan {
                members [ vlan10 ]:
            }
        }
    }
}

Okay, at first glance it looks about right, I presume the different routing tables have the appropriate default routes?

i.e. what does

show route table ri-a
show route table ri-b
show route table ri-c

 show you?

Actually

show route 0.0.0.0

 might be more useful as it should show you the default routes for all routing instances...

Ah, thanks for the response!  I hadn't seen it before today, my apologies for the delay in replying.

Here is show route table rib-b:

0.0.0.0/0          *[Static/5] 3d 23:41:34
                    > to 172.17.0.3 via vlan.10
172.17.0.0/21      *[Direct/0] 3d 23:41:34
                    > via vlan.10
172.17.7.240/32    *[Local/0] 3d 23:41:34
                      Local via vlan.10
172.17.99.0/24     *[Direct/0] 3d 23:41:34
                    > via vlan.99
172.17.99.240/32   *[Local/0] 3d 23:41:34
                      Local via vlan.99

rib-b looks to contain the correct routes...

 The problem I am seeing is as follows:

• Device 2 has 172.17.7.240 set as its default gateway
• Attempting to access 8.8.8.8, Device 2 is correctly routed to 172.17.7.240 and then 172.17.0.3 
• Attempting to access 172.17.99.99, Device 2 is correctly routed to 172.17.7.240 and then incorrectly 172.17.0.3, where the route fails, of course

If the config looks correct, and indeed, when the filters are applied, the correct gateway is used for 8.8.8.8 but not 172.17.99.99, what might be going on here?  Do I have to remove the static route to 0.0.0.0 that I've got defined under routing-options?

Again, my thanks.

 The problem I am seeing is as follows:

• Device 2 has 172.17.7.240 set as its default gateway
• Attempting to access 8.8.8.8, Device 2 is correctly routed to 172.17.7.240 and then 172.17.0.3 
• Attempting to access 172.17.99.99, Device 2 is correctly routed to 172.17.7.240 and then incorrectly 172.17.0.3, where the route fails, of course

---

Hmmm, I *thought* the issue was that Device 2 was routing to 8.8.8.8 via 172.17.0.1 as per your OP?

Okay, if that isn't the problem and routing from vlan 10 to 99 IS the problem, this makes (a little) more sense.

There is a bug in the implementation of RIB on the EX platform, where local routes shared across the routing instances, while routes look valid in the routing table, are not present in the PFE. In other words, if the only route to the destination in the route table of the routing instances you're forwarding via is the route put there as a result of the RIB import, then traffic won't reach it via the route. However, when we encountered the bug it resulted in traffic not reaching the destination at all.

Just to see if this bug is the problem add a term above your t2 term in your first-fbf-filter, we called ours "match_direct" so it was clear the term matched "direct" routes in the routing table. For you it would go along the lines of :

set firewall family inet filter first-fbf-filter term match_direct from destination address 172.17.99.0/24
set firewall family inet filter first-fbf-filter term match_direct then accept
insert firewall family inet filter first-fbf-filter term match_direct before term t2

So, for any traffic entering your first-fbf-filter, if the traffic is destined for 172.17.99.0/24 (you can make this 172.17.99.99/32 if you wish, obviously) then it will use the inet.0 routing table and should therefore be routed correctly.

I can't remember if "Local" routes were similarly affected by this bug, I don't *think* they were, so you could just try to add a static route to this routing instance for the individual host :

set routing-instances ri-b routing-options static route 172.17.99.99/32 next-hop 172.17.99.240

 Let me know how you get on...

Thank you again, you are an enormous help.

You're correct, in the OP I sad that the problem was that Device 2 was still using 0.1 rather than 0.3.  When I tested it just before my last posting, with the filter applied, the route to 8.8.8.8 was via 0.3 (!), which is what I had hoped.  The problem was actually that packets destined for the "directly connected" 99.x/24 subnet were also being routed to 0.3, rather than being routed over to the 99 VLAN.

Sorry if that's confusing... it certainly is to me.

I am running Junos 12.3R6.6 on 5 out of 6 switches in this network, and 12.3R3.4 on the last switch.  They are all EX3300s.  Is the "bug" you refer to fixed in some later version of Junos to which I should be upgrading?

I will test your term modifications and will report back asap.

My thanks again.

Under the firewall stanza I now have this:

family inet {
    filter first-fbf-filter {
        term t1 {
            from {
                source-address {
                    172.17.6.90/32;
                }
            }
            then {
                routing-instance ri-a;
            }
        }
        term match_direct {
            from {
                destination-address {
                    172.17.99.0/24;
                }
            }
            then accept;
        }
        term t2 {
            from {
                source-address {
                    172.17.6.91/32;
                }
            }
            then {
                routing-instance ri-b;
            }
        }
        term default {
            then accept;
        }
    }

 And Device 2 tracert output is now:

H:\>tracert 8.8.8.8

Tracing route to google-public-dns-a.google.com [8.8.8.8]
over a maximum of 30 hops:

  1     2 ms     1 ms     1 ms  switch-a.mydomain.ca [172.17.7.240]
  2    <1 ms    <1 ms    <1 ms  firewall-c.mydomain.ca [172.17.0.3]
  3     6 ms     8 ms     9 ms  myi-ips-dns-server.myisp.net [xxx.xxx.xxx.xxx]
  4    10 ms     8 ms     8 ms  ^C
H:\>tracert 172.17.99.99

Tracing route to device-3.mydomain.ca [172.17.99.99]
over a maximum of 30 hops:

  1     1 ms     1 ms     1 ms  switch-a.mydomain.ca [172.17.7.240]
  2    <1 ms    <1 ms    <1 ms  device-3.mydomain.ca [172.17.99.99]

Trace complete.

H:\>

 It works!!!  I changed some of there resolved DNS names in the output above, obviously, but the routing now behaves as intended.

Do you know, is that "bug" a documented one?  If I am to put this code into production, I will want to be sure that subsequent Junos upgrades don't break the work-around behaviour you've very helpfully provided here.

Again, sincerest thanks.

Excellent! Glad it worked out! Believe me, we went nuts figuring this out so I'm happy to pass on the experience!!!

Actually we had a JTAC case open for this, we were running 11.4R2.14 at the time, and the bug is still present in 12..R6.6 we're running now. The JTAC engineer actually never referred to it as a bug, he called it a "feature"!!!!

If they ever decide to pull that "feature" though no fear, that work around will be backawards compatible, as its just making sure the traffic flows according to how your configuration SHOULD work...

Thanks for marking solved too, my first on here!

Kind regards...