Switching

We're looking at what we can do to move away from a big flat L2 network to something scalable and sensible.

We're pretty comfortable that we can do standard OSPF between core and distribution (all EX4200).

Edge/access switches are EX2200 and we would likely just pull L2 into the distribution switches.

The sticking point is that we need to be able to span some L2 VLANs to some or all locations.

What do people suggest to overlay this please?  Ideally without having to buy additional licenses beyond the default functionality.

Use RVI's instead of layer 3 interfaces for your uplinks between edge and distribution and carry them over L2 trunks. Any vlans that need to span multiple IDF's through the distribution can ride over the same trunk. If you use VSTP or MSTP your uplink VLANs will both remain up regardless if an overlay vlan uses the same trunks and has one of the uplinks in blocking state.

--Paul

Thanks for that - and I hate to sound like I'm just saying this but I honestly was reading up on MSTP and did think it sounded like it ccould do what we need as it blocks per VLAN rather than on physical port.

Given the 4200 default feature set which would you suggest please?

Also how do you deal with "conflicts" between MSTP and OSPF on the RVI VLAN or do you simply not enable MSTP on that VLAN?

Ex4200 supports both vstp and mstp, I usually go with vstp because it seems less of a hassle to configure. Mstp is compatible with other vendors. 

Since your uplinks are point to point (and you can configure them as such in ospf to avoid neighbor election) you can skip stp on those vlans unless you think someone may come along later and misconfigure a second path for that vlan. 

If it's not too much trouble I'd be very interested in a command set for any of the more subtle stuff you're referred to.

We seem to have VSTP working, and we seem to have OSPF working, and I'd hope we couldn't have a loop because each VLAN should only exist in one place - the RVI it's being used for.

Curious what you mean by "Since your uplinks are point to point (and you can configure them as such in ospf to avoid neighbor election"?

Apologies if this all seems a bit basic - we're sure we have it working on our lab setup of 3x EX2200's but it's hard to mimic production.

I'm probably forgetting or typo'ing something but assuming this:

OSPF, vlans, vstp, (and bfd to speed up ospf on p2p) should look like this:

dist1:

system {
     host-name dist1;
}

interfaces {
    ge-0/0/0 {
        description "To edge";
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [51 100];
                }
            }
        }
    }
    ge-0/0/1 {
        description "To dist2";
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [ 100 ];
                }
            }
        }
    }
    vlan {
        unit 100 {
            description data;
            family inet {
                address 172.16.1.2/24 {
                    vrrp-group 1 {
                        virtual-address 172.16.1.1;
                        priority 200;
                    }
                }
            }
        }
        unit 51 {
            description "Edge uplink";
            family inet {
                address 10.0.0.0/31;
            }
        }
    }
}


protocols {
    ospf {
        area 0.0.0.0 {
            interface vlan.51 {
		interface-type p2p;
                bfd-liveness-detection {
                    minimum-interval 500;
                }
            }
            interface vlan.100 {
                passive;
            }
        }
    }
    vstp {
        vlan all {
            bridge-priority 4k;
        }
    }
}

vlans {
    data {
        vlan-id 100;
        l3-interface vlan.100;
    }
    vlan51 {
        vlan-id 51;
        l3-interface vlan.51;
    }
}

dist2:

system {
     host-name dist2;
}

interfaces {
    ge-0/0/0 {
        description "To edge";
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [52 100];
                }
            }
        }
    }
    ge-0/0/1 {
        description "To dist1";
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [ 100 ];
                }
            }
        }
    }
    vlan {
        unit 100 {
            description data;
            family inet {
                address 172.16.1.3/24 {
                    vrrp-group 1 {
                        virtual-address 172.16.1.1;
                        priority 100;
                    }
                }
            }
        }
        unit 52 {
            description "Edge uplink";
            family inet {
                address 10.0.1.0/31;
            }
        }
    }
}


protocols {
    ospf {
        area 0.0.0.0 {
            interface vlan.52 {
		interface-type p2p;
                bfd-liveness-detection {
                    minimum-interval 500;
                }
            }
            interface vlan.100 {
                passive;
            }
        }
    }
    vstp {
        vlan all {
            bridge-priority 8k;
        }
    }
}

vlans {
    data {
        vlan-id 100;
        l3-interface vlan.100;
    }
    vlan52 {
        vlan-id 52;
        l3-interface vlan.52;
    }
}

edge:

system {
     host-name edge;
}

interfaces {
    ge-0/0/0 {
        description "To dist1";
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [51 100];
                }
            }
        }
    }
    ge-0/0/1 {
        description "To dist2";
        unit 0 {
            family ethernet-switching {
                port-mode trunk;
                vlan {
                    members [ 52 100 ];
                }
            }
        }
    }
    ge-0/0/2 {
        description "vlan100 edge port";
        unit 0 {
            family ethernet-switching {
                port-mode access;
                vlan {
                    members [ 100 ];
                }
            }
        }
    }
    ge-0/0/3 {
        description "vlan200 edge port";
        unit 0 {
            family ethernet-switching {
                port-mode access;
                vlan {
                    members [ 200 ];
                }
            }
        }
    }
    ge-0/0/4 {
        description "vlan300 edge port";
        unit 0 {
            family ethernet-switching {
                port-mode access;
                vlan {
                    members [ 300 ];
                }
            }
        }
    }
    vlan {
        unit 100 {
            description data;
        }
        unit 200 {
            description data;
            family inet {
                address 172.16.2.1/24 {
                }
            }
        }
        unit 300 {
            description data;
            family inet {
                address 172.16.3.1/24 {
                }
            }
        }
        unit 51 {
            description "Edge uplink to dist1";
            family inet {
                address 10.0.0.1/31;
            }
        }
        unit 52 {
            description "Edge uplink to dist2";
            family inet {
                address 10.0.1.1/31;
            }
        }
    }
}

protocols {
    ospf {
        area 0.0.0.0 {
            interface vlan.51 {
		interface-type p2p;
                bfd-liveness-detection {
                    minimum-interval 500;
                }
            }
            interface vlan.52 {
		interface-type p2p;
                bfd-liveness-detection {
                    minimum-interval 500;
                }
            }
            interface vlan.200 {
		passive;
            }
            interface vlan.300 {
		passive;
            }
        }
    }
    vstp {
        vlan all {
            bridge-priority 16k;
        }
    }
}

vlans {
    data100 {
        vlan-id 100;
    }
    data200 {
        vlan-id 200;
        l3-interface vlan.200;
    }
    data300 {
        vlan-id 300;
        l3-interface vlan.300;
    }
    vlan51 {
        vlan-id 51;
        l3-interface vlan.51;
    }
    vlan52 {
        vlan-id 52;
        l3-interface vlan.52;
    }
}

Wow, that's amazingly comprehensive and very useful - thanks 🙂

Have you encountered any significant pitalls doing this?

We're stuck with a L2 requirement - some people have said "tunnel it" but I think that's a bit beyond us at this point.

We're struggling with why a hybrid design wouldn't work, whilst understanding that in a perfect world you wouldn't do it - but in a perfect world you don't have L2 legacy muck.

No pitfalls I can think of and has been working well for me for many years. As with any setup there's fine tuning that can be made, for example making sure you exclude the uplinks from storm control shutdown, otherwise a storm on the spanned vlan can unexpectedly take out both uplinks. VSTP also has a hard limit of ~256 vlans (in practice, less) so if you are running more than that you'll want to look at mstp and group some together.

We did spot the 256 VLAN limit (read it's about 190 in practise) and if it ever gets that bad I'll probably just start self-harming 🙂

Curious what you mean about disabling storm control?  Given what I think it does and that we're spanning L2 that seemed like something we'd want but that's just an assumption.

You want to enable storm control only on the edge ports, where a loop would form or where a bad device is connected. If you enable it everywhere you effectively lose the entire switch when the storm travels up your uplinks over vlan100 and the switch shuts those ports down.

---

@smicker wrote:

You want to enable storm control only on the edge ports, where a loop would form or where a bad device is connected. If you enable it everywhere you effectively lose the entire switch when the storm travels up your uplinks over vlan100 and the switch shuts those ports down.

---

Ahh that's an interesting shift from our mindset 🙂  We were assuming "Oh you need storm control on the uplinks to prevent a storm taking out a distribution or core switch".

You're saying you don't, but you do need it everywhere else to prevent it getting as far as the uplinks?

You want to block it as close to the source as possible, which is the edge port. If you include the uplinks then you are shutting down the entire switch with no real benefit.

Makes sense.  Have you much experience with broadcast storms?  We've only experienced a couple because they aren't something you intend to happen obviously but it rendered our 4200's useless - price you pay for L2 I guess but I'd be interested to know how much the stuff you've done can mitigate.

Thanks again btw - I've been doing a lot of digging in general terms but it's nice to get some real-world examples of how it specifically relates to the EX protocol set.

Yes--people frequently plug in small netgear or linksys switches in out on the floor and then accidentally loop them. BPDU guard catches most of these but occasionally something will get plugged in that doesn't generate or pass bpdu packets and we end up with a storm.

---

@smicker wrote:

Yes--people frequently plug in small netgear or linksys switches in out on the floor and then accidentally loop them. BPDU guard catches most of these but occasionally something will get plugged in that doesn't generate or pass bpdu packets and we end up with a storm.

---

Ah yes my old friend the Netgear 5 port switch - been there done that 🙂

I'm struggling to see any downsides of this so far.  We have to have L2 so assuming that compromise is going to exist regardless, being able to have L3 as well does seem to give us our cake and let us eat it.