Switching

Hello all,

 

I have a very confusing technical issue with my EX2200 POE switch (or entire network).  Picture this, our network has a default vlan and a guest vlan (18).  I have 4 EX2200 POE switches that manage the 2 wireless networks(internal and guest).  All the POE switches are configured similarly.  They all meet to a switch in my main switch closet that handles the guest vlan traffic, and the core to handle internal traffic.  

 

Port 0 is an Access port for the Guest Vlan.  A line goes from this port to the switch for the guest traffic.  Port 1-4 are trunk ports with Guest and default vlan for the access points.  Port 0 is limited to 10Mbps.

 

All areas worked great for several weeks.  I was out of the country on vacation, and apparently the entire network went down for 3 hours, then came back "by itself".  When I got back, I checked and couldn't determine the cause of the outage (this was 3 days after the issue occurred).  One thing I noticed was that the guest for one area was not working though.  I could not access anything on the guest from 3 access points, all going into the one juniper switch.  I tried plugging into the ethernet port with my pc with no luck as well (rules out wireless causing it).  Guest works in all other areas.

 

So here is where things get weirder...today I ran a new cable from the switch to the main switch closet to rule out a bad cable as the issue.  When I plug it into the Access port at 10Mbps, no change.  Out of curiosity, I created another Access port and did not limit it on the Guest vlan and plugged the cable in.  Success!  Guest network is back!!! (or so I thought)...

 

The ENTIRE network went down when that cable was plugged in...everything except the guest....unplug the cable, and the internal network came back immediately.

 

So I have been doing this a LONG time and have never seen this before.  I configured a new switch and threw it in to try, and the same issue.  I am at a loss and am hoping someone has some idea of what is causing this....

 

 

Thanks!

Can you post a simple topology diagram?  Your issue today, on the surface, looks to be a bridging loop or STP issue.  Do you have any logs in your switch that say something along the lines of 'blocked by STP'?  Can you post some scrubbed configs, too?

Thanks for your input.  What I am told is that nothing was being done to the network, and only a printer was being added at the time of the outage when I was out of town.

 

The logs show constant warnings saying can't get client adress: Bad file descriptor. See attached log image.  Port 4 (10M is the port where nothing happens when I plug in the guest line including no guest access.  Port 5 kills my internal network but guest works)

 

 

 

 

 

 

Please see attached basic topology and config.

 

The 'bad file descriptor' is a red herring, so to speak.  Nothing to do with your issue.

 

If you notice in the logs, you can see that your root bridge changed a few times.  I presume your spanning tree root bridge is supposed to be one of the 2 EX4300 switches (or the cluster) labeled 'CORE'?  Are you using RSTP throughout the network, including the HP switch?  If so, you're going to want to do a bit of reading up on spanning tree and configure your network appropriately.  STP is not a trivial thing.  RSTP really isn't meant for networks with more than one VLAN, which yours is.

 

Your issue with the ge-0/0/4 and ge-0/0/5 ports, I suspect, has to do with the fact that one is only negotiating at 10m, thereby reducing the possibility that it will become a root port (because the others are probably negotiating at 1G). 

 

Overall, you're dealing with spanning tree issues.  Read through some of the documentation regarding the different versions of STP on EX, as well as what your HP switch supports:

 

http://www.juniper.net/techpubs/en_US/junos12.3/information-products/pathway-pages/ex-series/spanning-tree-protocols.html

 

Ok thanks.  I have never really messed with STP.  I didn't set up RSTP at all.  I took over this job a few months ago with no documentation, so it is possible it was set up before, but I brought all the junipers in house, including the 2 core switches. So I did not configure anything in the way of RSTP. 

 

 

Any ideas why the other switches all work properly?

 

Thanks again for your help, I will research that a little bit.

I'm sorry there isn't much in the way of exact technical suggestions.  If you could modify your diagram to better indicate what ports are connected to what switch and the actual physical connections, perhaps we can make some better suggestions.

 

As I said, STP is not trivial.  Messing with it can cause your entire network to melt down if you don't know what you are doing.  Your best bet is to really ensure that:

 

1. There are as few physical loops in the network as possible.
2. The STP versions match between all switches, including the HP.
3. You have properly set up what you want your root bridge and secondary root bridge to be.

In response to your question about whether to shut off RSTP, the answer at this point is NO!  You need to get a better understanding of how all your switches are physically connected prior to making any STP changes, as well as know what your current root bridge is.

 

Can you do a 'show spanning-tree bridge' from the EX4300 and from an EX2200 and post the output here?

Please see attached Show Spanning-tree bridge command results.  The first 2 are from the core 4300 switches, the 2nd are 2 2200 POE switches, both of which have the vlans configured.  The first is the one that is mainly giving me problems right now.

As a follow-on to last message, and yes I agree looks very much like some STP situation, which therefore also means that while on vacation the network "did" chamge.

 

Not sure if this is the config for some EX2200 or potentially the config of the 'core' EX4300s?  I assume the EX4300s are set-up as a VC, yes?  What about the EX2200s, VC as well.  BTW, I assume you have contacted TAC, what do they have to say about this?

 

BTW, set rstp interface all will work differently between EX4300 (L2 CLI or ELS) and EX2200 (old CLI).  It is the "interface all" part that is not recognized on EX4300, and should commit error (?), which would lead me to believe this is some one of the EX2200 configs.

RSTP is on by default on all interfaces for all EX switches.

If RSTP is on by default, should I turn it off since it is not meant for more than one vlan?

 

Thanks again

I was going to contact them, but at $500 minimum, my company asked me to find a different avenue to solve the issue if possible.  

 

It looks like I need to read up on STP, because frankly, I never dealt with it so have no idea where to start there.

So you don't have a service contract, . . .  You can tell you bosses for about $25 per switch you could have had JTAC support for a year.

 

RSTP affects vlans, but RSTP operates at interface level, not vlan.  I would NOT suggest you disable RSTP, as based upon what you have stated, I would highly suspect your whole network will come crashing down.

Try reading through the most excellent Day One Book on configuring EX Series switches:

 

http://www.juniper.net/us/en/training/jnbooks/day-one/fabric-switching-tech-series/config-ex-series/

 

There's a brief overview of STP that compares the different types and how to configure them.  It's not in-depth stuff, but it will get you started on a basic understanding.

Ok thanks.  I will research how to identify and setup what switch is the root bridge.  My fear is the network will come crashing down on its own at any given time right now.  Since I inheritted a mess of this network and am trying to upgrade now, I will have to work fast. 

 

I don't know how it was configured previously, so I am trying to clear all that as well.

Ok, I read through the STP portion of the guide.  Thank you for that.  So it seems that STP is basically an 8 byte value that all switches use to identify what path to take in the event of a loop? Is that accurate?  

 

It kind of explains an issue I had when replacing the switches.  I upgraded to these from 3com 10/100 switches. There are 2 PCs and 1 printer that when plugged into the new gigabit switches, the ports showed no activity.  I thought maybe an issue with 10/100 vs gigabit, but it looks like the ports stopped listening due to an STP issue?  

 

Anyway, so it looks like I have a loop in my network based on what you guys are saying.  I will have to start there and try to trace it back.  I will read up more on STP as well.

 

Some very helpful commands that should be run on each switch in the network to gather data about what is plugged in where:

 

show spanning-tree bridge

show spanning-tree interface

show ethernet-switching table

show ethernet-switching interface

 

STP itself is not an 8-byte value, it's an entire protocol that runs on your layer 2 network to prevent bridging loops from occurring.  You technically do not need STP running if you can absolutely ensure that you will never have a physical loop in the network, but you really need to know your network well and have control over what gets plugged into it before even thinking about disabling it.  There are other methods that you can use for redundant links if you don't want to run STP, but these methods are often situation- or topology-dependent.

 

It would be good to know how you have your vertical stacks of EX2200s configured - are they single standalone switches or are they configured in a virtual chassis?  Same with the EX4300s.  I highly recommend using VC for the stacks of EX2200s, as it makes management much easier, if you can spare the ports on each switch.

 

Running show ethernet-switching interfaces shows a big issue, as you guys pointed out...question is, will fixing the root bridge to go to the core fix it?

 

root@ITBOTTOM> show ethernet-switching interfaces
Interface State VLAN members Tag Tagging Blocking
ge-0/0/0.0 up GUEST 18 untagged unblocked
ge-0/0/1.0 down default untagged blocked by STP
GUEST 18 tagged blocked by STP
ge-0/0/2.0 up default untagged unblocked
GUEST 18 tagged unblocked
ge-0/0/3.0 down default untagged blocked by STP
GUEST 18 tagged blocked by STP
ge-0/0/4.0 up default untagged unblocked
GUEST 18 tagged unblocked
ge-0/0/5.0 up GUEST 18 untagged unblocked
ge-0/0/6.0 up default untagged unblocked
ge-0/0/7.0 up default untagged unblocked
ge-0/0/8.0 up default untagged unblocked
ge-0/0/9.0 down default untagged blocked by STP
ge-0/0/10.0 down default untagged blocked by STP
ge-0/0/11.0 up default untagged unblocked
ge-0/0/12.0 up default untagged unblocked
ge-0/0/13.0 up default untagged unblocked
ge-0/0/14.0 up default untagged unblocked
ge-0/0/15.0 down default untagged blocked by STP
ge-0/0/16.0 up default untagged unblocked
ge-0/0/17.0 up default untagged unblocked
ge-0/0/18.0 up default untagged unblocked
ge-0/0/19.0 up default untagged unblocked
ge-0/0/20.0 up default untagged unblocked
ge-0/0/21.0 up default untagged blocked by STP
ge-0/0/22.0 down default untagged blocked by STP
ge-0/0/23.0 up default untagged unblocked
ge-0/0/24.0 down default untagged blocked by STP
ge-0/0/25.0 up default untagged unblocked
ge-0/0/26.0 down default untagged blocked by STP
ge-0/0/27.0 down default untagged blocked by STP
ge-0/0/28.0 up default untagged unblocked
ge-0/0/29.0 up default untagged unblocked
ge-0/0/30.0 up default untagged unblocked
ge-0/0/31.0 up default untagged unblocked
ge-0/0/32.0 up default untagged unblocked
ge-0/0/33.0 up default untagged unblocked
ge-0/0/34.0 up default untagged unblocked
ge-0/0/35.0 up default untagged unblocked
ge-0/0/36.0 up default untagged unblocked
ge-0/0/37.0 down default untagged blocked by STP
ge-0/0/38.0 down default untagged blocked by STP
ge-0/0/39.0 up default untagged unblocked
ge-0/0/40.0 up default untagged unblocked
ge-0/0/41.0 down default untagged blocked by STP
ge-0/0/42.0 up default untagged unblocked
ge-0/0/43.0 up default untagged unblocked
ge-0/0/44.0 up default untagged unblocked
ge-0/0/45.0 up default untagged unblocked
ge-0/0/46.0 up default untagged unblocked
ge-0/0/47.0 down default untagged blocked by STP
ge-0/1/0.0 up default untagged unblocked
me0.0 down mgmt untagged unblocked

My best guess is that you are someone needs to determine what your network design should be, based upon your current and potentially future needs, and then implement (and document) it properly.  You may need outside assistance, which is not going to cost you $500/hr, but will cost you over-all very much more.  There are many people out there who do this for a living, just for [unfortunate] folks like you.

While I agree with rccpgm that you may need professional assistance, there's no reason you can't learn how to do it yourself.  Granted, it may be much more painful, but that's the way some people learn best.

 

So you've gathered some initial information, which is good.  Now you need to have a detailed network map that shows physical connections between all your devices.  Before changing any configuration, this is absolutely necessary to have.  Once you have that and you are certain of its validity, you need to create another map of how you want it to look in the end, based upon how you want traffic to flow.  In your map, avoid using 'shared segment' lines so you know all the physical connections.

 

Regarding the information you sent, yes, you have some issues.  You appear to have at least two root bridges in your network, which is really not good if your network is all layer 2 and running RSTP.  The "Root ID" in your 'show spanning-tree bridge' output shows the current root ID and its priority in the format: Priority.RootID.  You should identify what each of those devices are once you have a good map.  I know I keep saying to have a map, and for good reason - you don't want to make any changes at all if you're not sure how they are going to affect traffic flow.  Also, I hope you have either management port access or console access to every switch.

 

 

There are a few switches in the main closet I have not changed yet, 3 3com switches and a cisco.  The 3com's are layer 3 unmanaged I believe and the cisco is managed but not showing up on my show lldp neighbors, which I am guessing is becuase it is not juniper so will not show.  They are on my list to get replaced, but could they be causing any issues as is?

Yes.  You probably do not see Cisco via LLDP (standard protocol) becuase lldp probably not enabled on it.  Cisco often times sets its STP Priority higher than default so that it will become a/the root bridge.  I might suspect the Cisco is one of your 2 root bridges, and evt is VERYright in that having 2 root bridges is generally not a good idea as you now have 2 separate STP domains, that somehow over-lap eash other.  Somewhat surprised that anything works at all -;)

So pulling the Cisco may automatically have the root bridges correct themselves?  (Rhetorical)  I am mapping the network now.  Thanks again for helping both of you.  We are a 24hr operation here so I am working fast to try to resolve this....

Another stupid question.  Ports blocked by STP, will they automatically unblock when I fix the issues?  Reason I ask is I already found one loop that I removed where a switch was looped back to itself.

 

Thanks

1.  Pulling the Cisco may or may not resolve *one* issue, but it also may introduce more issues.

2. The ports blocked by STP may or may not automatically unblock, it really all depends on why they were blocked in the first place.  The primary reasons a port goes into blocking mode are because of BPDU reception either where it is not expected (like an edge port) or it receives an inferior BPDU.  There are other reasons, but these are the most common.  If the reason the port is blocked is because it receives an inferior BPDU, then removing the loop *should* allow the port to come back up, but I honestly don't recall if there is a default bpdu-error timeout value or if it has to be manually reset by issuing a 'clear ethernet-switching bpdu-error' command.

 

It sounds like you have inherited a real mess there.  STP is about one of the worst things a novice should have to deal with.  If it could be weaponized, I'd use it to stop terrorism.

Ok.  Could running the clear command cause any issues, or if it needs to be blocked still, it will?  Something I noticed too is that every port that is down (nothing plugged in) says its blocked by STP.  Is that normal?

 

I have ran through all the switches in my area.  The Cisco I cannot do anything about becuase I do not have the user or password info to log into the cli.  It definitely looks like there is 2 clear root bridges.  I think the one may be the firewall, and the other may be the cisco switch.  My plan is to replace it with a 2200 switch, and hope that automatically clears up the 2 bridges.  Again, I fear it may make it worse, but it is something I must try.

 

When I was running through, there were 2 distinct loops I found.  One was a switch that had a cable from port 46 to 47 connected, plugged into itself.  The other had 2 ports connected to another switch, ports 17 and 21.  I removed those 2 bridges to itself, and the guest network I was having issues with came back up immediately.  

 

At least it seems I am making headway.  Thanks again for the help!

Good work!  It sounds like you're making some great progress.

 

Running the 'clear ethernet-switching bpdu-error' command should not cause any issues, because if a BPDU is seen it will immediately go back into blocking mode.

 

If I was to take a guess, the Cisco is likely running PVST+, which should be compatible with VSTP.  If you are trunking to the Cisco, you need to make sure that you are not tagging VLAN ID 1, because PVST sends all BPDUs untagged on VLAN1.  You can use the 'native-vlan-id' command on the trunk port to the Cisco if you are, in fact, trunking to it (recommended if the Cisco is to know about both VLANs).  That said, you don't want to make any changes until can get access to the Cisco to check the configuration. 

 

There's nothing inherently wrong with the Ciscos being the root bridges, as long as the rest of the network is properly configured.  We have a multivendor network with three different brands of switches, including Cisco, in some areas and we use MSTP and everything works fine.

 

Hey, I am back again...Today I removed the Cisco switch I was talking about earlier, and replaced with a few Juniper 2200EX switches.  The problem resurfaced.  I think I understand now but want clarification.  BTW, all the switches share the same root bridge now.

 

I attached my network map so you can see what I am talking about.  The RED lines are the lines that I ran for my VLAN to connect back.  What I am afraid of is that the Red lines are creating loops with the BLACK lines, which are my main network.  The ports that each RED and BLACK line run on are access ports.  Shouldn't the switches treat them as different networks essentially since the RED is on a GUEST VLAN and the BLACK is on the DEFAULT VLAN?

 

Thanks

Yes, these are creating loops, regardless of VLAN because, as I recall, your switches all run RSTP which only maintains a single RSTP instance for all VLANs.  What is the purpose of the physical loops?  If it's for a specific VLAN, could you not just trunk that VLAN through the network on the existing ports?

 

Also, I'd consider migrating away from RSTP and onto either VSTP or even MSTP, though MSTP requires a bit more understanding of STP, physical and logical topology, and end goals.

Ok, thats what I started thinking.

 

So, to trunk the lines through the normal network, I trunk each in/out port, and setup the vlan on each switch in the network, right?  I didn't do that because of the number of switches to make vlans on, I figured it would be quicker and easier to plug them in dedicated lines... I didn't realize they made loops.

 

I will try that.

 

I will have to research how to move to VSTP or MSTP as I am not familiar with them.

 

Thanks again for your help.

First off, yes, you would trunk them by including them as members on each of your interconnection ports and making the port into 'interface-mode trunk'.

 

Secondly, the configuration work required is not a lot if a) you just script it out in Notepad and paste it in to every switch or b) have a configuration storage software like Solarwinds NCM or Rancid or Kiwi CatTools, or c) put as many of these switches as you can into virtual chassis configurations.

 

Third, do not just change the STP protocol.  You will likely want console access to the switch in the very likely event that your only downstream port to that switch goes into blocking mode and you lose connection to it.  It would be very wise to do one switch at a time, perhaps starting at the far edge of the network and making your way towards the core, disabling your upstream port, making the change, then re-enabling the port after the change is made on the next switch upstream.

 

Of course, this all takes into account that you are doing this on your own, but like others suggested, it may be wise to enlist the help of a pro.  Unless of course you learn better by making mistakes and figuring out how to fix them. 

I am self taught on everything I have done, with the help of forums and google. I will probably just conquer it myself, unless I get in a real bind.  Thanks again for your help!

We are all set.  That took care of it. It was actually much less painful with the nice drawn out network map to go off of.  

 

Thanks a lot

I have a quick question.  I am running show lldp neighbors and show spanning-tree bridge on each switch then documenting the ports and to where connected, the root port, their id, and the root id.  

 

The question I have is that the show lldp neighbors gives a chassis ID of a different switch than it says its connected to...do I need to worry about that?  Thought it was weird...

Ok. That makes more sense. The problem is that I don't know how the cables were ran before me. There's no documentation.

I will run those commands and try to make sense of the results. Is it possible that this issue could cause a printer to suddenly lose network connectivity, but when you plug in another port, it works? Or is it an all or nothing type of issue?

I don't have VC set up on any of the switches. I'll research how to set that up as well if it will make a difference.

Thanks again for the help!

Your printer question - yes, STP can do all sorts of funky stuff you're not expecting.  My first STP problem back in 2000, before I ever knew what it was, was very similar.  I'd plug in or enable a port on one switch and another port on another switch somewhere else in the layer 2 domain would just stop working.

 

Those commands should help you figure out what's what and where, but also try to understand what each is telling you.  It can be daunting, for sure.  Continue to post questions as you go along, if you have any.

 

Also, fix one problem at a time.  First, get the STP issues resolved, then look at VC config.  Don't try to do everything all at once.

I do have one stupid question. This seems to have happened shortly after I introduced the guest network. The HP switch was set up before me and handled the vlan for another reason. It goes back into the original core which is not listed in my diagram. It consists of one Cisco 2620 and 3 3com switches. I piggyback off of this VLAN for my guest access. now there is a line from each vlan in each switch going to this HP switch which goes into the core eventually. I am now thinking that this is what is causing my loop. Because the main network goes into the core directly, the guest then goes into the core through the HP switch. I'm not at work right now but will run those first thing tomorrow. Again, Thanks for the help.

If your 2200s and your HP are connected to eachother and the core, there is a loop there. Loops are not bad and many times are necessary. It's just the lack of proper spanning tree config that's the issue.

Makes sense. I appreciate the clarification. This helped out a lot. I never encountered anything like this before but It's making sense. I'll post back if I have any questions as I go along tomorrow. Hopefully I can get this straightened out sooner than later. 

You might try and use show lldp neighbors (lldp is also enabled by default) to show you which switch is connected (and where/how) to what other switches.  This might help you figure out the cabling topology.

 

http://www.juniper.net/documentation/en_US/junos13.2/topics/reference/command-summary/show-lldp-neighbors-lldp-ex-series.html

 

 

Of course, use 'show lldp neighbor' if you want the easy solution. ☺