Ethernet Switching
Highlighted
Ethernet Switching

Confusing VLAN Issue

‎12-22-2014 10:39 AM

Hello all,

 

I have a very confusing technical issue with my EX2200 POE switch (or entire network).  Picture this, our network has a default vlan and a guest vlan (18).  I have 4 EX2200 POE switches that manage the 2 wireless networks(internal and guest).  All the POE switches are configured similarly.  They all meet to a switch in my main switch closet that handles the guest vlan traffic, and the core to handle internal traffic.  

 

Port 0 is an Access port for the Guest Vlan.  A line goes from this port to the switch for the guest traffic.  Port 1-4 are trunk ports with Guest and default vlan for the access points.  Port 0 is limited to 10Mbps.

 

All areas worked great for several weeks.  I was out of the country on vacation, and apparently the entire network went down for 3 hours, then came back "by itself".  When I got back, I checked and couldn't determine the cause of the outage (this was 3 days after the issue occurred).  One thing I noticed was that the guest for one area was not working though.  I could not access anything on the guest from 3 access points, all going into the one juniper switch.  I tried plugging into the ethernet port with my pc with no luck as well (rules out wireless causing it).  Guest works in all other areas.

 

So here is where things get weirder...today I ran a new cable from the switch to the main switch closet to rule out a bad cable as the issue.  When I plug it into the Access port at 10Mbps, no change.  Out of curiosity, I created another Access port and did not limit it on the Guest vlan and plugged the cable in.  Success!  Guest network is back!!! (or so I thought)...

 

The ENTIRE network went down when that cable was plugged in...everything except the guest....unplug the cable, and the internal network came back immediately.

 

So I have been doing this a LONG time and have never seen this before.  I configured a new switch and threw it in to try, and the same issue.  I am at a loss and am hoping someone has some idea of what is causing this....

 

 

Thanks!

40 REPLIES 40
Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

[ Edited ]
‎12-22-2014 10:59 AM

Can you post a simple topology diagram?  Your issue today, on the surface, looks to be a bridging loop or STP issue.  Do you have any logs in your switch that say something along the lines of 'blocked by STP'?  Can you post some scrubbed configs, too?

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

[ Edited ]
‎12-22-2014 11:51 AM

Thanks for your input.  What I am told is that nothing was being done to the network, and only a printer was being added at the time of the outage when I was out of town.

 

The logs show constant warnings saying can't get client adress: Bad file descriptor. See attached log image.  Port 4 (10M is the port where nothing happens when I plug in the guest line including no guest access.  Port 5 kills my internal network but guest works)

Log.png

 

 

 

 

 

 

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

[ Edited ]
‎12-22-2014 11:53 AM

Please see attached basic topology and config.

 

Attachments

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

[ Edited ]
‎12-22-2014 12:16 PM

The 'bad file descriptor' is a red herring, so to speak.  Nothing to do with your issue.

 

If you notice in the logs, you can see that your root bridge changed a few times.  I presume your spanning tree root bridge is supposed to be one of the 2 EX4300 switches (or the cluster) labeled 'CORE'?  Are you using RSTP throughout the network, including the HP switch?  If so, you're going to want to do a bit of reading up on spanning tree and configure your network appropriately.  STP is not a trivial thing.  RSTP really isn't meant for networks with more than one VLAN, which yours is.

 

Your issue with the ge-0/0/4 and ge-0/0/5 ports, I suspect, has to do with the fact that one is only negotiating at 10m, thereby reducing the possibility that it will become a root port (because the others are probably negotiating at 1G). 

 

Overall, you're dealing with spanning tree issues.  Read through some of the documentation regarding the different versions of STP on EX, as well as what your HP switch supports:

 

http://www.juniper.net/techpubs/en_US/junos12.3/information-products/pathway-pages/ex-series/spannin...

 

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:20 PM

Ok thanks.  I have never really messed with STP.  I didn't set up RSTP at all.  I took over this job a few months ago with no documentation, so it is possible it was set up before, but I brought all the junipers in house, including the 2 core switches. So I did not configure anything in the way of RSTP. 

 

 

Any ideas why the other switches all work properly?

 

Thanks again for your help, I will research that a little bit.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:20 PM

As a follow-on to last message, and yes I agree looks very much like some STP situation, which therefore also means that while on vacation the network "did" chamge.

 

Not sure if this is the config for some EX2200 or potentially the config of the 'core' EX4300s?  I assume the EX4300s are set-up as a VC, yes?  What about the EX2200s, VC as well.  BTW, I assume you have contacted TAC, what do they have to say about this?

 

BTW, set rstp interface all will work differently between EX4300 (L2 CLI or ELS) and EX2200 (old CLI).  It is the "interface all" part that is not recognized on EX4300, and should commit error (?), which would lead me to believe this is some one of the EX2200 configs.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:21 PM

RSTP is on by default on all interfaces for all EX switches.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:22 PM

I was going to contact them, but at $500 minimum, my company asked me to find a different avenue to solve the issue if possible.  

 

It looks like I need to read up on STP, because frankly, I never dealt with it so have no idea where to start there.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:25 PM

If RSTP is on by default, should I turn it off since it is not meant for more than one vlan?

 

Thanks again

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:27 PM

I'm sorry there isn't much in the way of exact technical suggestions.  If you could modify your diagram to better indicate what ports are connected to what switch and the actual physical connections, perhaps we can make some better suggestions.

 

As I said, STP is not trivial.  Messing with it can cause your entire network to melt down if you don't know what you are doing.  Your best bet is to really ensure that:

 

  1. There are as few physical loops in the network as possible.
  2. The STP versions match between all switches, including the HP.
  3. You have properly set up what you want your root bridge and secondary root bridge to be.

In response to your question about whether to shut off RSTP, the answer at this point is NO!  You need to get a better understanding of how all your switches are physically connected prior to making any STP changes, as well as know what your current root bridge is.

 

Can you do a 'show spanning-tree bridge' from the EX4300 and from an EX2200 and post the output here?

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:28 PM

So you don't have a service contract, . . .  You can tell you bosses for about $25 per switch you could have had JTAC support for a year.

 

RSTP affects vlans, but RSTP operates at interface level, not vlan.  I would NOT suggest you disable RSTP, as based upon what you have stated, I would highly suspect your whole network will come crashing down.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:32 PM

Try reading through the most excellent Day One Book on configuring EX Series switches:

 

http://www.juniper.net/us/en/training/jnbooks/day-one/fabric-switching-tech-series/config-ex-series/

 

There's a brief overview of STP that compares the different types and how to configure them.  It's not in-depth stuff, but it will get you started on a basic understanding.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:34 PM

Ok thanks.  I will research how to identify and setup what switch is the root bridge.  My fear is the network will come crashing down on its own at any given time right now.  Since I inheritted a mess of this network and am trying to upgrade now, I will have to work fast. 

 

I don't know how it was configured previously, so I am trying to clear all that as well.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 12:57 PM

Ok, I read through the STP portion of the guide.  Thank you for that.  So it seems that STP is basically an 8 byte value that all switches use to identify what path to take in the event of a loop? Is that accurate?  

 

It kind of explains an issue I had when replacing the switches.  I upgraded to these from 3com 10/100 switches. There are 2 PCs and 1 printer that when plugged into the new gigabit switches, the ports showed no activity.  I thought maybe an issue with 10/100 vs gigabit, but it looks like the ports stopped listening due to an STP issue?  

 

Anyway, so it looks like I have a loop in my network based on what you guys are saying.  I will have to start there and try to trace it back.  I will read up more on STP as well.

 

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

[ Edited ]
‎12-22-2014 01:47 PM

Some very helpful commands that should be run on each switch in the network to gather data about what is plugged in where:

 

show spanning-tree bridge

show spanning-tree interface

show ethernet-switching table

show ethernet-switching interface

 

STP itself is not an 8-byte value, it's an entire protocol that runs on your layer 2 network to prevent bridging loops from occurring.  You technically do not need STP running if you can absolutely ensure that you will never have a physical loop in the network, but you really need to know your network well and have control over what gets plugged into it before even thinking about disabling it.  There are other methods that you can use for redundant links if you don't want to run STP, but these methods are often situation- or topology-dependent.

 

It would be good to know how you have your vertical stacks of EX2200s configured - are they single standalone switches or are they configured in a virtual chassis?  Same with the EX4300s.  I highly recommend using VC for the stacks of EX2200s, as it makes management much easier, if you can spare the ports on each switch.

 

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 01:54 PM
Ok. That makes more sense. The problem is that I don't know how the cables were ran before me. There's no documentation.

I will run those commands and try to make sense of the results. Is it possible that this issue could cause a printer to suddenly lose network connectivity, but when you plug in another port, it works? Or is it an all or nothing type of issue?

I don't have VC set up on any of the switches. I'll research how to set that up as well if it will make a difference.

Thanks again for the help!
Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

[ Edited ]
‎12-22-2014 01:57 PM

Your printer question - yes, STP can do all sorts of funky stuff you're not expecting.  My first STP problem back in 2000, before I ever knew what it was, was very similar.  I'd plug in or enable a port on one switch and another port on another switch somewhere else in the layer 2 domain would just stop working.

 

Those commands should help you figure out what's what and where, but also try to understand what each is telling you.  It can be daunting, for sure.  Continue to post questions as you go along, if you have any.

 

Also, fix one problem at a time.  First, get the STP issues resolved, then look at VC config.  Don't try to do everything all at once.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

[ Edited ]
‎12-22-2014 02:05 PM

I do have one stupid question. This seems to have happened shortly after I introduced the guest network. The HP switch was set up before me and handled the vlan for another reason. It goes back into the original core which is not listed in my diagram. It consists of one Cisco 2620 and 3 3com switches. I piggyback off of this VLAN for my guest access. now there is a line from each vlan in each switch going to this HP switch which goes into the core eventually. I am now thinking that this is what is causing my loop. Because the main network goes into the core directly, the guest then goes into the core through the HP switch. I'm not at work right now but will run those first thing tomorrow. Again, Thanks for the help.

Highlighted
Ethernet Switching

Re: Confusing VLAN Issue

‎12-22-2014 02:34 PM
If your 2200s and your HP are connected to eachother and the core, there is a loop there. Loops are not bad and many times are necessary. It's just the lack of proper spanning tree config that's the issue.
Feedback