Switching

So about an hour ago, every AP (HP MSM410s) started rebooting every 2 minutes.  20 minutes later it stops.  Everything is normal. 

Background:  10 APs are connected through 5 different EX2200 POE switches.  All POE switches show similar to the attached log.  Basically, the ports that the APs are connected to have repeated Event IDs of SNMP_TRAP_LINK_DOWN.

HP looked at their logs from the APs and had no idea.  They are lost.  I don't know if I need to be concerned with the SNMP_TRAP_LINK_DOWN or if its just a warning that the port went down. 

It may (or may not be worth noting) that the server room was 92 degrees this morning.  There are 2 core 4300 switches in there, but none of the POE switches are in there.  Also, the wireless went down briefly this morning at 9ish, but only for 2 minutes.

I hope someone can help me because I don't know when it may happen again and am very confused.

I also attached the MSM410 log incase anyone has any HP expertise.

Attachment(s)

log during reboot.docx   32 KB 1 version

ex2200 log.docx   18 KB 1 version

You may want to open a JTAC case and see if anything pops up in the logs of EX2200, although I find it hard to believe that there is a direct relation between the EX2200 and the APs that could have caused this outage.

The logs you mentioned are just notifications that the interface went down which is expected if the APs rebooted.

Are the APs supposed to reboot in case they loose the control connection with the wireless controller? In case they do, then maybe there was a network connectivity which caused this and this might require to understand what happened in the network just before the event.

=====

If this worked for you please flag my post as an "Accepted Solution" so others can benefit. A kudo would be cool if you think I earned it.

Thanks for your response.  

There is no controller.  All APs are in Autonomous mode.  I came in this morning to it happening again.  The only thing that I noted that was similar to yesterday was that maintenance was running new power lines in the ceiling.  Is it possible if they were running the cable over another cable, it could cause enough noise to throw a POE switch's power out off?

Thanks

That's an interesting approach.

From top of my head, I have no idea if this is possible, but worth checking with JTAC.

=====

If this worked for you please flag my post as an "Accepted Solution" so others can benefit. A kudo would be cool if you think I earned it.

I will get in touch with them.  My company wants to avoid spending money on their support if possible.  I don't think it is a Juniper issue anyway, but figured I would ask here.  

Thanks

Ok, so it has been happening daily now since Monday.  I have found out the root cause of the rebooting is an error on each AP that says "ath_bstuck_tasklet: stuck beacon; hard resetting (bmiss count 18)".  HP hasn't come up with a solution, but someone on the HP forums has mentioned that they had an issue exactly like this, their fix was to enable IGMP snooping on their switches.  

From what I can tell, they are enabled by default on the EX2200 and EX4300 switches, can someone confirm this?  

Thanks

Not the same problem, but I had situations to DISABLE igmp snooping instead of enabling it - to connect non-igmp multicast traffic across swtiches. Default is enabled.

Regards,

/Charlie

Now I know the issue, but can't narrow down one thing.  Turns out an ipv6 address is spamming the network with multicast listener reports, killing the CPUs on the access points.  The question is, is there an IGMP snooping for ipv6?

Yes there is.  For IPv6 it is called MLD Snooping (https://www.ietf.org/proceedings/52/slides/magma-5.pdf).  For configuration info on EX look at:

http://www.juniper.net/documentation/en_US/junos12.1/topics/task/configuration/mld-snooping-vlan-cli.html

Thanks!  I will enable that and see where I get.

Got it all figured out now.  Turns out we put a few HP EliteDesk 800's out in our network and there is a huge issue with the Intel I217 ethernet cards in them.  See https://social.technet.microsoft.com/Forums/windows/en-US/945125a7-035f-4f33-a9d2-c274717d7706/icmpv6-multicast-listener-report-messages-are-flooding-the-local-network?forum=w7itpronetworking

Updated drivers to see if that takes care of it for now.  Thanks