Ethernet Switching
Highlighted
Ethernet Switching

EX2200 lost management interface after ethernet loop

‎08-15-2016 01:43 PM

Hi all

 

I had a loop on my network. Once I sorted that out, I discovered that three of my EX2200 switches were no longer reachable through the network. They are responsive to serial console connections.

 

One of those switches was in a lab, so I messed around with it for a bit but failed to get anything to work. I eventually rebooted it, and of course that works.

 

'restart management immediately' just seems to reload the active cli process. 'management' does not appear as a offered option when pressing ? after entering "restart".

 

The switch does report that the interface is up:

 

root@sa4-39> show interfaces me0
Physical interface: me0, Enabled, Physical link is Up
  Interface index: 1, SNMP ifIndex: 33
  Type: Ethernet, Link-level type: Ethernet, MTU: 1514, Speed: 100mbps
  Device flags   : Present Running
  Interface flags: SNMP-Traps
  Link type      : Full-Duplex
  Current address: f0:1c:2d:bc:f5:bf, Hardware address: f0:1c:2d:bc:f5:bf
  Last flapped   : 2016-08-15 14:41:15 EDT (01:45:26 ago)
    Input packets : 157777142
    Output packets: 18168943

  Logical interface me0.0 (Index 3) (SNMP ifIndex 34)
    Flags: SNMP-Traps Encapsulation: ENET2
    Input packets : 157777142
    Output packets: 18178196
    Protocol inet, MTU: 1500
      Flags: Is-Primary
      Addresses, Flags: Is-Default Is-Preferred Is-Primary
        Destination: 172.x.x/24, Local: 172.x.x.x, Broadcast: 172.x.x.x

All three affected switches are running 12.3R6.6.

 

All three affected switches are using out-of-band management, plugged into one of the ge- ports on the same switch

 

When the interface is plugged/unplugged, the switch records up/down activity on both the me interface and the ge interface that it is plugged in to.

 

Swapping the connection to another, similarly configured ge port has no effect; plugging other things into the ge port works normally. The switch appears to be switching properly; configuration changes made through the console appear to be taking effect properly.

 

Only one of the affected switches was directly involved in the network loop, although all switches saw the resulting storm.

 

Naturally, I would prefer not to reboot these switches. Does anyone know how I might resurrect the network connectivity of the management interface without a reboot?

6 REPLIES 6
Highlighted
Ethernet Switching

Re: EX2200 lost management interface after ethernet loop

‎08-15-2016 11:57 PM

When you said "not reachable" what actually did you mean?
1. Direct ip connectivity lost or lost access from remote networks via routing?
2. Try to ping yourself (me0.0 interface) during the problem.

Highlighted
Ethernet Switching

Re: EX2200 lost management interface after ethernet loop

‎08-16-2016 02:10 AM

HI Sir,

 

Couple of things to chck

 

1) From the switch "show interface terse | match inet"  >  Does this show your mgmt IP?

2) Can you ping the mgmt IP from switch itself. > ping mgmt ip address

3) show interface me0 extensive

4) show log messages | match me0

 

One more way to resolve this without reboot would be delete and reconfigure me0.

Before this try this in shell

ifconfig me0 down

ifconfig me0 up

 

Thanks

Partha

Highlighted
Ethernet Switching

Re: EX2200 lost management interface after ethernet loop

‎08-16-2016 06:09 AM

Sorry for the short lead time on this, but we ran out of time and were forced to eat the reboot. If/when it happens again I'll dig further into these items.

 

Thanks for the replys anyways.

Highlighted
Ethernet Switching

Re: EX2200 lost management interface after ethernet loop

[ Edited ]
‎03-27-2017 11:14 AM

Just in case someone's watching, I found another switch in this state.

 

From your list:

 

1) Yes

2) No, times out

3) See below

4) No relevant logs (I found this switch many weeks after it went missing on the network)

 

Also, the down/up cycle didn't help either.

 

Extensive output:

 

{master:0}
root@ex2200-03> show interfaces me0 extensive
Physical interface: me0, Enabled, Physical link is Up
  Interface index: 1, SNMP ifIndex: 33, Generation: 1
  Type: Ethernet, Link-level type: Ethernet, MTU: 1514, Clocking: Unspecified,
  Speed: 100mbps
  Device flags   : Present Running
  Interface flags: SNMP-Traps
  Link type      : Full-Duplex
  Physical info  : Unspecified
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 54:e0:32:8f:af:bf, Hardware address: 54:e0:32:8f:af:bf
  Alternate link address: Unspecified
  Last flapped   : 2017-03-27 18:07:59 UTC (00:03:03 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :          23344192147
   Output bytes  :                    0
   Input  packets:            305350394
   Output packets:              6395590
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0,
    Policed discards: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 9, Errors: 1, Drops: 0, MTU errors: 0,
    Resource errors: 0

  Logical interface me0.0 (Index 3) (SNMP ifIndex 34) (HW Token 4294967295)
   (Generation 1)
    Flags: SNMP-Traps Encapsulation: ENET2
    Bandwidth: 0
    Traffic statistics:
     Input  bytes  :          23344192147
     Output bytes  :            923271368
     Input  packets:            305350394
     Output packets:              8221098
    Local statistics:
     Input  bytes  :          23344192147
     Output bytes  :            923271368
     Input  packets:            305350394
     Output packets:              8221098
    Protocol inet, MTU: 1500, Generation: 141, Route table: 0
      Flags: Is-Primary

      Addresses, Flags: Is-Default Is-Preferred Is-Primary
        Destination: 10.17.0/20, Local: 10.17.15.250, Broadcast: 10.17.15.255,
        Generation: 1

 

Also interesting, the management interface seems to have ARP entries for remote systems even though it won't actually use them.

 

 

 

Highlighted
Ethernet Switching

Re: EX2200 lost management interface after ethernet loop

‎03-29-2017 03:03 AM

Can you provide the output of "show route" from the affected switch?

Ben Dale
JNCIP-ENT, JNCIP-SP, JNCIP-DC, JNCIE-SEC #63
Juniper Ambassador
Follow me @labelswitcher
Highlighted
Ethernet Switching

Re: EX2200 lost management interface after ethernet loop

‎03-29-2017 07:23 PM

HI

 

Could you please check this

 

1) Ping to the mgmt IP from the PC and run "monitor traffic interface me0 no-resolve" and see if ICMP packets are coming in.

2) Also get the output of show route.

3) Can you check the cpu occupancy.

4) show log messages | match me

5) show interfaces terse | match inet

6) Ping from the box itself to its own IP.

 

Thanks
Partha 

Feedback