Routing
Highlighted
Routing

Strange issue in mx960 bundle and Interface Flapping.

‎05-10-2020 09:56 AM

hi All,

 

we are facing some strange and random issue as Physical and Bundle flapping. Below are the logs we are getting.

 

May 10 11:58:29.806 kernel: %KERN-4: ae_linkstate_ifd_change: MDOWN received for interface xe-8/0/11, member of ae50
May 10 11:58:29.806 kernel: %KERN-4-: port status changed
May 10 11:58:29.806 kernel: %KERN-4: ae_bundlestate_ifd_change: bundle ae50: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1 Number of links (Current : Required) 0 : 1
May 10 11:58:29.807 cosd[14823]: %DAEMON-7: link protection 0 for intf ae50
May 10 11:58:29.816 kernel: %KERN-5-KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-8/0/11 - DETACHED state - will not carry traffic
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT <UpDown> ae50.0 index 361 <Broadcast Multicast> address #0 32.b6.4f.38.b0.ca
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT UpDown ae50.0 index 361 125.18.109.177/30 -> 125.18.109.179 <Broadcast Multicast Localup>
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT UpDown ae50.0 index 361 2404:a800:3a00:1::385/126 -> zero-len <Broadcast Multicast Localup>
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT UpDown ae50.0 index 361 fe80::32b6:4fff:fe38:b0ca/64 -> zero-len <Broadcast Multicast Localup>
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT <Bandwidth UpDown> ae50 index 330 <Broadcast Multicast> address #0 32.b6.4f.38.b0.ca
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT <Bandwidth> ae50.0 index 361 <Broadcast Multicast> address #0 32.b6.4f.38.b0.ca
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: KRT Ifstate: Received IPv6 address 2404:a800:3a00:1::385 on ifl ae50.0. Flag:2.
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: KRT Ifstate: Received IPv6 address fe80::32b6:4fff:fe38:b0ca on ifl ae50.0. Flag:2.
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT <UpDown> xe-8/0/11.0 index 475 <Broadcast Multicast> address #0 32.b6.4f.38.b0.ca
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: EVENT <UpDown> xe-8/0/11 index 316 <Broadcast Multicast> address #0 32.b6.4f.38.b0.ca
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: STP handler: Stp index=159, op=change, state=Discarding, Topo change generation=0
May 10 11:58:29.819 rpd[14819]: %DAEMON-6: *STP Change*, notify to other modules
May 10 11:58:29.838 mib2d[14851]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 894, ifAdminStatus up(1), ifOperStatus down(2), ifName ae50
May 10 11:58:29.838 mib2d[14851]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 877, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-8/0/11
May 10 11:58:29.967 rpd[14819]: %DAEMON-6: STP handler: Stp index=159, op=change, state=Discarding, Topo change generation=0
May 10 11:58:29.967 rpd[14819]: %DAEMON-6: *STP Change*, notify to other modules
May 10 11:58:30.266 rpd[14819]: %DAEMON-4: bgp_ifachange_group:7836: NOTIFICATION sent to 125.18.109.178 (External AS 55427): code 6 (Cease) subcode 6 (Other Configuration Change), Reason: Interface change for the peer-group
May 10 11:58:30.281 : %PFE-3: fpc1 PFE0_XF-chan-rx-75 read errors.crc_error[chan] 0x105410 <- 0x0 failed
May 10 11:58:30.281 : %PFE-3: fpc1 PFE0_XF-chan-rx-15 read errors.crc_error[chan] 0x101410 <- 0x0 failed
May 10 11:58:30.281 : %PFE-3: fpc1 PFE0_XF-chan-rx-90 read errors.crc_error[chan] 0x106410 <- 0x0 failed
May 10 11:58:30.266 rpd[14819]: %DAEMON-5-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 125.18.109.178 (External AS 55427) changed state from Established to Idle (event Stop) (instance master)
May 10 11:58:30.266 rpd[14819]: %DAEMON-3-BGP_WRITE_FAILED: bgp_send: sending 21 bytes to 2404:a800:3a00:1::386 (External AS 55427) failed: Can't assign requested address
May 10 11:58:30.266 rpd[14819]: %DAEMON-5-RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer 2404:a800:3a00:1::386 (External AS 55427) changed state from Established to Idle (event Stop) (instance master)
May 10 11:58:30.266 rpd[14819]: %DAEMON-4: bgp_ifachange_group:7836: NOTIFICATION sent to 2404:a800:3a00:1::386 (External AS 55427): code 6 (Cease) subcode 6 (Other Configuration Change), Reason: Interface change for the peer-group
May 10 11:58:30.406 : %PFE-3: fpc1 PFE0_XF-chan-rx-0 read errors.crc_error[chan] 0x100410 <- 0x0 failed
May 10 11:58:30.501 : %PFE-3: fpc1 PFE0_XF-chan-rx-105 read errors.crc_error[chan] 0x107410 <- 0x0 failed
May 10 11:58:30.542 : %PFE-3: fpc1 PFE0_XF-chan-rx-45 read errors.crc_error[chan] 0x103410 <- 0x0 failed
May 10 11:58:31.149 kernel: %KERN-4: ae_linkstate_ifd_change: MUP received for interface xe-8/0/11, member of ae50
May 10 11:58:31.149 kernel: %KERN-4-: port status changed
May 10 11:58:31.238 mib2d[14851]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 877, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-8/0/11
May 10 11:58:31.240 rpd[14819]: %DAEMON-6: EVENT <UpDown> xe-8/0/11.0 index 475 <Up Broadcast Multicast> address #0 32.b6.4f.38.b0.ca
May 10 11:58:31.240 rpd[14819]: %DAEMON-6: EVENT <UpDown> xe-8/0/11 index 316 <Up Broadcast Multicast> address #0 32.b6.4f.38.b0.ca
May 10 11:58:31.238 mib2d[14851]: %DAEMON-5-SNMP_TRAP_LINK_UP: ifIndex 908, ifAdminStatus up(1), ifOperStatus up(1), ifName xe-8/0/11.0

May 10 11:58:33.374 kernel: %KERN-5-KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-8/0/11 - ATTACHED state - acting as standby link
May 10 11:58:33.374 kernel: %KERN-5-KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-8/0/11 - CD state - ready to carry traffic

 

Please suggest what could be the reason and what kind of Mdown packet is?

Regards
Aman Goel
4 REPLIES 4
Highlighted
Routing

Re: Strange issue in mx960 bundle and Interface Flapping.

‎05-10-2020 06:28 PM

Hi Aman,

 

From the log message that you sent, it appears to me that this could be a prospective physical layer issue. The AE bundle is seen down every time the physical interface xe-8/0/11 goes down because the minimum links(child interfaces) that should be up, for an aggregated interface to remain up is "1" by default. In your case, I believe xe-8/0/11 is the only interface in the bundle. So if this interface goes down, the AE bundle also goes down. And that is why every time there is a flap in the physical interface, you will see the bundle flap.

 

I believe this could be a physical layer issue. In case the issue is seen again and your physical link goes down, please check w.r.t:

1) Optics/cables

2) Patch panels

3) Connection on the remote end

5) Optics on the remote end

6) Another port on the same local device (to verify if the issue lies with xe-8/0/11)

7) Check if the packet count entering and exiting the device is increasing.

8) Loopback testing( connecting TX and RX of the same port in a loop) and monitor the packet count 

 

If the above doesn't work, please take a look at the below document. It specifies step by step procedure to troubleshoot LAG ( you can refer the same for MX )

https://kb.juniper.net/InfoCenter/index?page=content&id=KB19798

 

In case you are seeing frequent interface flaps and the above link does not solve your issue, we might have to take a deeper look at the logs and collect some captures which might need you to open a JTAC case.

 

Hope this helps.

 

Please mark this "Accepted Solution" if this serves your purpose.

Kudos would be much appreciated too Smiley Happy

 

 

Highlighted
Routing

Re: Strange issue in mx960 bundle and Interface Flapping.

‎05-10-2020 06:50 PM

Hello Aman,

 

Greetings!

 

Basically, the issue started with the physical interface going down [xe-8/0/11] which was a part of AE50. AE50 went down as there was no other interface present in the AE to hold it up. After the AE bundle going down, the following events took place and which were expected:

1. STP topo change was observed.

2. BGP changed from Established to Idle state related to the AE bundle.

 

To resolve the issue need to fix the physical interface flap on xe-8/0/11.

 

Is the physical interface back up? if yes, check if the BGP session has changed to Established.

How often do you see a flap on xe-8/0/11?

To check the last flapped time, use the command: show interfaces xe-8/0/11 extensive |match "Last flapped"

 

If there is a continuous port flap, then I would recommend the below initial steps of troubleshooting to eradicate the optics issue:-

  • Remove the optics, clean it, and insert it back again.  
  • Perform a hard loop/soft loop and check the behavior of the optics.

https://www.juniper.net/documentation/en_US/junos/topics/topic-map/ethernet-fast-and-gigabit-loopbac...

 

I hope this helps. Please mark this post "Accept as solution" if this answers your query.

 

Kudos are always appreciated! Smiley Happy

 

Best Regards,

Lingabasappa H

 

Highlighted
Routing

Re: Strange issue in mx960 bundle and Interface Flapping.

‎05-11-2020 12:27 PM

Hi Aman,

 

Also share below command outputs to review along with the performing above steps shared by bmanvita and shlinga.

 

>show interface diagnostics optics xe-8/0/11

>show lacp interfaces ae50

>show configuration interfaces ae50

>show configuration interfaces xe-8/0/11

>show lacp timeouts ae50

 

 

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too

 

Kudos are appreciated too Smiley Happy

 

Regards,

Nadeem

Highlighted
Routing

Re: Strange issue in mx960 bundle and Interface Flapping.

‎05-15-2020 06:50 AM

Hello Aman,

 

Greetings,

 

Did you perform the steps suggested for troubleshooting? Did the physical interface xe-8/0/11 stop flapping?

Please update the thread and if the replies have helped you in troubleshooting.

 

Please do acknowledge the answers that help. If an answer solves your problem, post a short reply, and rate the posting. It will help others with the same or similar question, to know what information has been helpful.

 

If my posts have helped you resolve the issue, Please mark those posts "Accept as solution" if this answers your query.

 

Kudos are always appreciated! Smiley Happy

 

Best Regards,

Lingabasappa H