Junos
Junos

MX80 Route Reflector Sudden Flap

‎02-09-2015 02:08 PM

Hi Experts,

All of the sudden our two route reflector (MX80) started to flap. The BGP session with the clients are unaffected, but the session between the route reflectors flaps sometimes within an hour, other times two hours or at the most four hours.

The MTU is the same on both end, and the flap occurs at different intervals. No flaps in OSPF has been found. The RSVP LSP between both route reflectors has been unaffected, but we see that the LDP session get reset almost every time that the BGP session flaps. Below are the outputs.

 

noc@PHX-MX80-01# run show bgp neighbor X.X.X.103
Peer: X.X.X.103+179 AS 1350 Local: X.X.X.101+52827 AS 1350
Type: Internal State: Established Flags: <Sync>
Last State: OpenConfirm Last Event: RecvKeepAlive
Last Error: Hold Timer Expired Error
Export: [ next-hop ]
Options: <Preference LocalAddress LogUpDown AddressFamily Rib-group Refresh>
Address families configured: inet-vpn-unicast inet-labeled-unicast
Local Address: X.X.X.101 Holdtime: 90 Preference: 170
Number of flaps: 319
Last flap event: Restart
Error: 'Hold Timer Expired Error' Sent: 1 Recv: 0
Peer ID: X.X.X.103 Local ID: X.X.X.101 Active Holdtime: 90
Keepalive Interval: 30 Peer index: 0
BFD: disabled, down
NLRI for restart configured on peer: inet-vpn-unicast inet-labeled-unicast
NLRI advertised by peer: inet-vpn-unicast inet-labeled-unicast
NLRI for this session: inet-vpn-unicast inet-labeled-unicast
Peer supports Refresh capability (2)
Restart time configured on the peer: 120
Stale routes from peer are kept for: 300
Restart time requested by this peer: 120
NLRI that peer supports restart for: inet-vpn-unicast inet-labeled-unicast
NLRI that restart is negotiated for: inet-vpn-unicast inet-labeled-unicast
NLRI of received end-of-rib markers: inet-vpn-unicast inet-labeled-unicast
NLRI of all end-of-rib markers sent: inet-vpn-unicast inet-labeled-unicast
Peer supports 4 byte AS extension (peer-as 1350)
Peer does not support Addpath
Table inet.0 Bit: 10000
RIB State: BGP restart is complete
Send state: in sync
Active prefixes: 0
Received prefixes: 0
Accepted prefixes: 0
Suppressed due to damping: 0
Advertised prefixes: 1
Table bgp.l3vpn.0 Bit: 20000
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: in sync
Active prefixes: 15
Received prefixes: 326
Accepted prefixes: 326
Suppressed due to damping: 0
Advertised prefixes: 582
Table ipv4.inet.0
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: not advertising
Active prefixes: 11
Received prefixes: 254
Accepted prefixes: 254
Suppressed due to damping: 0
Table trucom.inet.0
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: not advertising
Active prefixes: 2
Received prefixes: 32
Accepted prefixes: 32
Suppressed due to damping: 0
Table gcta.inet.0
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: not advertising
Active prefixes: 0
Received prefixes: 9
Accepted prefixes: 9
Suppressed due to damping: 0
Table hickman.inet.0
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: not advertising
Active prefixes: 0
Received prefixes: 0
Accepted prefixes: 0
Suppressed due to damping: 0
Table jcl.inet.0
RIB State: BGP restart is complete
RIB State: VPN restart is complete
Send state: not advertising
Active prefixes: 0
Received prefixes: 8
Accepted prefixes: 8
Suppressed due to damping: 0
Last traffic (seconds): Received 4 Sent 4 Checked 86
Input messages: Total 9606 Updates 5616 Refreshes 0 Octets 881367
Output messages: Total 9640 Updates 5597 Refreshes 0 Octets 837692
Output Queue[0]: 0
Output Queue[1]: 0
Output Queue[2]: 0
Output Queue[3]: 0
Output Queue[4]: 0
Output Queue[5]: 0
Output Queue[6]: 0


BGP Trace
Feb 9 14:07:49.082502 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:07:49.082565 BGP SEND X.X.X.101+56143 -> X.X.X.103+179 Keepalive
Feb 9 14:08:14.625332 BGP RECV X.X.X.103+179 -> X.X.X.101+56143
Feb 9 14:08:14.625442 bgp_read_v4_message: done with X.X.X.103 (Internal AS 1350) received 19 octets 0 updates 0 routes
Feb 9 14:08:16.162341 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:08:16.162376 BGP SEND X.X.X.101+56143 -> X.X.X.103+179
Feb 9 14:08:39.972031 BGP RECV X.X.X.103+179 -> X.X.X.101+56143
Feb 9 14:08:39.972144 bgp_read_v4_message: done with X.X.X.103 (Internal AS 1350) received 19 octets 0 updates 0 routes
Feb 9 14:08:45.790718 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:08:45.790780 BGP SEND X.X.X.101+56143 -> X.X.X.103+179
Feb 9 14:09:06.548786 BGP RECV X.X.X.103+179 -> X.X.X.101+56143
Feb 9 14:09:06.548897 bgp_read_v4_message: done with X.X.X.103 (Internal AS 1350) received 19 octets 0 updates 0 routes
Feb 9 14:09:13.325329 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:09:13.325376 BGP SEND X.X.X.101+56143 -> X.X.X.103+179
Feb 9 14:09:35.830561 BGP RECV X.X.X.103+179 -> X.X.X.101+56143
Feb 9 14:09:35.830675 bgp_read_v4_message: done with X.X.X.103 (Internal AS 1350) received 19 octets 0 updates 0 routes
Feb 9 14:09:42.868167 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:09:42.868201 BGP SEND X.X.X.101+56143 -> X.X.X.103+179
Feb 9 14:10:01.876484 BGP RECV X.X.X.103+179 -> X.X.X.101+56143
Feb 9 14:10:01.876591 bgp_read_v4_message: done with X.X.X.103 (Internal AS 1350) received 19 octets 0 updates 0 routes
Feb 9 14:10:11.946676 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:10:11.946711 BGP SEND X.X.X.101+56143 -> X.X.X.103+179
Feb 9 14:10:23.705745 bgp_recv: read from peer X.X.X.103 (Internal AS 1350) failed: Connection reset by peer
Feb 9 14:10:23.705810 bgp_peer_close: closing peer X.X.X.103 (Internal AS 1350), state is 7 (Established)
Feb 9 14:10:23.706349 RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer X.X.X.103 (Internal AS 1350) changed state from Established to Idle (event Restart)
Feb 9 14:10:23.716242 bgp_event: peer X.X.X.103 (Internal AS 1350) old state Idle event Start new state Active
Feb 9 14:10:55.715906 bgp_event: peer X.X.X.103 (Internal AS 1350) old state Active event ConnectRetry new state Connect
Feb 9 14:10:55.717541 bgp_event: peer X.X.X.103 (Internal AS 1350) old state Connect event Open new state OpenSent
Feb 9 14:10:55.717589 advertising receiving-speaker only capabilty to neighbor X.X.X.103 (Internal AS 1350)
Feb 9 14:10:55.717630 bgp_send: sending 67 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:10:55.717661 BGP SEND X.X.X.101+52827 -> X.X.X.103+179
Feb 9 14:10:55.719525 BGP RECV X.X.X.103+179 -> X.X.X.101+52827
Feb 9 14:10:55.719604 BGP RECV version 4 as 1350 holdtime 90 id X.X.X.103 parmlen 38
Feb 9 14:10:55.719790 bgp_event: peer X.X.X.103 (Internal AS 1350) old state OpenSent event RecvOpen new state OpenConfirm
Feb 9 14:10:55.719819 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:10:55.719845 BGP SEND X.X.X.101+52827 -> X.X.X.103+179
Feb 9 14:10:55.719977 bgp_read_message: X.X.X.103 (Internal AS 1350): 0 bytes buffered
Feb 9 14:10:55.720555 BGP RECV X.X.X.103+179 -> X.X.X.101+52827
Feb 9 14:10:55.720712 RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer X.X.X.103 (Internal AS 1350) changed state from OpenConfirm to Established (event RecvKeepAlive)
Feb 9 14:10:55.720997 bgp_send: sending 19 bytes to X.X.X.103 (Internal AS 1350)
Feb 9 14:10:55.721035 BGP SEND X.X.X.101+52827 -> X.X.X.103+179
Feb 9 14:10:55.737677 BGP RECV X.X.X.103+179 -> X.X.X.101+52827
Feb 9 14:10:55.870807 bgp_read_v4_message: counted out X.X.X.103 (Internal AS 1350) received 19116 octets 105 updates 648 routes

 

BGP Config
group RR {
type internal;
local-address X.X.X.101;
family inet {
labeled-unicast;
}
family inet-vpn {
unicast;
}
export next-hop;
neighbor X.X.X.103;
}


RSVP LSP
X.X.X.103
From: X.X.X.101, State: Up, ActiveRoute: 0, LSPname: from-MX80.01-to-MX80.03
ActivePath: (primary)
LSPtype: Static Configured
LoadBalance: Random
Encoding type: Packet, Switching type: Packet, GPID: IPv4
*Primary State: Up
Priorities: 7 0
SmartOptimizeTimer: 180
Computed ERO (S [L] denotes strict [loose] hops): (CSPF metric: 1)
10.2.1.6 S
Received RRO (ProtectionFlag 1=Available 2=InUse 4=B/W 8=Node 10=SoftPreempt 20=Node-ID):
10.2.1.6
5 Jan 31 14:23:40.714 Selected as active path
4 Jan 31 14:23:40.512 Record Route: 10.2.1.6
3 Jan 31 14:23:40.511 Up
2 Jan 31 14:23:40.205 Originate Call
1 Jan 31 14:23:40.205 CSPF: computation result accepted 10.2.1.6
Created: Sat Jan 31 14:23:06 2015

 

LDP Trace
Feb 9 11:23:57.465433 Connection X.X.X.101 state Open -> Closed
Feb 9 11:23:57.465553 Session X.X.X.101 state Closing -> Nonexistent
Feb 9 11:23:58.467204 Session X.X.X.101 state Nonexistent -> Connecting
Feb 9 11:23:58.467290 Connection X.X.X.101 state Closed -> Opening
Feb 9 11:23:58.467468 Connecting to X.X.X.101
Feb 9 11:23:58.468255 Connection X.X.X.101 state Opening -> Open
Feb 9 11:23:58.468353 Session X.X.X.101 state Connecting -> Initialized
Feb 9 11:23:58.468392 Session X.X.X.101 state Initialized -> OpenSent
Feb 9 11:23:58.469601 Session X.X.X.101 GR state Nonexistent -> Operational
Feb 9 11:23:58.469629 Session X.X.X.101 state OpenSent -> OpenRec
Feb 9 11:23:58.470508 Session X.X.X.101 state OpenRec -> Operational
Feb 9 11:23:58.470582 RPD_LDP_SESSIONUP: LDP session X.X.X.101 is up
Feb 9 14:30:44.899563 Interface xe-0/0/3.0 (IFL 2684275848) up
Feb 9 14:30:44.899937 Interface xe-0/0/3.0 (IFL 2684275848) up
Feb 9 14:30:44.900227 Interface ge-1/0/0.0 (IFL 2684275848) up
Feb 9 14:30:44.900430 Interface ge-1/0/0.0 (IFL 2684275848) up
Feb 9 14:30:44.900544 Interface ge-1/0/1.0 (IFL 2684275848) up
Feb 9 14:30:44.900727 Interface ge-1/0/1.0 (IFL 2684275848) up
Feb 9 14:30:44.900838 Interface ge-1/0/2.0 (IFL 2684275848) up
Feb 9 14:30:44.901010 Interface ge-1/0/2.0 (IFL 2684275848) up
Feb 9 14:30:44.901602 Interface xe-0/0/1.0 (IFL 2684275848) up
Feb 9 14:30:44.901931 Interface xe-0/0/1.0 (IFL 2684275848) up

 

 

 

 

5 REPLIES 5
Junos

Re: MX80 Route Reflector Sudden Flap

‎02-09-2015 02:10 PM

Both route reflector are connected through a 10Gb fiber interface.

 

MX80.01 --------XE---------MX80.03

Junos

Re: MX80 Route Reflector Sudden Flap

‎02-09-2015 09:44 PM

It seems that one of my MX is not sending keepalives at some point. Any suggestion on why this might be happening

 

Feb 9 21:42:23.633010 bgp_send: sending 19 bytes to X.X.X.101 (Internal AS 1350)
Feb 9 21:42:23.633038 BGP SEND X.X.X.103+179 -> X.X.X.101+61856 No keepalive from neighbor
Feb 9 21:42:52.395568 bgp_send: sending 19 bytes to X.X.X.101 (Internal AS 1350)
Feb 9 21:42:52.395595 BGP SEND X.X.X.103+179 -> X.X.X.101+61856 
Feb 9 21:42:52.396444 bgp_recv: read from peer X.X.X.101 (Internal AS 1350) failed: Connection reset by peer
Feb 9 21:42:52.396489 bgp_peer_close: closing peer X.X.X.101 (Internal AS 1350), state is 7 (Established)
Feb 9 21:42:52.396613 RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer X.X.X.101 (Internal AS 1350) changed state from Established to Idle (event Restart)
Feb 9 21:42:52.409029 bgp_event: peer X.X.X.101 (Internal AS 1350) old state Idle event Start new state Active
Feb 9 21:43:07.104840 BGP RECV X.X.X.101+57147 -> X.X.X.103+179
Feb 9 21:43:07.104936 BGP RECV version 4 as 1350 holdtime 90 id X.X.X.101 parmlen 38
Feb 9 21:43:07.105229 bgp_event: peer X.X.X.101 (Internal AS 1350) old state Active event Open new state OpenSent
Feb 9 21:43:07.105261 advertising receiving-speaker only capabilty to neighbor X.X.X.101 (Internal AS 1350)
Feb 9 21:43:07.105294 bgp_send: sending 67 bytes to X.X.X.101 (Internal AS 1350)
Feb 9 21:43:07.105353 BGP SEND X.X.X.103+179 -> X.X.X.101+57147
Feb 9 21:43:07.105623 bgp_event: peer X.X.X.101 (Internal AS 1350) old state OpenSent event RecvOpen new state OpenConfirm
Feb 9 21:43:07.105650 bgp_send: sending 19 bytes to X.X.X.101 (Internal AS 1350)
Feb 9 21:43:07.105672 BGP SEND X.X.X.103+179 -> X.X.X.101+57147
Feb 9 21:43:07.106714 BGP RECV X.X.X.101+57147 -> X.X.X.103+179
Feb 9 21:43:07.106851 RPD_BGP_NEIGHBOR_STATE_CHANGED: BGP peer X.X.X.101 (Internal AS 1350) changed state from OpenConfirm to Established (event RecvKeepAlive)

Junos

Re: MX80 Route Reflector Sudden Flap

‎02-10-2015 02:51 AM

I don't see anything that really jumps out at me as an issue.  Maybe try setting up BFD on your BGP session to see if there is some sort of connectivity-loss between the boxes?

JNCIE-SEC #127
JNCIE-ENT #489
Junos

Re: MX80 Route Reflector Sudden Flap

‎05-17-2015 12:47 AM

[edit]
tcannon@SRX-B# run show interfaces xe-0/0/0 extensive | match Flap   
  Last flapped   : 2015-05-16 05:30:40 CEST (1d 12:13 ago)

 

If that date is changing, you have a hardware problem.

Junos

Re: MX80 Route Reflector Sudden Flap

‎02-12-2016 09:21 AM

Hi.

 

I have exactly the same problem. The bgp sessions between these 2 routers flap once a day.

Bgp sessions between other routers didnt flap.

 

Any suggestion?