Routing
Highlighted
Routing

Loss of OSPF & BGP due to high traffic

‎05-13-2009 03:36 AM

This morning at approx 06:30 we experienced a DDOS

We saw 1G of traffic coming in and being routed on, but this seems to have caused both OSPF and BGP to drop.

Network control traffic should be prioritized, so why when the physical pipe was full, did OSPF adjacencies drop?

 

It was on a M7i with 8.0

Diggers
11 REPLIES 11
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-13-2009 03:41 AM
Could you post the "show interface queue" output for both ends of the interfaces that dropped the OSPF neighborship. This could give an indication as to which queues had the traffic and why OSPF might have dropped.
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-13-2009 05:54 AM

Here we go

 

show interfaces queue ge-1/3/0 Physical interface: ge-1/3/0, Enabled, Physical link is Up

Interface index: 135, SNMP ifIndex: 29

Description:

Forwarding classes: 4 supported, 4 in use Egress queues: 4 supported, 8 in use

Queue: 0, Forwarding classes: best-effort

Queued:

Packets : 1134314941198 91590 pps

Bytes : 648814768203780 448164584 bps

Transmitted:

Packets : 1134314941276 91590 pps

Bytes : 669216841860446 461350760 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

Queue: 1, Forwarding classes: expedited-forwarding

Queued:

Packets : 9768512304 451 pps

Bytes : 1849439362897 608944 bps

Transmitted:

Packets : 9768512304 451 pps

Bytes : 2025272584369 673936 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

Queue: 2, Forwarding classes: assured-forwarding

Queued:

Packets : 1279261811 369 pps

Bytes : 841412916410 3711936 bps

Transmitted:

Packets : 1279261811 369 pps

Bytes : 864439629008 3765072 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

Queue: 3, Forwarding classes: network-control

Queued:

Packets : 1231695888 27 pps

Bytes : 179971549885 37952 bps

Transmitted:

Packets : 1231695888 27 pps

Bytes : 200709902755 41296 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

 

And the other end :-

show interfaces queue ge-0/1/0 Physical interface: ge-0/1/0, Enabled, Physical link is Up

Interface index: 129, SNMP ifIndex: 38

Description: 

Forwarding classes: 4 supported, 4 in use Egress queues: 4 supported, 4 in use

Queue: 0, Forwarding classes: best-effort

Queued:

Packets : 222765753691 27938 pps

Bytes : 121434808503804 130873816 bps

Transmitted:

Packets : 222765753691 27936 pps

Bytes : 124552873654238 134002896 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

Queue: 1, Forwarding classes: expedited-forwarding

Queued:

Packets : 6581966048 913 pps

Bytes : 1248536415860 1088856 bps

Transmitted:

Packets : 6581966048 913 pps

Bytes : 1340683940532 1191128 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

Queue: 2, Forwarding classes: assured-forwarding

Queued:

Packets : 12972389 8 pps

Bytes : 7110607511 7328 bps

Transmitted:

Packets : 12972389 8 pps

Bytes : 7292220957 8256 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

Queue: 3, Forwarding classes: network-control

Queued:

Packets : 310662055 15 pps

Bytes : 30498450022 10800 bps

Transmitted:

Packets : 310662055 15 pps

Bytes : 34771920160 12544 bps

Tail-dropped packets : 0 0 pps

RED-dropped packets : 0 0 pps

Low, non-TCP : 0 0 pps

Low, TCP : 0 0 pps

High, non-TCP : 0 0 pps

High, TCP : 0 0 pps

RED-dropped bytes : 0 0 bps

Low, non-TCP : 0 0 bps

Low, TCP : 0 0 bps

High, non-TCP : 0 0 bps

High, TCP : 0 0 bps

ianmac@NBG01-BDR-01>

Diggers
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-13-2009 06:01 AM
Looking at the output of these 2 interfaces there are no drops at all (assuming the counters have not been cleared). If the DDOS had completely saturated this link, than there should be some drops observable here somewhere. Maybe the log messages can give you more clues as to why OSPF and BGP dropped during the DDOS attack.
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-13-2009 06:10 AM

 

The only messages are of the timers expiring

 

May 13 06:09:04 rpd[3042]: bgp_read_v4_message: NOTIFICATION received from x.x.x.x (External AS x): code 4 (Hold Timer Expired Error), socket buffer sndcc: 57 rcvcc: 0 TCP state: 5, snd_una: 766943443 snd_nxt: 766943500 snd_wnd: 16384 rcv_nxt: 157418040 rcv_adv: 157434351, keepalive timer 26.508876

 

May 13 06:26:41 rpd[3042]: RPD_OSPF_NBRDOWN: OSPF neighbor x.x.x.x (ge-1/3/0.324) state changed from Full to Down due to InActiveTimer (event reason: neighbor was inactive and declared dead)

Diggers
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-13-2009 06:17 AM

I see that interface ge-1/3/0 is using vlans. Is it possible that it is connected to a switch and that the problems occured there?

 

What is also interasting is the time between the loss of BGP and OSPF. They did not occur at the same time which seem to rule out the possibility that the Routing Engine was under any extreem stress for a short period of time.

Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-13-2009 07:38 AM

The device conneacted is another M7i, which only had 3 ospf alarms

 

The alarms provided was just a snippet after those there were a lot more BGP timer expired alarms

Diggers
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-13-2009 07:42 AM

Do you see any lost packets under

 

 

show pfe statistics traffic

 

 

 

Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-14-2009 01:22 AM

Forwarding Engine traffic statistics:

 

Input packets: 1506264174732 68806 pps

Output packets: 1490420401209 68621 pps

 

Packet Forwarding Engine local traffic statistics:

 

Local packets input : 4340803041

Local packets output : 1242940223

Software input control plane drops : 0

Software input high drops : 0

Software input medium drops : 130244

Software input low drops : 0

Software output drops : 0

Hardware input drops : 10605363

 

Packet Forwarding Engine local protocol statistics:

 

HDLC keepalives : 0

ATM OAM : 0

Frame Relay LMI : 0

PPP LCP/NCP : 0

OSPF hello : 34566065

OSPF3 hello : 0

RSVP hello : 0

LDP hello : 0

BFD : 144 IS-IS

IIH : 0

 

Packet Forwarding Engine hardware discard statistics:

Timeout : 0

Truncated key : 0

Bits to test : 0

Data error : 0

Stack underflow : 0

Stack overflow : 0

Normal discard : 5277297962

Extended discard : 93943741

Invalid interface : 0

Info cell drops : 0

Fabric drops : 0

Diggers
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎05-14-2009 01:52 AM
Well from this output we can see some traffic drops associated with the PFE. The question  however is if these resulted in OSPF and BGP neighborships going done. Unfortunately this is not an easy question to answer. I would suggest you open a JTAC case to investigate this further.
Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎12-20-2013 06:26 AM

This has also happened to me recently and I have a ticket open with jtac for a month now trying to find a fix... so far there has been none.  Did you ever find a reason/fix?

Highlighted
Routing

Re: Loss of OSPF & BGP due to high traffic

‎12-21-2013 12:44 AM

Do you have any features like BFD configured? I am wondering if the ddos attack caused a longer than normal latency which could result in the neighbors and peer communication delyed thus resulting in the timeouts?

[KUDOS PLEASE! If you think I earned it!
If this solution worked for you please flag my post as an "Accepted Solution" so others can benefit..]