Switching

Hello all,

 

one of my EX4200-virtual-chassis just melt down. The chassis consists of two EX4200 switches connecting vice-versa over vcp0 and vcp1. 

 

no-split-detection;
member 0 {
    mastership-priority 255;
}
member 1 {
    mastership-priority 240;
}

 

My member0 started to log a few of this entries:

2:web 20:33:01 VC /kernel: Mbuf: High Utililization Level: (Low) Throttling low priority requests (10 ms)
2:web 20:34:10 VC /kernel: Mbuf: High Utililization Level: (Low) Throttling low priority requests (10 ms)
2:web 20:34:29 VC /kernel: Mbuf: High Utililization Level: (Low) Throttling low priority requests (10 ms)
2:web 22:33:14 VC /kernel: Mbuf: High Utililization Level: (Medium) Throttle low priority requests (150 ms)
0:web 23:45:49 VC /kernel: Mbuf: High Utililization Level: (Low) Throttling low priority requests (10 ms)

 

Three hours after the first log line, member0 became almost unresponsive and the link to member1 was broken. I could not login remotely and needed to connect over the serial interface. member0 was hardly printing out chars and needed to be rebooted to bring back life in the whole chassis. member1 was very responsive and showed member0 as missing link. 

 

My software is 12.3R6.6 and I am using NSR and GRES. I found a related PR https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR886083  , but it says it had been fixed a year ago. 

 

Somebody has another clue or witnessed this one before?

 

I checked the mbuf usage right now and from all my chassis clusters, the member0 is having a horrible high mbuf usage count:

 

fpc0:
--------------------------------------------------------------------------
76499/301/76800 mbufs in use (current/cache/total)
265/445/710/45600 mbuf clusters in use (current/cache/total/max)
256/128 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
24413K/965K/25378K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/4/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied            
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
0 mbuf requests delayed
0 cluster requests delayed

fpc1:
--------------------------------------------------------------------------
547/998/1545 mbufs in use (current/cache/total)
258/484/742/45600 mbuf clusters in use (current/cache/total/max)
256/256 mbuf+clusters out of packet secondary zone in use (current/cache)
0/0/0/0 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/0 9k jumbo clusters in use (current/cache/total/max)
0/0/0/0 16k jumbo clusters in use (current/cache/total/max)
664K/1217K/1882K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/4/6656 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
0 mbuf requests delayed
0 cluster requests delayed

 

 

Regards.

 

I have not seen that particular issue, but if it were me, I would setup some firewall filters on the loopback interface to see if I could identify some particular traffic hammering the RE.  Put counters on each term being allowed, and maybe apply some policers to rate-limit particular traffic if you identify something unusual.

 

Ron

Any chance you have a spanning tree loop that you might not be aware of? Or maybe an errant switch that is connected and speaking a different version of STP than your VC cluster?

We really did a change in our VSTP-configuration a few days ago and forgot to add one specific VLAN. We corrected that but saw no difference in leaking mbufs. We did a complete chassis reload then and now the issue seems gone. 

 

Thank you very much!