Junos
Junos

Process bbe-mibd with high cpu utilization

‎05-29-2019 03:57 AM

Model: mx104
Junos: 18.1R3-S4.2

 

Hello folks,

 

We have a MX104 w/ 18.1R3-S4.2 playing the role as subscriber management for our network. We have about 10k  end-users terminating they PPPoE tunnels on this box.

We are using LACP + DMUX interfaces. Some users come from normal dot1q vlans, and some users come from QinQ vlans. 

 

After a massive user disconnection  (eletric residential outage), the users weren't able to setup the pppoe connection.

 

Looking at the logs doesn't help much. The radius server was online, and reporting access-accept when using the "test aaa ppp" command. The only process that was "different" is bbe-mibd, consuming from 60% to 75% in the "show system processes extensive". 

 

To workaround, we shut down all vlans, and bring up it in a controlled manner.

 

Now , we have 10k users online, but the bbe-mibd process is still using +60% of the CPU.

 

 

Any ideas?

 

*I have attached the dynamic-profile configuration

Attachments

7 REPLIES 7
Junos

Re: Process bbe-mibd with high cpu utilization

‎05-29-2019 04:45 AM

Hi,

 

Did you noticed any DDoS violation during the time of issue? What about the CPU apart from bbe-mibd? 

Any other process reported high CPU like bbe-smgd or authd?

 

Regards,

Rahul

Junos

Re: Process bbe-mibd with high cpu utilization

‎05-29-2019 05:10 AM

Yes, some ddos-protection for protocol pppoe, and sample-pfe (this is still active, I've opened another thread relating it).

 

The other process are fine (for me):

 

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
6258 root 76 0 58568K 27776K RUN 172:46 57.91% bbe-mibd
3223 root 76 0 228M 138M RUN 85:58 7.71% bbe-smgd
1957 root 76 0 182M 67376K RUN 62:36 6.84% jdhcpd
3267 root 74 0 292M 174M RUN 37:06 3.56% authd
1933 root 43 0 109M 35980K select 43:33 2.88% chassisd
2182 root 70 0 259M 136M RUN 34:18 2.64% jpppd
2141 root 4 0 743M 619M kqread 30:30 1.22% rpd
3223 root 54 0 228M 138M RUN 85:58 1.12% bbe-smgd
1943 root 43 0 79440K 37928K select 14:52 1.03% pfed
Junos

Re: Process bbe-mibd with high cpu utilization

‎05-29-2019 06:19 AM

Can you please add following filter and check once?

 

user@router> show configuration snmp filter-interfaces 
interfaces {
    pp*;
    demux*;
    ".*\.[0-9][0-9][0-9][0-9][0-9][0-9]*";
    "ae[0-9]*";
    si-*;
}
all-internal-interfaces;
Junos
Solution
Accepted by topic author rganascim
‎05-31-2019 01:07 AM

Re: Process bbe-mibd with high cpu utilization

‎05-29-2019 06:21 AM

What is the subscriber scale you have on MX104? I can see you've configured interface-mib as well. Please disable it and monitor the bbe-mib process. Subscribers should relogin for changes to get effect.

Junos

Re: Process bbe-mibd with high cpu utilization

‎05-29-2019 11:56 AM

I applied the snmp interface filter, but the process continues to consume the cpu.

Junos

Re: Process bbe-mibd with high cpu utilization

‎05-29-2019 11:57 AM

l will try this night to disable interface-mib from dynamic-profile.

 

Thanks.

Junos

Re: Process bbe-mibd with high cpu utilization

‎05-31-2019 01:06 AM

I logged off all subscribers (removing the vlans from switch), then I disabled the interface-mib option in the dynamic-profile.

Without any online user, the process remained using +70% CPU. 

 

So, I killed the process via "kill PID", and the CPU fallen down to low usage.

 

Then, I put back the users online, and with less than 5 min, all the 11k subscribers was up and running. 

 

The cpu usage is ok now:

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 9262 root      -8    0   269M   177M biowr  169:22 23.44% bbe-smgd
   10 root     155   52     0K    16K RUN     19:30 16.89% idle
 1957 root      -4    0   183M 70868K getblk 322:41 15.38% jdhcpd
 9268 root      43    0   300M   182M select  69:30  9.13% authd
 2182 root      43    0   266M   145M select 153:17  8.15% jpppd
 1933 root      43    0   109M 35612K select 166:24  3.08% chassisd
 9262 root      59    0   269M   177M RUN    169:22  3.03% bbe-smgd
 2141 root       4    0   751M   625M kqread  95:23  2.64% rpd
   11 root     -56 -159     0K    16K WAIT    65:04  2.00% swi2: netisr 0
   30 root     -84 -187     0K    16K WAIT    52:43  1.71% irq96: fman0

Thanks for the help!