Junos OS

Model: mx104
Junos: 18.1R3-S4.2

Hello folks,

We have a MX104 w/ 18.1R3-S4.2 playing the role as subscriber management for our network. We have about 10k  end-users terminating they PPPoE tunnels on this box.

We are using LACP + DMUX interfaces. Some users come from normal dot1q vlans, and some users come from QinQ vlans. 

After a massive user disconnection  (eletric residential outage), the users weren't able to setup the pppoe connection.

Looking at the logs doesn't help much. The radius server was online, and reporting access-accept when using the "test aaa ppp" command. The only process that was "different" is bbe-mibd, consuming from 60% to 75% in the "show system processes extensive". 

To workaround, we shut down all vlans, and bring up it in a controlled manner.

Now , we have 10k users online, but the bbe-mibd process is still using +60% of the CPU.

Any ideas?

*I have attached the dynamic-profile configuration

Attachment(s)

dynamic-profile-BRAS.txt   1 KB 1 version

Hi,

Did you noticed any DDoS violation during the time of issue? What about the CPU apart from bbe-mibd? 

Any other process reported high CPU like bbe-smgd or authd?

Regards,

Rahul

Yes, some ddos-protection for protocol pppoe, and sample-pfe (this is still active, I've opened another thread relating it).

The other process are fine (for me):

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
6258 root 76 0 58568K 27776K RUN 172:46 57.91% bbe-mibd
3223 root 76 0 228M 138M RUN 85:58 7.71% bbe-smgd
1957 root 76 0 182M 67376K RUN 62:36 6.84% jdhcpd
3267 root 74 0 292M 174M RUN 37:06 3.56% authd
1933 root 43 0 109M 35980K select 43:33 2.88% chassisd
2182 root 70 0 259M 136M RUN 34:18 2.64% jpppd
2141 root 4 0 743M 619M kqread 30:30 1.22% rpd
3223 root 54 0 228M 138M RUN 85:58 1.12% bbe-smgd
1943 root 43 0 79440K 37928K select 14:52 1.03% pfed

Can you please add following filter and check once?

user@router> show configuration snmp filter-interfaces 
interfaces {
    pp*;
    demux*;
    ".*\.[0-9][0-9][0-9][0-9][0-9][0-9]*";
    "ae[0-9]*";
    si-*;
}
all-internal-interfaces;

What is the subscriber scale you have on MX104? I can see you've configured interface-mib as well. Please disable it and monitor the bbe-mib process. Subscribers should relogin for changes to get effect.

l will try this night to disable interface-mib from dynamic-profile.

Thanks.

I logged off all subscribers (removing the vlans from switch), then I disabled the interface-mib option in the dynamic-profile.

Without any online user, the process remained using +70% CPU. 

So, I killed the process via "kill PID", and the CPU fallen down to low usage.

Then, I put back the users online, and with less than 5 min, all the 11k subscribers was up and running. 

The cpu usage is ok now:

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 9262 root      -8    0   269M   177M biowr  169:22 23.44% bbe-smgd
   10 root     155   52     0K    16K RUN     19:30 16.89% idle
 1957 root      -4    0   183M 70868K getblk 322:41 15.38% jdhcpd
 9268 root      43    0   300M   182M select  69:30  9.13% authd
 2182 root      43    0   266M   145M select 153:17  8.15% jpppd
 1933 root      43    0   109M 35612K select 166:24  3.08% chassisd
 9262 root      59    0   269M   177M RUN    169:22  3.03% bbe-smgd
 2141 root       4    0   751M   625M kqread  95:23  2.64% rpd
   11 root     -56 -159     0K    16K WAIT    65:04  2.00% swi2: netisr 0
   30 root     -84 -187     0K    16K WAIT    52:43  1.71% irq96: fman0

Thanks for the help!

I applied the snmp interface filter, but the process continues to consume the cpu.