Junos
Junos

MX240 High Memory Usage

‎05-21-2020 03:37 PM

I have 3 Mx240 connected to my internet edges and one of them when doing SNMP polling is reporting 75% memory usage and alerting me all the time.  Is this to high for the MX240? Should I be concerned?  Is there a way to clear the inactive memory and zombie processes?

Here is the problem one

last pid: 82186; load averages: 0.01, 0.03, 0.00 up 1286+15:54:10 15:34:33
139 processes: 3 running, 121 sleeping, 1 zombie, 14 waiting

Mem: 1854M Active, 232M Inact, 353M Wired, 545M Cache, 69M Buf, 253M Free
Swap: 8192M Total, 8192M Free

PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
11 root 1 171 52 0K 12K RUN ??? 94.34% idle
1574 root 2 8 -88 115M 13968K nanslp 707.6H 1.22% chassisd
2404 root 1 111 15 134M 131M select 312.9H 0.00% sampled
2332 root 1 4 0 1085M 1030M kqread 109.9H 0.00% rpd

 

here is one of the others not reporting any alerts via my Nagios system

Mem: 1342M Active, 334M Inact, 294M Wired, 465M Cache, 69M Buf, 802M Free
Swap: 8192M Total, 8192M Free

 

PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
11 root 1 171 52 0K 12K RUN 4298.8 95.17% idle
1580 root 2 8 -88 115M 14172K nanslp 101.8H 1.32% chassisd
1586 root 1 4 0 926M 871M kqread 998:25 0.00% rpd

6 REPLIES 6
Highlighted
Junos

Re: MX240 High Memory Usage

‎05-21-2020 04:20 PM

Hi williamgeiger85,

 

Howdy, Based on what you just explained and provided it does not seem to be a problem with the zombie processes or a shortage/misused memory from the device, It seems to be something external, to be precise it might be related to an aggressive SNMP polling interval.

 

At the time you took these outputs the load averages were pretty low, your idle percentage was pretty high and there is no other process showing that is consuming a large amount of your CPU memory (spiking) which means that most of the times your device is in a really good stand. 

 

I would recommend you check and see if the mib2 or the snmpd process is going high when the issue is happening and if that is the case you can try by lowering the polling intervals because 75% of a CPU spike could leave some other processes without CPU memory and eventually crash them.

 

Regarding the zombie process, there is a way :

 

Use the “ps aux | grep -w Z ” from shell in order to list the zombie processes.

% ps aux | grep –Wz
Pid=17659
Pid=16898

Once you find the PID of the offender process, you can kill it from the shell with the command kill -9 + PID.

%kill -9 17659

%kill -9 16898

PID: process ID

  

 

 

If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \:)/

Regards,

Lil Dexx
JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

Highlighted
Junos

Re: MX240 High Memory Usage

‎05-21-2020 08:31 PM

HI williamgeiger85,

 

Good day!!

 

Could please answer below query and provide the requested output.

 

 

When started to appear this status?
Was there any software change prior the start of the malfunction?
Is the high CPU persistent or you are experiencing spikes?

Show chassis routing engine
Show system processes extensive
Set task accounting on
Show task accounting <<<< extract this output 10 times with 30 seconds of difference betweeng every shot.

Set task accounting off <<< disable task accounting once the outputs have been extracted.

 

Please mark "Accepted Solution" if this helps.

Kudos are always appreciated

 

Thanks 

Suraj  Rao

Highlighted
Junos

Re: MX240 High Memory Usage

‎05-27-2020 12:24 PM

Thx lil Dexx for the reply but this is related to memory usage not cpu usage and its a constant issue.  The reason I am aware of it at all is a SNMP poll that checks for memory usage and if its over 75% warn me.  I know its not a critical because my memory usage for weeks was going between 74% and 75%.  Its crept up to 75-76% now so I need to address this at some point. 

Highlighted
Junos

Re: MX240 High Memory Usage

[ Edited ]
‎05-27-2020 12:30 PM

Attached are the command outputs requested. 

this started about 1 -2 months ago.  It went from 74-75 for several weeks and now its at about 75-76%. 

There have been zero configuration changes since we noticed this but whatever caused thsi might have been done a very long time ago and the memory usage is just creeping up. 

No spikes in CPU load.  CPU is very low from what I can tell. 

https://drive.google.com/file/d/1MV0CiHrQ9ygRf3xGz81FeL3Uh6pBQVRj/view?usp=sharing

Command output.txt is shared here.

 

Highlighted
Junos

Re: MX240 High Memory Usage

‎05-27-2020 11:02 PM

Hi williamgeiger85,

 

Good day!!

 

Also, please provide the output of the follfwing command :

> show system processes extensive | match free
> show version

 

Please mark "Accepted Solution" if this helps.

Kudos are always appreciated

Highlighted
Junos

Re: MX240 High Memory Usage

‎05-28-2020 01:36 AM

Hi William,

 

You can use the below commands to check the current memory utilization per process : 

 

show system memory    <<< For system memory

show system virtual-memory  <<< For kernel memory

 

Please check the output of these commands to spot the processes that are utilizing highest memory and see if there is a pattern to it.

 

Since, aggresive SNMP polling looks to be a trigger for at least one such device, it is worth a try to reduce the polling interval and monitor the memory utilization.

 

BR,

Vishal

 

PS: Please accept my response as solution if it answers you query, kudos are appreciated too!

Feedback