NSM Upgrade More space needed AGAIN, but this time I can't find enough space!

[SomeITGuy]

ERROR: Need 2484MB more space on partition: /var/netscreen/GuiSvr for the following directories:  /var/netscreen/GuiSvr

So I cleaned up:

- logs

- Software library files

- ALL recent DB backups (this used to be the only fix needed)

- cores

- random backup files

df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda2              5036316    475224   4305260  10% /
none                   2075688         0   2075688   0% /dev/shm
/dev/sda9              4032092     53796   3773472   2% /tmp
/dev/sda7             20161172  10030332   9106700  53% /usr
/dev/sda6             85032152   4580884  76131796   6% /var
/dev/sda3             50394996     86084  47748956   1% /var/cores
/dev/sdb2            235215768  12485604 210781824   6% /var/netscreen/DevSvr
/dev/sda5             50394964   1779512  46055496   4% /var/netscreen/DevSvr/profiler_data
/dev/sda8             20161172  16620280   2516752  87% /var/netscreen/GuiSvr

A search for large ish files returns

find /var/netscreen/GuiSvr -type f -size +30000k -exec ls -lh -all {} \; | awk '{ print $9 ": " $5 }'
/var/netscreen/GuiSvr/Schemas-GDH/160/nsm.schema.bin: 102M
/var/netscreen/GuiSvr/Schemas-GDH/169/nsm.schema.bin: 108M
/var/netscreen/GuiSvr/Schemas-GDH/190/nsm.schema.bin: 135M
/var/netscreen/GuiSvr/Schemas-GDH/147/nsm.schema.bin: 98M
/var/netscreen/GuiSvr/Schemas-GDH/187/nsm.schema.bin: 135M
/var/netscreen/GuiSvr/be/schema-binary/nsm.schema.bin: 107M
/var/netscreen/GuiSvr/be/schemas/juniper-update-private/dmi/junos-es/releases/11.1/config.xsd: 31M
/var/netscreen/GuiSvr/be/schemas/juniper-update-private/dmi/junos-es/releases/10.4R1/config.xsd: 32M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-ex/releases/11.1R1.10/.svn/text-base/config.xsd.svn-base: 30M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-ex/releases/11.1R2.3/.svn/text-base/config.xsd.svn-base: 30M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-es/releases/10.2R2.11/.svn/text-base/config.xsd.svn-base: 31M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-es/releases/10.2R3.10/.svn/text-base/config.xsd.svn-base: 31M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-es/releases/10.3R1.9/.svn/text-base/config.xsd.svn-base: 31M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-es/releases/10.2R1.8/.svn/text-base/config.xsd.svn-base: 30M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-es/releases/10.3R2.11/.svn/text-base/config.xsd.svn-base: 31M
/var/netscreen/GuiSvr/dmi-schema-stage/nsm/dmi/junos-es/releases/10.4R1.9/.svn/text-base/config.xsd.svn-base: 33M
/var/netscreen/GuiSvr/xdb/init/schemafile-content.init: 218M
/var/netscreen/GuiSvr/xdb/xdb17188.tar.gz: 325M
/var/netscreen/GuiSvr/xdb/data/directive: 35M
/var/netscreen/GuiSvr/xdb/data/refDb: 61M
/var/netscreen/GuiSvr/xdb/data/idp-attack-grp: 33M
/var/netscreen/GuiSvr/xdb/data/auditlog: 75M
/var/netscreen/GuiSvr/xdb/data/attack: 237M
/var/netscreen/GuiSvr/xdb/data/referrer: 30M
/var/netscreen/GuiSvr/xdb/data/junos-es: 47M
/var/netscreen/GuiSvr/xdb/data/schemafile-content: 249M
/var/netscreen/GuiSvr/xdb/data/auditlogDetails: 261M
/var/netscreen/GuiSvr/xdb/data/auditlogReference: 65M
/var/netscreen/GuiSvr/sec-update/current/NSMFP14-DI-IDP.zip: 37M
/var/netscreen/GuiSvr/dmi-schema-stage.old/nsm/dmi/junos-es/releases/10.2R2.11/.svn/text-base/config.xsd.svn-base: 30M
/var/netscreen/GuiSvr/dmi-schema-stage.old/nsm/dmi/junos-es/releases/10.2R1.8/.svn/text-base/config.xsd.svn-base: 30M

I figure dmi-schema-stage.old is safe to nuke, but I am open to suggestions...

[vencour]

Is it NSM or NSM Express?

I gave 120GB for NSM (/var/ directory) and it's enough ... if not, then I connect usb harddrive and make link (for example "ln -s /media/sdd1/netscreen-linked /var/netscreen) and copy NSM files to free space.

This should be enough ...

Next thing, I reduced dbbackup days from default 7 to 3. I did not used dbbackup files anymore yet

Best regards

Vencour

[SomeITGuy]

-nsmexpress appliance so I don't have that kind of control over the drive size

- already stated I have cleaned up the backups, was already at three but manually removed those as well. Sent from my mobile.

[SomeITGuy · Accepted solution]

Solved it by moving all of /var/netscreen/GuiSvr/dmi-schema-stage.old to my /var/tmp just in case...

Cleared up something close to 20% of /var/netscreen/GuiSvr/

[cryptochrome]

---

SomeITGuy wrote:

Solved it by moving all of /var/netscreen/GuiSvr/dmi-schema-stage.old to my /var/tmp just in case...

Cleared up something close to 20% of /var/netscreen/GuiSvr/

---

Kind of worrying that they leave something this large and obviouly old in there. They need to seriously enhance their schema update scripts to do better hose cleaning. 

My new solution to this problem: Open a JTAC ticket EVERY time we want to update the schema or NSM itself and ask them to check if we have enough space. 

[smgg]

For people searching the forums with a NSMxpress appliance

and 

Upgrade was failing due to disk space issue in /usr partition.

JTAC did the following on my box

They deleted some files in /usr/netscreen/GuiSvr/lib/initVar/dmi-schema-stage/nsm/dmi/ which was taking too much space.

They used the following command :

rm -rf /usr/netscreen/GuiSvr/lib/initVar/dmi-schema-stage/nsm/dmi/*/releases/*/.svn/text-base/*

Hopefully this same someone some time and frustration.

[cryptochrome]

It's really quite unbelievable that this issue still exists AND that they *still* sell NSMxpress appliances with laughable 20 GB partitions, despite the fact they know exactly that this is way too small. Someone at Juniper definitely needs to be fired.

The same thing happened to me again about 2-3 months ago (after it happened with basically every NSM upgrade in the last couple of years). When I asked the JTAC engineer how many cases a day he works on solving NSM disk space issues he said at least 3-4 a day. Go figure.

You would believe Juniper would do something about it but as it seems, they can bear a lot of pain.

[gy]

There is a huge decrease in schema size starting with 235. For example:

234 compressed schema size: 467MB

235 compressed schema size: 55MB

if you are using an older one, it may be better to go above 234