Ethernet Switching
Ethernet Switching

EX4550-32F Host 0 /var partition is full

‎11-23-2016 09:09 PM

 I've got a strange problem with my EX4550

 

I get an error saying that the storage is full.

 

Show system storage displays this

 

root@ex4550-sheehan> show system storage
fpc0:
--------------------------------------------------------------------------
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 340M 129M 183M 41% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 68M 68M 0B 100% /packages/mnt/jbase
/dev/md1 5.8M 1.1M 4.2M 21% /packages/mfs-fips-mode-powerpc
/dev/md2 2.9M 2.9M 0B 100% /packages/mnt/fips-mode-powerpc-12.3R5.7
/dev/md3 9.0M 4.4M 3.9M 53% /packages/mfs-jcrypto-ex
/dev/md4 12M 12M 0B 100% /packages/mnt/jcrypto-ex-12.3R5.7
/dev/md5 8.1M 3.5M 4.0M 47% /packages/mfs-jdocs-ex
/dev/md6 6.2M 6.2M 0B 100% /packages/mnt/jdocs-ex-12.3R5.7
/dev/md7 43M 39M 700K 98% /packages/mfs-jkernel-ex
/dev/md8 108M 108M 0B 100% /packages/mnt/jkernel-ex-12.3R5.7
/dev/md9 12M 7.9M 3.6M 69% /packages/mfs-jpfe-ex45x
/dev/md10 22M 22M 0B 100% /packages/mnt/jpfe-ex45x-12.3R5.7
/dev/md11 17M 12M 3.2M 79% /packages/mfs-jroute-ex
/dev/md12 38M 38M 0B 100% /packages/mnt/jroute-ex-12.3R5.7
/dev/md13 12M 7.2M 3.6M 66% /packages/mfs-jswitch-ex
/dev/md14 21M 21M 0B 100% /packages/mnt/jswitch-ex-12.3R5.7
/dev/md15 14M 9.5M 3.4M 73% /packages/mfs-jweb-ex
/dev/md16 25M 25M 0B 100% /packages/mnt/jweb-ex-12.3R5.7
/dev/da0s3e 239M 239M -18.8M 109% /var
/dev/md17 126M 12K 116M 0% /tmp
/dev/da0s3d 702M 138K 646M 0% /var/tmp
/dev/da0s4d 116M 154K 106M 0% /config
/dev/md18 118M 22M 87M 20% /var/rundb
procfs 4.0K 4.0K 0B 100% /proc
/var/jail/etc 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/etc
/var/jail/run 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/run
/var/jail/tmp 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/tmp
/var/tmp 702M 138K 646M 0% /packages/mnt/jweb-ex-12.3R5.7/jail/var/tmp/uploads
devfs 1.0K 1.0K 0B 100% /packages/mnt/jweb-ex-12.3R5.7/jail/dev

 

Which seems to indicate the problem lies somewhere in the /var directory

 

du -h in /var shows this

 

root@ex4550-sheehan:RE:0% du -h
2.0K ./.snap
2.0K ./bin
2.0K ./crash
2.0K ./cron/tabs
4.0K ./cron
2.0K ./db/entropy
2.0K ./db/ext
2.0K ./db/fsad
2.0K ./db/gtpcd
2.0K ./db/certs/common/key-pair
2.0K ./db/certs/common/local
2.0K ./db/certs/common/certificate-request
2.0K ./db/certs/common/certification-authority
2.0K ./db/certs/common/crl
12K ./db/certs/common
4.0K ./db/certs/system-key-pair
4.0K ./db/certs/system-cert
22K ./db/certs
2.0K ./db/commit-queue
306K ./db/help
2.0K ./db/leases
2.0K ./db/scripts/commit
2.0K ./db/scripts/event
2.0K ./db/scripts/op
2.0K ./db/scripts/lib
10K ./db/scripts
2.0K ./db/zoneinfo
2.0K ./db/persistent_mac
2.0K ./db/dhcp_snoop
386K ./db
4.0K ./etc/filters
2.0K ./etc/ssl-/local
2.0K ./etc/ssl-/peer
2.0K ./etc/ssl-/certificate-authority
8.0K ./etc/ssl-
2.0K ./etc/ssl/local
2.0K ./etc/ssl/peer
2.0K ./etc/ssl/certificate-authority
8.0K ./etc/ssl
2.0K ./etc/ssl+/local
2.0K ./etc/ssl+/peer
2.0K ./etc/ssl+/certificate-authority
8.0K ./etc/ssl+
228K ./etc
2.0K ./etcroot/ssh
98K ./etcroot
2.0K ./home/b
4.0K ./home
2.0K ./log/ext
2.0K ./log/ggsn/gtppcdr
4.0K ./log/ggsn
2.0K ./log/flowc/failed
4.0K ./log/flowc
1.2M ./log
2.0K ./mfs/var/tmp
82K ./mfs/var/shmipc/if-rtsdb
84K ./mfs/var/shmipc
88K ./mfs/var
90K ./mfs
2.0K ./run/ext
2.0K ./run/named
2.0K ./run/ppp
154K ./run
2.0K ./sw/pkg
4.0K ./sw
2.0K ./tmp/.snap
2.0K ./tmp/vi.recover
2.0K ./tmp/install
2.0K ./tmp/pics
2.0K ./tmp/gres-tp
2.0K ./tmp/rtsdb
14K ./tmp
4.0K ./root
2.0K ./empty
2.0K ./account
2.0K ./at/jobs
2.0K ./at/spool
6.0K ./at
2.0K ./backups
2.0K ./transfer/config
4.0K ./transfer
2.0K ./heimdal
106K ./jail/etc
4.0K ./jail/run
2.0K ./jail/tmp/uploads
4.0K ./jail/tmp
116K ./jail
2.0K ./mail
2.0K ./msgs
2.0K ./named
2.0K ./preserve
2.0K ./validate
2.0K ./logical-systems
2.0K ./rwho
2.0K ./spool/lock
2.0K ./spool/lpd
2.0K ./spool/mqueue
2.0K ./spool/opielocks
2.0K ./spool/output/lpd
4.0K ./spool/output
14K ./spool
2.0K ./yp
2.0K ./rundb/.snap
2.0K ./rundb/private
10K ./rundb/sdb/shmem
520K ./rundb/sdb
22M ./rundb
24M .

 

Which doesn't appear to match in terms of size used

 

Earlier I did have a 230-240Mb httpd.log in /var/log which I deleted - but it doesn't appear to have made a difference in the reported size. Does this require a restart of the switch to update?

 

Switch is running 12.3R5.7 and has been up for 818 days, so it's probably due a reboot.

 

Any help would be greatly appreciated

5 REPLIES 5
Ethernet Switching

Re: EX4550-32F Host 0 /var partition is full

‎11-24-2016 12:37 AM

Have you tried request system storage cleanup ?

Ethernet Switching

Re: EX4550-32F Host 0 /var partition is full

‎11-24-2016 01:26 PM

Yes, tried that. 

 

root@ex4550-sheehan> request system storage cleanup
Please check the list of files to be deleted using the dry-run option. i.e.
request system storage cleanup dry-run
Do you want to proceed ? [yes,no] (no) yes

fpc0:
--------------------------------------------------------------------------
error: Rotating files failed: newsyslog: log_trim: fclose: No space left on devi ce

{master:0}

 

Is the response from this command.

Ethernet Switching

Re: EX4550-32F Host 0 /var partition is full

‎11-25-2016 04:03 AM

I assume you get the same error without the dry run option attempting the actual clean up.

 

I would agree that a restart is a next good option to see if the issue clears or at least releases whatever locks are preventing the storage cleanup from working.

Steve Puluka BSEET - Juniper Ambassador
IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
http://puluka.com/home
Ethernet Switching

Re: EX4550-32F Host 0 /var partition is full

‎11-28-2016 05:28 AM

I would rather manually delete old logs and then try request system storage cleanup again.

% rm /var/log/*.gz

With full filesystem there is risk that switch will not come up.

Ethernet Switching

Re: EX4550-32F Host 0 /var partition is full

‎11-13-2018 02:11 PM

Hey guys,

if someone still has the same issue try this:

 

du -sk /var/* | sort -nr | head

 

It finds files which was not clearly removed.