Switching

Hello,

 

we have started to take a look at the EX3400 series and encountered that is impossible to upgrade due to low storage space.

In first place transfering the package to the device is insanly slow at 140 kb/s. We have been testing via http, ftp and scp to see if it is not a protocol regression but transfer rates stayed at 140 kb/s. The upgrade itself fails due to low storage space but it's not possible to free up more storage since all the tmp directories are already empty.

 

Software upgrades have always been troublesome with Juniper due to low space, memory leakage and slow processing power but given the fact that the EX3400 is pretty new and already suffering from outdated hardware is not understandable and makes automatization of upgrade process impossible.

 

We never encountered upgrade problems with other vendors even with entry level gear.

 

Hello,

 

It is strange that you are experiencing very slow transfer rates.

Did you try other methods outlined here:

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB31201&actp=RSS

 

Regards,

 

Rushi

Hi Folks,
Please do a full SCAN of the box and delete any unwanted files,

 

start shell user root
cd /
find .

 

-rengar

HI

 

Please check this KB KB31633

 

Cause There is storage issues in EX2300 and EX3400 in /var directory. This has limited space which could be occupied by other files.

Solution We would be getting the error when we try normal upgrade   root@EX-2300> ...ar/tmp/junos-arm-32-15.1X53-D52.tgz force                  ERROR: not enough space to unpack /var/tmp/junos-arm-32-15.1X53-D52.tgz   Workaround to fix this issue is to move the file to /mfs directory and upgrade from there.   root@EX-2300:RE:0% mv junos-arm-32-15.1X53-D52.tgz /mfs root@EX-2300:RE:0% ls bcast.bdisp.log         gres-tp-tomcat          rtsdb bcast.disp.log          krt_rpf_filter.txt      sampled.pkts bcast.rstdisp.log       package.log             sd-upgrade bcast.undisp.log        pics                    stable ex_autod_config         pkg_cleanup.log         version_ex.xml ex_autod_rollback_cfg   pkg_cleanup.log.err root@EX-2300:RE:0% cli {master:0} root@EX-2300> ... add /mfs/junos-arm-32-15.1X53-D52.tgz                       NOTICE: Validating configuration against junos-arm-32-15.1X53-D52.tgz. NOTICE: Use the 'no-validate' option to skip this if desired. Verified junos-arm-32-15.1X53-D52 signed by PackageProductionEc_2016 Adding junos-arm-32-15.1X53-D52 ... Verified fips-mode signed by PackageProductionEc_2016 Adding fips-mode-arm-32-20161006.233631_builder_junos_151_x53_d52 ...

 

Thanks

Partha

Followed Partha advice this morning on 2300c and install works if you place the junos packages

in /mfs/

 

 

Before when i tried request system software add /var/tmp/junos-arm-32-18.3R1.9.tgz

it would fail

ERROR: insufficient space for /packages/db/junos-arm-32-18.3R1.9/contents/junos-runtime.tgz
ERROR: insufficient space
ERROR: Failed to add junos-arm-32-18.3R1.9.tgz

 

start shell user root

mv /var/tmp/junos-arm-32-18.3R1.9.tgz

exit

request system software add /mfs/junos-arm-32-18.3R1.9.tgz

Now  all is well.

I still get insufficient space erros when installing from /mfs/ even when using force option. 

 

Basically stuck on one code version. Juniper needs to address this issue asap, in my opinion.

The same issue exists on 2300/2300-C. One workaround is to install the upgrade from a mounted USB flash drive.

So in other words there is not fix. JTAC seems to be at a loss related to it, choosing instead to focus on system storage clean up not cleaning up 2 small 25KB log files. 

Even from usb still getting lack of space error. 

Did you make sure to delete recovery snapshots that might be taking up space?

Juniper just updated the “not enough storage” kb article with additional suggestions for freeing up space pre-install.

https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198

The only thing that worked for us was following this suggestion:

https://www.reddit.com/r/Juniper/comments/76w463/cleaning_up_old_packages_on_ex3400/

 

Which is basically removing all older versions from /packages/db, as all older versions seem to be kept there.

 

This is indeed extremely bad from all points, as it's a comepletly manual process that shouldn't have to exist, as it should be properly handled, and is prone to mistakes, to which I don't know the impact, since I don't know if these are temporary files or not. And of course, nothing in the KB about that either.

Still an issue for me under 18.4R1-S1. Was able to install by moving the installation package to /mfs and using

 

request system software add /mfs/junos-arm-32-18.4R1-S1.1.tgz no-copy no-validate force unlink reboot

We have a bunch of ex3400's and are experiencing the not enough storage space issue.  Manually deleting files sometimes works, but that does not scale when you have a lot of switches. 

 

I also see painfully slow transfer speeds with scp and ftp when copying the software package to the switch.  I routinely get 700 kB/s and it takes about 8 minutes to transfer a 300 mb file.  Anyone else see this and or find a solution?

Is it VC or stand alone device? is the traffic comming over the backup member? if is this the case, I would suggest to change the mastership of RE and transfer the traffic directly to the master.

The painfully slow scp/ftp traffic is due to the fixed PFE->CPU rate-limiter. Both ssh and ftp are limited to 100 pps towards the CPU/RE.

 

These rate-limiters are currently not configurable so no workaround.

To share our experience, we have a USB stick plugged into the master EX3400 and use it to store the update archive.

 

Usually this is still not enough to workaround space limitations and we also delete old packages manually. After our last update from 18.1R3-S5 to 18.1R3-S6, I put together the following command:

ls /packages/db/ | grep -vE `ls -l /packages/sets/active/ | grep '^l' | cut -d'/' -f 4 | tr '\n' '|' | sed 's/|$//'`

This should print all packages that are not referenced from /packages/sets/active/ which should be safe to remove. No guarantee that this works in all cases (it's your fault if you use it and it breaks), but might be easier than manually going through the entire directory manually 😉

Hi,

 

Please ensure that the destination file location of the "zip file for upgrade" has atleast two times more space than the size of the zip file. You can always keep a check on file storage capacity to prevent such errors. 

 

- M 🙂

Hi,

 

Please ensure that the destination file location of the "zip file for upgrade" has atleast two times more space than the size of the zip file. You can always keep a check on file storage capacity to prevent such errors. 

 

Hello,

 

As per the below KB,

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198

 

If you are currently running image is 18.1R2 or later:

 

1. root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force unlink no-copy

   The force option frees up storage space by removing the non-recovery snapshots if any. The unlink option is available starting 18.1R2. The unlink option will remove the .tgz file immediately after unpacking so that the available image size calculation takes into account the free 300M. Without using unlink, this space is not considered.

2. If the above step fails, run the software update with the force and unlink option again (it will work the second time in some situations).

 

The force unlink option has always worked for me if not once then the second time it has always worked. 

 

Thanks,

Puneet