Ethernet Switching
Highlighted
Ethernet Switching

Juniper EX3400 software upgrade not possible

‎06-25-2017 02:05 PM

Hello,

 

we have started to take a look at the EX3400 series and encountered that is impossible to upgrade due to low storage space.

In first place transfering the package to the device is insanly slow at 140 kb/s. We have been testing via http, ftp and scp to see if it is not a protocol regression but transfer rates stayed at 140 kb/s. The upgrade itself fails due to low storage space but it's not possible to free up more storage since all the tmp directories are already empty.

 

Software upgrades have always been troublesome with Juniper due to low space, memory leakage and slow processing power but given the fact that the EX3400 is pretty new and already suffering from outdated hardware is not understandable and makes automatization of upgrade process impossible.

 

We never encountered upgrade problems with other vendors even with entry level gear.

 

19 REPLIES 19
Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-25-2017 06:26 PM

Hello,

 

It is strange that you are experiencing very slow transfer rates.

Did you try other methods outlined here:

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB31201&actp=RSS

 

Regards,

 

Rushi

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-26-2017 10:07 PM

Hi Folks,
Please do a full SCAN of the box and delete any unwanted files,

 

start shell user root
cd /
find .

 

-rengar

-Python JNCIE 3X [SP|DC|ENT] JNCIP-SEC JNCDS 3X [ WAN | DC|SEC] JNCIS-Cloud JNCIS-DevOps CCIP ITIL
#Please mark my solution as accepted if it helped, Kudos are appreciated as well.
Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎07-03-2017 12:44 AM

HI

 

Please check this KB KB31633

 

Cause
There is storage issues in EX2300 and EX3400 in /var directory. This has limited space which could be occupied by other files.
 
Solution
We would be getting the error when we try normal upgrade
 
root@EX-2300> ...ar/tmp/junos-arm-32-15.1X53-D52.tgz force                 
ERROR: not enough space to unpack /var/tmp/junos-arm-32-15.1X53-D52.tgz
 
Workaround to fix this issue is to move the file to /mfs directory and upgrade from there.
 
root@EX-2300:RE:0% mv junos-arm-32-15.1X53-D52.tgz /mfs
root@EX-2300:RE:0% ls
bcast.bdisp.log         gres-tp-tomcat          rtsdb
bcast.disp.log          krt_rpf_filter.txt      sampled.pkts
bcast.rstdisp.log       package.log             sd-upgrade
bcast.undisp.log        pics                    stable
ex_autod_config         pkg_cleanup.log         version_ex.xml
ex_autod_rollback_cfg   pkg_cleanup.log.err
root@EX-2300:RE:0% cli
{master:0}
root@EX-2300> ... add /mfs/junos-arm-32-15.1X53-D52.tgz                      
NOTICE: Validating configuration against junos-arm-32-15.1X53-D52.tgz.
NOTICE: Use the 'no-validate' option to skip this if desired.
Verified junos-arm-32-15.1X53-D52 signed by PackageProductionEc_2016
Adding junos-arm-32-15.1X53-D52 ...
Verified fips-mode signed by PackageProductionEc_2016
Adding fips-mode-arm-32-20161006.233631_builder_junos_151_x53_d52 ...
 

 

Thanks

Partha

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

[ Edited ]
‎10-09-2018 05:29 AM

Followed Partha advice this morning on 2300c and install works if you place the junos packages

in /mfs/

 

 

Before when i tried request system software add /var/tmp/junos-arm-32-18.3R1.9.tgz

it would fail

ERROR: insufficient space for /packages/db/junos-arm-32-18.3R1.9/contents/junos-runtime.tgz
ERROR: insufficient space
ERROR: Failed to add junos-arm-32-18.3R1.9.tgz

 

start shell user root

mv /var/tmp/junos-arm-32-18.3R1.9.tgz

exit

request system software add /mfs/junos-arm-32-18.3R1.9.tgz

Now  all is well.

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎10-15-2018 06:33 AM

I still get insufficient space erros when installing from /mfs/ even when using force option. 

 

Basically stuck on one code version. Juniper needs to address this issue asap, in my opinion.

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎10-15-2018 07:04 AM

The same issue exists on 2300/2300-C. One workaround is to install the upgrade from a mounted USB flash drive.

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎10-15-2018 09:10 AM

So in other words there is not fix. JTAC seems to be at a loss related to it, choosing instead to focus on system storage clean up not cleaning up 2 small 25KB log files. 

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎10-16-2018 05:29 AM

Even from usb still getting lack of space error. 

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎10-16-2018 06:54 AM

Did you make sure to delete recovery snapshots that might be taking up space?

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎10-29-2018 12:06 PM
Juniper just updated the “not enough storage” kb article with additional suggestions for freeing up space pre-install.

https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198
Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎10-31-2018 01:15 PM

The only thing that worked for us was following this suggestion:

https://www.reddit.com/r/Juniper/comments/76w463/cleaning_up_old_packages_on_ex3400/

 

Which is basically removing all older versions from /packages/db, as all older versions seem to be kept there.

 

This is indeed extremely bad from all points, as it's a comepletly manual process that shouldn't have to exist, as it should be properly handled, and is prone to mistakes, to which I don't know the impact, since I don't know if these are temporary files or not. And of course, nothing in the KB about that either.

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎02-04-2019 04:44 PM

Still an issue for me under 18.4R1-S1. Was able to install by moving the installation package to /mfs and using

 

request system software add /mfs/junos-arm-32-18.4R1-S1.1.tgz no-copy no-validate force unlink reboot
Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎03-06-2019 03:44 PM

We have a bunch of ex3400's and are experiencing the not enough storage space issue.  Manually deleting files sometimes works, but that does not scale when you have a lot of switches. 

 

I also see painfully slow transfer speeds with scp and ftp when copying the software package to the switch.  I routinely get 700 kB/s and it takes about 8 minutes to transfer a 300 mb file.  Anyone else see this and or find a solution?

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-23-2019 04:41 PM

Is it VC or stand alone device? is the traffic comming over the backup member? if is this the case, I would suggest to change the mastership of RE and transfer the traffic directly to the master.

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-23-2019 10:17 PM

The painfully slow scp/ftp traffic is due to the fixed PFE->CPU rate-limiter. Both ssh and ftp are limited to 100 pps towards the CPU/RE.

 

These rate-limiters are currently not configurable so no workaround.


--
Best regards,

Jonas Hauge Klingenberg
Juniper Ambassador & Technology Architect, SEC DATACOM A/S (Denmark)
Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-24-2019 04:23 AM

To share our experience, we have a USB stick plugged into the master EX3400 and use it to store the update archive.

 

Usually this is still not enough to workaround space limitations and we also delete old packages manually. After our last update from 18.1R3-S5 to 18.1R3-S6, I put together the following command:

ls /packages/db/ | grep -vE `ls -l /packages/sets/active/ | grep '^l' | cut -d'/' -f 4 | tr '\n' '|' | sed 's/|$//'`

This should print all packages that are not referenced from /packages/sets/active/ which should be safe to remove. No guarantee that this works in all cases (it's your fault if you use it and it breaks), but might be easier than manually going through the entire directory manually ;-)

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-24-2019 07:31 PM

Hello,

 

As per the below KB,

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198

 

If you are currently running image is 18.1R2 or later:

 

  1. root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force unlink no-copy

    The force option frees up storage space by removing the non-recovery snapshots if any. The unlink option is available starting 18.1R2. The unlink option will remove the .tgz file immediately after unpacking so that the available image size calculation takes into account the free 300M. Without using unlink, this space is not considered.

  2. If the above step fails, run the software update with the force and unlink option again (it will work the second time in some situations).

 

The force unlink option has always worked for me if not once then the second time it has always worked. 

 

Thanks,

Puneet

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-26-2019 08:40 AM

Hi,

 

Please ensure that the destination file location of the "zip file for upgrade" has atleast two times more space than the size of the zip file. You can always keep a check on file storage capacity to prevent such errors. 

 

Ethernet Switching

Re: Juniper EX3400 software upgrade not possible

‎06-26-2019 08:41 AM

Hi,

 

Please ensure that the destination file location of the "zip file for upgrade" has atleast two times more space than the size of the zip file. You can always keep a check on file storage capacity to prevent such errors. 

 

- M Smiley Happy