Junos Automation (Scripting)
Junos Automation (Scripting)

juniper_junos_software and EX3400 Storage Limitations

‎07-08-2019 02:04 PM

Hey,

I want to upgrade our EX3400 switches off of the 15D series they are currently on. My problem, however, is that I have a lot of switches and want to use ansible to automate as many of them as possible. My playbook checks out, and I can see in my logs:

</ignore-signals>
</cli>
<output>
Verified junos-arm-32-19.2R1.8 signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-arm-32-19.2R1.8 ...
Verified fips-mode signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding fips-mode-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified jail-runtime signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding jail-runtime-arm-32-20190517.f0321c3_builder_stable_11 ...
Verified jdocs-ex signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding jdocs-ex-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified jpfe-EX34XX signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding jpfe-EX34XX-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified jphone-home signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding jphone-home-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified jsd-jet-1 signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding jsd-arm-32-19.2R1.8-jet-1 ...
Verified jsdn signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding jsdn-arm-32-19.2R1.8 ...
Verified junos-daemons signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-daemons-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-dp-crypto-support-ex signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-dp-crypto-support-ex-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-libs-ex signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-libs-ex-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-libs signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-libs-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-modules-ex signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-modules-ex-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-modules signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-modules-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-net-dcp-prd signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-net-dcp-prd-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-net-prd signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-net-prd-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-platform-ex signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-platform-ex-arm-32-20190621.152752_builder_junos_192_r1 ...
Verified junos-runtime-ex signed by PackageProductionEc_2019 method ECDSA256+SHA256
Adding junos-runtime-ex-arm-32-20190621.152752_builder_junos_192_r1 ...
ERROR: insufficient space for /packages/db/junos-arm-32-19.2R1.8/contents/junos-runtime.tgz
ERROR: insufficient space
</output>
<package-result>

I know this runs a system software clean-up by default, which I've specified directly now in my playbook, but I've also tried to running a software cleanup and deleting all snapshots (this is a fresh switch used explicitly for testing automation). I stumbled into the following KB: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198&actp=METADATA

For me, copying the tgz to the machine manually and then ansible with the remote package variable seems to defeat the point of this "no touch" deployment methodology. Anyone know a way around this? All the Juniper images are 300Mb or so, and my 3400 has only 700Mb spare once I do a cleanup in the gpt.

Thanks,

3 REPLIES 3
Junos Automation (Scripting)

Re: juniper_junos_software and EX3400 Storage Limitations

‎07-08-2019 05:33 PM

Hi David,

 

You need to add the following to the procedure:

 

root@juniper> request system storage cleanup
root@juniper> request system snapshot delete snap*
root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force no-copy

 

The "force option" frees up storage space by removing the non-recovery snapshots, if any.  The "no-copy" ensures there is no copy of install package stored on the box.  

 

Please also remember to try this command twice if the above fails the first time i.e. run the software update with the force and no-copy option again (it will work the second time in some situations).


request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force no-copy

 

Note: There's also an "unlink" option is available starting 18.1R2. The unlink option will remove the .tgz file immediately after un-packing so that the available image size calculation takes into account the free 300M. Without using unlink, this space is not considered. 

 

Hope this helps.

 

Regards,
-r.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution."
Kudos are always appreciated Smiley Happy.

Junos Automation (Scripting)

Re: juniper_junos_software and EX3400 Storage Limitations

‎07-09-2019 08:21 AM

Thanks for the response, but maybe I'm missing the point here...

1) The module itself purportedly runs the system storage cleanup. Regardless, I've run the snapshot removal command along with the storage cleanup manually on the switch and still failed once I run the playbook.

 

2) The solution you are proposing assumes the presence of the JunOS software in a tmp directory on the target device, but I would ideally like to install it remotely from the Ansible control machine.

I've tried running this playbook with the "no_copy" flag set to both true and false. Is there a way to preload the software from the playbook?

Junos Automation (Scripting)

Re: juniper_junos_software and EX3400 Storage Limitations

‎07-09-2019 09:19 AM
{master:0}
root> ...m software add /var/tmp/junos-arm-32-18.4R1.8.tgz no-copy force    
Error: not enough space to unpack /var/tmp/junos-arm-32-18.4R1.8.tgz
ERROR: Either use 'force' or consider deleting snapshots using 'request system snapshot delete <snap>'

{master:0}
root> ...m software add /var/tmp/junos-arm-32-18.4R1.8.tgz no-copy force    
Error: not enough space to unpack /var/tmp/junos-arm-32-18.4R1.8.tgz
ERROR: Either use 'force' or consider deleting snapshots using 'request system snapshot delete <snap>'

{master:0}
root> show system snapshot                                                  
fpc0:
--------------------------------------------------------------------------
No non-recovery snapshots available on the Junos volume

Recovery Snapshots:
Snapshots available on the OAM volume:
recovery.ufs
Date created: Sat Jul  7 01:33:26 UTC 2018
Junos version: 15.1X53-D58.3

Total recovery snapshots: 1

{master:0}

The commands don't seem to work manually once the image is loaded, either. I imagine this is a limitation with the EX3400s and their on-board memory. Maybe if we were able to use the "unlink" feature this would be easier, but we're on 15D.