Junos Automation (Scripting)
Junos Automation (Scripting)

EX2300-C ZTP upgrade with knobs (force unlink etc)

[ Edited ]
‎12-09-2019 04:49 AM

As the title describes, i have a big batch of EX2300-C switches that i want to upgrade in batches. I configured a linux server for both DHCP and TFTP and Zero Touch is working fine. The switch boots up out of the box, gets an IP from the DHCP server and then automatically fetches the junos package i included in my ZTP configuration on the server. My problem is with the EX2300-C not having enough space to unpack and install the new Junos. This is a known issue we encountered before and i can overcome that by adding the knobs "force" and "unlink" in "request system software add" when i am upgrading the switch manually.

Im therefore looking for a way to include these knobs when the switches are fetching the package automatically from the server.

Any ideas?

 

18 REPLIES 18
Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-09-2019 07:35 AM

Hi cpir0,

 

This issue is addressed on Junos 18.2R3-S1 or 18.3R3 and higher, please see the following:

https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1417441

 

For other Junos versions, although cleaning up the device storage can help as a workaround, but that needs to be done prior to ZTP so the workaround may not be a good fit for pure ZTP.  

 

Hope this helps.

 

Regards,
-r.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution."
Kudos are always appreciated Smiley Happy.

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-09-2019 07:37 AM

Hi cpir0,

 

there is no way to introduce those knobs from the ZTP configs on server, there should be a modification for EX2300 and EX3400 to have the unlink option default , could you please check what is the SW version on the device before the upgrade ? what you can do is maybe a python script to access the reachable IPs in DHCP server provided subnet and check the current version and if it is still not upgraded you can upgrade it using the script and utilizing pyez libraries.

 

If this solves your problem, please mark this post as "Accepted Solution

 

Regards,
A.A.
Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-09-2019 09:47 AM

I ended up using ZTP to push a SLAX script and event option, the SLAX script would do the install with all the options needed and also did storage cleanup for snapshot recoveries. Im willing to share if you want.

Kyle Quilliam
Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-11-2019 01:35 AM
That would be great! Thanks!!
Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

[ Edited ]
‎12-11-2019 11:55 PM

I would definitely appreciate it if you could share the script Kyle!

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-12-2019 12:13 AM

Ill get it sorted in my Github and post it up tomorrow.

Kyle Quilliam
Junos Automation (Scripting)
Solution
Accepted by topic author cpir0
a week ago

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-12-2019 12:39 PM

bonjour cpr0 

I am surprised that they do this to you right out of the box. I believe at least the first time I upgraded them using a similar ztp setup it worked fine. But that was a while ago. 

I have had to install images on a number of EX2300 , and been confronted with the space issue. Here are the 2 tricks I developped and that work especially well on switches that have been used and are not brand new. 

1) I have a bunch of autoboot usb keys with an old Image. I'll line up 10 switches, power them up , connect the vme (mgmt)  port to access the dhcp server, plug in the usb keys and just issue a request system reboot usb from the console:  the process will go on and put the switch back in a state where it accept the image coming from the DHCP without arguing (I am using 18.2R3S2 - which among other things solve the PR1442376 issue, and it's the recommended JTAC image) . It worked for me every time

2) on a connected switch, I will ftp the image not into the /var/tmp directory as recommended but in the /tmp directory. Then, issuing a  request system software add /tmp/junos-arm-32-18.2R3-S2.9.tgz force unlink no-copy reboot . for some reason, I don't get the no space failure. That also worked everytime, and I have 240 of these things  !!! I used secureCRT to access my switches, and load an .sh file on each switch that fetched the image and executed  it using "cli -c "blablabla" lines. I upgraded 240 switches in an afternoon 

 

once this is done, if I may suggest to disable the miniusb console port : go figure, but I haven't had a single swizzle reboot on the switches I did that ( set system port aux port-type rj45 disable) . You can try it , it will only disable the miniusb port console. If you get message like "getty request...", I strongly suggest you do it. 

 

more on this here 

https://forums.juniper.net/t5/Junos/this-can-not-be-good-getty-tcsetattr-dev-ttyu1-Invalid-argument/... 

 

have fun !

Michel

 

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-12-2019 11:32 PM

Bonjour Michel!

 

Thats exactly the issue we have. I have hundreds of 2300s and i need to get them to 18.2R3-S2.9 due to the PR you mentioned (and a couple of others like the LED issue etc..) !

 

I will definitely try the autoboot  procedure, havent htought of that. I did think of just plugging a bunch of usbs on a bunch of switches and then running some kind of a script to go through the procedure to mount it, copy the file and blah blah but it was still too cumbersome. Your method might do the trick!

 

Ill go through the thread you posted as well, try your suggested methods and come back.

 

Thank you very much for the awesome reply, appreciated!

 

Spyro

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-13-2019 05:34 AM

Bonjour Spyro, 

glad I can help.  We have had to postpone deploying 240 switches in the field due to 2 issues: the PR1442376, that I affectuously called "the zombie problem", and random reboot diagnosed as "swizzle reboot". Therefore, I have ended up with a big laboratory of ex2300 switches !!!

  • by all means, stick to 18.2R3-S2. Seems Juniper nailed it.  Other than that, you may end up with a switch that still forward trafic, but is totally unresponsive in any way, other than powercycle. 
  • if you have hundreds of 2300, I strongly suggest the disabling of the miniusb console port to prevent swizzle reboot. On my sample of 250 switches, I had about 30 that were susceptible to having it.  
  • Show Chassis routing-engine will show you the last reboot reason of a switch. if a switch swizzle, turn on messages (monitor start messages)  wait a minute to see if you get a deluge of getty request. JTAC answered to me at the time that it was related to bad consol usb cabling. Since I had none connected, we decided to simply turn it off, and it seems to fix the issue. They are working on it now and I was told a PR was issued 
  • you may want to read the zombie switch saga that ended up with the resolving of PR1442376    https://forums.juniper.net/t5/Junos/EX2300-48T-Stop-working-after-random-time-after-upgrade-to-19/m-...   it includes the various tests I did.
  • I went from taking 2 days of hard work  to upgrade images on my 240 switches to an afternoon sipping  coffee. the big difference is to have access remote to the switches or not. If no remote access, the bootable usb key is my favorite solution, and I still use it to reset a switch. If access, I use SecureCRT with customized command buttons that you can apply to all connected switches. It is extremely rare that I ended up on a switch that would not reboot. they all turned out to be rma cases. 
  • if you ftp the new image on the switches, I suggest doing a request system  storage cleanup first:  before systematically doing it, I ended up wasting time with the process going on until abruptly stopping because of space issue. I then had to manually go, erase the partially downloaded file, do the clean up and start again. :-(
  • still have not figured out why using /tmp works, but not /var/tmp.  But it works. for peace of mind, I tried a couple of /var/tmp procedure and it still failed, so I gave up entirely.  
  • I still have to check if 2 minor problems are still on: no leds up on the sfp ports, and the ports led half-full duplex indicator not working. may not seems like much, but it does not help reassuring field technicians that everything is fine. 

we will start deploying in january, so my lab will be dismantled. Right now, all my switches are running 18.2R3-S2.9 with the miniusb port disable. The 8 switches in production are running this setup too and I don't hear from them anymore. 

I want to get into the slax and writing scripts now. next challenge. 

Michel

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-15-2019 04:11 PM

I've been bashing my head aganst this the last few days and would greatly appreciate the slax script. I have 28 EX2300-C sitting on a table plugged up in the lab to do ZTP but cannot deploy because it gets stuck in the out of space loop. Really wanted those ready to go on Monday but here we are.

 

At least I'm not on this island of ZTP suck alone.

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-16-2019 05:13 AM

bonjour jgrider, 

   as far as the out os space issue, the only 2 ways I found out of this is ...

  1. use a bootable usb, then connect to a dhcp with the ztp parameters to download the image you want and a basic  configuration valid for any switch that get modified later on .
  2. on already configured switches, I do a system storage cleanup, copy the image on the /tmp directory instead of the /var/tmp as recommended in the procedures, and issue a add software with unlink, no-copy, force, reboot

it's no slax, but here is the content of an .sh file I sent to all my switches, then executed and went for coffe. 

root@vvb-valparadis-01-BYPASS:RE:0% cat GetImageAndUpgrade.sh 
cli -c "file copy ftp://10.10.10.4/ex2300/junos-arm-32-18.2R3-S2.9.tgz /tmp/junos-arm-32-18.2R3-S2.9.tgz"
cli -c "request system software add /tmp/junos-arm-32-18.2R3-S2.9.tgz force unlink no-copy reboot"

«hope it helps,

Michel

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-16-2019 05:45 AM

@LapointeMichel wrote:
  • ...
  • if you have hundreds of 2300, I strongly suggest the disabling of the miniusb console port to prevent swizzle reboot. On my sample of 250 switches, I had about 30 that were susceptible to having it.  
  • Show Chassis routing-engine will show you the last reboot reason of a switch. if a switch swizzle, turn on messages (monitor start messages)  wait a minute to see if you get a deluge of getty request. JTAC answered to me at the time that it was related to bad consol usb cabling. Since I had none connected, we decided to simply turn it off, and it seems to fix the issue. They are working on it now and I was told a PR was issued 
  • ...

 


well , 

I had a switch swizzle reboot friday the 13 ...   now it's only one in 240, on which the miniusb console port solution had been running successfully  for 2 months, but still ...  

I am sending the files to JTAC. I'll keep you posted. 

Michel

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

[ Edited ]
‎12-16-2019 05:47 AM

I worked out another option last night that seems to work for me. I boot the switch and do CTRL-C to stop uboot from loading the juniper partition. At the menu I go through the steps to get to the loader> prompt (5, then 5) and enter:

install --format tftp://<ip address>/junos-install-media-net-ex-arm-32-18.2R3.4.tgz

then walk away and let it reinstall using 18.2R3.4 (which seems to fix/ignore the space alert that 18.1 gives). Then on first boot it will go through the normal ZTP procedure to upgrade the switch to 18.2R3-S2.9 and get our template config on it.

 

The good thing is that loader is at least DHCP aware and as soon as you give it a TFTP path it starts the vme interface and gets an IP from the DHCP server so you don't have to setup the network parameters on each switch.

 

I think when Juniper starts shipping 2300s with 18.2 as the preload then we will not have to do these work arounds. But at least now I've got it workable solution for these 50 or so I'm getting in the next few weeks.

 

Thanks for sharing your procedure too; hopefully between the multiple ways here this will help others in the same boat.

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-16-2019 06:24 AM

merci !

that's worth a shot. I'll try that

Michel

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-18-2019 01:30 AM

Im in the middle of trying everything out right now and i will be reporting back with my experience Smiley Happy. But for the life of me, i cannot figure out dhcp options on windows. I had a setup working fine with an ISC DHCP server on linux but i want to use a windows laptop and i tried tftpd with dhcp but it seems options are not going through as i want them to in order for Junos to get the image file. 

 

Michel what kind of dhcp/tftp server are you using on windows?

 

Thanks!

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-18-2019 06:02 AM

 

Michel what kind of dhcp/tftp server are you using on windows?

 


Denis is managing this. I believe we have a Ubuntu server running dhcp. I just ssh to it and modify the /etc/dhcp/dhcpd.conf file depending on the config and image file I want installed when the new or reseted switch connect to it.

I don't believe we have a tftp server - I just send ftp requests. I do activate ftp on every Juniper switch. 

I can forward other questions to Denis if you need. 

Michel

by the way, I am trying another solution on the swizzle reboot problem. 240 switch have been running at least 18 hours now with no swizzle reboot. Hope it works. 

 

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

‎12-20-2019 12:58 PM

Ooof, Windows Server's DHCP Server does not make setting up the options easy. Can your run a small Linux instance in a VM in something like VirtualBox or maybe even a Docker instance for DHCP and TFTP? Configuring either one of those may be easier than converting the text values needed in Option 43 sub-options for ZTP to the hexadecimal strings required to be encapsulated in for option 43 on windows boxes. I did just find this at GitHub: https://github.com/JNPRAutomate/opt43builder I've not used it but that might be the way to get all those sub-options converted into a hex string for use in Windows DHCP.  

 

Also if you are trying to do this on a Windows 10 install instead of some version of Windows Server maybe http://tftpd32.jounin.net/ is something to look at.

 

Another option maybe Cygwin running dhcpd or maybe even Windows Services for Linux and the Ubuntu instance from the Windows Store.

 

I'm a Mac/Linux user so I can't offer much more than the results of some GoogleFu above. For something portable I'd probably grab a RaspberryPi or the like and get dhcpd, tftpd, rsyslog, and a http server running and call it done. Or maybe a inexpensive tiny laptop that would take a Linux distro.

 

Good luck!

 

Junos Automation (Scripting)

Re: EX2300-C ZTP upgrade with knobs (force unlink etc)

a week ago

good morning jgrider, 

just learned from JTAC that there is a PR related to the space issue.

PR1344512 

 

On EX2300 and EX3400 switches, when you perform multiple CLI upgrades, sometimes the software fails with "insufficient space" error

 

looking forward to a solve !

 

Still have to try your trick  

install --format tftp://<ip address>/junos-install-media-net-ex-arm-32-18.2R3.4.tgz

 

probably today:  I have to cleran up some switches. 

Michel