Automation
Automation

Scripting How-To: Automation of dual-AD software upgrade

by Juniper Employee on ‎08-02-2016 01:03 PM - edited on ‎08-23-2017 10:52 AM by Administrator Administrator
08.02.16   |   01:03 PM

Automation of dual-AD software upgrade

 

In a typical dual-aggregated Junos Fusion system software upgrade, a user needs to copy images to both aggregated devices and execute the software upgrade command. Consider a situation where each aggregated device has dual routing engines, and the complete process involves installing JUNOS software on all the four routing engines and rebooting all of them.

 

This can be simplified with SLAX automation. Using a SLAX script, the user can upgrade all the routing-engines by executing a single operational cli command. The user needs to copy the JUNOS software image to the aggregated device where this operational script is present.

 

tmp1.png

 

Figure: Dual aggregated Junos-Fusion system

 

To upgrade the JUNOS software on all the REs, the user needs to perform the following steps:

  • Copy image to aggregated devices
  • Issue:
    • ‘request system software add <image-location> re0’
    • ‘request system software add <image-location> re1’
    • ‘request system reboot both-routing-engines’
  • These steps need to be repeated on the other aggregated device that is the part of Junos Fusion system

 

SLAX Solution:

With SLAX automation, the given script performs the following steps:

 

  • Copies the JUNOS software image from one aggregation device to its peer aggregation device
  • Collects the chassis satellite information before upgrading
  • Starts installing the JUNOS software on the peer aggregation device’s routing engines and reboots those routing engines
  • If parallel option is not ‘true’, the script validates the chassis satellite state once the upgrade is completed by comparing it with the previously obtained information. If the satellite(s) state is not proper then installation is aborted.
  • Starts installing image on the current aggregation device’s routing engines and reboots those routing engines with a single op command

 

It is always recommended to upgrade one aggregated device after another in a dual aggregated system, as dual homed satellites can still be online in the aggregated device that is up, and traffic goes fine with minimal loss.

 

This operational script allows the user to specify if he/she wants to upgrade the entire system in parallel, or upgrade one aggregated device, wait till it comes up, then start upgrading the other.

 

To enable the op script, see: https://www.juniper.net/techpubs/en_US/junos15.1/topics/usage-guidelines/automation-enabling-an-op-s...

 

Usage:

Syntax:

op software-add-dual-ad

  Possible completions:

     filename  <path and package-name>

     host      <remote host name or IP address>

     force     <true to ignore warnings>

     validate  <true to check configuration compatibility>

     parallel  <true to upgrade all REs parallel>

 

Description:

Install a software package or bundle on routing-engines of both aggregation devices of a Junos Fusion dual AD setup

 

Options:

 

filename  required argument

          Value: Path to the image file

 

host      required argument

          Value: Host name or IP address in case of Junos Fusion peer ICCP address

          This value will be used to copy image and install on remote side

 

force     optional argument

          Value: true | false

          To force image installation to ignore warning, default value is true

 

validate  optional argument

          Value: true | false

          To check compatibility with current configuration, default value is false

parallel  optional argument

          Value: true | false

          true upgrades all REs parallel, false starts upgrading peer aggregated device and wait till it comes up with all FPCs then initiates upgrade locally

 

Sample output:

root@JUNOS-FUSION-01> op software-add-dual-ad filename /var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz host 10.1.1.2 force false validate false

 

hup

 

 

Pushing /var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz to re1:/var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz

hup

Verified junos-install-ex92xx-x86-64-16.1-20160622.1 signed by PackageDevelopmentEc_2016

Verified manifest signed by PackageDevelopmentEc_2016

Checking PIC combinations

Verified fips-mode signed by PackageDevelopmentEc_2016

Verified jail-runtime signed by PackageDevelopmentEc_2016

Verified jdocs signed by PackageDevelopmentEc_2016

Verified jpfe-X960 signed by PackageDevelopmentEc_2016

Verified jpfe-common signed by PackageDevelopmentEc_2016

Verified jpfe-wrlinux signed by PackageDevelopmentEc_2016

Verified jplatform-ex92xx signed by PackageDevelopmentEc_2016

Verified jsd signed by PackageDevelopmentEc_2016

Verified jsdn signed by PackageDevelopmentEc_2016

Verified jservices-crypto signed by PackageDevelopmentEc_2016

Adding jservices-crypto-x86-32-16.1-20160622.1 ...

Verified jservices-crypto-base signed by PackageDevelopmentEc_2016

Verified jservices-ipsec signed by PackageDevelopmentEc_2016

Verified jservices-ssl signed by PackageDevelopmentEc_2016

Verified jservices signed by PackageDevelopmentEc_2016

Adding jservices-x86-32-16.1-20160622.1 ...

Verified jservices-aacl signed by PackageDevelopmentEc_2016

Verified jservices-alg signed by PackageDevelopmentEc_2016

Verified jservices-appid signed by PackageDevelopmentEc_2016

Verified jservices-bgf signed by PackageDevelopmentEc_2016

Verified jservices-cpcd signed by PackageDevelopmentEc_2016

Verified jservices-hcm signed by PackageDevelopmentEc_2016

Verified jservices-idp signed by PackageDevelopmentEc_2016

Verified jservices-jdpi signed by PackageDevelopmentEc_2016

Verified jservices-jflow signed by PackageDevelopmentEc_2016

Verified jservices-llpdf signed by PackageDevelopmentEc_2016

Verified jservices-lrf signed by PackageDevelopmentEc_2016

Verified jservices-mobile signed by PackageDevelopmentEc_2016

Verified jservices-mss signed by PackageDevelopmentEc_2016

Verified jservices-nat signed by PackageDevelopmentEc_2016

Verified jservices-pcef signed by PackageDevelopmentEc_2016

Verified jservices-ptsp signed by PackageDevelopmentEc_2016

Verified jservices-rpm signed by PackageDevelopmentEc_2016

Verified jservices-sfw signed by PackageDevelopmentEc_2016

Verified jservices-voice signed by PackageDevelopmentEc_2016

Verified jsim-pfe signed by PackageDevelopmentEc_2016

Verified junos-daemons-mx signed by PackageDevelopmentEc_2016

Verified junos-daemons signed by PackageDevelopmentEc_2016

Verified junos-dp-crypto-support-mtx signed by PackageDevelopmentEc_2016

Verified junos-dp-crypto-support signed by PackageDevelopmentEc_2016

Verified junos-libs-compat32-mx signed by PackageDevelopmentEc_2016

Verified junos-libs-compat32 signed by PackageDevelopmentEc_2016

Verified junos-libs-mx signed by PackageDevelopmentEc_2016

Verified junos-libs signed by PackageDevelopmentEc_2016

Verified junos-modules-mx signed by PackageDevelopmentEc_2016

Verified junos-modules signed by PackageDevelopmentEc_2016

Verified junos-net-prd signed by PackageDevelopmentEc_2016

Verified junos-platform signed by PackageDevelopmentEc_2016

Verified junos-runtime-mx signed by PackageDevelopmentEc_2016

Verified junos-runtime signed by PackageDevelopmentEc_2016

Verified junos-vmguest-mtx signed by PackageDevelopmentEc_2016

Verified jweb signed by PackageDevelopmentEc_2016

Verified oam signed by PackageDevelopmentEc_2016

Verified os-compat32 signed by PackageDevelopmentEc_2016

Verified os-crypto signed by PackageDevelopmentEc_2016

Verified os-kernel-prd signed by PackageDevelopmentEc_2016

Verified os-libs-compat32 signed by PackageDevelopmentEc_2016

Verified os-libs signed by PackageDevelopmentEc_2016

Verified os-runtime signed by PackageDevelopmentEc_2016

Verified os-vmguest signed by PackageDevelopmentEc_2016

Verified os-zoneinfo signed by PackageDevelopmentEc_2016

Verified py-base signed by PackageDevelopmentEc_2016

Verified py-extensions signed by PackageDevelopmentEc_2016

Verified vrr-mx signed by PackageDevelopmentEc_2016

NOTICE: 'pending' set will be activated at next reboot...

Rebooting. Please wait ...

shutdown: [pid 26555]

Shutdown NOW!

Shutdown NOW!

 

System shutdown time has arrived

0

xnm:rpc results:86:(7) PCDATA invalid Char value 7

xnm:rpc results:86:(7) PCDATA invalid Char value 7

 

hup

 

Verified junos-install-ex92xx-x86-64-16.1-20160622.1 signed by PackageDevelopmentEc_2016

Verified manifest signed by PackageDevelopmentEc_2016

Checking PIC combinations

Verified fips-mode signed by PackageDevelopmentEc_2016

Verified jail-runtime signed by PackageDevelopmentEc_2016

Verified jdocs signed by PackageDevelopmentEc_2016

Verified jpfe-X960 signed by PackageDevelopmentEc_2016

Verified jpfe-common signed by PackageDevelopmentEc_2016

Verified jpfe-wrlinux signed by PackageDevelopmentEc_2016

Verified jplatform-ex92xx signed by PackageDevelopmentEc_2016

Verified jsd signed by PackageDevelopmentEc_2016

Verified jsdn signed by PackageDevelopmentEc_2016

Verified jservices-crypto signed by PackageDevelopmentEc_2016

Adding jservices-crypto-x86-32-16.1-20160622.1 ...

Verified jservices-crypto-base signed by PackageDevelopmentEc_2016

Verified jservices-ipsec signed by PackageDevelopmentEc_2016

Verified jservices-ssl signed by PackageDevelopmentEc_2016

Verified jservices signed by PackageDevelopmentEc_2016

Adding jservices-x86-32-16.1-20160622.1 ...

Verified jservices-aacl signed by PackageDevelopmentEc_2016

Verified jservices-alg signed by PackageDevelopmentEc_2016

Verified jservices-appid signed by PackageDevelopmentEc_2016

Verified jservices-bgf signed by PackageDevelopmentEc_2016

Verified jservices-cpcd signed by PackageDevelopmentEc_2016

Verified jservices-hcm signed by PackageDevelopmentEc_2016

Verified jservices-idp signed by PackageDevelopmentEc_2016

Verified jservices-jdpi signed by PackageDevelopmentEc_2016

Verified jservices-jflow signed by PackageDevelopmentEc_2016

Verified jservices-llpdf signed by PackageDevelopmentEc_2016

Verified jservices-lrf signed by PackageDevelopmentEc_2016

Verified jservices-mobile signed by PackageDevelopmentEc_2016

Verified jservices-mss signed by PackageDevelopmentEc_2016

Verified jservices-nat signed by PackageDevelopmentEc_2016

Verified jservices-pcef signed by PackageDevelopmentEc_2016

Verified jservices-ptsp signed by PackageDevelopmentEc_2016

Verified jservices-rpm signed by PackageDevelopmentEc_2016

Verified jservices-sfw signed by PackageDevelopmentEc_2016

Verified jservices-voice signed by PackageDevelopmentEc_2016

Verified jsim-pfe signed by PackageDevelopmentEc_2016

Verified junos-daemons-mx signed by PackageDevelopmentEc_2016

Verified junos-daemons signed by PackageDevelopmentEc_2016

Verified junos-dp-crypto-support-mtx signed by PackageDevelopmentEc_2016

Verified junos-dp-crypto-support signed by PackageDevelopmentEc_2016

Verified junos-libs-compat32-mx signed by PackageDevelopmentEc_2016

Verified junos-libs-compat32 signed by PackageDevelopmentEc_2016

Verified junos-libs-mx signed by PackageDevelopmentEc_2016

Verified junos-libs signed by PackageDevelopmentEc_2016

Verified junos-modules-mx signed by PackageDevelopmentEc_2016

Verified junos-modules signed by PackageDevelopmentEc_2016

Verified junos-net-prd signed by PackageDevelopmentEc_2016

Verified junos-platform signed by PackageDevelopmentEc_2016

Verified junos-runtime-mx signed by PackageDevelopmentEc_2016

Verified junos-runtime signed by PackageDevelopmentEc_2016

Verified junos-vmguest-mtx signed by PackageDevelopmentEc_2016

Verified jweb signed by PackageDevelopmentEc_2016

Verified oam signed by PackageDevelopmentEc_2016

Verified os-compat32 signed by PackageDevelopmentEc_2016

Verified os-crypto signed by PackageDevelopmentEc_2016

Verified os-kernel-prd signed by PackageDevelopmentEc_2016

Verified os-libs-compat32 signed by PackageDevelopmentEc_2016

Verified os-libs signed by PackageDevelopmentEc_2016

Verified os-runtime signed by PackageDevelopmentEc_2016

Verified os-vmguest signed by PackageDevelopmentEc_2016

Verified os-zoneinfo signed by PackageDevelopmentEc_2016

Verified py-base signed by PackageDevelopmentEc_2016

Verified py-extensions signed by PackageDevelopmentEc_2016

Verified vrr-mx signed by PackageDevelopmentEc_2016

NOTICE: 'pending' set will be activated at next reboot...

Rebooting. Please wait ...

shutdown: [pid 29823]

Shutdown NOW!

Shutdown NOW!

 

System shutdown time has arrived

 

0

 

]

 

hup

 

Pushing /var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz to re1:/var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz

hup

Verified junos-install-ex92xx-x86-64-16.1-20160622.1 signed by PackageDevelopmentEc_2016

Verified manifest signed by PackageDevelopmentEc_2016

Checking PIC combinations

Verified fips-mode signed by PackageDevelopmentEc_2016

Verified jail-runtime signed by PackageDevelopmentEc_2016

Verified jdocs signed by PackageDevelopmentEc_2016

Verified jpfe-X960 signed by PackageDevelopmentEc_2016

Verified jpfe-common signed by PackageDevelopmentEc_2016

Verified jpfe-wrlinux signed by PackageDevelopmentEc_2016

Verified jplatform-ex92xx signed by PackageDevelopmentEc_2016

Verified jsd signed by PackageDevelopmentEc_2016

Verified jsdn signed by PackageDevelopmentEc_2016

Verified jservices-crypto signed by PackageDevelopmentEc_2016

Adding jservices-crypto-x86-32-16.1-20160622.1 ...

Verified jservices-crypto-base signed by PackageDevelopmentEc_2016

Verified jservices-ipsec signed by PackageDevelopmentEc_2016

Verified jservices-ssl signed by PackageDevelopmentEc_2016

Verified jservices signed by PackageDevelopmentEc_2016

Adding jservices-x86-32-16.1-20160622.1 ...

Verified jservices-aacl signed by PackageDevelopmentEc_2016

Verified jservices-alg signed by PackageDevelopmentEc_2016

Verified jservices-appid signed by PackageDevelopmentEc_2016

Verified jservices-bgf signed by PackageDevelopmentEc_2016

Verified jservices-cpcd signed by PackageDevelopmentEc_2016

Verified jservices-hcm signed by PackageDevelopmentEc_2016

Verified jservices-idp signed by PackageDevelopmentEc_2016

Verified jservices-jdpi signed by PackageDevelopmentEc_2016

Verified jservices-jflow signed by PackageDevelopmentEc_2016

Verified jservices-llpdf signed by PackageDevelopmentEc_2016

Verified jservices-lrf signed by PackageDevelopmentEc_2016

Verified jservices-mobile signed by PackageDevelopmentEc_2016

Verified jservices-mss signed by PackageDevelopmentEc_2016

Verified jservices-nat signed by PackageDevelopmentEc_2016

Verified jservices-pcef signed by PackageDevelopmentEc_2016

Verified jservices-ptsp signed by PackageDevelopmentEc_2016

Verified jservices-rpm signed by PackageDevelopmentEc_2016

Verified jservices-sfw signed by PackageDevelopmentEc_2016

Verified jservices-voice signed by PackageDevelopmentEc_2016

Verified jsim-pfe signed by PackageDevelopmentEc_2016

Verified junos-daemons-mx signed by PackageDevelopmentEc_2016

Verified junos-daemons signed by PackageDevelopmentEc_2016

Verified junos-dp-crypto-support-mtx signed by PackageDevelopmentEc_2016

Verified junos-dp-crypto-support signed by PackageDevelopmentEc_2016

Verified junos-libs-compat32-mx signed by PackageDevelopmentEc_2016

Verified junos-libs-compat32 signed by PackageDevelopmentEc_2016

Verified junos-libs-mx signed by PackageDevelopmentEc_2016

Verified junos-libs signed by PackageDevelopmentEc_2016

Verified junos-modules-mx signed by PackageDevelopmentEc_2016

Verified junos-modules signed by PackageDevelopmentEc_2016

Verified junos-net-prd signed by PackageDevelopmentEc_2016

Verified junos-platform signed by PackageDevelopmentEc_2016

Verified junos-runtime-mx signed by PackageDevelopmentEc_2016

Verified junos-runtime signed by PackageDevelopmentEc_2016

Verified junos-vmguest-mtx signed by PackageDevelopmentEc_2016

Verified jweb signed by PackageDevelopmentEc_2016

Verified oam signed by PackageDevelopmentEc_2016

Verified os-compat32 signed by PackageDevelopmentEc_2016

Verified os-crypto signed by PackageDevelopmentEc_2016

Verified os-kernel-prd signed by PackageDevelopmentEc_2016

Verified os-libs-compat32 signed by PackageDevelopmentEc_2016

Verified os-libs signed by PackageDevelopmentEc_2016

Verified os-runtime signed by PackageDevelopmentEc_2016

Verified os-vmguest signed by PackageDevelopmentEc_2016

Verified os-zoneinfo signed by PackageDevelopmentEc_2016

Verified py-base signed by PackageDevelopmentEc_2016

Verified py-extensions signed by PackageDevelopmentEc_2016

Verified vrr-mx signed by PackageDevelopmentEc_2016

NOTICE: 'pending' set will be activated at next reboot...

Rebooting. Please wait ...

shutdown: [pid 27181]

Shutdown NOW!

Shutdown NOW!

 

System shutdown time has arrived

0

                                                                              

*** FINAL System shutdown message from root@JUNOS-FUSION-01 ***            

 

System going down IMMEDIATELY                                                 

 

                                                                              

Stopping cron.

Waiting for PIDS: 17816.

.

Jul 14 02:24:00 jlaunchd: clksyncd-service (PID 18457) terminate signal 15 sent

Jul 14 02:24:01 jlaunchd: ethernet-link-fault-management (PID 18706) terminate signal 15 sent

Jul 14 02:24:01 jlaunchd: mib-process (PID 18725) terminate signal 15 sent

Jul 14 02:24:05 jlaunchd: sflow-service (PID 18715) exited with status=0 Normal Exit

Jul 14 02:24:05 jlaunchd: l2-learning (PID 18699) exited with status=0 Normal Exit

Jul 14 02:24:05 jlaunchd: app-engine-virtual-machine-management-service (PID 17828) exited with status=0 Normal Exit

Waiting (max 60 seconds) for system process `vnlru' to stop...done

Waiting (max 60 seconds) for system process `bufdaemon' to stop...done

Waiting (max 60 seconds) for system process `syncer' to stop...

Syncing disks, vnodes remaining...0 0 0 done

All buffers synced.

Uptime: 1h31m12s

Khelp module "jsocket" can't unload until its refcount drops from 2 to 0.

 

<<<<<<<<<<<<<<<<<<<<< BOOT LOG messages >>>>>>>>>>>>>>>>>>>>>>

 

FreeBSD/amd64 (JUNOS-FUSION-01) (ttyu0)

 

login:

 

SLAX script details:

Arguments:

$filename        - **REQUIRED** Path to the image to be installed.

$host            - **REQUIRED** Host Name/IP address or ICCP IP

$force           - *OPTIONAL* force addition of package (ignore warnings)

                 - true/false [true]

$validate        - *OPTIONAL* check compatibility with current configuration

                 - true/false [false]

Templates:

generate-sshkey         generate ssh public key on local box

copy-sshkey             copy ssh public key to remote server for authentication

copy-file               copy junos image to remote box

install-image           install junos image on the box

build-upgrade-string    build the cli to install junos image on given re        

check-satellite         check the status of chassis satellite

determine-local-re      determine the master re

get-satellite-info      get the chassis satellite information

check-ping              check the reachability of the host

 

Implementation details:

  • The script takes two necessary arguments – filename and host.
    • filename should point to the JUNOS software image that is expected to be downloaded on the local device by the user before beginning the script execution.
    • host is the IP address / name of the remote device to install the JUNOS software image. In the case of Junos Fusion, the ICCP address can be used as a host IP.
  • For installing the software image there are two optional arguments – force and validate.
    • force would ignore the warnings at the time of installation. Default force is false.
    • validate would check compatibility with current configuration. Default validate is true. 
  • The script generates ssh key on local device using generate-sshkey
  • The id_rsa.pub key generated is then copied to the list of known users on the remote device using copy-sshkey. The copy-sshkey template uses file-put rpc to copy the content of the key on the remote server. This step is required to facilitate image copy without using a password.
  • The image present at path passed as filename is then copied to the /var/tmp of the remote device using copy-file template which uses file-copy rpc.
  • The chassis satellite information of the peer aggregation device is obtained using the template get-satellite-info and stored in a variable locally.
  • The script then determines the backup RE of the remote device using determine-local-re template and installs the image on it. After that, the image is installed on the master RE of the remote device.
  • If the parallel option is not set to ‘true’,
    • The script sleeps for 5 minutes, which is the expected time for the reboot of the aggregation device and waits for the interfaces to come up using the template check-ping.
    • The script then checks for the satellite information and matches with the one obtained before upgrading.
    • In case of mismatch the script sleeps for 10 sec and tries again to obtain the status of satellite(s). There are 10 such trials, after which the script is aborted from further upgrading.
    • In the satellite(s) state(s) is proper the script goes ahead with upgrading the local device.

           All this is taken care of by template check-satellite.

 

  • The JUNOS software image is then installed on the local device REs in similar order.
  • The local and remote devices are rebooted after successful installation.

 

Though this is written for Junos Fusion, it can be used to upgrade MC-LAG peer boxes. Also, this script can be extended to use for single RE boxes as well and to upgrade multiple devices (standalone and VCs) using a single point.

 

/**************************************************************************
Script 			- 	software-add-dual-ad.slax
Author 			- 	Dharmik Thakkar  (dharmikt@juniper.net)
		 		Anil Kumar A 	 (anilak@juniper.net)
		 		Rushi Trivedi	 (rtrivedi@juniper.net)
Functionality 	        -	This slax script installs JUNOS software on all the 
				routing engines in aggregation devices of Junos Fusion.
***************************************************************************/

version 1.0;

ns junos = "http://xml.juniper.net/junos/*/junos";
ns xnm = "http://xml.juniper.net/xnm/1.1/xnm";
ns jcs = "http://xml.juniper.net/junos/commit-scripts/1.0";

import "../import/junos.xsl";

/**************************************************************************
List of arguments for the script - 
$filename 	- **REQUIRED** Path to the image to be installed.
$host 		- **REQUIRED** Host Name/IP address or ICCP IP
$force 		- *OPTIONAL* force addition of package (ignore warnings) - true/false [true]
$validate 	- *OPTIONAL* check compatibility with current configuration - true/false [false]
***************************************************************************/

var $arguments = {
     <argument> {	
          <name> "filename";
          <description> "**REQUIRED** Path to the image to be installed."; 
     }
     <argument> {	
          <name> "host";
          <description> "**REQUIRED** Host Name/IP address or ICCP IP"; 	
     }
     <argument>{
     	  <name> "force";
     	  <description> "*OPTIONAL* force addition of package (ignore warnings) - true/false [true]";
     }
     <argument>{
     	  <name> "validate";
     	  <description> "*OPTIONAL* check compatibility with current configuration - true/false [false]";
     }
     <argument>{
     	  <name> "parallel";
     	  <description> "*OPTIONAL* install junos software on both ADs parellely, traffic might get dropped - true/false [false]";
     }
}

param $host;
param $filename;
param $force = "true";
param $validate = "false";
param $parallel = "false";
var $username = "root";
var $password = "Embe1mpls";
/*
var $filename = "/var/tmp/junos-install-ex92xx-x86-64-16.1-20160622.1.tgz";
var $host="10.1.1.2";
*/


/*
template get-iccp-link($local){
		var $iccp = <get-inter-chassis-control-protocol-information>;
		var $out = jcs:execute($local,$iccp);
		var $pattern = "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}";
		var $peer_ip = jcs:regex($pattern,$out);
		expr $peer_ip;
}
*/

/**************************************************************************
template 		-	generate-sshkey
arguments 		- 	*$local* ->	"handle to local connection"
functionality	        -	generate ssh public key on local box
***************************************************************************/

template generate-sshkey($local){
	
	var $remove_gen = {
    	<request-shell-execute> {
				<command> "rm -f /root/.ssh/id_rsa";
		  }
	}
	var $add_gen = {
	 	<request-shell-execute> {
			<command> 'ssh-keygen -t rsa -f /root/.ssh/id_rsa -N ""';
		}
	}
	var $cat = {
		<request-shell-execute>{
			<command> 'cat /root/.ssh/id_rsa.pub';
		}
	}

	var $rem-key = jcs:execute($local,$remove_gen);
    var $add-key = jcs:execute($local,$add_gen);
	var $key = jcs:execute($local,$cat);
	expr $key;

}


/**************************************************************************
template 		-	copy-sshkey
arguments 		- 	*$local* 	  -> "handle to local connection"
				*$connection*     -> "handle to remote server connection"
functionality	        -	copy ssh public key to remote server for authentication
***************************************************************************/

template copy-sshkey($local, $connection){
	
	var $get-key = {
		<request-shell-execute> {
			<command> "cat /root/.ssh/id_rsa.pub";
		}
	}  
	var $key = jcs:execute($local,$get-key);
	if ($key//xnm:error) {
		 copy-of ($out//xnm:error);
	}
	
	var $fileput = {
		   <file-put> {
				<filename>"/root/.ssh/authorized_keys";
				<encoding>"ascii";
				<permission>"0644";
				<delete-if-exist>;
				<file-contents> $key;
			}
	}
	var $out = jcs:execute($connection, $fileput);
	if ($out//xnm:error) {
		 copy-of ($out//xnm:error);
	}
	
	var $cat = {
	<request-shell-execute>{
		<command> 'cat /root/.ssh/authorized_keys';
		}
	}
	var $temp = jcs:execute($connection,$cat);
	if ($temp//xnm:error) {
		 copy-of ($out//xnm:error);
	}
	expr $temp;
}

/**************************************************************************
template 		-	copy-file
arguments 		- 	*$local* -> "handle to local connection"
				*$host*  -> "remote server name/ip"
functionality	        -	copy junos image to remote server
***************************************************************************/

template copy-file($local, $host){
	var $dest_filename = 'root@'_ $host _ ':/var/tmp';
    	var $copy-rpc = <file-copy> {
        	<source> $filename;
        	<destination> $dest_filename;
    	};
		var $out = jcs:execute($local, $copy-rpc);
		expr $out;
}

/**************************************************************************
template 		-	install-image
arguments 		- 	*$connection* -> "handle to remote server connection"
				*$filename*   -> "path to junos image to be installed"
				*$force*      -> "force addition of package"
				*$validate*   -> "check compatibility with configuration"
functionality	        -	install junos image on the box
***************************************************************************/

template install-image($connection, $filename, $force, $validate) {
    
    if( not( $filename ) ) {
        <xsl:message terminate="yes"> "You must specify the JUNOS upgrade file.";
    }
    var $local-re = { call determine-current-re($connection); }
    var $other-re = {
        if( $local-re == "re0" ) {
            expr "re1";
        }
        else {
            expr "re0";
        }
    }

    var $other-re-upgrade-string = { 
        call build-command-string( $re=$other-re, $filename, $force, $validate ); 
    }
    var $other-re-command = <command> $other-re-upgrade-string;
    
    var $other-result = jcs:execute($connection, $other-re-command );

    for-each( $other-result//text() ) {
        expr jcs:output( . );
    }
    
    var $local-re-upgrade-string = { 
        call build-command-string( $re=$local-re, $filename, $force, $validate ); 
    }
    var $local-re-command = <command> $local-re-upgrade-string;
	
    var $local-result = jcs:execute($connection, $local-re-command);
    
    for-each( $local-result//text() ) {
        expr jcs:output( . );
    }
}

/**************************************************************************
template 		-	build-upgrade-string
arguments 		- 	*$re*	      -> "routing engine to install image"
				*$filename*   -> "path to junos image to be installed"
				*$force*      -> "force addition of package"
				*$validate*   -> "check compatibility with configuration"
functionality	        -	build the cli to install junos image on given re
***************************************************************************/

template build-command-string($re, $filename, $force, $validate) {

    if( $force == "true" ) {
        if( $validate == "false" ) {
            expr "request system software add " _ $filename _ " reboot " _ $re _ " force " _ " no-validate";
        }
        else {
            expr "request system software add " _ $filename _ " reboot " _ $re _ " force";
        }
    }
    else {
        if( $validate == "false" ) {
            expr "request system software add " _ $filename _ " reboot " _ $re _ " no-validate";
        }
        else {
            expr "request system software add " _ $filename _ " reboot " _ $re;
        }
    }
}

/**************************************************************************
template 		-	determine-local-re
arguments 		- 	*$connection* -> "handle to server connection"
functionality	        -	determine the master re
***************************************************************************/

template determine-current-re($connection) {
	
    var $get-routing-engine = <get-route-engine-information>;
    var $routing-engine-info = jcs:execute( $connection, $get-routing-engine );
	
	if( not( $routing-engine-info/route-engine/slot ) ) {
		expr "re0";
	}
	else if( $routing-engine-info/route-engine[slot == "0"]/load-average-one ) {
		expr "re0";
	}
	else if( $routing-engine-info/route-engine[slot == "1"]/load-average-one ) {
		expr "re1";
	}
	else {
		expr "re0";
	}
}

/**************************************************************************
template 		-	check-ping
arguments 		- 	*$local*  -> "handle to local device connection"
				*host*	  -> "peer AD ip / iccp ip"
				*counter* -> "counter number of retries for ping"
functionality	        -	check the reachability of the host
***************************************************************************/

template check-ping($local, $host, $counter=0){
	var $ping-rpc = {
		<ping>{
			<host> $host;
			<count> 1;
		}
	}
	var $ping-out = jcs:execute($local, $ping-rpc);
	if($ping-out/ping-success){
		expr jcs:output("!!! Ping Success. !!!");
		expr "true";
	}else{
		if($counter < 20){
			expr jcs:output("!!! Ping Failed. !!!");
			expr jcs:output("!!! Sleeping for 10 sec. !!!");
			expr jcs:sleep(10);
			call check-ping($local, $host, $counter = $counter+1);
		}else{
			expr jcs:output("!!! The host is not reachable after upgradation. !!!");
			expr jcs:output("!!! Aborting further upgradation of local device. !!!");
			expr jcs:close($local);
		}
	}
}

/**************************************************************************
template 		-	get-satellite-info
arguments 		- 	*$connection*  -> "handle to peer AD connection"
functionality	        -	get the chassis satellite information
***************************************************************************/

template get-satellite-info($connection){
	var $satellite-rpc = <get-chassis-satellite-information>;
		var $satellite := jcs:execute($connection, $satellite-rpc);
		if(jcs:empty($satellite)){
			expr "false";
		}else{
			var $each_satellite := $satellite/satellite;
			mvar $satellite-info;
			for-each($each_satellite){
				append $satellite-info += <satellite>{
					<slot> ./slot-id;
					<state> ./operation-state;
				}                                                                          
			}
			expr jcs:output("slot-id \tsatellite state");
			for-each($satellite-info/satellite){ 
				expr jcs:output(./slot _ "\t\t" _ ./state);
			}
			expr $satellite-info;
		}
} 


/**************************************************************************
template 		-	check-satellite
arguments 		- 	*$connection*  	 -> "handle to peer AD connection"
				*satellite-info* -> "satellite info before upgrade"
					*counter* 		 -> "counter number of retries"
functionality	-	check the status of chassis satellite
***************************************************************************/

template check-satellite($connection, $satellite-info, $counter=0){
	var $satellite-new-info := {call get-satellite-info($connection);}
	if($satellite-new-info == "false"){
		expr jcs:output("!!! Satellites are not up yet. !!!");
		expr jcs:output("!!! Sleeping for 10 sec. !!!");
		expr jcs:sleep(10);
		if($counter < 15){
			call check-satellite($connection, $satellite-info, $counter = $counter+1);
		}else{
			expr jcs:output("!!! The satellites did not come up even after trying 15 times. !!!");
			expr "false";
		}
	}else{
		if($satellite-new-info == $satellite-info){
			expr jcs:output("!!! The satellites are in proper state after peer aggregation device upgrade. !!!");
			expr "true"; 
		}else{
			if($counter < 15){
				expr jcs:output("!!! The satellites are not in proper state after peer aggregation device upgrade. !!!");
				expr jcs:output("!!! Trying again. !!!");
				expr jcs:output("!!! Sleeping for 10 sec. !!!");
				expr jcs:sleep(10);
				call check-satellite($connection, $satellite-info, $counter = $counter+1);
			}else{
				expr jcs:output("!!! The satellites are not in proper state even after trying 10 times. !!!");
				expr "false";
			}
		}	
	}
}
/**************************************************************************
template 		-	match
functionality	-	main template to start script execution
***************************************************************************/

match /{
	<op-script-results> {
		var $local = jcs:open();
		var $connection = jcs:open($host, $username, $password);
		if ($connection) {
			expr jcs:output("!!! Generating the ssh public key on the local device. !!!");
			var $cat-key := {call generate-sshkey($local);}
			expr jcs:output("!!! Copying the ssh public key to the remote devices' list of known hosts. !!!");
			var $copy := {call copy-sshkey($local,$connection);}
			expr jcs:output("!!! Copying the JUNOS software image to the remote device. !!!");
			var $copyfile := {call copy-file($local, $host);}
			expr jcs:output("!!! Getting the satellite state before upgrading the aggregation device. !!!");
			var $satellite-info := {call get-satellite-info($connection);}
			expr jcs:output("!!! Installing JUNOS software on the remote device. !!!");
			expr jcs:output("!!! Please wait while the installation is completed. !!!");
			var $other-re := {call install-image($connection, $filename, $force, $validate);}
			expr jcs:close($connection);
			if($parallel == "false"){
				expr jcs:output("!!! Sleeping for 450 sec. !!!");
				expr jcs:output("!!! Waiting for interfaces to come up. !!!");
				expr jcs:sleep(450);
				var $ping-success = {call check-ping($local, $host);}
				var $new_connection = jcs:open($host, $username, $password);
				if($new_connection){
					expr jcs:output("!!! Checking the satellite status before going ahead !!!");
					var $satellite-status = {call check-satellite($connection = $new_connection, $satellite-info);}
					if($satellite-status == "true"){
						expr jcs:output("!!! Going ahead with local aggregation device upgradation. !!!");
						expr jcs:close($new_connection);
					}else{
						expr jcs:output("!!! Aborting the upgradation for local aggregation device. !!!");
						expr jcs:output("!!! There is some problem after upgradation with peer aggregation device. !!!");
						expr jcs:output("!!! Manually debug the peer aggregation device. !!!");
						expr jcs:close($new_connection);
						expr jcs:close($local);
						<xsl:message terminate="yes"> "!!! Script Aborted. !!!";
					}
				}else{
					expr jcs:output("!!! No connection to host. !!!");
					expr jcs:output("!!! Aborting further upgradation. !!!");
					expr jcs:close($local);
					<xsl:message terminate="yes"> "!!! Script Aborted. !!!";
				}
			}
			expr jcs:output("!!! Installing the JUNOS software on the local device. !!!");
			expr jcs:output("!!! Please wait while the installation is completed. !!!");
			var $re := {call install-image($connection=$local, $filename, $force, $validate);}
		}else{
			expr jcs:output("!!! No connection to host. !!!");
			<xsl:message terminate="yes"> "!!! Script Aborted. !!!";
		}
		expr jcs:close($local);
	}
}