SRX

Hi guys, 

I´m trying to upgrade the Junos on a SRX340 chassis cluster, currently running 15.1X49-D170.4.

Attending to the upgrade path shown here https://www.juniper.net/documentation/en_US/junos/topics/topic-map/upgrading-and-downgrading-to-upgraded-bsd.html#id-upgrading-junos-os-with-upgraded-freebsd , I should be able to upgrade from 15.1 to 17.3, so I downloaded 17.3R1.10 and run the upgrade with

request system software add /var/tmp/junos-srxsme-17.3R1.10.tgz

But it failed verifying the configuration, so I made some research and found that this error could be be bypassed by adding "no-validate" at the end of the command. After that, rebooted both nodes but none of them booted up and entered in panic mode with the prompt db>

Then made a cold reboot and fortunatelly both nodes boot up fine, but with the previous Junos version, 15.1X49-D170.4

Any help? 

Thanks

Hello,

This should work. I checked some reported issues and did not find anything similar. We had some kernel panic issues earlier, but honestly the code you are running should not be impacted.

https://kb.juniper.net/InfoCenter/index?page=content&id=TSB17581

I would isolate one of the nodes. Only have console and mgt connected and attempt the upgrade again. If you still see the node going into db prompt, you may open a JTAC case to investigate, else try installing the image using a bootable usb.

https://kb.juniper.net/InfoCenter/index?page=content&id=KB16652  (Method 3)

I hope this helps. Regards,

Vikas

Hi Trasgu,

If you ever stuck in db> prompt, please issue db> cont or db> reset.

In case if the above command, doesn't help we can proceed with the reboot of the device. If you were unable to upgrade the device from the CLI, you can proceed with the USB Bootloader upgrade method - https://www.juniper.net/documentation/en_US/junos12.1x44/topics/task/installation/security-junos-os-boot-loader-usb-storage-device-srx-series-device-installing.html

A friend of mine of mine recommended to try the upgrade directly to version 18, with the no-validate , so I tried but got exactly the same result, the cluster went to hell.... No way to upgrade it!!!???

root@SPCFW-BRAVO> ... add /var/tmp/junos-srxsme-18.1R3.3.tgz no-validate    
Formatting alternate root (/dev/da0s1a)...
/dev/da0s1a: 588.2MB (1204616 sectors) block size 16384, fragment size 2048
        using 4 cylinder groups of 147.06MB, 9412 blks, 18944 inodes.
super-block backups (for fsck -b #) at:
 32, 301216, 602400, 903584
saving package file in /var/sw/pkg ...
Installing package '/altroot/cf/packages/install-tmp/junos-18.1R3.3' ...
Verified junos-boot-srxsme-18.1R3.3.tgz signed by PackageProductionEc_2018 method ECDSA256+SHA256
Verified junos-srxsme-18.1R3.3-domestic signed by PackageProductionEc_2018 method ECDSA256+SHA256
JUNOS 18.1R3.3 will become active at next reboot
WARNING: A reboot is required to load this software correctly
WARNING:     Use the 'request system reboot' command
WARNING:         when software installation is complete
Saving state for rollback ...

{secondary:node1}
root@SPCFW-BRAVO>                                                                                
*** FINAL System shutdown message from root@SPCFW-BRAVO ***                  

System going down IMMEDIATELY                                                  

                                                                               
Sep 10 00:01:54 init: usb-control (PID 1353) exited with status=0 Normal Exit
SSWaiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `vnlru_mem' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 0 0 0 done

syncing disks... All buffers synced.
Uptime: 34m51s
Rebooting...
cpu_reset: Stopping other CPUs


SPI stage 1 bootloader (Build time: May  3 2016 - 23:48:30)
early_board_init: Board type: SRX_340

U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:48:31)

SRX_340 board revision major:1, minor:10, serial #: CY2519AF0258
OCTEON CN7130-AAP pass 1.2, Core clock: 1200 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10fc00000, size: 0x400000
DRAM: 4 GiB
Clearing DRAM...... done
Using default environment

SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
Found valid SPI bootloader at offset: 0x90000, size: 1481840 bytes


U-Boot 2013.07-JNPR-3.1 (Build time: May 03 2016 - 23:50:19)

Using DRAM size from environment: 4096 MBytes
checkboard siege 
SATA0: not available
SATA1: not available
SATA BIST STATUS = 0x0
SRX_340 board revision major:1, minor:10, serial #: CY2519AF0258
OCTEON CN7130-AAP pass 1.2, Core clock: 1200 MHz, IO clock: 600 MHz, DDR clock: 667 MHz (1334 Mhz DDR)
Base DRAM address used by u-boot: 0x10f000000, size: 0x1000000
DRAM: 4 GiB
Clearing DRAM...... done
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
PCIe: Port 0 link active, 1 lanes, speed gen2
PCIe: Link timeout on port 1, probably the slot is empty
PCIe: Port 2 not in PCIe mode, skipping
Net:   octrgmii0
octeon_fdt_broadcom_config: Unknown broadcom phy for octrgmii0
Interface 4 has 1 ports (AGL)
Type the command 'usb start' to scan for USB storage devices.

Boot Media: eUSB usb 
Found TPM SLB9660 TT 1.2 by Infineon
TPM initialized
Hit any key to stop autoboot:  0 
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
SF: 1048576 bytes @ 0x200000 Read: OK
## Starting application at 0x8f0000a0 ...
Consoles: U-Boot console  
Found compatible API, ver. 3.1
USB1:   
Starting the controller
USB XHCI 1.00
scanning bus 1 for devices... 2 USB Device(s) found
USB0:   
Starting the controller
USB XHCI 1.00
scanning bus 0 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found

FreeBSD/MIPS U-Boot bootstrap loader, Revision 2.8
(slt-builder@svl-ssd-build-vm06.juniper.net, Tue Feb 10 00:32:30 PST 2015)
Memory: 4096MB
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
[0]Booting from eUSB slice 1
/boot/init.4th loaded.
Loading /boot/defaults/loader.conf 
/kernel data=0xfcbef0+0x1a62bc syms=[0x4+0xb29e0+0x4+0x10ff75]


Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]...               
Kernel entry at 0x801000c0 ...
init regular console
Primary ICache: Sets 16 Size 128 Asso 39
Primary DCache: Sets 8 Size 128 Asso 32
Secondary DCache: Sets 1024 Size 128 Asso 4
CIU_FUSE 0xf/0xf
GDB: debug ports: uart
GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
kld_map_v: 0x8ff80000, kld_map_p: 0x0
Running in PARTITIONED TLB MODE
Copyright (c) 1996-2018, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
JUNOS 18.1R3.3 #0: 2018-08-30 04:35:30 UTC
    builder@monarth:/volume/build/junos/18.1/release/18.1R3.3/obj/octeon/junos/bsd/kernels/JSRXNLE/kernel
can't re-use a leaf (debug)!
JUNOS 18.1R3.3 #0: 2018-08-30 04:35:30 UTC
    builder@monarth:/volume/build/junos/18.1/release/18.1R3.3/obj/octeon/junos/bsd/kernels/JSRXNLE/kernel
real memory  = 4294967296 (4194304K bytes)
avail memory = 2370215936 (2260MB)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
Security policy loaded: Junos MAC/veriexec (mac_veriexec)
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
MAC/veriexec fingerprint module loaded: SHA1
MAC/veriexec fingerprint module loaded: SHA256
random: <Software, Yarrow> initialized
cpu0 on motherboard
: CAVIUM's OCTEON 70XX/71XX CPU Rev. 0.2 with no FPU implemented
        L1 Cache: I size 78kb(128 line), D size 32kb(128 line), thirty two way.
        L2 Cache: Size 512kb, 4 way
obio0 on motherboard
uart0: <Octeon-16550 channel 0> on obio0
uart0: console (9600,n,8,1)
twsi0 on obio0
set clock 0x58
xhci0: <Cavium Octeon 7xxx xHCI Host Driver> on obio0
usb0: <USB bus for xHCI Controller> on xhci0
usb0: USB revision 3.0
uhub0: vendor 0x0000 XHCI root hub, class 9/0, rev 3.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
xhci1: <Cavium Octeon 7xxx xHCI Host Driver> on obio0
usb1: <USB bus for xHCI Controller> on xhci1
usb1: USB revision 3.0
uhub1: vendor 0x0000 XHCI root hub, class 9/0, rev 3.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
cpld0 on obio0
pcib0: <Cavium on-chip PCIe HOST bridge> on obio0
Disabling Octeon big bar support
pcib0: Initialized controller
pci0: <PCI bus> on pcib0
pci0: <network, ethernet> at device 0.0 (no driver attached)
pci0: <network, ethernet> at device 0.1 (no driver attached)
ahci0: <Cavium Octeon AHCI> on obio0
ahci0: AHCI v1.30 controller with 2 6Gbps ports, PM supported
ata0: <Cavium Octeon AHCI Channel> on ahci0
ata1: <Cavium Octeon AHCI Channel> on ahci0
gblmem0 on obio0
octpkt0: <Octeon RGMII> on obio0
cfi0: <Macronix MX25L64 - 8MB> on obio0
cfi1: <Macronix MX25L64 - 8MB> on obio0
octagl0: <Octeon AGL> on obio0
umass0: Swissbit USB Flash Module, rev 2.00/1.00, addr 2
miibus0: <MII bus> on octagl0
brgphy0: <BCM54616S 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
Timecounter "mips" frequency 1200000000 Hz quality 0
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
random: unblocking device.
Kernel thread "wkupdaemon" (pid 60) exited prematurely.
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
xhci0: ERROR! udev1-2 Ring, not running! endpoint state 2
(da0:umass-sim0:0:0:0): got CAM status 0x4
(da0:umass-sim0:0:0:0): fatal error, failed to attach to device
(da0:umass-sim0:0:0:0): lost device
(da0:umass-sim0:0:0:0): removing device entry
Opened disk da0 -> 6
Trying to mount root from ufs:/dev/da0s1a

Manual root filesystem specification:
  <fstype>:<device>  Mount <device> using filesystem <fstype>
                       eg. ufs:/dev/da0a
  ?                  List valid disk boot devices
  <empty line>       Abort manual input

mountroot> Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot
Ignoring watchdog timeout during boot/reboot

panic: Root mount failed, startup aborted.
cpuid = 0
KDB: stack backtrace:
SP 0: not in kernel
uart_sab82532_class+0x0 (0,0,0,0) ra 0 sz 0
pid 1, process: swapper
###Entering boot mastership relinquish phase
KDB: enter: panic
[thread pid 1 tid 100006 ]
Stopped at      kdb_enter+0x810:        lui     a0,0x80e5
db> 

Hi Trasgu,

Found the problem, see the below TSB:

https://kb.juniper.net/InfoCenter/index?page=content&id=TSB17581

Due to a required Junos eUSB flash driver update, SRX3xx devices purchased or received via RMA after June 14, 2019 require a minimum Junos version of:

• Junos 15.1X49-D150 and higher
• Junos 17.4R3 and higher
• Junos 18.2R2 and higher
• Junos 18.3R1 and all subsequent releases

The only time you should use no validate is when you have a kb article that says to as part of a larger procedure.

The key here will likely be to determine why validation of the config failed.

Use just the validate option next time and I think you add either extensive or detail to get the reasons.

What was the configuration which failed during validation?

The configuration was really really simple and basic, as this is a new cluster for a branch office that I´m configuring when I have some spare time. No objects, no firewall rules, only one reth if... nothing at all. That made me surprised when saw the configuration validation error... !!!??? crazy. The current config below:

root@SPCFW-ALPHA> show configuration 
## Last commit: 2019-09-07 20:14:18 CEST by root
version 15.1X49-D170.4;
groups {
    node0 {
        system {
            host-name SPCFW-ALPHA;
        }
        interfaces {
            fxp0 {
                unit 0 {
                    family inet {
                        address 10.101.44.1/24;
                    }
                }
            }
        }
    }
    node1 {
        system {
            host-name SPCFW-BRAVO;
        }
        interfaces {
            fxp0 {
                unit 0 {
                    family inet {
                        address 10.101.44.2/24;
                    }
                }
            }
        }
    }
}
apply-groups "${node}";
system {
    time-zone Europe/Madrid;
    root-authentication {
        encrypted-password "$"; ## SECRET-DATA
    }
    name-server {
        8.8.8.8;
        8.8.4.4;
    }
    services {
        ssh;
        netconf {
            ssh;
        }
        web-management {
            https {
                system-generated-certificate;
            }
        }
    }
    syslog {
        archive size 100k files 3;
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    max-configurations-on-flash 5;
    max-configuration-rollbacks 5;
    license {
        autoupdate {                    
            url https://ae1.juniper.net/junos/key_retrieval;
        }
    }
    ntp {
        server 69.164.198.192 prefer;
        server 216.239.35.8 prefer;
    }
    phone-home {
        server https://redirect.juniper.net;
        rfc-complaint;
    }
}
chassis {
    alarm {
        management-ethernet {
            link-down ignore;
        }
    }
    cluster {
        control-link-recovery;
        reth-count 5;
        redundancy-group 0 {
            node 0 priority 200;
            node 1 priority 100;
        }
        redundancy-group 1 {
            node 0 priority 200;
            node 1 priority 100;
            preempt;
        }
    }
}
security {
    log {
        mode stream;
        report;
    }
    screen {
        ids-option untrust-screen {
            icmp {
                ping-death;
            }
            ip {
                source-route-option;
                tear-drop;
            }
            tcp {
                syn-flood {
                    alarm-threshold 1024;
                    attack-threshold 200;
                    source-threshold 1024;
                    destination-threshold 2048;
                    timeout 20;
                }
                land;
            }
        }
    }
    zones {
        security-zone Internal {
            host-inbound-traffic {
                system-services {
                    all;
                }
                protocols {
                    all;
                }
            }
            interfaces {                
                reth1.0;
            }
        }
        security-zone External;
        security-zone VPN;
        security-zone DMZ;
    }
}
interfaces {
    ge-0/0/3 {
        gigether-options {
            redundant-parent reth1;
        }
    }
    ge-5/0/3 {
        gigether-options {
            redundant-parent reth1;
        }
    }
    fab0 {
        fabric-options {
            member-interfaces {
                ge-0/0/2;
            }
        }
    }
    fab1 {
        fabric-options {
            member-interfaces {
                ge-5/0/2;
            }
        }
    }
    reth1 {
        description MGMT;
        redundant-ether-options {
            redundancy-group 1;
        }
        unit 0 {
            family inet {
                address 10.101.40.254/24;
            }
        }
    }
}
protocols {
    l2-learning {
        global-mode switching;
    }
    rstp {
        interface all;
    }
}
access {
    address-assignment {
        pool junosDHCPPool1 {
            family inet {
                network 192.168.1.0/24;
                range junosRange {
                    low 192.168.1.2;
                    high 192.168.1.254;
                }
                dhcp-attributes {
                    router {
                        192.168.1.1;
                    }
                    propagate-settings ge-0/0/0.0;
                }
            }                           
        }
        pool junosDHCPPool2 {
            family inet {
                network 192.168.2.0/24;
                range junosRange {
                    low 192.168.2.2;
                    high 192.168.2.254;
                }
                dhcp-attributes {
                    router {
                        192.168.2.1;
                    }
                    propagate-settings ge-0/0/0.0;
                }
            }
        }
    }
}
vlans {
    vlan-trust {
        vlan-id 3;
        l3-interface irb.0;
    }
}

Hi Trasgu,

Please take a look at the TSB as suggested by other community members - https://kb.juniper.net/InfoCenter/index?page=content&id=TSB17581

Instead of upgrading to Junos 18.1R3.3, try upgrading to Junos 18.2R2 and higher or Junos 18.3R1.

If it still stuck in db> prompt, please perform the below steps.

db> cont

If the above didn't work then,

db> reset.

In case if the above command, doesn't help we can proceed with the reboot of the device. If you were unable to upgrade the device from the CLI, you can proceed with the USB Bootloader upgrade method assuming you can stop at loader prompt- https://www.juniper.net/documentation/en_US/junos/topics/topic-map/install-software-on-srx.html#id-installing-junos-os-on-srx-series-devices-from-the-boot-loader-using-a-usb-storage-device

If none of the above works, try zeroizing the box without any configuration except root password and lets check the behaviour.

Thanks lpaniagua and noobmaster. I tried to upgrade now to 18.3R2.7 built 2019-05-03, and failed again in the first attempt, so repeated with "no-validate" and then the nodes rebooted with the new version running, didn´t go to panic mode. 

However something went wrong, as now the HA led keeps in amber. Below a few show outputs:

root@SPCFW-BRAVO> show chassis cluster status 
Monitor Failure codes:
    CS  Cold Sync monitoring        FL  Fabric Connection monitoring
    GR  GRES monitoring             HW  Hardware monitoring
    IF  Interface monitoring        IP  IP monitoring
    LB  Loopback monitoring         MB  Mbuf monitoring
    NH  Nexthop monitoring          NP  NPC monitoring              
    SP  SPU monitoring              SM  Schedule monitoring
    CF  Config Sync monitoring      RE  Relinquish monitoring
 
Cluster ID: 1
Node   Priority Status               Preempt Manual   Monitor-failures

Redundancy group: 0 , Failover count: 0
node0  200      primary              no      no       None           
node1  100      secondary            no      no       None           

Redundancy group: 1 , Failover count: 0
node0  0        primary              yes     no       CS             
node1  0        secondary            yes     no       CS     

root@SPCFW-BRAVO> show chassis cluster information 
node0:
--------------------------------------------------------------------------
Redundancy Group Information:

    Redundancy Group 0 , Current State: primary, Weight: 255

        Time            From                 To                   Reason
        Sep 10 14:06:51 hold                 secondary            Hold timer expired
        Sep 10 14:06:52 secondary            primary              Better priority (200/100)

    Redundancy Group 1 , Current State: primary, Weight: 0

        Time            From                 To                   Reason
        Sep 10 14:06:51 hold                 secondary            Hold timer expired
        Sep 10 14:06:53 secondary            primary              Remote yield (0/0)

Chassis cluster LED information:
    Current LED color: Amber
    Last LED change reason: Monitored objects are down
Control port tagging:                   
    Disabled

Failure Information:

    Coldsync Monitoring Failure Information:
        Statistics:
            Coldsync Total SPUs: 1
            Coldsync completed SPUs: 0
            Coldsync not complete SPUs: 1

    Fabric-link Failure Information:
        Fabric Interface: fab0
          Child interface   Physical / Monitored Status     
          ge-0/0/2              Up   / Down 

node1:
--------------------------------------------------------------------------
Redundancy Group Information:

    Redundancy Group 0 , Current State: secondary, Weight: 255

        Time            From                 To                   Reason
        Sep 10 14:06:52 hold                 secondary            Hold timer expired

    Redundancy Group 1 , Current State: secondary, Weight: 0

        Time            From                 To                   Reason
        Sep 10 14:06:53 hold                 secondary            Hold timer expired

Chassis cluster LED information:
    Current LED color: Amber
    Last LED change reason: Monitored objects are down
Control port tagging:
    Disabled

Failure Information:

    Coldsync Monitoring Failure Information:
        Statistics:
            Coldsync Total SPUs: 1
            Coldsync completed SPUs: 0
            Coldsync not complete SPUs: 1

root@SPCFW-BRAVO> show chassis cluster interfaces    
Control link status: Up

Control interfaces: 
    Index   Interface   Monitored-Status   Internal-SA   Security
    0       fxp1        Up                 Disabled      Disabled  

Fabric link status: Up

Fabric interfaces: 
    Name    Child-interface    Status                    Security
                               (Physical/Monitored)
    fab0    ge-0/0/2           Up   / Down               Disabled   
    fab0   
    fab1    ge-5/0/2           Up   / Up                 Disabled   
    fab1   

Redundant-ethernet Information:     
    Name         Status      Redundancy-group
    reth0        Down        Not configured   
    reth1        Up          1                
    reth2        Down        Not configured   
    reth3        Down        Not configured   
    reth4        Down        Not configured   
                                        
Redundant-pseudo-interface Information:
    Name         Status      Redundancy-group
    lo0          Up          0                

The funny thing is that ge-0/0/2 is up and blinking green, both nodes are connected with a single cable, no switch in between.

So a new chapter to write in this wonderful Juniper technology...

Thanks

Update: I rebooted the node 0 and everything seems to be fine now

THanks guys

Trasgu,

Im glad its working as expected now!

Hi,

Try the "request system software add /var/tmp/junos-srxsme-17.3R1.10.tgz" again and let us know what are the commands that made the validation failed.

Also you could make a backup of the configuration, then delete the whole configuration from the SRX and finally try the upgrade. If the configuration is causing the problem then you will be upgrading a SRX with no configuration. When the upgrade is done, you can just paste the old configuration. You could try this type of upgrade with the "partition" option to format both partitions and install the new junos:

https://kb.juniper.net/InfoCenter/index?page=content&id=KB32192&cat=&actp=LIST