Junos OS

Hi all,

I had very strange things happen this weekend to our EdgeRouter and Juniper EX4300 switches. Our EdgeRouter seems to have completely lost networking capabilities and will refuse to factory reset even though the unit powers on and the lights show activity.

Our Juniper switches (configured in virtual chassis) which is connected to the EdgeRouter and connected to the same UPS are also having issues. Both switches power on and show activity on the ports, but networking does not work. I am not even able to hit the juniper switch address when plugged into a port that is configured to allow access to it. I am able to access the console port though. My configuration is still there and everything looks correct. I am able to "show interface" and see the activity on a port with my computer plugged in, but when I do "show arp" these are the only entries I get even though I have multiple things plugged into my switch and showing activity:

show arp
MAC Address       Address         Name                      Interface               Flags
02:00:00:00:00:0a 128.0.0.5       128.0.0.5                 bme0.0                  permanent
02:00:00:00:00:0a 128.0.0.6       128.0.0.6                 bme0.0                  permanent
02:00:00:00:00:0a 128.0.0.16      128.0.0.16                bme0.0                  permanent
Total entries: 3

When this happened, I hard restarted both switches, upon restart, it looks like the switches reset itself to version 13.2X51-D35.3. The switch actually shows a commit made by root that was made when the siwtch was restarted: 

show system commit
0   2019-05-04 09:19:22 EDT by root via other

And here is the one difference from my last commit:

how system rollback 0 compare 1
- version 18.1R1.9;
+ version 13.2X51-D35.3;

I'm not sure where to go from here. Has anyone heard of this happening before? Again, my configuration seems to be OK and the ports on the switches show activity, but my version has been set back and L2 networking seems completely broken. Any help is greatly appreciated.

Your switch has booted from the backup partition, which is running JUNOS 13. Your v18 configuration is incompatible with v13 so it is not getting loaded at boot. You can either try booting from the v18 partition with

request system reboot slice alternate media internal

or reinstall v18 over the current v13 and reboot. You should grab a copy of your configuration in case something goes wrong and you need to reinstall from scratch.

In either case remember to snapshot to your backup partition so they are running the same code.

request system snapshot slice alternate

Thanks for your response smicker. Unfortuntely, the switches were not able to boot to the internal partition and continue to boot from the backup. Here is what it looks like when loading:

U-Boot 2011.06 (Jan 10 2014 - 02:08:51)

Unicore software on multiprocessor system!!
To enable mutlticore build define CONFIG_MP
CPU0:  P20BJE, Version: 2.0, (0x82190120)
Core:  E500MC, Version: 3.2, (0x80230032)
Clock Configuration:
       CPU0:1500 MHz, CPU1:1500 MHz,
       CCB:600  MHz,
       DDR:600  MHz (1200 MT/s data rate) (Asynchronous), LBC:75   MHz
       FMAN1: 500 MHz
L1:    D-cache 32 kB enabled
       I-cache 32 kB enabled
Reset Configuration Word (RCW):
       00000000: 4c580000 00000000 1e140000 00440000
       00000010: 648e20c1 ffc02000 ce000000 41000000
       00000020: 00000000 00000000 00000000 f05b4101
       00000030: 00000000 00000000 00000000 00000000
Board: EX4300-24T 3.18
EPLD:  Version 10.0 (0x0a)
I2C:   ready
DRAM:  Initializing
Detected UDIMM TS7K8EDSQ-0000
    DDR: 2 GiB (DDR3, 64-bit, CL=8, ECC on)
FLASH bank: 0
Flash: 8 MiB
L2:    128 KB enabled
Corenet Platform Cache: 1024 KB enabled
SERDES: bank 2 disabled
SERDES: timeout resetting bank 2

Firmware Version: 01.00.00
PCIe2: Root Complex, x2, regs @ 0xfe201000
PCIe2: Bus 00 - 01
In:    serial
Out:   serial
Err:   serial
USB:
Register 10011 NbrPorts 1
USB EHCI 1.00
Register 10011 NbrPorts 1
USB EHCI 1.00
scanning bus for devices...
Root Hub 0: 2 USB Device(s) found
Root Hub 1: 1 USB Device(s) found
       scanning bus for storage devices... 1 Storage Device(s) found
Net:   Initializing Fman
Uploaded FMAN microcode version 101.8
FM1@DTSEC1 connected to Broadcom BCM54616S
FM1@DTSEC1 [PRIME]
Consoles: U-Boot console

FreeBSD/PowerPC U-Boot bootstrap loader, Revision 2.4
(hbuilder@svl-junos-pool103.juniper.net, Fri Jan 10 02:08:46 PST 2014)
Memory: 2048MB
bootsequencing is enabled
bootsuccess is set
new boot device = disk0s1:
Loading /boot/defaults/loader.conf
/kernel data=0xb26ef0+0xd5090 syms=[0x4+0x96090+0x4+0xda8ac]


Hit [Enter] to boot immediately, or space bar for command prompt.
Booting [/kernel]...
Kernel entry at 0x800000c0 ...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1996-2015, Juniper Networks, Inc.
All rights reserved.
Copyright (c) 1992-2006 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
JUNOS 13.2X51-D35.3 #0: 2015-04-09 20:48:22 UTC
    builder@svl-junos-pool88.juniper.net:/volume/build/junos/13.2/service/13.2X51-D35.3/obj-powerpc/junos/bsd/kernels/JUNIPER-EX-4300/kernel
can't re-use a leaf (all_slot_serialid)!
Timecounter "decrementer" frequency 18750000 Hz quality 0
cpu0: Freescale e500mc core revision 3.2
cpu0: HID0 c0000080<EMCP,EN_L2MMU_MHD,EN_MAS7_UPDATE>
real memory  = 1992294400 (1900 MB)
avail memory = 1949667328 (1859 MB)
Security policy loaded: Junos MAC/veriexec (mac_veriexec)
Security policy loaded: JUNOS MAC/pcap (mac_pcap)
Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
MAC/veriexec fingerprint module loaded: SHA256
MAC/veriexec fingerprint module loaded: SHA1
ETHERNET SOCKET BRIDGE initialising
Initializing EXSERIES properties ...
nexus0: <Powerpc Nexus device>
ocpbus0: <on-chip peripheral bus> on nexus0
openpic0: <OpenPIC in on-chip peripheral bus> iomem 0xfe040000-0xfe06ffff on ocpbus0
lbc0: <Freescale 8533 Local Bus Controller> iomem 0xfe124000-0xfe12410f,0xff000000-0xffffffff irq 41 on ocpbus0
cb fpga: un-supported RE/board:148
cfi0: <AMD/Fujitsu - 8MB> iomem 0xff800000-0xffffffff on lbc0
cb fpga: un-supported RE/board:148
syspld0 iomem 0xff000000-0xff00ffff on lbc0
i2c0: <MPC85XX OnChip i2c Controller> iomem 0xfe118000-0xfe118014 irq 54 on ocpbus0
ds1672 rtc0: <DS1672 RTC> on i2c0
i2c1: <MPC85XX OnChip i2c Controller> iomem 0xfe118100-0xfe118114 irq 54 on ocpbus0
poe0: <POE> on i2c1
poe_attach: re-types method failed
device_attach: poe0 attach returned 19
uart0: <16550 or compatible> iomem 0xfe11c500-0xfe11c50f irq 52 on ocpbus0
uart0: console (9600,n,8,1)
uart1: <16550 or compatible> iomem 0xfe11c600-0xfe11c60f irq 52 on ocpbus0
pcib0: <Freescale MPCP20BJE PCI Express host controller> iomem 0xfe201000-0xfe201fff,0xc0000000-0xc7ffffff on ocpbus0
pci0: <PCI bus> on pcib0
pcib1: <PCI-PCI bridge> at device 0.0 on pci0
pci1: <PCI bus> on pcib1
besw0: <Broadcom BCM56649J GigE Switch> mem 0xc0000000-0xc003ffff irq 57 at device 0.0 on pci1
BCM device 0xb649 has been attachedbman0 iomem 0xfe31a000-0xfe31afff,0xfd800000-0xfd803fff,0xfd804000-0xfd804fff irq 32,121,123,125,127,129,131,133,135,137,139
on ocpbus0
bman0: Hardware version: 1.0.
qman0 iomem 0xfe318000-0xfe318fff,0xfd808000-0xfd80bfff,0xfd80c000-0xfd80cfff irq 32,120,122,124,126,128,130,132,134,136,138 on ocpbus0
qman0: Hardware version: 1.2.
fman0 iomem 0xfe400000-0xfe4fffff irq 112,32 on ocpbus0
fman0: Hardware version: 3.0.
dtsec0 on fman0
dtsec0: Configured for independent mode.
dtsec0: hardware MAC address 80:ac:ac:5d:7f:c0
miibus0: <MII bus> on dtsec0
brgphy0: <BCM54616S 10/100/1000baseTX PHY> on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
ehci0: <Frescale Integrated USB 2.0 controller> iomem 0xfe210000-0xfe210fff irq 60 on ocpbus0
usb0: EHCI version 1.0
usb0 on ehci0
usb0: USB revision 2.0
uhub0: Freescale EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub0: 1 port with 1 removable, self powered
umass0: Transcend TS2GUFM-H, rev 2.00/11.00, addr 2
ehci1: <Frescale Integrated USB 2.0 controller> iomem 0xfe211000-0xfe211fff irq 61 on ocpbus0
usb1: EHCI version 1.0
usb1 on ehci1
usb1: USB revision 2.0
uhub1: Freescale EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
uhub1: 1 port with 1 removable, self powered
Initializing product: 143 ..
Loading the NETPFE fc module
###PCB Group initialized for udppcbgroup
###PCB Group initialized for tcppcbgroup
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <TS TS2GUFM-H 1100> Fixed Direct Access SCSI-4 device
da0: 40.000MB/s transfers
da0: 1968MB (4030464 512 byte sectors: 255H 63S/T 250C)
Kernel thread "wkupdaemon" (pid 46) exited prematurely.
Trying to mount root from ufs:/dev/da0s1a
Attaching /packages/junos via /dev/mdctl...
Mounted junos-ex package on /dev/md0...
Verified manifest signed by PackageProduction_13_2_0
Verified jboot signed by PackageProduction_13_2_0
Verified junos-ex-13.2X51-D35.3 signed by PackageProduction_13_2_0
Mounted fips-mode-powerpc package on /dev/md2...
Verified manifest signed by PackageProduction_13_2_0
Verified fips-mode-powerpc-13.2X51-D35.3 signed by PackageProduction_13_2_0
Mounted jdocs-ex package on /dev/md4...
Verified manifest signed by PackageProduction_13_2_0
Verified jdocs-ex-13.2X51-D35.3 signed by PackageProduction_13_2_0
Mounted junos-ex-4300 package on /dev/md6...
Verified manifest signed by PackageProduction_13_2_0
Verified junos-ex-4300-13.2X51-D35.3 signed by PackageProduction_13_2_0
Mounted jweb-ex package on /dev/md8...
Verified manifest signed by PackageProduction_13_2_0
Verified jweb-ex-13.2X51-D35.3 signed by PackageProduction_13_2_0
Executing /packages/mnt/jweb-ex-13.2X51-D35.3/mount.post..
Mounted py-base-powerpc package on /dev/md10...
Verified manifest signed by PackageProduction_13_2_0
Verified py-base-powerpc-13.2X51-D35.3 signed by PackageProduction_13_2_0
Automatic reboot in progress...
** /dev/da0s1a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 71503 free (23 frags, 8935 blocks, 0.0% fragmentation)
** /dev/da0s3e
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 78584 free (320 frags, 9783 blocks, 0.4% fragmentation)
Computing slice and partition sizes for /dev/da0 ...
savecore: Router rebooting after a normal shutdown...
No dump exists
** /dev/da0s3d
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 411532 free (28 frags, 51438 blocks, 0.0% fragmentation)
** /dev/da0s4d
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 59052 free (36 frags, 7377 blocks, 0.1% fragmentation)
rm: /var/etc/pam.conf: Operation not permitted
mkdir: /mfs/var/sdb: No such file or directory
mkdir: /mfs/var/sdb: No such file or directory
Creating initial configuration...mgd: Running FIPS Self-tests
veriexec: no fingerprint for file='/sbin/kats/cannot-exec' fsid=69 fileid=51404 uid=0 pid=401
mgd: FIPS Self-tests Passed
mgd: commit complete
Setting initial options:  debugger_on_panic=NO debugger_on_break=NO.
Starting optional daemons: .
Doing initial network setup:.
Initial interface configuration:
additional daemons:.
Additional routing options:kern.module_pathLoading the CHMIC module
Loading the EX-series platform NETPFE modulees;/modules/peertype;/modules/ifpfe_drv;/modules/platform;/modules;

kld netpfe drv: ifpfed_chmic ifpfed_eth ifpfed_irb ifpfed_ml_cmn ifpfed_svcskld platform: ex_ifpfe if_vcpkld peertype: peertype_hcm peertype_pfem peertype_sfi p
eertype_slavere peertype_xdpc grat_arp_on_ifup=YES: net.link.ether.inet.grat_arp_on_ifup: 1 -> 1
 ipsec kld.
Doing additional network setup:.
Starting final network daemons:.
starting local daemons:set cores for group access
.
PFEX runs as JUNOS process...
Boot media /dev/da0 has dual root support
** /dev/da0s2a
FILE SYSTEM CLEAN; SKIPPING CHECKS
clean, 75455 free (31 frags, 9428 blocks, 0.0% fragmentation)
setting ldconfig path: /usr/lib /opt/lib
starting standard daemons: cron.
Local package initialization:.
Initialize /var subdirs
Tue May  7 12:49:53 EDT 2019

sw-latham-core (ttyu0)

login: 
Logging to master
Password:


***********************************************************************
**                                                                   **
**  WARNING: THIS DEVICE HAS BOOTED FROM THE BACKUP JUNOS IMAGE      **
**                                                                   **
**  It is possible that the primary copy of JUNOS failed to boot up  **
**  properly, and so this device has booted from the backup copy.    **
**                                                                   **
**  The primary copy will be recovered by auto-snapshot feature now  **
**                                                                   **

Here is what my partitions look like:

show system storage partitions
fpc0:
--------------------------------------------------------------------------
Boot Media: internal (da0)
Active Partition: da0s2a
Backup Partition: da0s1a
Currently booted from: backup (da0s1a)

Partitions information:
  Partition  Size   Mountpoint
  s1a        316M   /
  s2a        324M   altroot
  s3d        887M   /var/tmp
  s3e        170M   /var
  s4d        116M   /config

fpc1:
--------------------------------------------------------------------------
Boot Media: internal (da0)
Active Partition: da0s1a
Backup Partition: da0s2a
Currently booted from: backup (da0s2a)

Partitions information:
  Partition  Size   Mountpoint
  s1a        316M   altroot
  s2a        324M   /
  s3d        887M   /var/tmp
  s3e        170M   /var
  s4d        116M   /config

Any other recommendations? What would cause something like this? 

Smicker is right, the switches are booting from the backup partition which wasn't upgraded

This is shown in show system storage partitions

show system storage partitions
fpc0:
--------------------------------------------------------------------------
Boot Media: internal (da0)
Active Partition: da0s2a
Backup Partition: da0s1a
Currently booted from: backup (da0s1a)

Partitions information:
  Partition  Size   Mountpoint
  s1a        316M   /
  s2a        324M   altroot
  s3d        887M   /var/tmp
  s3e        170M   /var
  s4d        116M   /config

fpc1:
--------------------------------------------------------------------------
Boot Media: internal (da0)
Active Partition: da0s1a
Backup Partition: da0s2a
Currently booted from: backup (da0s2a)

Partitions information:
  Partition  Size   Mountpoint
  s1a        316M   altroot
  s2a        324M   /
  s3d        887M   /var/tmp
  s3e        170M   /var
  s4d        116M   /config

 You need to follow this procedure https://kb.juniper.net/InfoCenter/index?page=content&id=KB23180

I was able to fix this by copying over the backup aprtition to the primary partition and then performing an upgrade to the latest JUNOS. Then made sure my backup parititon was up to date as well.