Switching

After recovering from a bad upgrade by reinstalling clean from usb, EX3400 standalone (new install) can't reach login: prompt from serial console.

Seems like I am missing something simple as all the ports are active. Couldn't find any solutions in my searching. Hoping someone has seen this before.

Full details:

I ran an upgrade on my EX3400 switch (new switch, was setting up first time) using

request system software add /tmp/usb/junos-arm-32-18.2R3.4.tgz

which failed because the configuration didnt match this version. I looked that up and found a thread stating this was a problem introduced in version 17 for certain configs and I could use the `no-validate` option to ignore. I tried that, install worked, but caused a kernel panic on restart (hindsight tells me I should have been more careful with this).

So I went ahead and followed the steps to recover here (usb boot): https://kb.juniper.net/InfoCenter/index?page=content&id=KB31265&actp=METADATA

Worked great, rebooted and no kernel panic. I tried this with an 18.x version, got the no serial console login prompt issue.

So, I decided to reinstall to the original version as well just for sanity, 15.1X53-D55.5.

Still got the no login prompt when connected to serial console. I am certain my serial console is fine because it logs output right up untill here every time I reboot.

I tried running a factory-reset on config by holding the reset button on the front, no luck.

Last few lines of output always look ike this:

Starting casperd.
Starting jlaunchhelperd.
/etc/rc: WARNING: run_rc_command: cannot run 
kern.rtc_retries: 0 -> 5
Starting cron.

Thu Jan  1 00:02:18 UTC 1970
                 

Can't interact with console at all from here. Enter does nothing nor does pressing any combination of keys I could think of. My other switch shows `FreeBSD/arm (TEST) (ttyu1)` right after this and then the login prompt which this switch never reaches.

Hi nickaielloftx,

Please verify if the below is the correct image type that you tried to run on the device:

junos-arm-32-18.2R3.4.tgz

Did you see normal boot logs while the device was booting up?

I am suspecting an issue with the file system or hardware issue. Does not seem to me like a miss with respect to something simple. 

In case this is a critical box, you can try opening a JTAC case for quicker remote troubleshooting.

Hope this helps 🙂

Please mark "Accepted Solution" if this helps you solve your query.

Kudos are always appreciated!

Hi,

junos-arm-32-18.2R3.4.tgz

Was the version I installed that caused the issue. It was actually a mistake on my part as I had grabbed the wrong version. I had intended to use version 18.1R3.3, but got mixed up when downloading the version.

For the bad versio, boot logs were fine right up until a kernel panic at

kern.module_path: /packages/sets/active/boot/junFatal kernel mode data abort: 'Translation Fault (L2)' on read

The odd part is that after installing other version from USB, everything seems to install and then boot fine, just can't get into it because of no login prompt over serial.

I did open a JTAC case, but unfortunately the only option from that side was an RMA. Wanted to see if anyone in the community recognized the issue.

Appreciate the feedback!

My experience shows that install formats are not successful from time to time, it helps to do it a second time after it fails completely or partially.

Can you please confirm that you used the image file "junos-install-media-usb-ex-arm-32-1X.XRX.X.img.gz" to deploy it to an USB device and to boot the EX3400 from it? It may be worth a second try.

You can try to use a very recent version for a recovery, in case you are running into any software issue, e.g. "junos-install-media-usb-ex-arm-32-19.1R3-S1.img.gz".

Hi F1ght3r,

I tried installing 2 different versions via the format USB method (3 total installs).

The first was "junos-install-media-usb-arm-32-18.1R3.3-limited.img"

Second, I tried the non-limited of that same version "junos-install-media-usb-arm-32-18.1R3.3.img"

Both of these were downloaded from the "Install media" section of the downloads page for EX3400.

Finally, I decided to just try going back to the original version (I actually had it saved locally, no longer available to Download from what I saw):

extracting "junos-install-media-usb-arm-32-15.1X53-D55.5.img.gz" gave me

"junos-install-media-usb-arm-32.img"

For all 3 attempts, the output indicates success after booting to USB (shows success then reboots), but then fails to give the login prompt once booted to the new version. During boot, I was always able to see it loading the modules ending in the version.

Example: "junos-net-prd-arm-32-20161230.114620_builder_junos_151_x53_d55"

Where the version matched whatever I installed last.

---

You can try to use a very recent version for a recovery, in case you are running into any software issue, e.g. "junos-install-media-usb-ex-arm-32-19.1R3-S1.img.gz".

---

I will give this a try.

Hi Nick,

> Second, I tried the non-limited of that same version "junos-install-media-usb-arm-32-18.1R3.3.img"

Just to be safe, are we talking about the same package? The correct one for 18.1R3 would be "junos-install-media-usb-arm-32-18.1R3.3.img.gz"

Yeah, "junos-install-media-usb-arm-32-18.1R3.3.img.gz" was the downloaded package, I extracted and used

dd if=./junos-install-media-usb-arm-32-18.1R3.3.img of=/dev/<myusbdevice>

Then plugged into the switch and at next boot, I interrupted with ctrl+c and selected boot menu -> Boot from USB

Process came from: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31265&actp=METADATA

Ok, sounds good, please tell me when you have tested it with 19.1R3-S1 install media.

Tried with version 19.1R3 but same problem. Though one thing I noticed now as I am replying, the media I downloaded doesn't have the "-S1" as you mentioned. Not sure if that is an issue/where to get those "-S1" versions from.

"junos-install-media-usb-ex-arm-32-19.1R3.9.img.gz"

The end of the output is

Creating a recovery snapshot
Creating image ...
Compressing image ...
Image size is 335MB
Recovery snapshot created successfully
@ 946685718 [2000-01-01 00:15:18 UTC] snapshot done

 Lock Manager
RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
Copyright (c) 1992-2006 Birdstep Technology, Inc.  All Rights Reserved.

Unix Domain sockets Lock manager
Lock manager 'lockmgr' started successfully.

Database Initialization Utility
RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
Copyright (c) 1992-2006 Birdstep Technology, Inc.  All Rights Reserved.

Profile database initialized
Enhanced arp scale is disabled
/etc/rc: WARNING: No dump device.  Savecore not run.
Prefetching /usr/sbin/rpd ...
Prefetching /usr/sbin/lacpd ...
Prefetching /usr/sbin/chassisd ...
Starting jlaunchhelperd.
sysctl: unknown oid 'kern.rtc_retries'
Starting cron.
cron: cron already running, pid: 12700
/etc/rc: WARNING: failed to start cron

Thu Mar 26 00:00:00 UTC 2020

> Though one thing I noticed now as I am replying, the media I downloaded doesn't have the "-S1" as you mentioned.

> Not sure if that is an issue/where to get those "-S1" versions from.

No, this is not an issue, "-S1" is just an additional bugfixing service release. To download service releases, you have to choose "OS: Junos SR" instead of "Junos".

Just to make sure, when you connect your console cable to any other device, your input is recognized normally when you press Enter or any other key, right?

Console worked fine, just switched it to another EX3400 switch and was able to interact normally with that switch, no changes to console settings required.

Then I'm afraid that you have to RMA this device as suggested by JTAC, unless JTAC don't find another solution for that. Maybe some internal code was overwritten by using the wrong install package at the beginning, which cannot be solved by a recovery installation.

Thanks for the support here! Appreciate the help.

In case anyone else comes across this in the future, I was actually able to get it working.

First thing was installing clean by booting from USB installer described here: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31265&actp=METADATA

Don't know if the version matters, but I chose to return to the original version the switch started with as I fortunately had the install media for it.

After that I still got the no login prompt, so rebooted and followed these steps: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31403&actp=METADATA

Except, instead of just setting password, I replaced the whole config with the config pulled from my other switch.

Additionaly, I did not run reboot from the recovery cli. Instead I exited from both configuration and operational mode which quickly loaded to a "login: " prompt. I logged in with the password I had just set and did some other config/testing before rebooting. The next boot was normal and reached the "login: " prompt fine.

Hope this saves someone in the future!