Ethernet Switching
Highlighted
Ethernet Switching

Unable to upgrade EX3300 stack members to JunOS 12.3R12.

[ Edited ]
‎01-16-2020 04:17 AM

Hello everyone,

Yesterday I tried to upgrade our EX3300 stack with Junos 12.3R12. The current fw is 12.3R6.6. I have downloaded fw to: /var/tmp/jinstall-ex-3300-12.3R12.4-domestic-signed.tgz on master switch, after that I have issed command to validate new fw:

request system software validate /var/tmp/jinstall-ex-3300-12.3R12.4-domestic-signed.tgz

And finally I have started upgrade of last virtual-chassis member (9th), to avoid a downtime of whole VC:

request system software add /var/tmp/jinstall-ex-3300-12.3R12.4-domestic-signed.tgz member 9

Unfortunately after the member 9 has been rebooted it fails to load Junos. In the console some disquieting things has appeared during start of this switch (as below). After a while I have managed to take a snapshot and start switch with previous version of Junos (12.3R12). The affected switch status of virtual-chassis was 'not present'.

Can anybody explain me what happened and what I have done wrong?

 

MISSING '.'  I=4097  OWNER=0 MODE=40755
SIZE=512 MTIME=Mar 17 12:26 2014
DIR=?

UNEXPECTED SOFT UPDATE INCONSISTENCY

FIX? yes

DIRECTORY CORRUPTED  I=4098  OWNER=0 MODE=40755
SIZE=512 MTIME=Jun 14 08:27 2013
DIR=?

UNEXPECTED SOFT UPDATE INCONSISTENCY

SALVAGE? yes

MISSING '.'  I=4098  OWNER=0 MODE=40755
SIZE=512 MTIME=Jun 14 08:27 2013
DIR=?

...

UNREF DIR  I=4131  OWNER=0 MODE=40700
SIZE=512 MTIME=Feb 12 13:49 2015
RECONNECT? yes

NO lost+found DIRECTORY
CREATE? yes

CG 0: BAD MAGIC NUMBER

UNEXPECTED SOFT UPDATE INCONSISTENCY
SORRY. CANNOT CREATE lost+found DIRECTORY
UNEXPECTED SOFT UPDATE INCONSISTENCY


UNREF DIR  I=4130  OWNER=0 MODE=40700
SIZE=512 MTIME=Feb 12 13:49 2015
RECONNECT? yes

NO lost+found DIRECTORY
CREATE? yes

CG 0: BAD MAGIC NUMBER

UNEXPECTED SOFT UPDATE INCONSISTENCY
SORRY. CANNOT CREATE lost+found DIRECTORY
UNEXPECTED SOFT UPDATE INCONSISTENCY

...

UNEXPECTED SOFT UPDATE INCONSISTENCY
SORRY. CANNOT CREATE lost+found DIRECTORY
UNEXPECTED SOFT UPDATE INCONSISTENCY



CLEAR? yes

UNREF FILE  I=4106  OWNER=0 MODE=120755
SIZE=25 MTIME=Feb 12 13:50 2015
RECONNECT? yes

NO lost+found DIRECTORY
CREATE? yes

CG 0: BAD MAGIC NUMBER

UNEXPECTED SOFT UPDATE INCONSISTENCY
SORRY. CANNOT CREATE lost+found DIRECTORY
UNEXPECTED SOFT UPDATE INCONSISTENCY

 

 

4 REPLIES 4
Highlighted
Ethernet Switching

Re: Unable to upgrade EX3300 stack members to JunOS 12.3R12.

‎01-16-2020 05:25 AM

Hi,

 

That output shows file system corruption.  You need to format install that device.

 

Follow those steps to format install.

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB20643

 

Regards,

 

Randall

 

Highlighted
Ethernet Switching

Re: Unable to upgrade EX3300 stack members to JunOS 12.3R12.

[ Edited ]
‎01-16-2020 07:09 AM

Thank you very much for response. Is that means that it was just a coincidence that the file system of the switch was corrupted?
As I mentioned the switch is now working fine. But this knowledge will be helpful.

Highlighted
Ethernet Switching

Re: Unable to upgrade EX3300 stack members to JunOS 12.3R12.

‎01-16-2020 07:37 AM

Due to the flash technology available back in 2008/9 the file systems on the legacy EX product lines (EX2200/3300/4200) are susceptible to flash corruption over time.  Longer switch is in-service with no reboot, the more likely this may happen.  It is hit or miss, from my experience.  I believe the most often occurance is with the Master RE switch in an EX4200 VC.  Since EX4200 is the oldest product, it is most likely one to see this.  Flash corruption does not negatively affect the operation of the switch.  Despite Unix/Linux tools being present, pre-identifying is difficult at best, and (again from experience) not of the tools to fix this work on an operating switch.

 

I have seen this most often with EX4200, few and far better on EX3300/EX2200 - you look like one of the lucky ones!  For EX4200VC I have found that if you do a partition switch (affects Master RE) that things seem to go much better.

 

As @randero stated, once a switch is re-boot/reset and has this state (flash corruption present) generally only recovery is via a USB install which would also require local console access.

 

HTH and Good Luck

Highlighted
Ethernet Switching
Solution
Accepted by topic author DawidK
‎01-21-2020 03:49 AM

Re: Unable to upgrade EX3300 stack members to JunOS 12.3R12.

‎01-16-2020 08:49 AM

There are even KB articles for the affected platforms:

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB20570

 

I have lots of experience with EX4200 (VC) flash pain. The main reason for flash corruption are locally written syslog messages, which corrupt the internal flash. And the only way to recover completely is to perform an "install --format". The only way to prevent this corruption is to disable locally written syslog messages completely and never use traceoptions or anything else that writes to the local flash.

 

By the way, flash corruption actually does affect the operation of the Switch. We saw very strange behavior and if you perform a commit on such a device, the device may crash and creates a vmcore dump.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution".
If you think that my answer was helpful, please spend some Kudos.