Ethernet Switching
Highlighted
Ethernet Switching

ex2300 zombi state where 2 48 port switch in virtual chassis

‎02-06-2020 04:04 PM

Hi,

 

I also got hit by ex2300 zombi state where 2 48 port switch in virtual chassis, Only power cycle reboot solved the issue I am still in Junos: 18.2R3-S2.9

fpc0:
--------------------------------------------------------------------------
Hostname: kannurcore
Model: ex2300-48t
Junos: 18.2R3-S2.9
JUNOS OS Kernel 32-bit [20191022.14c2ad5_builder_stable_11]
JUNOS OS libs [20191022.14c2ad5_builder_stable_11]
JUNOS OS runtime [20191022.14c2ad5_builder_stable_11]
JUNOS OS time zone information [20191022.14c2ad5_builder_stable_11]
JUNOS py extensions [20191115.190104_builder_junos_182_r3_s2]
JUNOS py base [20191115.190104_builder_junos_182_r3_s2]
JUNOS OS crypto [20191022.14c2ad5_builder_stable_11]
JUNOS network stack and utilities [20191115.190104_builder_junos_182_r3_s2]
JUNOS libs [20191115.190104_builder_junos_182_r3_s2]
JUNOS runtime [20191115.190104_builder_junos_182_r3_s2]
JUNOS Web Management Platform Package [20191115.190104_builder_junos_182_r3_s2]
JUNOS ex libs [20191115.190104_builder_junos_182_r3_s2]
JUNOS ex runtime [20191115.190104_builder_junos_182_r3_s2]
JUNOS ex platform support [20191115.190104_builder_junos_182_r3_s2]
JUNOS dcp network modules [20191115.190104_builder_junos_182_r3_s2]
JUNOS modules [20191115.190104_builder_junos_182_r3_s2]
JUNOS ex modules [20191115.190104_builder_junos_182_r3_s2]
JUNOS ex Data Plane Crypto Support [20191115.190104_builder_junos_182_r3_s2]
JUNOS daemons [20191115.190104_builder_junos_182_r3_s2]
JUNOS SDN Software Suite [20191115.190104_builder_junos_182_r3_s2]
JUNOS Extension Toolkit [20191115.190104_builder_junos_182_r3_s2]
JUNOS Phone-home [20191115.190104_builder_junos_182_r3_s2]
JUNOS Packet Forwarding Engine Support (EX34XX) [20191115.190104_builder_junos_182_r3_s2]
JUNOS jdocs ex [20191115.190104_builder_junos_182_r3_s2]
JUNOS jail runtime [20191022.14c2ad5_builder_stable_11]
JUNOS FIPS mode utilities [20191115.190104_builder_junos_182_r3_s2]

fpc1:
--------------------------------------------------------------------------
Hostname: kannurcore
Model: ex2300-48t
Junos: 18.2R3.4
JUNOS OS Kernel 32-bit [20190605.30b921f_builder_stable_11]
JUNOS OS libs [20190605.30b921f_builder_stable_11]
JUNOS OS runtime [20190605.30b921f_builder_stable_11]
JUNOS OS time zone information [20190605.30b921f_builder_stable_11]
JUNOS py extensions [20190620.215446_builder_junos_182_r3]
JUNOS py base [20190620.215446_builder_junos_182_r3]
JUNOS OS crypto [20190605.30b921f_builder_stable_11]
JUNOS network stack and utilities [20190620.215446_builder_junos_182_r3]
JUNOS libs [20190620.215446_builder_junos_182_r3]
JUNOS runtime [20190620.215446_builder_junos_182_r3]
JUNOS Web Management Platform Package [20190620.215446_builder_junos_182_r3]
JUNOS Web Management Application package [18.2A1]
JUNOS ex libs [20190620.215446_builder_junos_182_r3]
JUNOS ex runtime [20190620.215446_builder_junos_182_r3]
JUNOS ex platform support [20190620.215446_builder_junos_182_r3]
JUNOS dcp network modules [20190620.215446_builder_junos_182_r3]
JUNOS modules [20190620.215446_builder_junos_182_r3]
JUNOS ex modules [20190620.215446_builder_junos_182_r3]
JUNOS ex Data Plane Crypto Support [20190620.215446_builder_junos_182_r3]
JUNOS daemons [20190620.215446_builder_junos_182_r3]
JUNOS SDN Software Suite [20190620.215446_builder_junos_182_r3]
JUNOS Extension Toolkit [20190620.215446_builder_junos_182_r3]
JUNOS Phone-home [20190620.215446_builder_junos_182_r3]
JUNOS Packet Forwarding Engine Support (EX34XX) [20190620.215446_builder_junos_182_r3]
JUNOS jdocs ex [20190620.215446_builder_junos_182_r3]
JUNOS jail runtime [20190605.30b921f_builder_stable_11]
JUNOS FIPS mode utilities [20190620.215446_builder_junos_182_r3]

 

 

and the curious thing is one switch automatically rolled back to problematic version Junos: 18.2R3.4 can you any suggest me the 15.1 software train where this issue is not there, I can't take the risk I narrowly escaped from production outage.

 

 

What is the alternate command to make the backup partition also in 18.2R3-S2.9 to avoid rollbacks?

request system snapshot slice alternate is not working any more in new Junos.

Thanks,
Jsree

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
5 REPLIES 5
Highlighted
Ethernet Switching

Re: ex2300 zombi state where 2 48 port switch in virtual chassis

‎02-06-2020 08:10 PM

Hi

 

It looks like the software upgrade for fpc1 failed rather than the rollback.

for 15.1 the recommended version would be Junos 15.1X53-D591 however without investigating this properly we cannot say this issue will not be there on 15.1.

As far as system snapshot is concerned are you seeing any error while doing the "request system snapshot slice alternate member all" ?

 

Thanks

 

Highlighted
Ethernet Switching

Re: ex2300 zombi state where 2 48 port switch in virtual chassis

[ Edited ]
‎02-06-2020 09:29 PM

Hi,

 

root@kannurcore> request system snapshot sl
^
syntax error.
root@kannurcore> request system snapshot sl
^
syntax error.
root@kannurcore> request system snapshot ?
Possible completions:
<[Enter]> Execute this command
all-members Archive data and executable areas of all virtual chassis members
delete Delete an existing snapshot
load Load a given snapshot and activate it upon reboot
local Archive data and executable areas of local virtual chassis member
media Media to snapshot to
member Archive data and executable areas of specific virtual chassis member
partition Partition the media
recovery Create a recovery snapshot
| Pipe through a command

 

 

There is no option in these version of junos.

 

I made sure that both had same os was on both switch

Thanks,
Jsree

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
Ethernet Switching

Re: ex2300 zombi state where 2 48 port switch in virtual chassis

‎02-06-2020 11:00 PM

For the snapshot i believe there has been changes in the New FreeBSD version.

You can simply try 

 

>request system snapshot

 

{master:1}
root@EX3400> request system snapshot
fpc1:
--------------------------------------------------------------------------
NOTICE: Snapshot snap.20170223.125901 created successfully

{master:1}
root@EX3400> file list /packages/sets/

/packages/sets/:
active/
optional/
previous/
snap.20170223.125901/

Now for more details you can refer to the following

 

 

https://kb.juniper.net/InfoCenter/index?page=content&id=KB32170&actp=METADATA

 

Hope this helps.

Highlighted
Ethernet Switching

Re: ex2300 zombi state where 2 48 port switch in virtual chassis

‎02-07-2020 08:33 AM

Hi 

 

Thank you for the post, I have gone through earlier, but in legacy Junose there was resilient Juno's partition that is main and backup. In the event of the main Junos gets corrupt it will automatically load from backup Junos. when we install new Junos version the existing Junos version gets replaced and become backup partition, got me in trouble once I forgot to slice the main Junos partition to backup. So how this works in the new Junos version?

 

what happened to existing  Junos version when we upgrade to the new one, for example, upgrading from 18.3r3 to 18.4, is there any chance the 18.3 will active once the current 18.4 is corrupt as it was happened in the legacy junos. To avoid that I used to run request system snapshot slice alternate command. what is the equivalent command in new Junos? How to avoid the automatical rollback of Junos in the vent of corruption.

 

 

 

Thanks,
Jsree

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too .....
Highlighted
Ethernet Switching
Solution
Accepted by topic author jsree
‎02-09-2020 09:09 PM

Re: ex2300 zombi state where 2 48 port switch in virtual chassis

‎02-09-2020 08:44 PM

Hello 

 

From the article we can see that in the new junos we have 2 type of snapshots. 

The one you are looking for is the recovery snapshot 

 

 

root@EX34xx> request system snapshot recovery
fpc0:
--------------------------------------------------------------------------
Creating image ...
Compressing image ...
Image size is 243MB
Recovery snapshot created successfully

Recovery snapshot does the same thing as request system snapshot slice alternate on the older platforms.

 

 

So if you do the recovery snapshot after the upgrade then you can recover the junos from the OAM volume as follows

 

root@EX3400> request system recover oam-volume

NOTICE: Recovering the OAM volume ...
da0p1 deleted
da0p1 added
/dev/gpt/oam: 500.0MB (1024000 sectors) block size 32768, fragment size 4096
    using 4 cylinder groups of 125.03MB, 4001 blks, 16128 inodes.
super-block backups (for fsck_ffs -b #) at:
 192, 256256, 512320, 768384
Verified oam signed by PackageProductionEc_2016 method ECDSA256+SHA256
Installing OAM volume contents ...
The OAM volume is now installed
NOTICE: Creating a recovery snapshot on the OAM volume ...
Creating image ...
Compressing image ...
Image size is 243MB
Recovery snapshot created successfully
NOTICE: OAM volume recovered  >>> Recovered
Feedback