Junos OS

Hi, I'm trying to perform a recovery snapshot on a MX150 in 18.2R2.6, but I don't understand if it's successful or not:

root> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 1008MB
df: /dev/gpt/oam: Invalid argument
mount: /dev/gpt/oam: Invalid argument
umount: /mnt: not a file system root directory
test: -ge: argument expected
mount: /dev/gpt/oam: Invalid argument
ERROR: 'oam' package needs to be updated in order to use OAM functionality
mount: /dev/gpt/oam: Invalid argument
ERROR: 'oam' package needs to be updated in order to use OAM functionality
Recovery snapshot created successfully

root> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190121.112934:
Location: /packages/sets/snap.20190121.112934
Creation date: Jan 21 11:29:34 2019
Junos version: 18.2R2.6

Total non-recovery snapshots: 1

Recovery Snapshots:
mount: /dev/gpt/oam: Invalid argument
ERROR: 'oam' package needs to be updated in order to use OAM functionality

Is recovery snapshot supported on this platform, aside from creating a USB stick?

OAM volume may need to be recovered, please try "request system recover oam-volume" and then re-try "request system snapshot recovery"

root> request system recover oam-volume
NOTICE: Recovering the OAM volume ...
ERROR: unknown or unsupported RE: Juniper
warning: unable to create volume: oam
warning: the storage device that holds it is not present
ERROR: oam recovery error: failed to create oam-volume

I noticed that on another identical device, with same release, recovery snapshot is possible. The only difference is that the first was zeroized after the upgrade to 18.2, the second only upgraded.

On the twin device:

root> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190220.165750:
Location: /packages/sets/snap.20190220.165750
Creation date: Feb 20 16:57:50 2019
Junos version: 18.2R2.6

Total non-recovery snapshots: 1

Recovery Snapshots:
Snapshots available on the OAM volume:
recovery.ufs
Date created: Wed Feb 20 16:59:39 UTC 2019
Junos version: 18.2R2.6

Total recovery snapshots: 1

Maybe the format has deleted some file?

Can you share "show system storage" output from both devices?

device WITH snapshot:

root> show system storage detail
Filesystem 1024-blocks Used Avail Capacity Mounted on
/dev/gpt/junos 20901946 1202192 18027599 6% /.mount
/dev/gpt/config 812260 36 747244 0% /.mount/config
/dev/gpt/var 7308076 93276 6630156 1% /.mount/var
tmpfs 8762920 12 8762908 0% /.mount/tmp
tmpfs 1325764 1600 1324164 0% /.mount/mfs

--------------------------------------

device WITHOUT snapshot:

root> show system storage detail
Filesystem 1024-blocks Used Avail Capacity Mounted on
/dev/gpt/junos 20901946 1202145 18027646 6% /.mount
/dev/gpt/config 812260 28 747252 0% /.mount/config
/dev/gpt/var 7308076 86016 6637416 1% /.mount/var
tmpfs 8761544 8 8761536 0% /.mount/tmp
tmpfs 1325764 1596 1324168 0% /.mount/mfs

They look basically the same to me

Hi DByte,

Aside from recovering RE with USB stick, you can try to play with gpart to re-create oam

patrition, from your outputs it looks corrupted.

!!! Please keep in mind that inacurate use of gpart may result in FS coruption on your box.

!!! And you would need to recover it with USB media (recuires physical access/remote hands).

On the OK box check how it should be:

> start shell

% gpart list vtbd0



<-- should be something like this:

# gpart list vtbd0
Geom name: vtbd0
<snip>
2. Name: vtbd0p2
Mediasize: 2147483648 (2.0G)
Sectorsize: 512
Stripesize: 0
Stripeoffset: 20480
Mode: r0w0e0
...
label: oam
length: 2147483648
offset: 20480
type: freebsd-ufs
index: 2
end: 4194343
start: 40
<snip>

Then on the problem box, check oam:

% gpart list vtbd0

If it is present than first you need to delete it (if you don't have it on the problem box just skip):

> start shell user root
# gpart delete -i 2 vtbd0
vtbd0p2 deleted

 And create it again with parameters from OK box, in my example it was:

# gpart add -t freebsd-ufs -i 2 -b 40 -l oam vtbd0
vtbd0p2 added

You may want to check man page for gpart.

After that recovery snapshot was created sucsessfuly: 

> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 1300MB
Recovery snapshot created successfully

Thanks,

Alex

Thanks for the suggestion, I'll try tomorrow

Hi,

unfortunately the shell commands didn't work:

root> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 1008MB
df: /dev/gpt/oam: Invalid argument
mount: /dev/gpt/oam: Invalid argument
umount: /mnt: not a file system root directory
test: -ge: argument expected
mount: /dev/gpt/oam: Invalid argument
ERROR: 'oam' package needs to be updated in order to use OAM functionality
mount: /dev/gpt/oam: Invalid argument
ERROR: 'oam' package needs to be updated in order to use OAM functionality
Recovery snapshot created successfully

root> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190220.165915:
Location: /packages/sets/snap.20190220.165915
Creation date: Feb 20 16:59:15 2019
Junos version: 18.2R2.6

Total non-recovery snapshots: 1

Recovery Snapshots:
mount: /dev/gpt/oam: Invalid argument
ERROR: 'oam' package needs to be updated in order to use OAM functionality

Hi DByte,

Soory, I don't have any more thoughts on the subject. 

As I wrote, recovery from USB media probably would help here. 

!! Just don't forgot to back up your MX150's config and any important file/script you have on it.

!! Recovery will erase box FS.

Also if you have coverage opening Jtac case would be a good idea.

Thanks,

Alex

Hi, using a usb key did the trick:

root> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190222.091433:
Location: /packages/sets/snap.20190222.091433
Creation date: Feb 22 09:14:35 2019
Junos version: 18.2R2.6

Total non-recovery snapshots: 1

Recovery Snapshots:
Snapshots available on the OAM volume:
recovery.ufs
Date created: Fri Feb 22 09:16:37 UTC 2019
Junos version: 18.2R2.6

Total recovery snapshots: 1

However, I'd like to know why a zeroize ended up to delete the OAM data needed for snapshot? Is there a better way to bring device to factory-default without using a usb key?

Hi DByte,

>I'd like to know why a zeroize ended up to delete the OAM data needed for snapshot?

It's unexpected behaviour and I would assume a bug here, if you have an active contract you can report it to Jtac.

If not I would recommend to not zeroize the box.

Also I believe the trigger here is not only zeroize, but prior SW upgrade as well.

Probably if you zeroize it after recovery from USB (FS re-partition) it would not resulted in the subject issue.

> Is there a better way to bring device to factory-default without using a usb key?

 Yes, if you just want to restore config to factory-default you can use "load factory-default":

> configure
# load factory-default

Because it's factory default Junos would require "root-authentication" password to be set

before the commit. Also you probably would like to set some basic mgmt  configuration after

the "load factory-default".

Thanks,

Alex

Thanks Alex,

factory-default command loads a factory configuration without root password, mandatory for a commit.

What I mean is to create a brand-new device, even without root password, as in factory default config.

Hi,

Indeed if you want to return the device in the brand-new state "load factory default" is not an

option. It does not clean FS on the box, just reset the config.

To clean the box (including FS) to the default state you have two options (as you already aware):

- "request system zeroize";

- recovery from USB media.

I don't avare of any other option. 

Regarding the initial issue I believe you can try following ooo:

- upgrade;

- create recovery snapshot (request system snapshot recovery);

- zeroize;

- check is the recovery snapshot exist and if not, try to creati it again.

But I'm not sure if this order of operations helps, because as I understand zeroize process

also touches oam partition.

Thanks,

Alex