Junos OS

Happy new year 🙂

I just upgraded an EX9214  from 17.1R1.8 to 17.2R3.4 and wanted to backup the software via snapshot recovery:

root@boejuniper6> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 2110MB
ERROR: The OAM volume is too small to store a snapshot

root@boejuniper6> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190103.112901:
Location: /packages/sets/snap.20190103.112901
Creation date: Jan 3 11:29:01 2019
Junos version: 17.1R1.8

Total non-recovery snapshots: 1

Recovery Snapshots:
Snapshots available on the OAM volume:
recovery.ufs
Date created: Thu Jan 3 11:27:16 CET 2019
Junos version: 17.1R1.8

Total recovery snapshots: 1

I didn't find any hint (neither Software Installation and Upgrade Guide nor tech pubs or knowledge base) how to delete or overwrite the existing OAM volume...

Can anyone advice me?

Try the request system storage cleanup and run the snapshot again to see if that frees up space on the right volume.

Unfortunately not:

rhellwig@boejuniper6> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190103.112901:
Location: /packages/sets/snap.20190103.112901
Creation date: Jan 3 11:29:01 2019
Junos version: 17.1R1.8

Snapshot snap.20190104.101954:
Location: /packages/sets/snap.20190104.101954
Creation date: Jan 4 10:19:54 2019
Junos version: 18.3R1.9

Total non-recovery snapshots: 2

Recovery Snapshots:
Snapshots available on the OAM volume:
recovery.ufs
Date created: Thu Jan 3 11:27:16 CET 2019
Junos version: 17.1R1.8

Total recovery snapshots: 1

{master}
rhellwig@boejuniper6> request system storage cleanup

List of files to delete:

Size Date Name
11B Jan 4 13:41 /var/jail/tmp/alarmd.ts
....
0B Jan 4 10:18 /var/tmp/rtsdb/if-rtsdb
Delete these files ? [yes,no] (no) yes
{master}
rhellwig@boejuniper6> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 2459MB
ERROR: The OAM volume is too small to store a snapshot

In the meantime I upgraded the SW to an actual level (18.3R1.9) but this didn't help.

On the second RE I also tried the request system recover oam-volume oper cmd as described in Software Installation and Upgrade Guide (p.503), but this went wrong and made the /oam unavailable...

rhellwig@boejuniper6> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190103.143713:
Location: /packages/sets/snap.20190103.143713
Creation date: Jan 3 14:37:13 2019
Junos version: 17.2R3.4

Snapshot snap.20190104.100147:
Location: /packages/sets/snap.20190104.100147
Creation date: Jan 4 10:01:47 2019
Junos version: 18.3R1.9

Total non-recovery snapshots: 2

Recovery Snapshots:
mount: /dev/oam: No such file or directory

Reboot, offline/online and upgrade didn't solve this 2nd problem.

Hello,

Please see if this KB article https://kb.juniper.net/InfoCenter/index?page=content&id=KB32176 helps to recover OAM volume.

HTH

Thx

Alex

Hello,

thx for that advice but it didn't help:

root@boejuniper6:~ # fsck_ufs /dev/ada1p2
Can't stat /dev/ada1p2: No such file or directory
Can't stat /dev/ada1p2: No such file or directory

or the other try:

root@boejuniper6:~ # fsck_ufs /dev/gpt/oam

Can't stat /dev/gpt/oam: No such file or directory
Can't stat /dev/gpt/oam: No such file or directory

It seems the RE has lost it's mind:
root@boejuniper6> request system recover oam-volume
NOTICE: Recovering the OAM volume ...
(ada0:ata0:0:1:0): WRITE_DMA. ACB: ca 00 28 00 00 40 00 00 00 00 22 00
(ada0:ata0:0:1:0): CAM status: ATA Status Error
(ada0:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )
(ada0:ata0:0:1:0): RES: 41 04 28 00 00 00 00 00 00 22 00
(ada0:ata0:0:1:0): Retrying command
...
(ada0:ata0:0:1:0): Error 5, Retries exhausted
Requesting switchover due to disk failure on ada0
Requesting reboot due to disk failure on ada0
gpart: /dev/ada0p1: Input/output error
gpart: autofill: No space left on device
test: -lt: unexpected operator
^C[abort]

Hi,

I have almost the same issue like you have, this is in a MX960 with RE-S-1800X4 Routing Engine.

After doing a system zeroize, i got these logs:

netdaemon_disable_timer:ISSU/GRES in progress. Stopping timer for netdaemon
(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
(ada1:ata0:0:1:0): CAM status: Command timeout
(ada1:ata0:0:1:0): Retrying command
(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
(ada1:ata0:0:1:0): CAM status: Command timeout
(ada1:ata0:0:1:0): Retrying command
(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
(ada1:ata0:0:1:0): CAM status: Command timeout
(ada1:ata0:0:1:0): Retrying command
(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
(ada1:ata0:0:1:0): CAM status: Command timeout
(ada1:ata0:0:1:0): Retrying command
(ada1:ata0:0:1:0): WRITE_DMA. ACB: ca 00 e0 91 13 40 00 00 00 00 00 00
(ada1:ata0:0:1:0): CAM status: Command timeout
(ada1:ata0:0:1:0): Error 5, Retries exhausted
Requesting switchover due to disk failure on ada1
Requesting reboot due to disk failure on ada1

After reboot, the unit has booted via SATA SSD1 and disabled the CF, see below

ada0: <TOSHIBA THNSFC256GAMJ CJLA0201> ATA8-ACS SATA 2.x device
ada0: Serial Number Y1FS1002TLLZ
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 244198MB (500118192 512 byte sectors)
ada0: Previously was known as ad0
ada1 at ata0 bus 0 scbus0 target 1 lun 0
ada1: <SMART CF 20110218> ATA-4 device
ada1: Serial Number SPG2013050701129
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 512bytes)
ada1: 3807MB (7797888 512 byte sectors)
ada1: Previously was known as ad1
GEOM: ada1: corrupt or invalid GPT detected.
GEOM: ada1: GPT rejected -- may not be recoverable.

Is there anyone who know how to fix this? I am facing this problem with 4 different RE-S-1800X4.

Hi Iliass020,

You seem to be facing a different issue on the MX960. Seems you had a hard disk failure on ada1, and upon restart, the GPT was unable to be found. My recommendation is to format install using a bootable USB, and check following command output to see if you have the ada1 listed there. If not, you may need to replace the RE.

>> Check "show chassis hardware detail" output to check if ada1 is the hardrive, based on the RE rev number, the HD model may be different.

Routing Engine 1 REV 07 740-031116 9009120212 RE-S-1800x4
ada0 30533 MB UGB94BPH32H0S1-KCI 11000049731 Compact Flash
ada1 3998 MB Virtium - TuffDrive VC P1T0200278561011 14
usb0 (addr 1) EHCI root HUB 0 Intel uhub0
usb1 (addr 2) product 0x0020 32 vendor 0x8087 uhub1
DIMM 0 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80
DIMM 1 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80
DIMM 2 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80
DIMM 3 SGU04G72H1BD2SA-BB DIE REV-52 PCB REV-54 MFR ID-ce80

You may try deleting the non-recovery snapshot on yoru system first first using following command- 

request system snapshot delete snap.20190103.112901

Sample - 

Hi Pranavs,

thx for your advice. But the command "request system snapshot delete snap.xx.yy" can handle only non-recovery snapshots. This works fine. My serious problem is handling the recovery snapshot. Even if I delete all non-recovery snapshots the "request system snapshot recovery" command fails:

rhellwig@boejuniper6> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 2459MB
ERROR: The OAM volume is too small to store a snapshot

Hi RTHellwig,

Can you please check the contents of following directory -

/packages/sets/active/optional

There may be some unecessary packages here from previous upgrades. If you delete those, the snap size may go down and you should be able to succesfully create the recovery snap.

For example -

labroot@s07-1-re0> start shell
% cd /packages/sets/active/optional
% ls -ltr
total 8
lrwxrwxrwx 1 root wheel 36 Feb 16 09:27 jsdn -> /packages/db/jsdn-x86-32-17.3R3-S2.2
lrwxrwxrwx 1 root wheel 76 Feb 16 09:27 junos-daemons-mx -> /packages/db/junos-daemons-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
lrwxrwxrwx 1 root wheel 87 Feb 16 09:27 junos-dp-crypto-support-mtx -> /packages/db/junos-dp-crypto-support-mtx-x86-32-20181106.231519_builder_junos_173_r3_s2
lrwxrwxrwx 1 root wheel 82 Feb 16 09:27 junos-libs-compat32-mx -> /packages/db/junos-libs-compat32-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
lrwxrwxrwx 1 root wheel 73 Feb 16 09:27 junos-libs-mx -> /packages/db/junos-libs-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
lrwxrwxrwx 1 root wheel 76 Feb 16 09:27 junos-modules-mx -> /packages/db/junos-modules-mx-x86-64-20181106.231519_builder_junos_173_r3_s2
drwxrwxrwx 2 root wheel 512 Feb 16 09:27 boot
lrwxrwxrwx 1 root wheel 76 Feb 16 09:27 junos-runtime-mx -> /packages/db/junos-runtime-mx-x86-32-20181106.231519_builder_junos_173_r3_s2
lrwxrwxrwx 1 root wheel 67 Feb 16 09:27 jweb-ex -> /packages/db/jweb-ex-x86-32-20181106.231519_builder_junos_173_r3_s2
lrwxrwxrwx 1 root wheel 37 Feb 16 09:27 junos-install-x86-64 -> /packages/db/junos-x86-64-17.3R3-S2.2
lrwxr-xr-x 1 root wheel 67 Feb 16 09:28 jail-runtime -> /packages/db/jail-runtime-x86-32-20180911.6c98660_builder_stable_10
lrwxr-xr-x 1 root wheel 66 Feb 16 09:29 jpfe-X -> /packages/db/jpfe-X-x86-32-20181106.231519_builder_junos_173_r3_s2
lrwxr-xr-x 1 root wheel 69 Feb 16 09:29 jpfe-X960 -> /packages/db/jpfe-X960-x86-32-20181106.231519_builder_junos_173_r3_s2
lrwxr-xr-x 1 root wheel 72 Feb 16 09:29 jpfe-wrlinux -> /packages/db/jpfe-wrlinux-x86-32-20181106.231519_builder_junos_173_r3_s2
lrwxr-xr-x 1 root wheel 76 Feb 16 09:29 jplatform-ex92xx -> /packages/db/jplatform-ex92xx-x86-32-20181106.231519_builder_junos_173_r3_s2

I am able to succesfully create snaps on my EX9214. Can you check this output on your device and delete the packages that are present from an older code, and hence not required? May be save the previous packages mentioned above by creating a optional directory under : 

/packages/sets/previous/

Hey RTHellwig,

I checked on your original question as well. Currently there is no manual method to delete recovery snapshot. That may get introduced in later Junos releases. In order to be able to succefully create a recovery snap, you need to decrease the size of your snap. One way to do that is to delete the optional packages that get zipped into the recovery snap by default, but they may not be needed. Please see my previous post and share the requested output so we can try to determine which packages you can delete.

========

Kudos are appreciated.

Mark the solution as accepted if it works.

I'm not sure if you are still having this issue, but if not then just consider my post as documenting a possible solution for the next guy that has this issue and searches for it :  )

It is actually possible to manually delete a recovery snapshot, for this you need root access to the box.

Start the shell as user root:

user@router> start shell user root
Password:

Create a temporary folder in the /var/tmp directory to use as mounting point for the OAM volume:

root@router:/var/home/user # mkdir /var/tmp/oam
root@router:/var/home/user # mount /dev/gpt/oam /var/tmp/oam

CD into the mounting point just created and go into the snapshot folder:

root@router:/var/home/user # cd /var/tmp/oam/snapshot/
root@router:/var/tmp/oam/snapshot # ls
VERSION recovery.ufs.uzip

From here remove the recovery.ufs.uzip and VERSION files:

root@router:/var/tmp/oam/snapshot # rm recovery.ufs.uzip

root@router:/var/tmp/oam/snapshot # rm VERSION

Unmount the OAM filesystem:

root@router:/var/tmp/oam # cd /
root@router:/ # umount /dev/gpt/oam

Hope this helpes

Hello pranavs, jminnaar,

my last advantures...:

I run in trouble while playing around - I don't know when and why, but sometimes I lost the /oam on RE0:

Recovery Snapshots:
mount: /dev/oam: No such file or directory
ERROR: 'oam' package needs to be updated in order to use OAM functionality

I tried to resolve this via upgrading to 19.1R1.6 but it didn't help, /oam is still missing. So I tried on RE1 the "alternative delete procedure" with good results:

rhellwig@boejuniper6> show system snapshot

Non-recovery snapshots:

Snapshot snap.20190709.075643:
Location: /packages/sets/snap.20190709.075643
Creation date: Jul 9 07:56:43 2019
Junos version: 19.1R1.6

Total non-recovery snapshots: 2

Recovery Snapshots:
No recovery snapshots available on the OAM volume

But nevertheless the software seems to be too big to be stored:

rhellwig@boejuniper6> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 2691MB
ERROR: The OAM volume is too small to store a snapshot

I' not sure if 19.1 is so much bigger than 18.3 or there is still any pointer anywhere to the oam...

I checked the content of /packages/sets/active/optional and deleted all that didn't match with the current release. The result is:

root@boejuniper6:/packages/sets/active/optional # ls -lkr

total 15
lrwxr-xr-x 1 root wheel 65 Jul 9 07:47 sflow-mx -> /packages/db/sflow-mx-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 64 Jul 9 07:47 jweb-ex -> /packages/db/jweb-ex-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 73 Jul 9 07:47 junos-runtime-mx -> /packages/db/junos-runtime-mx-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 74 Jul 9 07:47 junos-net-mtx-prd -> /packages/db/junos-net-mtx-prd-x86-64-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 73 Jul 9 07:47 junos-modules-mx -> /packages/db/junos-modules-mx-x86-64-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 70 Jul 9 07:47 junos-libs-mx -> /packages/db/junos-libs-mx-x86-64-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 79 Jul 9 07:47 junos-libs-compat32-mx -> /packages/db/junos-libs-compat32-mx-x86-64-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 41 Jul 9 07:47 junos-install-ex92xx-x86-64 -> /packages/db/junos-ex92xx-x86-64-19.1R1.6
lrwxr-xr-x 1 root wheel 84 Jul 9 07:47 junos-dp-crypto-support-mtx -> /packages/db/junos-dp-crypto-support-mtx-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 73 Jul 9 07:47 junos-daemons-mx -> /packages/db/junos-daemons-mx-x86-64-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 33 Jul 9 07:46 jsdn -> /packages/db/jsdn-x86-32-19.1R1.6
lrwxr-xr-x 1 root wheel 73 Jul 9 07:46 jplatform-ex92xx -> /packages/db/jplatform-ex92xx-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 70 Jul 9 07:51 jpfe-wrlinux9 -> /packages/db/jpfe-wrlinux9-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 66 Jul 9 07:46 jpfe-X960 -> /packages/db/jpfe-X960-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 63 Jul 9 07:46 jpfe-X -> /packages/db/jpfe-X-x86-32-20190321.051058_builder_junos_191_r1
lrwxr-xr-x 1 root wheel 67 Jul 9 07:46 jail-runtime -> /packages/db/jail-runtime-x86-32-20190305.df99236_builder_stable_11
drwxr-xr-x 2 root wheel 512 Jul 9 07:47 boot

Hi RTHellwig,

If the oam partition doesn't exist or is corrupted, then a recovery snapshot will probably fail as well as it needs to be stored on this partition. Did you try to recover the OAM volume?

user@router> request system recover oam-volume

Regards,

John

Unfortunately it doesn't help:

root@boejuniper6> request system recover oam-volume
NOTICE: Recovering the OAM volume ...
(ada0:ata0:0:1:0): WRITE_DMA. ACB: ca 00 28 00 00 40 00 00 00 00 20 00
(ada0:ata0:0:1:0): CAM status: ATA Status Error
(ada0:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )
(ada0:ata0:0:1:0): RES: 41 04 28 00 00 00 00 00 00 20 00
(ada0:ata0:0:1:0): Retrying command
...
(ada0:ata0:0:1:0): WRITE_DMA. ACB: ca 00 28 00 00 40 00 00 00 00 20 00
(ada0:ata0:0:1:0): CAM status: ATA Status Error
(ada0:ata0:0:1:0): ATA status: 41 (DRDY ERR), error: 04 (ABRT )
(ada0:ata0:0:1:0): RES: 41 04 28 00 00 00 00 00 00 20 00
(ada0:ata0:0:1:0): Error 5, Retries exhausted
Requesting switchover due to disk failure on ada0
Requesting reboot due to disk failure on ada0
gpart: /dev/ada0p1: Input/output error
gpart: autofill: No space left on device
test: -lt: unexpected operator

^C[abort]

I had this problem and couldn't find a solution anywhere, no matter what I did, there was never enough space to do a recovery snapshot. I finally found something that works, though it's undocumented. This doesn't APPEAR to cause any problems, but who knows..

From shell prompt, I ran these two commands:

pkg setop rm previous
pkg delete old

And several hundred MB was freed up. I was able to create a recovery snapshot. YMMV

I tried it on both REs (on backup with good OAM, on master with bad OAM) without success (nothing happened):

{backup}
root@boejuniper6:~ # pkg setop rm previous
root@boejuniper6:~ # pkg delete old
root@

{backup}
root@boejuniper6> request system snapshot recovery
Creating image ...
Compressing image ...
Image size is 2691MB
ERROR: The OAM volume is too small to store a snapshotboejuniper6:~ #

-----------------------------------------------------------------------

{master}
root@boejuniper6:~ # pkg setop rm previous
root@boejuniper6:~ # pkg delete old
root@boejuniper6:~ #

{master}
root@boejuniper6> show system snapshot

Non-recovery snapshots:
Snapshot snap.20190103.143713:
Location: /packages/sets/snap.20190103.143713
Creation date: Jan 3 14:37:13 2019
Junos version: 17.2R3.4

Snapshot snap.20190104.100147:
Location: /packages/sets/snap.20190104.100147
Creation date: Jan 4 10:01:47 2019
Junos version: 18.3R1.9

Total non-recovery snapshots: 2

Recovery Snapshots:
mount: /dev/oam: No such file or directory
ERROR: 'oam' package needs to be updated in order to use OAM functionality