Junos OS

Hello Expert,

I have these issue. Unable to commit sync. the error are below:

I checked the pastitions and seems to be ok for both RE

User@GGSN> edit
Entering configuration mode
The configuration has been changed but not committed

{master}[edit]
User@GGSN#

{master}[edit]
User@GGSN# commit check | display detail
re0:
2012-08-28 12:22:53 WAT: obtaining db lock on re1
error: timeout waiting for response from re1
error: timeout waiting for response from re1
error: remote lock-configuration failed on re1

note: consider using 'commit synchronize force' to
terminate remote edit sessions and force the commit
2012-08-28 12:23:53 WAT: failed to lock re1
error: timeout waiting for response from re1

{master}[edit]
User@GGSN# commit synchronize force
re0:
error: timeout waiting for response from re1
error: timeout waiting for response from re1
error: remote lock-configuration failed on re1
error: timeout waiting for response from re1

In the mastership logs I can see the following:

Master RE

ug 24 23:11:08 failed to send RE info/keepalive: errno=50, total=1 in the last 20 sec
Aug 24 23:11:09 failed to receive keepalives from other RE for the last 20 sec
Aug 24 23:11:29 failed to send RE info/keepalive: errno=50, total=3 in the last 20 sec
Aug 24 23:11:29 failed to send RE info/keepalive: errno=65, total=18 in the last 20 sec
Aug 24 23:11:30 failed to receive keepalives from other RE for the last 40 sec
Aug 24 23:11:49 failed to send RE info/keepalive: errno=65, total=21 in the last 20 sec
Aug 24 23:11:50 failed to receive keepalives from other RE for the last 60 sec
Aug 24 23:12:10 failed to send RE info/keepalive: errno=65, total=22 in the last 20 sec
Aug 24 23:12:11 failed to receive keepalives from other RE for the last 80 sec
Aug 24 23:12:31 failed to send RE info/keepalive: errno=65, total=21 in the last 20 sec
Aug 24 23:12:32 failed to receive keepalives from other RE for the last 100 sec
Aug 24 23:12:52 failed to send RE info/keepalive: errno=65, total=22 in the last 20 sec
Aug 24 23:12:53 failed to receive keepalives from other RE for the last 120 sec
Aug 24 23:13:07 event = E_ORE_B, state = master, param = 0x0x8be3008
Aug 24 23:13:07 Currentstate master NextState master reason_code 1
Aug 24 23:13:07 new state = master
Aug 24 23:13:12 failed to send RE info/keepalive: errno=65, total=5 in the last 20 sec
Aug 24 23:15:01 event = E_CMD_S, state = master, param = 0x0x0
Aug 24 23:15:01 send "you are the master" request
Aug 24 23:15:01 Currentstate master NextState giveup reason_code 1
Aug 24 23:15:01 timestamp: Fri Aug 24 23:15:01 2012

Aug 24 23:15:01 new state = giveup
Aug 24 23:15:01 received version 1, "you are the master" acknowledgement
Aug 24 23:15:01 event = E_ACK_Y, state = giveup, param = 0x0x0
Aug 24 23:15:01 mcontrol_shutdown
Aug 24 23:15:01 mcontrol_notmaster
Aug 24 23:15:01 Currentstate giveup NextState disabled reason_code 1
Aug 24 23:15:01 new state = disabled
Aug 24 23:15:01 received version 1, "claim mastership" request
Aug 24 23:15:01 event = E_REQ_C, state = disabled, param = 0x0x0
Aug 24 23:15:01 send "claim mastership" acknowledgement
Aug 24 23:15:01 Currentstate disabled NextState disabled reason_code 1
Aug 24 23:15:01 new state = disabled
Aug 24 23:15:01 event = E_ORE_B, state = disabled, param = 0x0x8be3008
Aug 24 23:15:01 Currentstate disabled NextState disabled reason_code 1
Aug 24 23:15:01 new state = disabled
Aug 24 23:15:14 event = E_ORE_M, state = disabled, param = 0x0x8be3008
Aug 24 23:15:14 mcontrol_disabled_exit
Aug 24 23:15:14 mcontrol_shutdown
Aug 24 23:15:14 mcontrol_notmaster

Backup RE:

Aug 24 23:13:03 *** mcontrol init V01 ***cknowledgement edgement ages to 844ent
Aug 24 23:13:03 soft-restart: is not a masterte disabled reason_code 1to 728
Aug 24 23:13:03 Socket = 0x00000017dNextState backup reason_code 008s to 204
Aug 24 23:13:03 mcontrol hipri thread createdbled, param = 0x0x8be2008to 16ment
Aug 24 23:13:03 *** re_priority is 2***extState disabled reason_code 1to 844
Aug 24 23:13:03 event = E_CFG_B, state = init, param = 0x0x0x0n_code 1to 940
Aug 24 23:13:03 Currentstate init NextState backup reason_code 0be2008
Aug 24 23:13:03 new state = backupexitxtState master reason_code 1e008to 844
Aug 24 23:13:06 reallocated memory used for reading mcontrol messages to 16
Aug 24 23:13:06 reallocated memory used for reading mcontrol messages to 844ent
Aug 24 23:13:06 event = E_ORE_M, state = backup, param = 0x0x8be0008
Aug 24 23:13:06 Currentstate backup NextState backup reason_code 0ec08ledgement
Aug 24 23:13:06 new state = backupe24 22:58:54 2012try = 1.code 0ages to 16
Aug 24 23:15:01 received version 1, "you are the master" request0ages to 716
Aug 24 23:15:01 event = E_REQ_Y, state = backup, param = 0x0x0edgement
Aug 24 23:15:01 send "you are the master" acknowledgement x0nowledgement 716
Aug 24 23:15:01 send "claim mastership" request up reason_code 0ec08s to 940
Aug 24 23:15:01 Currentstate backup NextState claim reason_code 0ages to 940ent
Aug 24 23:15:01 new state = claimy used for reading mcontrol messages to 16
Aug 24 23:15:01 received version 1, "claim mastership" acknowledgementto 844
Aug 24 23:15:01 event = E_ACK_C, state = claim, param = 0x0x08be9008 1
Aug 24 23:15:10 The local RE becomes the master, retry = 0._code 0ges to 940
Aug 24 23:15:10 Currentstate claim NextState master reason_code 0ages to 160
Aug 24 23:15:10 timestamp: Fri Aug 24 23:15:10 2012ment edgement ages to 844ent
Aug 24 23:13:03 soft-restart: is not a masterte disabled reason_code 1to 728
Aug 24 23:15:10 new state = master7dNextState backup reason_code 008s to 204
Aug 24 23:15:15 received version 1, "claim mastership" requestx8be2008to 16ment
Aug 24 23:15:15 event = E_REQ_C, state = master, param = 0x0x0n_code 1to 844
Aug 24 23:15:15 send "claim mastership" negative acknowledgement ode 1to 940
Aug 24 23:15:15 Currentstate master NextState master reason_code 12008
Aug 24 23:15:15 new state = masterexitxtState master reason_code 1e008to 844
Aug 24 23:15:16 event = E_ORE_B, state = master, param = 0x0x8be0008s to 16
Aug 24 23:15:16 Currentstate master NextState master reason_code 1ges to 844ent
Aug 24 23:15:16 new state = master
Aug 24 23:15:16 reallocated memory used for reading mcontrol messages to 940ent
Aug 24 23:15:16 event = E_ORE_B, state = master, param = 0x0x8be0008s to 16
Aug 24 23:15:16 Currentstate master NextState master reason_code 1ges to 716
Aug 24 23:15:16 new state = mastertate = backup, param = 0x0x0edgement
Aug 24 23:20:37 event = E_NO_IPC, state = master, param = 0x0x0ledgement 716
Aug 24 23:20:37 No response from the other routing engine for the last 2 seconds
.ug 24 23:15:01 Currentstate backup NextState claim reason_code 0ages to 940ent
Aug 24 23:15:01 new state = claimy used for reading mcontrol messages to 16
Aug 24 23:20:37 Currentstate master NextState master reason_code 1mentto 844
Aug 24 23:20:37 new state = mastertate = claim, param = 0x0x08be9008 1
Aug 24 23:20:58 failed to receive keepalives from other RE for the last 20 sec
Aug 24 23:21:13 failed to send RE info/keepalive: errno=50, total=3 in the last
20 sec 23:15:10 timestamp: Fri Aug 24 23:15:10 2012ment edgement ages to 844ent
Aug 24 23:21:13 failed to send RE info/keepalive: errno=65, total=9 in the last
20 sec 23:15:10 new state = master7dNextState backup reason_code 008s to 204
Aug 24 23:21:19 failed to receive keepalives from other RE for the last 40 sect
Aug 24 23:21:34 failed to send RE info/keepalive: errno=65, total=22 in the last
20 sec23:15:15 send "claim mastership" negative acknowledgement ode 1to 940
Aug 24 23:21:40 failed to receive keepalives from other RE for the last 60 sec
Aug 24 23:21:55 failed to send RE info/keepalive: errno=65, total=22 in the last
20 sec23:15:16 event = E_ORE_B, state = master, param = 0x0x8be0008s to 16
Aug 24 23:22:01 failed to receive keepalives from other RE for the last 80 sect
Aug 24 23:22:16 failed to send RE info/keepalive: errno=65, total=23 in the last
20 sec23:15:16 reallocated memory used for reading mcontrol messages to 940ent
Aug 24 23:22:22 failed to receive keepalives from other RE for the last 100 sec
Aug 24 23:22:37 failed to send RE info/keepalive: errno=65, total=22 in the last
20 sec23:15:16 new state = mastertate = backup, param = 0x0x0edgement
Aug 24 23:22:43 failed to receive keepalives from other RE for the last 120 sec
Aug 24 23:22:54 received version 1, "claim mastership" requestthe last 2 seconds
Aug 24 23:22:54 event = E_REQ_C, state = master, param = 0x0x0e 0ages to 940ent
Aug 24 23:22:54 send "claim mastership" negative acknowledgement ages to 16
Aug 24 23:22:54 Currentstate master NextState master reason_code 1mentto 844
Aug 24 23:22:54 new state = mastertate = claim, param = 0x0x08be9008 1
Aug 24 23:22:57 event = E_ORE_B, state = master, param = 0x0x8be0008ast 20 sec
Aug 24 23:22:57 Currentstate master NextState master reason_code 13 in the last
Aug 24 23:22:57 new state = master 24 23:15:10 2012ment edgement ages to 844ent
Aug 24 23:22:57 event = E_ORE_B, state = master, param = 0x0x8be0008in the last
Aug 24 23:22:57 Currentstate master NextState master reason_code 108s to 204
Aug 24 23:22:57 new state = masterkeepalives from other RE for the last 40 sect
Aug 24 23:22:58 failed to send RE info/keepalive: errno=65, total=9 in the last
20 secc23:15:15 send "claim mastership" negative acknowledgement ode 1to 940
Aug 24 23:24:02 event = E_CMD_S, state = master, param = 0x0x0 the last 60 sec
Aug 24 23:24:02 send "you are the master" request errno=65, total=22 in the

Appreciate if you can help me determine the cause why its cant commit sync

From the logs it looks like RE1 isn't getting messages from RE0.  Had something similar happen once or twice although I couldn't tell you how it got to that state.  From the logs it looks like RE1 is the confused one.

To recover:

1) Reboot RE1

2) While RE1 is down, "commit force" on RE0

- you will get a warning that it failed to contact RE1 but the commit should continue

- your changes will go live

3) Once RE1 comes back online, "commit synchronize full" on RE0

If they still can't communicate then there is something funky on RE0.  Restarting the chassis-control subsystem may clear it but I'm not certain.

-Chad

Agreed, also after a reboot of RE1 if it still cannot communicate appropriately using Chad's directions, it may be worth halting it and re-seating it.  I've had it in the past where it got into some funky state like this and removing all power from the confused RE solved it.