No Cluster functionality after Upgrade
Hello,
after upgrade (In-Service Software Upgrade ) our Juniper Cluster (Two SRX240) from version 12.1X46-D40 to 12.1X55-D40 our cluster no longer works. No connection between the nodes.
Before upgrade we make a snapshot on a usb stick.
Actualy we boot from one node of the usb stick, this works.
--- JUNOS 12.1X46-D40.2 built 2015-09-26 02:25:28 UTC
---
--- NOTICE: System is running on alternate media device (/dev/da1s1a).
---
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Some information of the running cluster:
root@FW> show chassis cluster status
Cluster ID: 1
Node Priority Status Preempt Manual Monitor-failures
Redundancy group: 0 , Failover count: 1
node0 254 primary no no None
node1 0 lost n/a n/a n/a
Redundancy group: 1 , Failover count: 1
node0 0 primary no no CS
node1 0 lost n/a n/a n/a
root@FW> show chassis cluster interfaces
Control link status: Up
Control interfaces:
Index Interface Status Internal-SA
0 fxp1 Down Disabled
Fabric link status: Down
Fabric interfaces:
Name Child-interface Status
(Physical/Monitored)
fab0 ge-0/0/2 Up / Down
fab0
root@FW> show chassis cluster statistics
Control link statistics:
Control link 0:
Heartbeat packets sent: 135504
Heartbeat packets received: 0
Heartbeat packet errors: 0
Fabric link statistics:
Child link 0
Probes sent: 271216
Probes received: 0
Child link 1
Probes sent: 0
Probes received: 0
Services Synchronized:
Service name RTOs sent RTOs received
Translation context 0 0
Incoming NAT 0 0
Resource manager 0 0
DS-LITE create 0 0
Session create 8403648 0
IPv6 session create 0 0
Session close 870398 0
IPv6 session close 0 0
Session change 247087 0
IPv6 session change 0 0
ALG Support Library 3318 0
Gate create 0 0
Session ageout refresh requests 0 0
IPv6 session ageout refresh requests 0 0
Session ageout refresh replies 0 0
IPv6 session ageout refresh replies 0 0
IPSec VPN 0 0
Firewall user authentication 0 0
MGCP ALG 0 0
H323 ALG 0 0
SIP ALG 0 0
SCCP ALG 0 0
PPTP ALG 0 0
JSF PPTP ALG 0 0
RPC ALG 0 0
RTSP ALG 0 0
RAS ALG 0 0
MAC address learning 0 0
GPRS GTP 0 0
GPRS SCTP 0 0
GPRS FRAMEWORK 0 0
JSF RTSP ALG 0 0
JSF SUNRPC MAP 0 0
JSF MSRPC MAP 0 0
DS-LITE delete 0 0
JSF SLB 0 0
APPID 0 0
JSF MGCP MAP 0 0
JSF H323 ALG 0 0
JSF RAS ALG 0 0
JSF SCCP MAP 0 0
JSF SIP MAP 0 0
PST_NAT_CREATE 0 0
PST_NAT_CLOSE 0 0
PST_NAT_UPDATE 0 0
JSF TCP STACK 0 0
JSF IKE ALG 0 0
root@FW> show chassis cluster information
node0:
--------------------------------------------------------------------------
Redundancy Group Information:
Redundancy Group 0 , Current State: primary, Weight: 255
Time From To Reason
Nov 27 19:54:44 hold secondary Hold timer expired
Nov 27 19:55:00 secondary primary Only node present
Redundancy Group 1 , Current State: primary, Weight: 0
Time From To Reason
Nov 27 19:54:44 hold secondary Hold timer expired
Nov 27 19:55:00 secondary primary Only node present
Chassis cluster LED information:
Current LED color: Red
Last LED change reason: Peer node: node1 is not present
Control port tagging:
Disabled
Failure Information:
Coldsync Monitoring Failure Information:
Statistics:
Coldsync Total SPUs: 1
Coldsync completed SPUs: 0
Coldsync not complete SPUs: 1
Fabric-link Failure Information:
Fabric Interface: fab0
Child interface Physical / Monitored Status
ge-0/0/2 Up / Down
Control-link Failure Information:
Link Status: Up
Dual Control Link Status: Unsupported
Interface Physical / Monitored Status
fxp1 Up / Down
root@FW> show chassis hardware
node0:
--------------------------------------------------------------------------
Hardware inventory:
Item Version Part number Serial number Description
Chassis SRX240H
Routing Engine REV 38 750-021793 AAAV6932 RE-SRX240H
FPC 0 FPC
PIC 0 16x GE Base PIC
FPC 1 750-029144 082009700389 FPC
PIC 0 1x ADSL 2/2+ B
Power Supply 0
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Some information of the second node:
root@FW> show chassis cluster status
Monitor Failure codes:
CS Cold Sync monitoring FL Fabric Connection monitoring
GR GRES monitoring HW Hardware monitoring
IF Interface monitoring IP IP monitoring
LB Loopback monitoring MB Mbuf monitoring
NH Nexthop monitoring NP NPC monitoring
SP SPU monitoring SM Schedule monitoring
Cluster ID: 1
Node Priority Status Preempt Manual Monitor-failures
Redundancy group: 0 , Failover count: 0
node0 0 lost n/a n/a n/a
node1 1 secondary no no None
Redundancy group: 1 , Failover count: 0
node0 0 lost n/a n/a n/a
node1 0 secondary no no IF CS
root@FW> show chassis cluster statistics
Control link statistics:
Control link 0:
Heartbeat packets sent: 134771
Heartbeat packets received: 0
Heartbeat packet errors: 0
Fabric link statistics:
Child link 0
Probes sent: 0
Probes received: 0
Child link 1
Probes sent: 0
Probes received: 0
Services Synchronized:
Service name RTOs sent RTOs received
Translation context 0 0
Incoming NAT 0 0
Resource manager 0 0
DS-LITE create 0 0
Session create 0 0
IPv6 session create 0 0
Session close 0 0
IPv6 session close 0 0
Session change 0 0
IPv6 session change 0 0
ALG Support Library 0 0
Gate create 0 0
Session ageout refresh requests 0 0
IPv6 session ageout refresh requests 0 0
Session ageout refresh replies 0 0
IPv6 session ageout refresh replies 0 0
IPSec VPN 0 0
Firewall user authentication 0 0
MGCP ALG 0 0
H323 ALG 0 0
SIP ALG 0 0
SCCP ALG 0 0
PPTP ALG 0 0
JSF PPTP ALG 0 0
RPC ALG 0 0
RTSP ALG 0 0
RAS ALG 0 0
MAC address learning 0 0
GPRS GTP 0 0
GPRS SCTP 0 0
GPRS FRAMEWORK 0 0
JSF RTSP ALG 0 0
JSF SUNRPC MAP 0 0
JSF MSRPC MAP 0 0
DS-LITE delete 0 0
JSF SLB 0 0
APPID 0 0
JSF MGCP MAP 0 0
JSF H323 ALG 0 0
JSF RAS ALG 0 0
JSF SCCP MAP 0 0
JSF SIP MAP 0 0
PST_NAT_CREATE 0 0
PST_NAT_CLOSE 0 0
PST_NAT_UPDATE 0 0
JSF TCP STACK 0 0
JSF IKE ALG 0 0
root@FWG> show chassis cluster information
node1:
--------------------------------------------------------------------------
Redundancy Group Information:
Redundancy Group 0 , Current State: secondary, Weight: 255
Time From To Reason
Nov 27 19:09:55 hold secondary Hold timer expired
Redundancy Group 1 , Current State: secondary, Weight: -3315
Time From To Reason
Nov 27 19:09:56 hold secondary Hold timer expired
Chassis cluster LED information:
Current LED color: Red
Last LED change reason: Peer node: node0 is not present
Control port tagging:
Disabled
Failure Information:
Coldsync Monitoring Failure Information:
Statistics:
Coldsync Total SPUs: 1
Coldsync completed SPUs: 0
Coldsync not complete SPUs: 1
Interface Monitoring Failure Information:
Redundancy Group 1, Monitoring status: Failed
Interface Status
ge-5/0/15 Down
ge-5/0/14 Down
ge-5/0/13 Down
ge-5/0/12 Down
ge-5/0/11 Down
ge-5/0/10 Down
ge-5/0/9 Down
ge-5/0/8 Down
ge-5/0/7 Down
ge-5/0/6 Down
ge-5/0/5 Down
ge-5/0/4 Down
ge-5/0/3 Down
Control-link Failure Information:
Link Status: Up
Dual Control Link Status: Unsupported
Interface Physical / Monitored Status
fxp1 Up / Down
root@FW> edit
warning: Clustering enabled; using private edit
error: shared configuration database modified
Please temporarily use 'configure shared' to commit
outstanding changes in the shared database, exit,
and return to configuration mode using 'configure'
root@FW> configure shared
Entering configuration mode
The configuration has been changed but not committed
{secondary:node1}[edit]
root@FW# commit
warning: ISSU in progress, commit disallowed
{secondary:node1}[edit]
root@FW> show chassis hardware
node1:
--------------------------------------------------------------------------
Hardware inventory:
Item Version Part number Serial number Description
Chassis SRX240H
FPC 0 FPC
root@FW> show chassis fpc pic-status
node1:
--------------------------------------------------------------------------
Slot 0 Present FPC
{secondary:node1}
------
It seems that node1 is in the ISSU progress. How can i cancel this? i try this: https://kb.juniper.net/InfoCenter/index?page=content&id=KB26324&cat=JUNOS&actp=LIST&smlogin=true but no success.
In Addition no Hardware Information and no network moduls on node1. A problem because of "ISSU mode"?
If you need log files i can send you.
Best regards