Thanks for the reply. Here are the requested outputs.
me@mgmtfw01-b> show version
node0:
--------------------------------------------------------------------------
Hostname: mgmtfw01-a
Model: srx240h2
JUNOS Software Release [12.3X48-D45.6]
node1:
--------------------------------------------------------------------------
Hostname: mgmtfw01-b
Model: srx240h2
JUNOS Software Release [12.3X48-D45.6]
{secondary:node1}
me@mgmtfw01-b> show chassis alarms
node0:
--------------------------------------------------------------------------
No alarms currently active
node1:
--------------------------------------------------------------------------
No alarms currently active
{secondary:node1}
me@mgmtfw01-b> show system core-dumps
node0:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
/var/tmp/*core*: No such file or directory
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
node1:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
/var/tmp/*core*: No such file or directory
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
{secondary:node1}
me@mgmtfw01-b> show chassis routing-engine | no-more
node0:
--------------------------------------------------------------------------
Routing Engine status:
Temperature 39 degrees C / 102 degrees F
CPU temperature 39 degrees C / 102 degrees F
Total memory 2048 MB Max 1126 MB used ( 55 percent)
Control plane memory 1072 MB Max 557 MB used ( 52 percent)
Data plane memory 976 MB Max 566 MB used ( 58 percent)
CPU utilization:
User 91 percent
Background 0 percent
Kernel 9 percent
Interrupt 0 percent
Idle 1 percent
Model RE-SRX240H2
Serial ID ACMX4357
Start time 2018-10-11 11:15:51 CEST
Uptime 60 days, 2 hours, 42 minutes, 5 seconds
Last reboot reason Router rebooted after a normal shutdown.
Load averages: 1 minute 5 minute 15 minute
1.01 1.15 1.19
node1:
--------------------------------------------------------------------------
Routing Engine status:
Temperature 40 degrees C / 104 degrees F
CPU temperature 39 degrees C / 102 degrees F
Total memory 2048 MB Max 389 MB used ( 19 percent)
Control plane memory 1072 MB Max 386 MB used ( 36 percent)
Data plane memory 976 MB Max 0 MB used ( 0 percent)
CPU utilization:
User 20 percent
Background 0 percent
Kernel 74 percent
Interrupt 0 percent
Idle 6 percent
Model RE-SRX240H2
Serial ID ACMZ8364
Start time 2018-12-10 10:21:21 CET
Uptime 2 hours, 13 minutes, 45 seconds
Last reboot reason Router rebooted after a normal shutdown.
Load averages: 1 minute 5 minute 15 minute
1.46 1.83 1.83
{secondary:node1}
me@mgmtfw01-b> show chassis fpc pic-status
node0:
--------------------------------------------------------------------------
Slot 0 Online FPC
PIC 0 Online 16x GE Base PIC
node1:
--------------------------------------------------------------------------
Slot 0 Present FPC
{secondary:node1}
me@mgmtfw01-b> show chassis fpc detail | no-more
node0:
--------------------------------------------------------------------------
Slot 0 information:
State Online
Total CPU DRAM ---- CPU less FPC ----
Start time 2018-12-05 09:46:59 CET
Uptime 5 days, 3 hours, 11 minutes, 23 seconds
node1:
--------------------------------------------------------------------------
Slot 0 information:
State Present
Total CPU DRAM ---- CPU less FPC ----
{secondary:node1}
me@mgmtfw01-b> show chassis cluster status
Monitor Failure codes:
CS Cold Sync monitoring FL Fabric Connection monitoring
GR GRES monitoring HW Hardware monitoring
IF Interface monitoring IP IP monitoring
LB Loopback monitoring MB Mbuf monitoring
NH Nexthop monitoring NP NPC monitoring
SP SPU monitoring SM Schedule monitoring
CF Config Sync monitoring
Cluster ID: 1
Node Priority Status Preempt Manual Monitor-failures
Redundancy group: 0 , Failover count: 0
node0 200 primary no no None
node1 0 secondary no no CF
Redundancy group: 1 , Failover count: 0
node0 0 primary no no CS
node1 0 secondary no no IF CS CF
Redundancy group: 2 , Failover count: 0
node0 0 primary no no CS
node1 0 secondary no no IF CS CF
Redundancy group: 3 , Failover count: 0
node0 0 primary no no CS
node1 0 secondary no no IF CS CF
Redundancy group: 4 , Failover count: 0
node0 0 primary no no CS
node1 0 secondary no no IF CS CF
{secondary:node1}
me@mgmtfw01-b> show chassis cluster interfaces | no-more
Control link status: Up
Control interfaces:
Index Interface Monitored-Status Internal-SA
0 fxp1 Up Disabled
Fabric link status: Down
Fabric interfaces:
Name Child-interface Status
(Physical/Monitored)
fab0
fab0
fab1
fab1
Redundant-pseudo-interface Information:
Name Status Redundancy-group
lo0 Up 0
Interface Monitoring:
Interface Weight Status Redundancy-group
ge-5/0/14 128 Down 1
ge-5/0/15 128 Down 1
ge-0/0/15 128 Down 1
ge-0/0/14 128 Down 1
ge-5/0/13 255 Down 2
ge-0/0/13 255 Down 2
ge-5/0/12 128 Down 3
ge-5/0/11 128 Down 3
ge-0/0/12 128 Down 3
ge-0/0/11 128 Down 3
ge-5/0/10 255 Down 4
ge-0/0/10 255 Down 4
{secondary:node1}
me@mgmtfw01-b> show chassis cluster information detail | no-more
node0:
--------------------------------------------------------------------------
Redundancy mode:
Configured mode: active-active
Operational mode: active-active
Cluster configuration:
Heartbeat interval: 1000 ms
Heartbeat threshold: 3
Control link recovery: Enabled
Fabric link down timeout: 66 sec
Node health information:
Local node health: Not healthy
Remote node health: Not healthy
Redundancy group: 0, Threshold: 255, Monitoring failures: none
Events:
Oct 11 11:15:13.709 : hold->secondary, reason: Hold timer expired
Oct 25 15:39:14.955 : secondary->primary, reason: Only node present
Dec 5 09:39:39.985 : primary->secondary-hold, reason: Manual failover
Dec 5 09:39:49.702 : secondary-hold->primary, reason: Only node present
Dec 5 09:41:40.678 : primary->secondary-hold, reason: Manual failover
Dec 5 09:42:05.439 : secondary-hold->primary, reason: Only node present
Dec 5 09:43:12.740 : primary->secondary-hold, reason: Manual failover
Dec 5 09:43:38.498 : secondary-hold->primary, reason: Only node present
Dec 5 09:45:16.073 : primary->secondary-hold, reason: Manual failover
Dec 5 09:45:42.458 : secondary-hold->primary, reason: Only node present
Redundancy group: 1, Threshold: 0, Monitoring failures: cold-sync-monitoring
Events:
Oct 11 11:15:13.773 : hold->secondary, reason: Hold timer expired
Oct 25 15:39:14.907 : secondary->ineligible, reason: Fabric link down
Oct 25 15:39:15.106 : ineligible->primary, reason: Only node present
Redundancy group: 2, Threshold: 0, Monitoring failures: cold-sync-monitoring
Events:
Oct 11 11:15:13.812 : hold->secondary, reason: Hold timer expired
Oct 25 15:39:14.911 : secondary->ineligible, reason: Fabric link down
Oct 25 15:39:15.138 : ineligible->primary, reason: Only node present
Redundancy group: 3, Threshold: 0, Monitoring failures: cold-sync-monitoring
Events:
Oct 11 11:15:13.849 : hold->secondary, reason: Hold timer expired
Oct 25 15:39:14.916 : secondary->ineligible, reason: Fabric link down
Oct 25 15:39:15.142 : ineligible->primary, reason: Only node present
Redundancy group: 4, Threshold: 0, Monitoring failures: cold-sync-monitoring
Events:
Oct 11 11:15:13.888 : hold->secondary, reason: Hold timer expired
Oct 11 17:32:32.836 : secondary->primary, reason: Remote is in secondary hold
Oct 25 15:39:14.917 : primary->ineligible, reason: Fabric link down
Oct 25 15:39:15.169 : ineligible->primary, reason: Only node present
Control link statistics:
Control link 0:
Heartbeat packets sent: 5185608
Heartbeat packets received: 4956247
Heartbeat packet errors: 0
Duplicate heartbeat packets received: 0
Control recovery packet count: 0
Sequence number of last heartbeat packet sent: 5185628
Sequence number of last heartbeat packet received: 5231
Fabric link statistics:
Child link 0
Probes sent: 886973
Probes received: 0
Child link 1
Probes sent: 533133
Probes received: 0
Switch fabric link statistics:
Probe state : DOWN
Probes sent: 0
Probes received: 0
Probe recv errors: 0
Probe send errors: 0
Probe recv dropped: 0
Sequence number of last probe sent: 0
Sequence number of last probe received: 0
Chassis cluster LED information:
Current LED color: Amber
Last LED change reason: Monitored objects are down
Control port tagging:
Disabled
Cold Synchronization:
Status:
Cold synchronization completed for: N/A
Cold synchronization failed for: N/A
Cold synchronization not known for: N/A
Current Monitoring Weight: 255
Progress:
CS Prereq 0 of 1 SPUs completed
1. if_state sync 1 SPUs completed
2. fabric link 0 SPUs completed
3. policy data sync 1 SPUs completed
4. cp ready 0 SPUs completed
5. VPN data sync 0 SPUs completed
6. Dynamic addr sync 0 SPUs completed
CS RTO sync 0 of 1 SPUs completed
CS Postreq 0 of 1 SPUs completed
Statistics:
Number of cold synchronization completed: 0
Number of cold synchronization failed: 0
Events:
Oct 11 11:17:27.928 : Cold sync for PFE is RTO sync in process
Oct 11 11:17:27.929 : Cold sync for PFE is Post-req check in process
Oct 11 11:17:27.936 : Cold sync for PFE is Completed
Dec 5 09:54:17.411 : Cold sync for PFE is Not complete
Loopback Information:
PIC Name Loopback Nexthop Mbuf
-------------------------------------------------
Success Success Success
Interface monitoring:
Statistics:
Monitored interface failure count: 303
Events:
Dec 6 12:50:19.901 : Interface ge-0/0/14 monitored by rg 1, changed state from Down to Up
Dec 6 12:50:22.364 : Interface ge-0/0/15 monitored by rg 1, changed state from Down to Up
Dec 6 12:50:36.740 : Interface ge-0/0/11 monitored by rg 3, changed state from Up to Down
Dec 6 12:50:36.855 : Interface ge-0/0/12 monitored by rg 3, changed state from Up to Down
Dec 6 12:50:39.944 : Interface ge-0/0/11 monitored by rg 3, changed state from Down to Up
Dec 6 12:50:40.046 : Interface ge-0/0/12 monitored by rg 3, changed state from Down to Up
Dec 6 12:50:44.296 : Interface ge-0/0/14 monitored by rg 1, changed state from Up to Down
Dec 6 12:50:46.808 : Interface ge-0/0/15 monitored by rg 1, changed state from Up to Down
Dec 6 12:50:48.643 : Interface ge-0/0/14 monitored by rg 1, changed state from Down to Up
Dec 6 12:50:49.966 : Interface ge-0/0/15 monitored by rg 1, changed state from Down to Up
Fabric monitoring:
Status:
Fabric Monitoring: Enabled
Activation status: Suspended by local node and other node
Fabric Status reported by data plane: Down
JSRPD internal fabric status: Down
Fabric link events:
Dec 10 12:54:42.405 : Fabric link fab1 is down
Dec 10 12:54:42.429 : Fabric link fab1 is down
Dec 10 12:54:42.450 : Fabric link fab1 is deleted
Dec 10 12:54:42.488 : Fabric link fab0 is up
Dec 10 12:55:39.375 : Fabric link fab1 is down
Dec 10 12:55:39.412 : Fabric link fab1 is down
Dec 10 12:55:39.465 : Fabric link fab1 is down
Dec 10 12:55:39.510 : Fabric link fab0 is up
Dec 10 12:55:39.549 : Fabric link fab1 is down
Dec 10 12:55:39.575 : Fabric link fab1 is down
Control link status: Up
Server information:
Server status : Connected
Server connected to 130.16.0.1/52793
Client information:
Client status : Inactive
Client connected to None
Control port tagging:
Disabled
Control link events:
Dec 10 12:43:01.319 : Control link up, link status timer
Dec 10 12:43:33.079 : Control link fxp1 is up
Dec 10 12:48:27.022 : Control link down, link status timer
Dec 10 12:48:39.021 : Control link fxp1 is up
Dec 10 12:49:04.788 : Control link up, link status timer
Dec 10 12:49:36.445 : Control link fxp1 is up
Dec 10 12:54:30.529 : Control link down, link status timer
Dec 10 12:54:42.497 : Control link fxp1 is up
Dec 10 12:55:08.238 : Control link up, link status timer
Dec 10 12:55:39.522 : Control link fxp1 is up
Hardware monitoring:
Status:
Activation status: Enabled
Redundancy group 0 failover for hardware faults: Enabled
Hardware redundancy group 0 errors: 0
Hardware redundancy group 1 errors: 0
Schedule monitoring:
Status:
Activation status: Disabled
Schedule slip detected: None
Timer ignored: No
Statistics:
Total slip detected count: 31
Longest slip duration: 25(s)
Events:
Dec 7 10:57:17.079 : Detected schedule slip
Dec 7 10:58:17.170 : Cleared schedule slip
Dec 7 12:23:33.217 : Detected schedule slip
Dec 7 12:24:33.330 : Cleared schedule slip
Dec 8 07:07:46.401 : Detected schedule slip
Dec 8 07:08:46.859 : Cleared schedule slip
Dec 8 11:52:31.165 : Detected schedule slip
Dec 8 11:53:31.260 : Cleared schedule slip
Dec 9 03:51:24.219 : Detected schedule slip
Dec 9 03:52:24.317 : Cleared schedule slip
Configuration Synchronization:
Status:
Activation status: Enabled
Last sync operation: Auto-Sync
Last sync result: Succeeded
Last sync mgd messages:
mgd: rcp: /config/juniper.conf: No such file or directory
Non-existant dump device /dev/bo0s1b
mgd: commit complete
Events:
Oct 11 11:15:35.406 : Auto-Sync: In progress. Attempt: 1
Oct 11 11:18:22.218 : Auto-Sync: Clearing mgd. Attempt: 1
Oct 11 11:18:31.062 : Auto-Sync: Succeeded. Attempt: 1
Cold Synchronization Progress:
CS Prereq 0 of 1 SPUs completed
1. if_state sync 1 SPUs completed
2. fabric link 0 SPUs completed
3. policy data sync 1 SPUs completed
4. cp ready 0 SPUs completed
5. VPN data sync 0 SPUs completed
6. Dynamic addr sync 0 SPUs completed
CS RTO sync 0 of 1 SPUs completed
CS Postreq 0 of 1 SPUs completed
Command history:
Dec 5 09:44:15.890 : Manual failover of RG-0 to node0
Dec 5 09:44:28.871 : Manual failover reset of RG-0
Dec 5 09:44:33.187 : Manual failover of RG-0 to node0
Dec 5 09:45:02.494 : Manual failover of RG-0 to node0
Dec 5 09:45:38.155 : Manual failover reset of RG-0
Dec 5 09:45:52.176 : Manual failover of RG-0 to node0
Dec 5 15:28:26.029 : Manual failover reset of RG-4
Dec 5 15:28:39.491 : Manual failover reset of RG-3
node1:
--------------------------------------------------------------------------
Redundancy mode:
Configured mode: active-active
Operational mode: unknown
Cluster configuration:
Heartbeat interval: 1000 ms
Heartbeat threshold: 3
Control link recovery: Enabled
Fabric link down timeout: 66 sec
Node health information:
Local node health: Not healthy
Remote node health: Not healthy
Redundancy group: 0, Threshold: 0, Monitoring failures: config-sync-monitoring
Events:
Dec 10 10:24:57.009 : hold->secondary, reason: Hold timer expired
Redundancy group: 1, Threshold: -511, Monitoring failures: interface-monitoring, cold-sync-monitoring, config-sync-monitoring
Events:
Dec 10 10:24:57.102 : hold->secondary, reason: Hold timer expired
Redundancy group: 2, Threshold: -510, Monitoring failures: interface-monitoring, cold-sync-monitoring, config-sync-monitoring
Events:
Dec 10 10:24:57.585 : hold->secondary, reason: Hold timer expired
Redundancy group: 3, Threshold: -511, Monitoring failures: interface-monitoring, cold-sync-monitoring, config-sync-monitoring
Events:
Dec 10 10:24:57.622 : hold->secondary, reason: Hold timer expired
Redundancy group: 4, Threshold: -510, Monitoring failures: interface-monitoring, cold-sync-monitoring, config-sync-monitoring
Events:
Dec 10 10:24:57.671 : hold->secondary, reason: Hold timer expired
Control link statistics:
Control link 0:
Heartbeat packets sent: 5211
Heartbeat packets received: 4757
Heartbeat packet errors: 0
Duplicate heartbeat packets received: 0
Control recovery packet count: 0
Sequence number of last heartbeat packet sent: 5237
Sequence number of last heartbeat packet received: 5185634
Fabric link statistics:
Child link 0
Probes sent: 0
Probes received: 0
Child link 1
Probes sent: 0
Probes received: 0
Switch fabric link statistics:
Probe state : DOWN
Probes sent: 0
Probes received: 0
Probe recv errors: 0
Probe send errors: 0
Probe recv dropped: 0
Sequence number of last probe sent: 0
Sequence number of last probe received: 0
Chassis cluster LED information:
Current LED color: Amber
Last LED change reason: Monitored objects are down
Control port tagging:
Disabled
Cold Synchronization:
Status:
Cold synchronization completed for: N/A
Cold synchronization failed for: N/A
Cold synchronization not known for: N/A
Current Monitoring Weight: 255
Progress:
CS Prereq 0 of 1 SPUs completed
1. if_state sync 0 SPUs completed
2. fabric link 0 SPUs completed
3. policy data sync 0 SPUs completed
4. cp ready 0 SPUs completed
5. VPN data sync 0 SPUs completed
6. Dynamic addr sync 0 SPUs completed
CS RTO sync 0 of 1 SPUs completed
CS Postreq 0 of 1 SPUs completed
Statistics:
Number of cold synchronization completed: 0
Number of cold synchronization failed: 0
Loopback Information:
PIC Name Loopback Nexthop Mbuf
-------------------------------------------------
Success Success Success
Interface monitoring:
Statistics:
Monitored interface failure count: 0
Fabric monitoring:
Status:
Fabric Monitoring: Enabled
Activation status: Suspended by local node and other node
Fabric Status reported by data plane: Down
JSRPD internal fabric status: Down
Fabric link events:
Dec 10 11:30:58.765 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:31:06.997 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:31:13.229 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:37:02.814 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:37:10.999 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:43:05.846 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:43:14.009 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:49:09.882 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 11:49:18.072 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Dec 10 12:34:13.129 : Fabric monitoring is suspended due to USPIPC CONNECTION failure
Control link status: Up
Server information:
Server status : Inactive
Server connected to None
Client information:
Client status : Connected
Client connected to 129.16.0.1/62845
Control port tagging:
Disabled
Control link events:
Dec 10 11:37:43.889 : Control link up, link status timer
Dec 10 11:43:05.929 : Control link fxp1 is down
Dec 10 11:43:05.929 : Control link down, flowd is down
Dec 10 11:43:14.746 : Control link fxp1 is up
Dec 10 11:43:47.390 : Control link up, link status timer
Dec 10 11:49:09.964 : Control link fxp1 is down
Dec 10 11:49:09.965 : Control link down, flowd is down
Dec 10 11:49:19.221 : Control link fxp1 is up
Dec 10 11:49:50.982 : Control link up, link status timer
Dec 10 12:34:13.119 : Control link fxp1 is up
Hardware monitoring:
Status:
Activation status: Enabled
Redundancy group 0 failover for hardware faults: Enabled
Hardware redundancy group 0 errors: 0
Hardware redundancy group 1 errors: 0
Schedule monitoring:
Status:
Activation status: Disabled
Schedule slip detected: None
Timer ignored: No
Statistics:
Total slip detected count: 16
Longest slip duration: 2578(s)
Events:
Dec 10 11:31:11.706 : Detected schedule slip
Dec 10 11:32:13.095 : Cleared schedule slip
Dec 10 11:37:15.779 : Detected schedule slip
Dec 10 11:38:17.021 : Cleared schedule slip
Dec 10 11:43:17.973 : Detected schedule slip
Dec 10 11:44:19.238 : Cleared schedule slip
Dec 10 11:49:22.655 : Detected schedule slip
Dec 10 11:50:24.368 : Cleared schedule slip
Dec 10 12:34:13.090 : Detected schedule slip
Dec 10 12:35:13.349 : Cleared schedule slip
Configuration Synchronization:
Status:
Activation status: Enabled
Last sync operation: Auto-Sync
Last sync result: In progress
Last sync mgd messages:
mgd: rcp: /config/juniper.conf: No such file or directory
Events:
Dec 10 10:25:23.643 : Auto-Sync: In progress. Attempt: 1
Dec 10 12:34:13.078 : Auto-Sync: Retry needed. Attempt: 1
Dec 10 12:34:18.930 : Auto-Sync: In progress. Attempt: 2
Cold Synchronization Progress:
CS Prereq 0 of 1 SPUs completed
1. if_state sync 0 SPUs completed
2. fabric link 0 SPUs completed
3. policy data sync 0 SPUs completed
4. cp ready 0 SPUs completed
5. VPN data sync 0 SPUs completed
6. Dynamic addr sync 0 SPUs completed
CS RTO sync 0 of 1 SPUs completed
CS Postreq 0 of 1 SPUs completed
{secondary:node1}