Ethernet Switching
Ethernet Switching

EX9200 MC-LAG Failover Recovery Times

‎05-31-2017 11:25 PM

Hello All,

 

I would appreciate any answers to gain some sort of consensus for my current situation.  I currently have two EX9204 chassis configured in an MC-LAG along with VRRP and when simulating a failover situation by rebooting any one of the members, I experience a network disruption somewhere in the realm of 20 seconds before connectivity resumes internally and externally.  What I mean by that is multi homed nodes (switches and servers) connected directly to the MC-LAG and external nodes upstream from the EX9204 switches.

 

I’ve been working with support for the past month and we have not been able to reduce this recovery time.  What I’m trying to find out is if everyone else believes this is expected behavior from this platform?  Our company went with this platform and are anxious to put this in production to replace antiquated equipment from another vendor and although 20 seconds is not a lot of time, but this failover and recovery period is somewhat unacceptable in my opinion if it were to ever happen during business hours.

 

Please, anyone, inform me of your experiences, guidance, and/or opinions.  Thanks!

11 REPLIES 11
Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎06-22-2017 12:02 AM

Hello,

 

Could you please use the below configuration and update the status?

 

set protocols iccp local-ip-addr 1.1.1.2
set protocols iccp peer 1.1.1.1 session-establishment-hold-time 50
set protocols iccp peer 1.1.1.1 backup-liveness-detection backup-peer-ip 30.30.30.2
set protocols iccp peer 1.1.1.1 liveness-detection minimum-interval 300
set protocols iccp peer 1.1.1.1 liveness-detection transmit-interval minimum-interval 300

 

set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 aggregated-ether-options lacp system-id 00:00:00:00:00:01
set interfaces ae0 aggregated-ether-options lacp admin-key 1
set interfaces ae0 aggregated-ether-options mc-ae mc-ae-id 1
set interfaces ae0 aggregated-ether-options mc-ae chassis-id 1
set interfaces ae0 aggregated-ether-options mc-ae mode active-active
set interfaces ae0 aggregated-ether-options mc-ae status-control standby
set interfaces ae0 aggregated-ether-options mc-ae init-delay-time 30
set interfaces ae0 aggregated-ether-options mc-ae events iccp-peer-down

 

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎06-27-2017 02:30 PM

I have no experience with that technology, however do you have the latest version?

Starting with Junos OS Release 14.2R3 on MX Series routers, enhanced convergence improves Layer 2 and Layer 3 convergence time when a multichassis aggregated Ethernet (MC-AE) link goes down or comes up in a bridge domain or VLAN. Convergence time is improved because the traffic on the MC-AE interface is switched to the interchassis link (ICL) without waiting for a MAC address update.

If you have configured an IRB interface over an MC-AE interface that has enhanced convergences enabled, then you must configure enhanced convergence on the IRB interface as well. Enhanced convergence must be enabled for both Layer 2 and Layer 3 interfaces.

[KUDOS PLEASE! If you think I earned it!
If this solution worked for you please flag my post as an "Accepted Solution" so others can benefit..]
Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎06-29-2017 01:01 PM

Hello @srinireddy,

 

I will try to make those changes but I have a few questions:

 

For the iccp settings, you don't specifiy redundancy-group-id-list settting.  Is this not needed? Here is what I currently have on the active member:

 

set protocols iccp local-ip-addr 10.20.127.1
set protocols iccp peer 10.20.127.2 session-establishment-hold-time 50
set protocols iccp peer 10.20.127.2 redundancy-group-id-list 1
set protocols iccp peer 10.20.127.2 backup-liveness-detection backup-peer-ip 10.20.18.5
set protocols iccp peer 10.20.127.2 liveness-detection minimum-interval 2000
set protocols iccp peer 10.20.127.2 liveness-detection multiplier 4

 

Also, with ae0, you are referring to a typical mc-ae and not the ICCP link correct?  Further for mc-ae events iccp-peer-down, should  prefer-status-control-active be set or not set?  Thanks for your help,

 

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎06-29-2017 01:06 PM

Hi, I'm currently running 17.2.   Can you elaborate on how enhanced convergence is obtained on L2/L3 interfaces?

 

As an update to the situation, this is the progress so far:

 

1) For software upgrade, reboots, graceful shutdowns for maintenance scenarios, there the downing of all ae’s on the desired member works with little to no loss of traffic.  Although it’s not a procedure I expect for this class of networking equipment, but at least there is a method I suppose.

 

2) For complete fail, hangs, or power failure on a member of the core pair:

 

If the standby member suffers from a complete unexpected sudden outage, there will be no to little (sub-second) loss of traffic.

 

However if the active member suffers from a similar failure, there will be about a 10 second outage to layer 3 traffic only.  It’s not a lot but it’s still not acceptable for this level of network switching. I believe this is happening due to the fact that I've noticed the LACP downstream links have to reconverge where once completed, traffic then begins to resume.

 

Is this expected behavior with MC-LAG on EX platforms in a power-down failure scenario?  Thanks,

 

 

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎11-01-2018 01:58 PM

you just need to add prefer-status-control active konb on all mc-ae interfaces on both MC_LAG peers.

 

set interfaces ae0 aggregated-ether-options mc-ae events iccp-peer-down prefer-status-control-active

 

it will make sure that LACP system ID will be retained to configured vlaues once MC_LAG peer is getting reboot or shutdown.   You will get maximum 1 or 2 ping drops once knob is applied.

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎12-13-2018 12:14 PM

In version 16 with a similar configuration, when you have mc-ae events iccp-peer-down prefer-status-control-active configured on both peers, you will get a warning like this one for every MC-AE when you try to commit the configuration:

 

[edit interfaces ae83 aggregated-ether-options]

'mc-ae'

    warning: prefer-status-control-active is used with status-control standby. Use this command only if BLD is configured

 

However, when we removed the command, we experience the same situation described here - 60 seconds of outage while the downstream portchannels go down and back up due to LACP issues.  And another 60 seconds when the primary chassis comes back online.  No issues when the standby chassis goes down/up.

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎12-13-2018 04:15 PM

Please add hold timers greater than your BFD interval to the physical interfaces that make up your ISL and test failover again.

 

For example with BFD = 3x1000ms

 

set interfaces et-0/0/52 hold-time up 100
set interfaces et-0/0/52 hold-time down 4000
set interfaces et-0/0/52 ether-options 802.3ad ae0
set interfaces et-0/0/53 hold-time up 100
set interfaces et-0/0/53 hold-time down 4000
set interfaces et-0/0/53 ether-options 802.3ad ae0

 

 

 

 

 

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎12-19-2018 11:46 AM

After implementing BLD and prefer-status-control-active on both MC-LAG peers, as well as the hold timers on the ISL link suggested above, I am still experiencing the same situation.  If we manually shutdown all the interfaces on either chassis, or reboot the standby chassis, we see no traffic interruption.  However, issuing a "request system reboot" command on the primary node causes the VRRP gateways to become unreachable immediately, and endpoints on downstream switches become unreachable after 10 seconds.  Everything comes back up at the same time, 50 seconds after issuing the reboot command.  I'm guessing the same would be true in a hard failure situation, although I haven't tried physically powering down the node without the reboot command.

 

I'm still a little confused over which link to adjust the hold-down timers on (ISL vs ICCP).  In our environment we have an ICCP link configured, but we're peering with an IP address on an IRB.  Is the link described as "ICCP" actually doing any MC-LAG related for us, since we're peering with an IP associated with an IRB?  See attached config for more details:

 

admin@TEST-GSJA-re0> show configuration interfaces ae0
apply-groups-except global-AE-PARAM;
description "[TEST-GSJA-to-TEST-GSJB ICCP Inter-chassis communications link ae0 ]";
aggregated-ether-options {
    lacp {
        active;
    }
}
unit 0 {
    family inet {
        address 10.144.200.1/30;
    }
}

{master}
admin@TEST-GSJA-re0> show configuration interfaces ae10
apply-groups-except global-AE-PARAM;
description "[TEST-GSJA-to-TEST-GSJB ICL Inter-chassis communications link ae10 ]";
mtu 1518;
aggregated-ether-options {
    lacp {
        active;
    }
}
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members all;
        }
    }
}

admin@TEST-GSJA-re0> show configuration interfaces irb.10
family inet {
    address 10.144.100.1/30 {
        arp 10.144.100.2 l2-interface ae10.0 mac ec:13:db:11:8f:f0;
    }
}

admin@TEST-GSJA-re0> show configuration protocols iccp
local-ip-addr 10.144.100.1;
peer 10.144.100.2 {
    session-establishment-hold-time 50;
    redundancy-group-id-list 10;
    backup-liveness-detection {
        backup-peer-ip 10.121.121.11;
    }
    liveness-detection {
        minimum-receive-interval 500;
        transmit-interval {
            minimum-interval 500;
        }
    }
}

admin@TEST-GSJA-re0> show configuration groups global-AE-PARAM
interfaces {
    "<ae[1-9]*>" {
        aggregated-ether-options {
            lacp {
                active;
                system-id 00:01:02:03:04:05;
                admin-key 5;
            }
            mc-ae {
                redundancy-group 10;
                chassis-id 0;
                mode active-active;
                status-control active;
                events {
                    iccp-peer-down {
                        prefer-status-control-active;
                    }
                }
            }
        }
    }
}

 

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎12-19-2018 01:31 PM

 

Does it work if you remove lacp from ae10? I’ve not seen lacp on the ISL in any of the juniper recommended configs for mclag—perhaps it interacts in some way with mcae?

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎08-29-2019 10:27 AM

AE10 is ICL.

 

From 9214 MC-Lag document:

https://www.juniper.net/documentation/en_US/release-independent/nce/information-products/pathway-pag...

 

set interfaces ae1 description ICL-LINK

set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 aggregated-ether-options lacp periodic fast
set interfaces ae1 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae1 unit 0 family ethernet-switching vlan members all

Ethernet Switching

Re: EX9200 MC-LAG Failover Recovery Times

‎08-29-2019 05:09 PM

@jhosee - I'll give you my 2 cents worth.  For MC-LAG type technology, which is now close to 15 years old, failback/recovery ALWAYS takes longer, and in general is never sub-second.  For these systems, it is easy to note and stop forwarding for a link or switch down event.

 

For a recovery event, lots of things are going on for which all must be noted and learned by the system to start forwarding traffic properly.  L2 only should be faster than L3, but system must learn link is back up and good (LACP), add link back into AE hash configuration, and restart sending while expecting (hoping) remote end has also adjusted properly.

 

In general I have found recovery times of 3-5 seconds are "normal" but depending upon size/scale of network, and exactly what is being measured some flows are very likely to recover faster than others.  Scale is a BIG determining factor, as the more items the systems needs to learn, the longer it is bound to take.  Not sure if you have tested say 1 L2 MAC on a link recovery and seen what the recovery time is?

 

I am not sure of what your exact MC-LAG configuration you are useing, as Juniper documentation for MC-LAG although greatly improved, has never been 100% consistent across products.

 

As someone else pointed out there have also been some SW changes that depreciated some commands, and made other commands (prefer-status-control?) require additional associated knob settings.

 

In general I believe most common recovery situation is for a SW upgrade, where whole switch must be re-boot and brought back on line.  For this situation, I recommend:

 

#1 - complete isolation of the switch to be re-booted to start with; deactivate all interfaces.

#2 - then active just interfaces associated with ICL/ICCP, which it my network designs are always a single common AE

#3 - let this sit for a while, and make sure MACs/ARPs are being synced between 2 nodes

#4 - then bring up links, either one at a time or all at once, and at this time measure recovery.  This recovery should be way less than 10-20 seconds, from my experience.

 

Just FYI, I have been working with MC-LAG type technology since 2005/6 and I am quite aware of what is required SW wise to get recovery to happen quicker.

 

If you really want faster recovery, I would suggest looking at EVPN/ESI based network designs, vs MC-LAG.  At this point in networking, MC-LAG technologies should be considered as really being out-dated, IMHO.

 

Again, my 2 cents worth.

 

HTH and good luck.