Switching

Hello,

Thanks for reading my question.

I have an EVPN setup with 2x MX960 and 2x EX9200. We are having some problems with this setup (see other topic) so we decided to update our MX firmware from 14.2R5.8 to 14.2R7.5 and also upgrade the EX9200's in a later stage.

After the update of the first MX we noticed that EVPN has problems switching/routing traffic to the other devices.

L2 traffic is supported to flow from 34:6a:c2:47:48:70 to 00:30:17:0b:69:a8

The initial mac table of each device with the local mac learned:

user@MX> show bridge mac-table vlan-id 352

Bridging domain : VL352-net52_download, VLAN : 352
addresssss flags interface Index ID
34:6a:c2:47:48:70 D ae70.352

user@EX> show ethernet-switching table vlan-id 352

Ethernet switching table : 1 entries, 1 learned
name address flags interface Index ID
34:6a:c2:47:48:70 DC,SE - pip-14.740010000000 1048692 1048692

At this point i run a ping from the device connected to the MX to the device connected to the EX and MAC table is learned properly:

user@MX> show bridge mac-table vlan-id 352

Bridging domain : VL352-net52_download, VLAN : 352
addresssss flags interface Index ID
34:6a:c2:47:48:70 D ae70.352

00:30:17:0b:69:a8 DC 1048645 1048645

user@EX> show ethernet-switching table vlan-id 352

Ethernet switching table : 1 entries, 1 learned
name address flags interface Index ID
34:6a:c2:47:48:70 DC,SE - pip-14.740010000000 1048692 1048692

00:30:17:0b:69:a8 D,SE - ae11.352

This looks good, but only the first 1 or 2 pings worked. After that its a timeout.

-bash-4.1$ ping6 evs-web-01-352
PING evs-web-01-352(evs-web-01-352) 56 data bytes
64 bytes from evs-web-01-352: icmp_seq=1 ttl=64 time=1.84 ms
64 bytes from evs-web-01-352: icmp_seq=2 ttl=64 time=0.768 ms

timeout

timeout

timeout

Configuration has not changed and is still working fine on the other devices that have not been upgraded.

EVPN trace log does not show any irregular entries compared to the working devices.

Could anyone point me in the right direction?

Related config:

user@MX> show configuration routing-instances HOSTING-EVPN
instance-type virtual-switch;
vrf-target target:101:1009;
protocols {
evpn {
traceoptions {
file evpntrace size 1m;
flag all;
}
extended-vlan-list [ 352 833 ];
default-gateway do-not-advertise;
}
}
bridge-domains {
VL352-net52_download {
domain-type bridge;
vlan-id 352;
interface ae77.352;
interface ae70.352;

user@EX> show configuration routing-instances HOSTING-EVPN
instance-type virtual-switch;
vrf-target target:101:1009;
protocols {
evpn {
traceoptions {
file evpntrace size 1m;
flag all;
}
extended-vlan-list [ 352-353 ];
default-gateway do-not-advertise;
}
}
switch-options {
mac-statistics;
}
vlans {
VL352-net52_download {
vlan-id 352;
interface ae10.352;
interface ae11.352;
domain-type bridge;

Hi !

Before 14.2R7.5 there were a lot of bugs and problems in the EVPN code and also some visible and invisible behaviour changes from version to version.

According to personal experience and experiences from two bigger comapnies in Germany the R7.5 is the first and only version stable and recommended for EVPN

Therefore I assume that you trap into such behaviour change and I think you have to upgrade all MXes and maybe also the EX9200 junos to have the same features ( but I have never used that EX9200 one for EVPN).

But I know that this is somewhat risky in a production environment.

With best regards

Alexander Marhold

JNCIP-DC

Hi Alexander,

Thank you for the information. As you state it might as well be a version incompatibility issue between the devices. 

It might also be a configuration change that i don't know about. But i checked the configuration page (https://www.juniper.net/documentation/en_US/junos/topics/topic-map/evpn-irb-solution-configuring.html) and this is pretty much how i configured it except ISIS instead of OSPF and i don't have "set routing-options forwarding-table chained-composite-next-hop ingress evpn"  But from what i can find this is just a command for helping the evironment scale, not for any function.

Do you have any idea if there might be any other configuration changes?

But then again.. the first few pings work fine so traffic flows. There is just something somewhere that decides that the forwarding should stop after a few packets.

as far as I know the chained composite command is in a hidden group and thus set automatically.

for EVPN over MPLS there are no major command changes and additions

the only nice thing is the posibility to have individual addresses on the irb with a common VIP-gateway address, which means that you can sucessfully ping from the irb to the CE, however currently there is no "accept-data" feature implemented, that means you can ping from the CE the individual irb addresses but not the common gateway address

However I have no information regarding the EX9200,

Else the problem could also lie in an shared all-active ESI between the MXs and different behaviour there ? But thsi is a wild guess, as there is no information about that in your config snippets.

with best regards

Alexander

Thanks again for the input.

EX9200 comes with some limitations yes. It can only be used as a MPLS edge switch (MES) For example: LSP's are not supported.

I do not use any multi-homing. Its 4 datacenters and one MX or EX in each datacenter as the core network device. Datacenters are fully standalone. Therefor i do not use any ESI configuration in any way. 

I have tried to use it in all possible scenarios (single active, both active) but it does not help. In all scenario's the port is in Up/Forwarding mode (show evpn instance HOSTING-EVPN extensive) but it still does not work.

The EVPN works between the two MX'es where one is running the old firmware and the other the new. But it does not work in a MX+EX combination with old+new firmware.

Tonight we will upgrade the other MX so that both MX'es run the same version. Luckely we can do this as the EVPN part is not yet in production on the MX'es.

In the coming week i will try to find out what the best Junos version is for the EX9200 with the best EVPN support and then upgrade those. Fingers crossed!

If anyone has any information about the best software for EX9200 running EVPN, please let me know.

I left a ping running al day. It seems to allow traffic sometimes spontaniously. 

Look at the sequence nr numbers.

-bash-4.1$ ping6 evs-web-01-352
PING evs-web-01-352(evs-web-01-352) 56 data bytes
64 bytes from evs-web-01-352: icmp_seq=1 ttl=64 time=2.21 ms
64 bytes from evs-web-01-352: icmp_seq=2 ttl=64 time=0.854 ms
64 bytes from evs-web-01-352: icmp_seq=2295 ttl=64 time=2.22 ms
64 bytes from evs-web-01-352: icmp_seq=10717 ttl=64 time=1.91 ms
64 bytes from evs-web-01-352 icmp_seq=12105 ttl=64 time=1.93 ms
64 bytes from evs-web-01-352: icmp_seq=13319 ttl=64 time=0.794 ms

I have a feeling it works for a second during the MAC learning, but when learning it done, it stops forwarding.

More verification that learning works fine:

user@MX> show route forwarding-table family bridge

Routing table: HOSTING-EVPN.evpn
Bridging domain: VL352-net52_download.evpn
EVPN:
Destination Type RtRef Next hop Type Index NhRef Netif
00:30:17:0b:69:a8/48 user 0 chain 15695876 19
34:6a:c2:47:48:70/48 user 0 ucst 2207 5 ae70.352

user@EX> show route forwarding-table family bridge

Routing table: HOSTING-EVPN.evpn
Bridging domain: VL352-net52_download.evpn
EVPN:
Enabled protocols: , MAC Stats
Destination Type RtRef Next hop Type Index NhRef Netif
00:30:17:0b:69:a8/48 user 0 ucst 1882 5 ae11.352
34:6a:c2:47:48:70/48 user 0 chain 15695876 3

Hi all,

Thanks for thinking with me on this case.

We have solved the problem by deactivating & activating the mBGP sessions between the devices. After the sessions reestablished the EVPN functionality was working again.

We did not find anything during diagnosing. Resetting the BGP session was just guessing and trying everything. 

We think we hit a bug during the updating and switching the active/inactive RE.

We are using 14.2R7.5 on MX960 with 32x10G and 2x100/8x10G linecards.

Allthough we have not tried to reproduce the issue (yet), i reccomend everything running EVPN features to upgrades thier devices by fully rebooting both RE's and not use features like Routing Engine Redundancy, Gracefull Routing Engine failover or similar functions. Better to go through the dark for a few minutes.