Contrail Platform Developers
Contrail Platform Developers

VXLAN between TSN and TOR QFX 5100 doesn't work

‎10-13-2017 06:30 AM

Hello Community,

 

I'm trying to connect BMs to contrail network as described here [0]. I'm using the following versions:

OpenContrail - 3.1.1
QFX 5100 - QFX5100-48S-6Q (JunOS 14.1X53-D45.3)
 
TNS node (172.16.175.61) is connected directly to TOR switch (interface:ge-0/0/21, ip address of lo0.0 192.168.10.1) 
I have 1 VM (ip: 10.20.30.68 mac:02:b0:aa:8d:0b:9d) running on compute node (compute node ip: 172.16.175.103). And 1 BM (mac: ac:de:48:6b:e4:e4 ip: 10.20.30.11 connected to TOR xe0/0/1 port).
VTEPs are being created fine on ToR switch to both compute nodes and TSN see attach. Hovewer sometimes ARP resolution is failed on BM server, when I add static ARP entries on VM and BM ping works. 
On TSN node I see both arp requests from BM and replies to BM, but for unknown reason they never reach BM.
 
There is tcpdump output:
 
08:42:27.447492 64:64:9b:45:19:00 > 0c:c4:7a:6c:89:6c, ethertype IPv4 (0x0800), length 106: 192.168.10.1.45105 > 172.16.175.61.4789: VXLAN, flags [I] (0x08), vni 8 
ac:de:48:6b:e4:e4 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 56: Request who-has 10.20.30.68 tell 10.20.30.11, length 42 
08:42:27.447549 0c:c4:7a:6c:89:6c > 64:64:9b:45:19:00, ethertype IPv4 (0x0800), length 106: 172.16.175.61.49983 > 192.168.10.1.4789: VXLAN, flags [I] (0x08), vni 8 
02:b0:aa:8d:0b:9d > ac:de:48:6b:e4:e4, ethertype ARP (0x0806), length 56: Reply 10.20.30.68 is-at 02:b0:aa:8d:0b:9d, length 42
 

[0] https://www.juniper.net/documentation/en_US/contrail2.2/topics/concept/using-tor-ovsdb-contrail.html

Attachments

3 REPLIES 3
Contrail Platform Developers

Re: VXLAN between TSN and TOR QFX 5100 doesn't work

‎02-10-2018 09:52 PM

I am also having a similar kind of issue. Were you able to get a solution for it ?

Contrail Platform Developers

Re: VXLAN between TSN and TOR QFX 5100 doesn't work

‎02-15-2018 08:27 PM

From your packet capture, it is evident that TSN (acting as ARP responder in this case) is sending the ARP Reply by providing the MAC address of VM to the BMS. The DIP of the outer packet indicates that the ARP reply is being re-directed to lo0 of TOR. 

 

You may have to check why QFX is not forwarding that to the bare metal server. This is not a Contrail side issue. 

 

I see that the same issue is reported for QFX 14.1X53-D45 and is being worked upon by our engineering folks. A quick fix would be to try out a different QFX code that doesn't have this behavior. I am running 14.1X53-D40.8 in my lab and I don't see this problem there.

 

Hope this helps.

 

Greetings,

Sandeep.

Contrail Platform Developers

Re: VXLAN between TSN and TOR QFX 5100 doesn't work

[ Edited ]
‎02-15-2018 10:11 PM

I am working on opencontrail and have implemented the below mentioned scenario.

I have connected my VM that is running on ESXI host vswitch1 to qfx5100 switch and want to ping the VM created by openstack from the ESXI Host VM. I have also configured the opencontrail physical device interface configuration with the physical interface that is connected to the ESXI Host VM, the dhcp request is not able to provide ip address to my VM at the ESXI host. The port of ESXI Host is trunk and the QFX5100 port is in flexible vlan tagging with encapsulation vlan-bridge.

The below mentioned commands on the TOR Compute Node (which is connected to vswitch0 of the same ESXI host) shows that the request somehow is being dropped.

 

The new stats are mentioned below.

I am not able to get the ip for Bare Metal Server through DHCP request nor I am able to ping the Bare metal server and the VM present within the openstack.

------------------------------------------------------------------------------------------------------------
{master:0}
mirai.neeku@sw2> show ovsdb controller
VTEP controller information:
Controller IP address: 10.21.222.126
Controller protocol: ssl
Controller port: 9999
Controller connection: up
Controller seconds-since-connect: 9321
Controller seconds-since-disconnect: 0
Controller connection status: active

 

{master:0}
mirai.neeku@sw2> show ovsdb mac
Logical Switch Name: Contrail-74672427-fdf0-4d28-ad5e-25808b7bf01c
  Mac IP Encapsulation Vtep
  Address Address Address
  ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.21.222.115
  00:0c:29:83:f2:3f 0.0.0.0 Vxlan over Ipv4 10.21.222.115
  02:43:97:3e:80:7b 0.0.0.0 Vxlan over Ipv4 10.21.222.124
  02:a6:86:14:52:2e 0.0.0.0 Vxlan over Ipv4 10.21.222.125
  ff:ff:ff:ff:ff:ff 0.0.0.0 Vxlan over Ipv4 10.21.222.126

Ethernet switching table : 3 entries, 1 learned
Routing instance : default-switch
    Vlan MAC MAC Age Logical
    name address flags interface
    Contrail-74672427-fdf0-4d28-ad5e-25808b7bf01c 00:0c:29:83:f2:3f D - ge-0/0/10.11
    Contrail-74672427-fdf0-4d28-ad5e-25808b7bf01c 02:43:97:3e:80:7b SO - vtep.32769
    Contrail-74672427-fdf0-4d28-ad5e-25808b7bf01c 02:a6:86:14:52:2e SO - vtep.32770

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)

 

mirai.neeku@sw2> show ethernet-switching vxlan-tunnel-end-point remote mac-table

MAC flags (S -static MAC, D -dynamic MAC, L -locally learned, C -Control MAC
           SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

Logical system : <default>
Routing instance : default-switch
 Bridging domain : Contrail-74672427-fdf0-4d28-ad5e-25808b7bf01c, VLAN : NA, VNID : 5
   MAC MAC Logical Remote VTEP
   address flags interface IP address
   02:43:97:3e:80:7b SO vtep.32769 10.21.222.124
   02:a6:86:14:52:2e SO vtep.32770 10.21.222.125

 

{master:0}
mirai.neeku@sw2> show ethernet-switching vxlan-tunnel-end-point source
Logical System Name Id SVTEP-IP IFL L3-Idx
<default> 0 10.21.222.115 lo0.0 0
    L2-RTT Bridge Domain VNID MC-Group-IP
    default-switch Contrail-74672427-fdf0-4d28-ad5e-25808b7bf01c 5 0.0.0.0

 

{master:0}
mirai.neeku@sw2> show ovsdb logical-switch
Logical switch information:
Logical Switch Name: Contrail-74672427-fdf0-4d28-ad5e-25808b7bf01c
Flags: Created by both
VNI: 5
Num of Remote MAC: 3
Num of Local MAC: 2

 

{master:0}
mirai.neeku@sw2> show ovsdb virtual-tunnel-end-point
Encapsulation Ip Address Num of MAC's
VXLAN over IPv4 10.21.222.115 2
VXLAN over IPv4 10.21.222.124 1
VXLAN over IPv4 10.21.222.125 1
VXLAN over IPv4 10.21.222.126 1

------------------------------------------------------------------------------------------------------------
root@compute-3:/etc/contrail# rt --dump 1 --family bridge
Kernel L2 Bridge table 0/1

Flags: L=Label Valid, Df=DHCP flood

Index DestMac Flags Label/VNID Nexthop
24280 ec:3e:f7:1:59:d - 1
42268 0:c:29:d1:a:29 Df - 3
94920 0:c:29:83:f2:3f LDf 5 21
97192 ff:ff:ff:ff:ff:ff LDf 5 27
176248 2:a6:86:14:52:2e LDf 18 24
234012 2:43:97:3e:80:7b LDf 18 13
252916 0:0:5e:0:1:0 Df - 3

 

root@compute-3:/etc/contrail# nh --get 27
Id:27 Type:Composite Fmly:AF_BRIDGE Rid:0 Ref_cnt:4 Vrf:1
              Flags:Valid, Multicast, L2,
              Sub NH(label): 23(0) 26(0) 19(0)

root@compute-3:/etc/contrail# nh --get 23
Id:23 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1
              Flags:Valid, Tor,
              Sub NH(label): 21(5)

root@compute-3:/etc/contrail# nh --get 21
Id:21 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:3 Vrf:0
              Flags:Vxlan,
              Oif:0 Len:0 Flags Vxlan, Data:
              Vrf:0 Sip:0.0.0.0 Dip:0.0.0.0

root@compute-3:/etc/contrail# nh --get 26
Id:26 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1
              Flags:Valid, Evpn,
              Sub NH(label):

root@compute-3:/etc/contrail# nh --get 19
Id:19 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1
              Flags:Valid, Fabric,
              Sub NH(label): 13(4610)

root@compute-3:/etc/contrail# nh --get 13
Id:13 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:4 Vrf:0
              Flags:Valid, MPLSoUDP,
              Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:00 0c 29 fb c2 55 00 0c 29 d1 0a 29 08 00
              Vrf:0 Sip:10.21.222.126 Dip:10.21.222.124

root@compute-3:/etc/contrail# nh --get 24
Id:24 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:3 Vrf:0
              Flags:Valid, MPLSoUDP,
              Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:00 0c 29 f6 d6 30 00 0c 29 d1 0a 29 08 00
              Vrf:0 Sip:10.21.222.126 Dip:10.21.222.125

root@compute-3:/etc/contrail#

 

maheen.iqbal@sw2> show interfaces vtep.32771
Logical interface vtep.32771 (Index 553) (SNMP ifIndex 648)
Flags: Up SNMP-Traps Encapsulation: ENET2
VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 10.21.222.126, L2 Routing Instance: default-switch, L3 Routing Instance: default
Input packets : 0
Output packets: 0
Protocol eth-switch, MTU: 1600
Flags: Is-Primary, Trunk-Mode

{master:0}
maheen.iqbal@sw2> show interfaces vtep.32770
Logical interface vtep.32770 (Index 552) (SNMP ifIndex 647)
Flags: Up SNMP-Traps Encapsulation: ENET2
VXLAN Endpoint Type: Remote, VXLAN Endpoint Address: 10.21.222.124, L2 Routing Instance: default-switch, L3 Routing Instance: default
Input packets : 0
Output packets: 0
Protocol eth-switch, MTU: 1600
Flags: Trunk-Mode

 

Also not receiving any traffic at the Vtep interfaces and not able to ping the source vtep ip address from the TSN Node.

 

-Mirai

Attachments