EVPN and VXLAN Episode 3: Symmetric IRB
In the previous episode we saw how two hosts, on different VLANs (VXLANs), can talk each other if they are on the same VRF. In details, we talked about the Asymmetric Routing model (because different VNIs are used for the two traffic directions: routing is performed on the ingress leaf, and switching on the egress one).
However, another method exists for achieving routing between two VXLAN segments, which is… guess what? Symmetric Routing (or Symmetric IRB, where IRB = Integrated Routing and Bridging).
With the Symmetric Routing model, every VRF on our fabric will be associated to an additional VNI (“Layer 3 VNI” - which can be seen more or less like the Service Label on MPLS) which will be used to send the routed packets over the fabric (and route over VTEPs).
A single VNI for the packets going back and forth is not the only benefit of the Symmetric Routing model: with the Asymmetric model, you should create/propagate the different VXLAN “segments” on every device of the fabric where you want to route your packets, even if there are no hosts for that VXLAN connected on the device. With the Symmetric Routing model, this is not needed anymore: only the Layer 3 VNI is propagated on the different fabric devices.
Let’s see how this works with our previous topology:
Imagine you have to ping 192.168.20.2 from 192.168.0.1.
The ICMP Echo starting from 192.168.0.1 arrives at the Leaf 1 (AR-SW-1). Leaf 1 does a route table lookup, and sees that 192.168.20.0/24 is reachable using the Symmetric Routing model, using the L3 VNI (in example) 6000.
Leaf 1 encapsulates the packet on VXLAN VNI 6000.
But… VXLAN requires an Ethernet header to be present. For that, Leaf 1 forges the Ethernet header using the Leaf 2 “Router MAC Address” as destination MAC, and its “Router MAC Address” as source MAC.
Now the Leaf 2 receives the packet, it looks at the VNI and the destination MAC address, and understands that it has to route it. It does a routing table lookup, and defines how to send out the packet.
On the way back, Leaf 2 does exactly the same operation (inverting the two VXLAN MAC addresses, of course).
The different “Router MAC Address” are signaled using BGP extended communities: we can see that our Type-2 announcements are a bit different:
(The first VNI carried on the message is the Layer 2 VNI, and the second one is the Layer 3 VNI)
Let’s verify this directly in the Lab: we can use a topology similar to the previous one, but entirely based on two Arista Leafs. We should start with porting and adapting the configuration we did on the previous episode to include the L3 VNI info. At the same time, we can start using OSPF to announce our loopback addresses (and improve a bit the potential scalability).
The following configuration example is referred to Leaf-1:
service routing protocols model multi-agent
vrf instance TEST
switchport access vlan 10
ip address 10.255.255.1/30
ip ospf area 0.0.0.0
ip address 10.0.0.1/32
ip ospf area 0.0.0.0
ip address 192.168.0.201/24
ip virtual-router address 192.168.0.254
vxlan source-interface Loopback0
vxlan udp-port 4789
vxlan vlan 10 vni 10
vxlan vrf TEST vni 6000
vxlan learn-restrict any
ip virtual-router mac-address 00:00:aa:bb:02:cc
ip routing vrf TEST
router ospf 1
router bgp 65000
no bgp default ipv4-unicast
distance bgp 20 200 200
neighbor 10.0.0.2 remote-as 65000
neighbor 10.0.0.2 next-hop-self
neighbor 10.0.0.2 update-source Loopback0
neighbor 10.0.0.2 bfd
neighbor 10.0.0.2 send-community extended
route-target both 10:10
neighbor 10.0.0.2 activate
route-target import evpn 65000:6000
route-target export evpn 65000:6000
We can verify that the information about VNI 6000 is propagated:
And host /32 routes are created on the VRF routing table to allow Inter-VXLAN routing:
We can try a ping between two hosts on two different Layer 2 domains, connected to two different switches:
As you can see, the VXLAN VNI is set to 6000.
To recap: When using Symmetric Routing, the ingress VTEP does not require L3 presence on the destination subnet, and the egress VTEP does not require L3 presence on the source subnet.
In one of their documents, Arista also says:
Note: Symmetric IRB is recommended, as it does not require that all VLANs and VRFs exist everywhere to guarantee host reachability. Additionally, with Symmetric IRB, Remote ARP entries are maintained in software instead of consuming hardware resources. Thus making it a more scalable solution.