Cisco ACI GARP based Learning

I recently ran across an interesting issue in my ACI fabric and there was not that much information available about it yet, so I thought I would share it.

The issue was related to a load balancer  configured in HA mode as a concrete device on the ACI fabric.  When the load balancer failed over traffic to the passive node traffic going to the VIP would start failing and it would take a couple minutes for the VIP to become responsive again.

In troubleshooting the issue we looked at a packet capture and saw that immediately after the failover the load balancer would send out a gratuitous ARP as expected, however on the leaf switch it showed no GARP’s being received.

The default behavior of an ACI fabric is to do all learning via UDP unicast lookups in the endpoint database located in the spines and as such there is no need to broadcast or flood an ARP.  However in order to get things like HA on load balancers and firewalls or like OS level clustering like Microsoft Windows Failover Clustering or Linux Heartbeat we need to be able to learn based on GARP.  A GARP (Gratuitous ARP) is used by devices on the network as a way to proactively update the ARP cache to let other devices know that the location of a MAC address has changed (advanced notification).  In order for the fabric to be able to learn endpoint moves via GARP, we need to enable some non-default features on the Bridge Domain (BD) associated with the End Point Group (EPG).  Those are “ARP Flooding” and EP Move Detection Mode” (GARP Detection Mode).  Below is a screenshot of the settings I am referring to:

The first screenshot of enabling ARP Flooding is from “Tenant>Networking>Bridge Domains>YOUR-BD”

Bridge Domain EPG Fabric

The second screenshot of enabling GARP based detection is also from “Tenant>Networking>Bridge Domains>YOUR-BD”, but you then need to goto the L3 Configurations tab on the BD.

Cisco ACI and GARP based Learning Bridge Domain EPG Fabric

These screenshots are from an APIC running on the 1.2 codebase.


Advertise Management IP via LLDP

Coming from a Cisco background this was a little bit of a change.  I am now using Dell Force10 switches and I have been trying to figure out how to get LLDP to advertise the management IP to its neighbors like CDP does.  CDP does this by default and there is non-default configuration that must be done in order to get LLDP to do it.  With LLDP you must add LLDP configuration to each neighbor facing interface.  Below is an example of what that configuration would look like:

In configuration mode:




Network DRS: What?

We all know that VMware NSX brings L2–L4 network services up into the logical space, things like Layer 2 switching, distributed layer 3 routing, and distributed firewalling can all now be processed within the hypervisor.

Yeah so? That is so 2013.

What is 2014 going to bring?

What else did NSX enable that is not as broadly spoken about? Network visibility for the hypervisor.

Once you have lifted your network up into the logical space (software) you then have end to end visibility into the logical traffic flows.  When DRS is given insight into these flows it can then be able to use that information (in addition to traditional compute level information) to more intelligently place VM’s in the environment.  Enter Network DRS.  A very simplistic scenario of this would be if you have 2 VM’s that are on separate hosts and those VM’s are generating a significant amount of traffic between each other.  Since DRS would now have insight into the logical network traffic flows it would detect this and migrate one of the VM’s onto the same host as the other.  Now, instead of having the network traffic go out onto the physical wire, it would be moving between the two VM’s at bus speed.

This is just one very brief example of the exciting things that NSX enables.  VMware very briefly mentioned this as a technology preview at VMworld 2013.  I, for one, am very excited to see what progress has been made to bring this to a generally available reality.