Understanding basic OTV configuration

What is OTV and what problem does it solve ?
OTV stands for Overlay Transport Protocol and is a part of Cisco’s Nexus data center networking technology to allow vlans to extend across multiple data centers.
Virtual Machine (VM) administrators often move VMs between hosts to perform maintenance, host upgrades or as part of DEV Ops testing.

On VMware this process is called VMotion and on Microsoft’s Hyper-V it is called Live Migration.
Moving VMs to another host requires the target host to be part of the same VLAN as the source host.

Prior OTV if these hosts reside in different data centers, then a special Layer 2 (L2) connection is required between data centers.

OTV removes this restriction by extending the same VLAN over any IP network between data centers.
This is achieved by encapsulating the frames of the extended VLANs in IP.

This article explains the concepts behind a basic OTV setup.

OTV encapsulation
OTV encapsulates MAC-in-IP and adds a 42 byte header to a normal IP packet.

otvencap

When you run OTV over your data center IP interconnection links, the network must be capable of supporting a minimum MTU of 1542 bytes.

Therefore you should run a ping test with the DF bit set to confirm your network supports the required packet size.

Understanding OTV interfaces
At each site that participates in OTV, you must set up an OTV Edge Device that connects to the IP network.
The Edge Device is responsible for handling the OTV encapsulation, discovery of other OTV Edge devices residing and maintaining the OTV logical domain.

An Edge device has a number of interfaces which are relevant to setting up OTV.

otvinterfacedesc2

The interface which the OTV edge device connects to the IP network is called the ‘join interface’.
This is a standard IP enabled interface which can be any physical, logical (subinterface) or L2 (or L3) port channel interface.
And you should add the mtu command to the join interface as well as an appropriate description for future reference.

— on edge device at site 1
interface e1/41
description “join-interface” for Overlay1
no switchport
ip address 10.1.1.1/24
mtu 9216

— on edge device at site 2
interface e1/41
description “join-interface” for Overlay1
no switchport
ip address 10.1.1.2/24
mtu 9216

— on edge device at site 3
interface e1/41
description “join-interface” for Overlay1
no switchport
ip address 10.1.1.3/24
mtu 9216

All Edge devices connect logically to form the OTV overlay network. Each Edge device has a logical OTV interface.

On the Edge device you will configure and declare this logical interface as an “overlay” interface.

After creating the overlay interface you should bind the join interface to the overlay interface as follows:

— on edge devices for each site
interface overlay 1
description otv interface
otv join-interface e1/41

Each overlay interface must have a join interface associated with it.

Since the purpose of the Edge device is to extend VLANs it must have L2 interfaces (access or trunks ports) that receive traffic from the VLANs to be extended.
Otherwise the overlay interface would not know which IP interface to use to encapsulate the layer 2 frames.
These interfaces are known as internal interfaces to distinguish them from the join or OTV interface.

There is no special configuration to be done on internal interfaces except to ensure these vlans are seen on your edge device.

To define the vlans you wish to extend across the OTV network you configure these vlans under the interface overlay configuration mode. For example,

— on edge device at site 1
otv-extend vlans 10-12, 15

— on edge device at site 2
otv-extend vlans 11, 12

— on edge device at site 3
otv-extend vlans 10, 15

Now that we have covered the basic nomenclature for describing the OTV configuration we can move on to describing the 2 flavors of OTV implementation.

OTV control plane implementation using either multicast or unicast mode
OTV requires a mechanism to discover all edge devices on the OTV network. This neighbor discovery defines the boundary for the OTV domain and it happens within the control plane.

When OTV was first introduced it was implemented using multicast.

OTV edge devices discover each other by treating the OTV network as a multicast network which relies on running IGMP and multicast aware routing protocols such as PIM across the IP network.

From a practical implementation point this means the customer must go to the Telco and Service Provider to ensure that multicast traffic is supported over their IP network.

This usually increase costs as multicast support is offered as an added value service. Coordination is required with the IP network supplier to implement multicast in the core.

Nexus devices from NX-OS version 5.2 onwards and ASRs running IOS-XE version 3.9 onwards has OTV unicast mode. Unicast mode removes the necessity for multicast capability (and configuration) in the IP network.

Let us look at that next.

OTV unicast mode
Most customers will likely prefer OTV unicast because it is does not require knowledge of multicast configuration and is much simpler to set up and to understand.
I will describe it first because this will be the preferred setup for most 2 – 3 site OTV networks.

In unicast OTV, edge devices discover each other by designating at least one edge OTV device as an OTV adjacency server.
Each Edge device in the OTV is pointed (manually) to the adjacency server and registers to it when they initially join the OTV network.

The Adjacency server therefore learns about all the OTV edge devices on the network and in turn publish these neighbor addresses dynamically to the all edge devices.

unicastreplication

Once an edge OTV device learns where the other edge devices are (by identifying the device with the ip address of the join interface) , it can replicate its control plane packet for each peer and unicast the traffic to each neighbor it discovers from the adjacency server.

Therefore when you configure an the OTV edge device for unicast you will first select and configure the adjacency server;

— on Edge device at site 1
interface overlay 1
otv adjacency-server unicast-only

— On other remote edge devices i.e. site 2 and site 3

interface overlay 1
otv adjacency-server unicast-only
otv use-adjacency-server 10.1.1.1 unicast-only

OTV multicast mode
For larger networks with more than 5 DCs the original multicast mode for OTV provides better scalability.

Unlike OTV unicast which replicates every control packet to every other neighbor, an edge device in muticast mode sends its control packets to a multicast group.
By relying on the multicasting abilities within the IP network the edge device only has to update to the group address.

 

I will assume an understanding of multicast; if you need a refresher please refer to Peter Revill’s excellent articles on this topic (see bottom of article for urls).

For multicast OTV these are the essentials points to note.

– the Provider IP network must support PIM Sparse Mode with RP or PIM-Bidir multicast traffic
– you do not have to configure PIM on your edge devices
– the Edge devices for OTV multicast acts as IGMP hosts and joins an Any Source Multicast (ASM) group.

An Edge device will send the join interface’s ip address to an ASM multicast group as its source address.

Since it is the join interface that has to be identified as an IGMP host you will have to enable IGMP on it. Version 3 is required because this is the only version with a Source address field built into it.

— on all edge devices
interface e1/41
description “join-interface” for Overlay1
igmp version 3

IANA has scoped the 239.0.0.0/8 as the private address range for organisations to use. This address range is similar to the 10.0.0.0/8 in the private unicast space.

With OTV the ASM configuration is again under the interface configuration mode and is called the otv control-group since it controls the dissemination of MAC-to-IP between edge devices.

Since all edge devices usually belong to only 1 OTV domain the control-group address will normally be a specific ip address.

— on all edge devices
interface overlay 1
otv control-group 239.7.7.7

ASM is most suitable because everytime there is a MAC table update from any edge device, it only has to send 1 update to the ASM group. The multicast core will replicate the update to all receivers in the group.

Multicast traffic over the extended VLANs i.e. the data plane
If there is multicast traffic to be encapsulated over the VLAN – how do we deal with it ?

For example a host at site 1 may have multicast traffic for hosts at a different data center in the same extended vlan.

— for unicast mode transport
For unicast OTV mode there is no further configuration to do to handle multicast traffic for the extended vlans.
Everything happens behind the scenes. Essentially when a host at a site joins a multicast group it signals to the edge device.

The OTV edge device will record this host against the multicast group and also inform other neighbor edge devices within the OTV domain.
In this way all devices (MAC addresses–Multicast Group-IP mapping) for a particular multicast group is synchronised.

Any edge device receiving traffic for the multicast group will replicate copies and unicast the information to neighbor edge devices based on its mac-multicast-ip group table.

— for multicast mode transport
In the case of OTV multicast we will have to create a separate multicast handling channel in the data plane.

In the data plane, not every edge device needs to know about the a multicast. Multicast receivers in a multicast group may only reside at certain sites on certain extended vlans.

Therefore Source Specific Multicast (SSM) is more suitable for replicating multicast traffic that comes from hosts because it prevents unnecessarily ‘flooding’ of traffic sources that receivers have no interest in.
The address for SSM groups must come from 232.0.0.0/8.

To configure the data plane multicast transport,

— on all edge devices in otv multicast domain
interface overlay 1
otv data-group 232.7.7.0/24

The data-group is a subnet range because conceivably end hosts could join different SSM channels where you have a range of source addresses and Group addresses (S,G).

Site identifier and edge device redundancy
In an OTV domain every site must be unique and this is represented by an otv site identifier.

You configure this on every edge device in the global configuration mode. The otv site identifier in represented in MAC address format.

— on all edge device at site 1
otv site-identifier 0000.0000.000a

— on all edge device at site 2
otv site-identifier 0000.0000.000b

— on all edge device at site 3
otv site-identifier 0000.0000.000c

All edge devices on the same site must have the same site identifier but site ids must be unique for each site.

siteid

Since the edge device is the gateway to the OTV network, it is often best practice to implement a redundant edge device for each site.

When you have a redundant setup, you have to decide which of these will be the authoritative edge device (AED) that actively participates in OTV hellos, neighbor MAC-ip address updates and OTV database synchronisation.

Cisco has mandated a special site vlan for all local edge devices to elect an AED

The site vlan is defined in the global configuration mode.

— on edge devices
otv site-vlan 13

Although the site vlan is only locally significant, you should make the site vlans the same for consistency and for ease of support.
Even if you only have 1 edge device at a site you still need to configure a site vlan.

What about arp ?
When you extend vlans with OTV, arp traffic must be allowed between sites to discover the MAC address of hosts when a client is looking for arp resolution for an ip addresss.

To reduced effect of ARP traffic the OTV control plane can allow edge devices to build a local ARP table via arp address snooping.
Initial arp requests are allowed over the overlay and once learnt subsequent arp requests are fulfilled locally by the edge device.

— on all edge devices
interface overlay 1
no otv suppress-arp-nd

This command has a curious form and I am not sure why arp suppression for otv is not the default condition.

Miscellaneous
OTV encapsulation is done in hardware by ASICs. Currently OTV is only supported on N7k and ASRs.

You cannot mix N7k and ASRs on a single site but you can mix platforms within your domain.
Performance and limitations within a mixed environment is down to the lowest common denominator.

Summary
You can read the Cisco documentation for specific OTV configuration scenarios.

With OTV there is a lot happening under the hood and there are other tweaks for OTV which I will cover in future.

I hope this post has saved you a lot of reading helped clarify how a basic OTV configuration is put together.

Addendum
Links to Peter Revill’s multicast tutorials.

http://www.ccierants.com/2013/02/ccie-dc-multicast-part-1.html
http://www.ccierants.com/2013/02/ccie-dc-multicast-part-2.html
http://www.ccierants.com/2013/02/ccie-dc-multicast-part-3.html
http://www.ccierants.com/2013/02/ccie-dc-multicast-part-4.html