Introduction

Cisco DNA Center gives us the flexibility to configure multiple fabric sites and connect them using different Transits. In this paper, we are going to talk about how we can connect multiple fabric sites using IP as a transit and how can we carry SGTs over IP Transit.

What is a Fabric Site ?

A “Fabric site” is a portion of the fabric which has its own set of Control Plane Nodes, Border Nodes and Edge Nodes. Different levels of redundancy and scale can be designed per Siteby including local resources: DHCP, AAA, DNS, Internet, etc. A Fabric Site may cover a single physical location, multiple locations, or just a subset of a location.

Single Location Branch, Campus or Metro Campus

Multiple Locations Metro Campus + Multiple Branches

Subset of a Location Building or Area within a Campus

What is a Fabric Domain ?

SD-Access fabric domain is hierarchal representation of fabric sites managed by Cisco DNAC SD-Access fabric domain can consist of multiple fabric sites and each site has its own devices which helps us achieve the benefits such as scale, resiliency and survivability. A Fabric domain should be able to scale horizontally by having state specific information within each site.

What is a Transit ?

Multiple Fabric sites in a single Fabric domain will be interconnected using a Transit.

IP- Based Transit - Leverages a traditional IP-based (VRF-LITE, MPLS) network, which requires remapping of VRFs and SGTs between sites.

IP Transit :

Traffic between sites will use control and data plane of the transit network/WAN.

The SD-Access Border will hand off the traffic to the directly connected WAN/External router and the traffic is delivered across the Wan/external domain to other site. No Lisp lookup with the transit control node is performed here as the border will hand off traffic to the WAN/External router using VRF-LITE.

If SD-WAN is deployed in the transit network/WAN then performance based routing, Encryption etc. in the WAN is possible.

End to end policy can only be maintained by Manual configuration.

Why IP Based Transit ?

IP Based Transit can be used if customers are already using existing WAN or have adapted SD-WAN. Tends to be many remote branch offices connected via traditional IP WAN/MPLS or SD-WAN. Also when there is a requirement for site-to-site encryption and traffic engineering and policy based routing.

Typical use cases

Internet Handoff

P2P IPSEC encryption

Policy Based Routing

WAN Accelerators

Traffic engineering

Mobile Backhaul LTE

SD-Access Distributed Campus Fabric IP as Transit

IP transits provides IP connectivity without native Cisco SD-Access encapsulation (SD-Access transit provides).Using IP based transits we can connect the SD-Access fabric to external networks using VRF lite for ip connectivity. We need to make sure that SGT information is also carried between sites using below methods in “SGT Propagation between sites” section.

Important to note that each fabric site has its own set of control plane nodes, edge nodes, border nodes, WLC. Cisco DNAC automates the BGP configuration for all the VRFs on the border nodes and it can be seen in the UI workflow below.

When client gets authenticated and gets assigned an SGT, that packet is traversed to the destination in data plane using VXLAN encapsulation which carries the SGT in its header and when the client is trying to reach to destination outside of the fabric then the packet is de encapsulated at the border from VXLAN header where SGT preservation is also gone. The same applies for returning traffic as well which enters the border switch and as border does not know about SGT for that source or destination.

In a SD-Access distributed deployment, you can have ISE PSNs dedicated to each of the fabric sites or for larger deployments where you have more than 2 PSNs at a site, we can have PSNs centrally located at a data center behind a load balancer and point the VIP of load balancer from DNAC settings

Supported Platforms for Border (as of Cisco DNAC 1.2.10)

Cisco SD-Access Border Node

Cisco SD-Access Transit

IP Transit

C9K

YES

YES

ASR1K/ISR4K

YES

YES

C6K

NO

YES

N7K

NO

YES

Fusion Device

As Cisco SD-Access achieves macro segmentation using vrfs, Users in those vrfs would want to talk to shared services residing out of the fabric which is in global routing table and we use a fusion devices which can be either router/switch/firewall to do route leaking leveraging the L3 handoff on the Border.

If the fusion device is a cisco firewall then ASA supports both SXP and inline tagging. Cisco FTD supports pxgrid and inline tagging and it will know about SGTs via pxgrid from ise. Now ,we will look at how we can propagate SGTs.

SGT Propagation between sites

Note: Assuming Cisco ISE is deployed in the network and end users are authenticating to Cisco Ise via 802.1x or any other method. Cisco ISE will assign Security Group Tag (SGT) once they authenticate to your wired/wireless network.

There are two ways in which we can make sure that SGTs are propagated across Fabric Sites.

Device Inline Tagging.

Using SXP or Scalable group tag exchange Protocol

Device Inline Tagging

Inline tagging is the process where the Security Group Tag is carried within a special field known as CMD (Cisco Meta Data) that can be inserted in an L2 header, whether that be Ethernet, IPsec or Tunnel as in GRE/DMVPN. EtherType:0x8909 16-Bit SGT encapsulated within Cisco Meta Data (CMD) payload. We have to make sure all the devices in the path support Inline tagging. Cisco TrustSec-capable devices have built-in hardware capabilities than can send and receive packets with SGT embedded in the MAC (L2) layer. It allows Ethernet interfaces on the device to be enabled for L2-SGT imposition so that the device can insert an SGT in the packet to be carried to its next hop Ethernet neighbor. SGT-over-Ethernet is a method of hop-by-hop propagation of SGT embedded in Ethernet packets. The inline identity propagation is scalable, provides near line-rate performance and avoids control plane overhead.

Configs that need to be enabled on each and every device in the path on both ingress/egress interface is :

Border# configure terminal

Border(config)# interface gigabitethernet 1/0/1

Border(config-if)# cts manual

Border(config-if-cts-manual)# propagate sgt

Border(config-if-cts-manual)# policy static sgt 54 trusted

Enable this on all the sub interfaces on routers and physical interfaces on switches on the interface facing outside of the fabric from Border.

SXP or Scalable group tag exchange Protocol is a TCP-based, control plane protocol used to exchange IP to SGT bindings that have been created on a network device through either dynamic assignment (802.1X, MAB, WebAuth) or statically via CLI with peer networking devices. SXP is supported on most Cisco Routers, Switches, Firewalls, and the Cisco Identity Services Engine or ISE. IP-to-SGT binding exchange over 64999/TCP

SXP is typically used in scenarios where a network devices do not support inline tagging.

SXP is used to propagate the IP-SGT information over the above mentioned network.

Here's how we create SXP sessions between border switches and ISE. The SXP tunnel should be created individually for each VN. By using this you have to build SXP for each vrf onto the border device. Alternatively, we can create sxp domains on ise which allows you to put certain mappings in each domain and only send once you want to each VN using ISE SXP Domain Filters.

An SXP Domain is a collection of SXP Devices and the administrator can decide which domain to send IP-to-SGT mappings to. This is not mandatory as a Default Domain exists and this is used by default for all SXP Devices and all IP-to-SGT mappings. If using SXP Domains to control the distribution of mappings, add the required Domains from Work Centers > TrustSec > SXP > SXP Devices:

Each domain is a unique table of IP/SGT bindings which can be peered independently from other domains and you can chose to have each domain pointing to each VN or Site.

SXP isn’t orchestrated from Cisco DNAC today so we can use template editor in DNAC to configure SXP tunnel. Once the destination SGTs are known to Border via SXP and source SGTs are already known, we can do enforcement on the border manually by entering cts role based enforcement for those vlans on the trunk interface egressing towards the WAN.

Now go to ISE, to the "Work Center > TrustSec > Settings > SXP Settings" and

Make sure the password is matched on ISE as well

Now go to "Work Center > Trustsec > SXP > SXP Devices" and add a session each for each VN on each border.

We can check the status of SXP connection using below commands:

show cts sxp connections vrf Campus

show cts sxp connections brief

We can perform Policy enforcement at the Firewall as well by building SXP from ISE to Firewall and sending the mappings to the firewall and sending SGTs from border to firewall via inline tagging as ASA firewall supports both types of SGT propagation.

Which Border to Pick ?

Cisco SD-Access Network Requirements

Latency Requirements (RTT)

In Summary, device latency should be around 100 msec RTT, you can go up to 200 msec RTT but there could be a performance hit. Anything beyond 200 msec is not recommended by Cisco at this time

The RTT (round-trip time) between Cisco DNA Center and network devices should be taken into consideration. The optimal RTT should be less than 100 milliseconds to achieve optimal performance

for base automation and assurance. When RTT is between 100 milliseconds and 200 milliseconds, longer execution time could be experienced for certain events including Inventory Collection, Fabric Provision and Image Update, ranging from a few minutes to tens of minutes. Cisco does not recommend RTT more than 200 milliseconds.

Hello.I'm having an issue with SSH console slowness once connected into a network device on GNS3. It generally slow on any router/switch regardless of the configuration to ssh/vty lines.If using Telnet it's perfectly fine and very responsive but ssh is sl...
view more

I'm trying to connect 2 sites to the main office in the middle using VPN I already configured the left side and It works fine but I can't seem to connect the right side to the middle network, here are all the configuration commands i used, !on ...
view more

I'm trying to connect 2 sites to the main office in the middle using VPN I already configured the left side and It works fine but I can't seem to connect the right side to the middle network, here are all the configuration commands i used, !on router...
view more

I'm having an issue with a single VLAN. I have a few VLANs on this switch: 1,4,25,26,30,45. The connected ports are configured thusly: int range ge1-10switchport mode trunkswitchport trunk allowed vlan all My uplink port is ge10My testing port i...
view more

Hello there. I am attempting to setup multiple VLAN's at my church using two SG300-10-10PP managed switches. However, after several attempts and searching the web for examples and instructions, only the default VLAN can access the Internet.&nbs...
view more