BESS Workgroup A. Sajassi, Ed.
INTERNET-DRAFT A. Banerjee
Intended Status: Standards Track S. Thoria
D. Carrel
Cisco
B. Weis
Individual
J. Drake
Juniper
Expires: January 8, 2020 July 8, 2019
Secure EVPNdraft-sajassi-bess-secure-evpn-02
Abstract
The applications of EVPN-based solutions ([RFC7432] and [RFC8365])
have become pervasive in Data Center, Service Provider, and
Enterprise segments. It is being used for fabric overlays and inter-
site connectivity in the Data Center market segment, for Layer-2,
Layer-3, and IRB VPN services in the Service Provider market segment,
and for fabric overlay and WAN connectivity in Enterprise networks.
For Data Center and Enterprise applications, there is a need to
provide inter-site and WAN connectivity over public Internet in a
secured manner with same level of privacy, integrity, and
authentication for tenant's traffic as IPsec tunneling using IKEv2.
This document presents a solution where BGP point-to-multipoint
signaling is leveraged for key and policy exchange among PE devices
to create private pair-wise IPsec Security Associations without IKEv2
point-to-point signaling or any other direct peer-to-peer session
establishment messages.
Status of this Memo
This Internet-Draft is submitted to IETF in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF), its areas, and its working groups. Note that
other groups may also distribute working documents as
Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months
Sajassi et al. Expires January 8, 2020 [Page 1]

INTERNET DRAFT Secure EVPN July 8, 2019
SBD: Supplementary Broadcast Domain. A BD that does not have any ACs,
only IRB interfaces, and it is used to provide connectivity among all
the IP-VRFs of the tenant. The SBD is only required in IP-VRF- to-IP-
VRF use-cases (see Section 4.4.).
SN: Subnet.
TS: Tenant System.
VA: Virtual Appliance.
VNI: Virtual Network Identifier. As in [RFC8365], the term is used as
a representation of a 24-bit NVO instance identifier, with the
understanding that VNI will refer to a VXLAN Network Identifier in
VXLAN, or Virtual Network Identifier in GENEVE, etc. unless it is
stated otherwise.
VTEP: VXLAN Termination End Point, as in [RFC7348].
VXLAN: Virtual Extensible LAN, as in [RFC7348].
This document also assumes familiarity with the terminology of
[RFC7432], [RFC8365] and [RFC7365].
Sajassi et al. Expires January 8, 2020 [Page 5]

INTERNET DRAFT Secure EVPN July 8, 20191 Introduction
The applications of EVPN-based solutions have become pervasive in
Data Center, Service Provider, and Enterprise segments. It is being
used for fabric overlays and inter-site connectivity in the Data
Center market segment, for Layer-2, Layer-3, and IRB VPN services in
the Service Provider market segment, and for fabric overlay and WAN
connectivity in the Enterprise networks. For Data Center and
Enterprise applications, there is a need to provide inter-site and
WAN connectivity over public Internet in a secured manner with the
same level of privacy, integrity, and authentication for tenant's
traffic as used in IPsec tunneling using IKEv2. This document
presents a solution where BGP point-to-multipoint signaling is
leveraged for key and policy exchange among PE devices to create
private pair-wise IPsec Security Associations without IKEv2 point-to-
point signaling or any other direct peer-to-peer session
establishment messages.
EVPN uses BGP as control-plane protocol for distribution of
information needed for discovery of PEs participating in a VPN,
discovery of PEs participating in a redundancy group, customer MAC
addresses and IP prefixes/addresses, aliasing information, tunnel
encapsulation types, multicast tunnel types, multicast group
memberships, and other info. The advantages of using BGP control
plane in EVPN are well understood including the following:
1) A full mesh of BGP sessions among PE devices can be avoided by
using Route Reflector (RR) where a PE only needs to setup a single
BGP session between itself and the RR as opposed to setting up N BGP
sessions to N other remote PEs; therefore, reducing number of BGP
sessions from O(N^2) to O(N) in the network. Furthermore, RR
hierarchy can be leveraged to scale the number of BGP routes on the
RR.
2) MP-BGP route filtering and constrained route distribution can be
leveraged to ensure that the control-plane traffic for a given VPN is
only distributed to the PEs participating in that VPN.
For setting up point-to-point security association (i.e., IPsec
tunnel) between a pair of EVPN PEs, it is important to leverage BGP
point-to-multipoint singling architecture using the RR along with its
route filtering and constrain mechanisms to achieve the performance
and the scale needed for large number of security associations (IPsec
tunnels) along with their frequent re-keying requirements. Using BGP
signaling along with the RR (instead of peer-to-peer protocol such as
IKEv2) reduces number of message exchanges needed for SAs
establishment and maintenance from O(N^2) to O(N) in the network.
Sajassi et al. Expires January 8, 2020 [Page 6]

INTERNET DRAFT Secure EVPN July 8, 20192 Requirements
The requirements for secured EVPN are captured in the following
subsections.
2.1 Tenant's Layer-2 and Layer-3 data & control traffic
Tenant's layer-2 and layer-3 data and control traffic must be
protected by IPsec cryptographic methods. This implies not only
tenant's data traffic must be protected by IPsec but also tenant's
control and routing information that are advertised in BGP must also
be protected by IPsec. This in turn implies that BGP session must be
protected by IPsec.
2.2 Tenant's Unicast & Multicast Data Protection
Tenant's layer-2 and layer-3 unicast traffic must be protected by
IPsec. In addition to that, tenant's layer-2 broadcast, unknown
unicast, and multicast traffic as well as tenant's layer-3 multicast
traffic must be protected by IPsec when ingress replication or
assisted replication are used. The use of BGP P2MP signaling for
setting up P2MP SAs in P2MP multicast tunnels is for future study.
2.3 P2MP Signaling for SA setup and Maintenance
BGP P2MP signaling must be used for IPsec SAs setup and maintenance.
The BGP signaling must follow P2MP signaling framework per
[CONTROLLER-IKE] for IPsec SAs setup and maintenance in order to
reduce the number of message exchanges from O(N^2) to O(N) among the
participant PE devices.
2.4 Granularity of Security Association Tunnels
The solution must support the setup and maintenance of IPsec SAs at
the following level of granularities:
1) Per PE: A single IPsec tunnel between a pair of PEs to be used for
all tenants' traffic supported by the pair of PEs.
2) Per tenant: A single IPsec tunnel per tenant per pair of PEs. For
example, if there are 1000 tenants supported on a pair of PEs, then
1000 IPsec tunnels are required between that pair of PEs.
3) Per subnet: A single IPsec tunnel per subnet (e.g., per VLAN/EVI)
of a tenant on a pair of PEs.
4) Per IP address: A single IPsec tunnel per pair of IP addresses of
a tenant on a pair of PEs.
Sajassi et al. Expires January 8, 2020 [Page 7]

INTERNET DRAFT Secure EVPN July 8, 2019
5) Per MAC address: A single IPsec tunnel per pair of MAC addresses
of a tenant on a pair of PEs.
6) Per Attachment Circuit: A single IPsec tunnel per pair of
Attachment Circuits between a pair of PEs.
2.5 Support for Policy and DH-Group List
The solution must support a single policy and DH group for all SAs as
well as supporting multiple policies and DH groups among the SAs.
3 BGP Component
The architecture that encompasses device-to-controller trust model,
has several components among which is the signaling component. Secure
EVPN Signaling, as defined in this document, is the BGP signaling
component of the overall Architecture. We will briefly describe this
Architecture here to further facilitate understanding how Secure EVPN
fits into the overall architecture. The Architecture describes the
components needed to create BGP based SD-WANs and how these
components work together. Our intention is to list these components
here along with their brief description and to describe this
Architecture in details in a separate document where to specify the
details for other parts of this architecture besides the BGP
signaling component which is described in this document.
The Architecture consists of four components. These components are
Zero Touch Bring-up, Configuration Management, Orchestration, and
Signaling. In addition to these components, secure communications
must be provided between the edge nodes and all servers/devices
providing the architecture components.
3.1 Zero Touch Bring-up (ZTB)
The first component is a zero touch capability that allows an edge
device to find and join its SD-WAN with little to no assistance other
than power and network connectivity. The goal is to use existing
work in this area. The requirements are that an edge device can
locate its ZTB server/component of its SD-WAN controller in a secure
manner and to proceed to receive its configuration.
3.2 Configuration Management
After an edge device joins its SD-WAN, it needs to be configured.
Sajassi et al. Expires January 8, 2020 [Page 8]

INTERNET DRAFT Secure EVPN July 8, 2019
Configuration covers all device configuration, not just the
configuration related to Secure EVPN. The previous Zero Touch Bring-
up component will have directed the edge device, either directly or
indirectly, to its configuration server/component. One example of a
configuration server is the I2NSF Controller. After a device has been
configured, it can engage in the next two components. Configuration
may include updates over time and is not a one time only component.
3.3 Orchestration
This component is optional. It allows for more dynamic updates of
configuration and statistics information. Orchestration can be more
dynamic than configuration.
3.4 Signaling
Signaling is the component described in this document. The
functionality of a Route Reflector is well understood. Here we
describe the signaling component of BGP SD-WAN Architecture and the
BGP extension/signaling for IPsec key management and policy.
4 Solution Description
This solution uses BGP P2MP signaling where an originating PE only
send a message to the Route Reflector (RR) and then the RR reflects
that message to the interested recipient PEs. The framework for such
signaling is described in [CONTROLLER-IKE] and it is referred to as
device-to-controller trust model. This trust model is significantly
different than the traditional peer-to-peer trust model where a P2P
signaling protocol such as IKEv2 [RFC7296] is used in which the PE
devices directly authenticate each other and agree upon security
policy and keying material to protect communications between
themselves. The device-to-controller trust model leverages P2MP
signaling via the controller (e.g., the RR) to achieve much better
scale and performance for establishment and maintenance of large
number of pair-wise Security Associations (SAs) among the PEs.
This device-to-controller trust model first secures the control
channel between each device and the controller using peer-to-peer
protocol such as IKEv2 [RFC7296] to establish P2P SAs between each PE
and the RR. It then uses this secured control channel for P2MP
signaling in establishment of P2P SAs between each pair of PE
devices.
Sajassi et al. Expires January 8, 2020 [Page 9]

INTERNET DRAFT Secure EVPN July 8, 2019
Each PE advertises to other PEs via the RR the information needed in
establishment of pair-wise SAs between itself an every other remote
PEs. These pieces of information are sent as Sub-TLVs of IPSec tunnel
type in BGP Tunnel Encapsulation attribute. These Sub-TLVs are
detailed in section 5 and are based on the DIM message components
from [CONTROLLER-IKE] and the IKEv2 specification [RFC7296]. The
IPsec tunnel TLVs along with its Sub-TLVs are sent along with the BGP
route (NLRI) for a given level of granularity.
If only a single SA is required per pair of PE devices to multiplex
user traffic for all tenants, then IPsec tunnel TLV is advertised
along with IPv4 or IPv6 NLRI representing loopback address of the
originating PE. It should be noted that this is not a VPN route but
rather an IPv4 or IPv6 route.
If a SA is required per tenant between a pair of PE devices, then
IPsec tunnel TLV can be advertised along with EVPN IMET route
representing the tenant or can be advertised along with a new EVPN
route representing the tenant.
If a SA is required per tenant's subnet (e.g., per VLAN) between a
pair of PE devices, then IPsec tunnel TLV is advertised along with
EVPN IMET route.
If a SA is required between a pair of tenant's devices represented by
a pair of IP addresses, then IPsec tunnel TLV is advertised along
with EVPN IP Prefix Advertisement Route or EVPN MAC/IP Advertisement
route.
If a SA is required between a pair of tenant's devices represented by
a pair of MAC addresses, then IPsec tunnel TLV is advertised along
with EVPN MAC/IP Advertisement route.
If a SA is required between a pair of Attachment Circuits (ACs) on
two PE devices (where an AC can be represented by <VLAN, port>), then
IPsec tunnel TLV is advertised along with EVPN Ethernet AD route.
4.1 Inheritance of Security Policies
Operationally, it is easy to configure a security association between
a pair of PEs using BGP signaling. This is the default security
association that is used for traffic that flows between peers.
However, in the event more finer granularity of security association
is desired on the traffic flows, it is possible to set up SAs between
a pair of tenants, a pair of subnets within a tenant, a pair of IPs
between a subnet, and a pair of MACs between a subnet using the
appropriate EVPN routes as described above. In the event, there are
no security TLVs associated with an EVPN route, there is a strict
Sajassi et al. Expires January 8, 2020 [Page 10]

INTERNET DRAFT Secure EVPN July 8, 2019
order in the manner security associations are inherited for such a
route. This results in an EVPN route inheriting the security
associations of the parent in a hierarchical fashion. For example,
traffic between an IP pair is protected using security TLVs announced
along with the EVPN IP Prefix Advertisement Route or EVPN MAC/IP
Advertisement route as a first choice. If such TLVs are missing with
the associated route, then one checks to see if the subnets the IPs
are associated with has security TLVs with the EVPN IMET route. If
they are present, those associations are used in securing the
traffic. In the absence of them, the peer security associations are
used. The order in which security associations are inherited are from
the granular to the coarser, namely, IP/MAC associated TLVs with the
EVPN route being the first preference, and the subnet, the tenant,
and the peer associations preferred in that fashion.
It should be noted that when a security association is made it is
possible for it to be re-used by a large number of traffic flows. For
example, a tenant security association may be associated with a
number of child subnet routes. Clearly it is mandatory to keep a
tenant security association alive, if there are one or more subnet
routes that want to use that association. Logically, the security
associations between a pair of entities creates a single secure
tunnel. It is thus possible to classify the incoming traffic in the
most granular sense {IP/MAC, subnet, tenant, peer} to a particular
secure tunnel that falls within its route hierarchy. The policy that
is applied to such traffic is independent from its use of an existing
or a new secure tunnel. It is clear that since any number of
classified traffic flows can use a security association, such a
security association will not be torn down, if at least there is one
policy using such a secure tunnel.
4.2 Distribution of Public Keys and Policies
One of the requirements for this solution is to support a single DH
group and a single policy for all SAs as well as to support multiple
DH groups and policies among the SAs. The following subsections
describe what pieces of information (what Sub-TLVs) are needed to be
exchanged to support a single DH group and a single policy versus
multiple DH groups and multiple policies.
4.2.1 Minimal DIM
For SA establishment, at the minimum, a PE needs to advertise to
other PEs, its DIM values as specified in [CONTROLLER-IKE]. These
include:
ID Tunnel ID
N Nonce
Sajassi et al. Expires January 8, 2020 [Page 11]

INTERNET DRAFT Secure EVPN July 8, 2019
RC Rekey Counter
I Indication of initial policy distribution
KE DH public value.
When this minimal set of DIM values is sent, then it is assumed that
all peer PEs share the same policy for which DH group to use, as well
as which IPSec SA policy to employ. Section 5.1 defines the Minimal
DIM sub-TLV as part of IPsec tunnel TLV in BGP Tunnel Encapsulation
Attribute.
4.2.2 Multiple Policies
There can be scenarios for which there is a need to have multiple
policy options. This can happen when there is a need for policy
change and smooth migration among all PE devices to the new policy is
required. It can also happen if different PE devices have different
capabilities within the network. In these scenarios, PE devices need
to be able to choose the correct policy to use for each other. This
multi-policy scheme is described in section 6 of [CONTROLLER-IKE]. In
order to support this multi-policy feature, a PE device MUST
distribute a policy list. This list consists of multiple distinct
policies in order of preference, where the first policy is the most
preferred one. The receiving PE selects the policy by taking the
received list (starting with the first policy) and comparing that
against its own list and choosing the first one found in common. If
there is no match, this indicates a configuration error and the PEs
MUST NOT establish new SAs until a message is received that does
produce a match.
4.2.2.1 Multiple DH-groups
It can be the case that not all peers use the same DH group. When
multiple DH groups are supported, the peer may include multiple KE
Sub-TLVs. The order of the KE Sub-TLVs determines the preference.
The preference and selection methods are specified in Section 6 of
[CONTROLLER-IKE].
4.2.2.2 Multiple or Single ESP SA policies
In order to specify an ESP SA Policy, a DIM may include one or more
SA Sub-TLVs. When all peers are configured by a controller with the
same ESP SA policy, they MAY leave the SA out of the DIM. This
minimizes messaging when group configuration is static and known.
However, it may also be desirable to include the SA. If a single SA
is included, the peer is indicating what ESP SA policy it uses, but
is not willing to negotiate. If multiple SA Sub-TLVs are included,
the peer is indicating that it is willing to negotiate. The order of
Sajassi et al. Expires January 8, 2020 [Page 12]

INTERNET DRAFT Secure EVPN July 8, 2019
the SA Sub-TLVs determines the preference. The preference and
selection methods are specified in Section 6 of [CONTROLLER-IKE].
4.3 Initial IPsec SAs Generation
The procedure for generation of initial IPsec SAs is described in
section 3 of [CONTROLLER-IKE]. This section gives a summary of it in
context of BGP signaling. When a PE device first comes up and wants
to setup an IPsec SA between itself and each of the interested remote
PEs, it generates a DH pair along for each [what word here?
"tennant"?] using an algorithm defined in the IKEv2 Diffie-Hellman
Group Transform IDs [IKEv2-IANA]. The originating PE distributes the
DH public value along with the other values in the DIM (using IPsec
Tunnel TLV in Tunnel Encapsulation Attribute) to other remote PEs via
the RR. Each receiving PE uses this DH public number and the
corresponding nonce in creation of IPsec SA pair to the originating
PE - i.e., an outbound SA and an inbound SA. The detail procedures
are described in section 5.2 of [CONTROLLER-IKE].
4.4 Re-Keying
A PE can initiate re-keying at any time due to local time or volume
based policy or due to the result of cipher counter nearing its final
value. The rekey process is performed individually for each remote
PE. If rekeying is performed with multiple PEs simultaneously, then
the decision process and rules described in this rekey are performed
independently for each PE. Section 4 of [CONTROLLER-IKE] describes
this rekeying process in details and gives examples for a single
IPsec device (e.g., a single PE) rekey versus multiple PE devices
rekey simultaneously.
4.5 IPsec Databases
The Peer Authorization Database (PAD), the Security Policy Database
(SPD), and the Security Association Database (SAD) all need to be
setup as defined in the IPsec Security Architecture [RFC4301].
Section 5 of [CONTROLLER-IKE] gives a summary description of how
these databases are setup for the controller-based model where key is
exchanged via P2MP signaling via the controller (i.e., the RR) and
the policy can be either signaled via the RR (in case of multiple
policies) or configured by the management station (in case of single
policy).
5 EncapsulationSajassi et al. Expires January 8, 2020 [Page 13]

INTERNET DRAFT Secure EVPN July 8, 2019
Vast majority of Encapsulation for Network Virtualization Overlay
(NVO) networks in deployment are based on UDP/IP with UDP destination
port ID indicating the type of NVO encapsulation (e.g., VxLAN, GPE,
GENEVE, GUE) and UDP source port ID representing flow entropy for
load-balancing of the traffic within the fabric based on n-tuple that
includes UDP header. When encrypting NVO encapsulated packets using
IP Encapsulating Security Payload (ESP), the following two options
can be used: a) adding a UDP header before ESP header (e.g., UDP
header in clear) and b) no UDP header before ESP header (e.g.,
standard ESP encapsulation). The following subsection describe these
encapsulation in further details.
5.1 Standard ESP Encapsulation
When standard IP Encapsulating Security Payload (ESP) is used
(without outer UDP header) for encryption of NVO packets, it is used
in transport mode as depicted below. When such encapsulation is used,
for BGP signaling, the Tunnel Type of Tunnel Encapsulation TLV is set
to ESP-Transport and the Tunnel Type of Encapsulation Extended
Community is set to NVO encapsulation type (e.g., VxLAN, GENEVE, GPE,
etc.). This implies that the customer packets are first encapsulated
using NVO encapsulation type and then it is further encapsulated &
encrypted using ESP-Transport mode.
Sajassi et al. Expires January 8, 2020 [Page 14]

INTERNET DRAFT Secure EVPN July 8, 20196.3.1 Transform Substructure
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Transform Attr Length |Transform Type | Reserved. |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Transform ID | Reserved |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
~ Transform Attributes ~
| |
+---------------------------------------------------------------+
Figure 9: Transform Substructure Sub-TLV
The Transform Attr Length is the length of the Transform Attributes
field.
The Transform Type is from Section 3.3.2 of [RFC7296] and
[IKEV2IANA]. Only the values ENCR, INTEG, and ESN are allowed.
The Transform ID specifies the transform identification value from
[IKEV2IANA].
Reserved is unused and MUST be zero on transmit and MUST be ignored
on receipt.
The Transform Attributes are taken directly from 3.3.5 of [RFC7296].
7 Applicability to other VPN types
Although P2MP BGP signaling for establishment and maintenance of SAs
among PE devices is described in this document in context of EVPN,
there is no reason why it cannot be extended to other VPN
technologies such as IP-VPN [RFC4364], VPLS [RFC4761] & [RFC4762],
and MVPN [RFC6513] & [RFC6514] with ingress replication. The reason
EVPN has been chosen is because of its pervasiveness in DC, SP, and
Enterprise applications and because of its ability to support SA
establishment at different granularity levels such as: per PE, Per
tenant, per subnet, per Ethernet Segment, per IP address, and per
MAC. For other VPN technology types, a much smaller granularity
levels can be supported. For example for VPLS, only the granularity
of per PE and per subnet can be supported. For per-PE granularity
level, the mechanism is the same among all the VPN technologies as
IPsec tunnel type (and its associated TLV and sub-TLVs) are sent
along with the PE's loopback IPv4 (or IPv6) address. For VPLS, if
Sajassi et al. Expires January 8, 2020 [Page 19]