FAT or Entropy Label?

In this article we are going to see two features that improve the load-balancing in the MLPS Core.

Why?
In the networking when dealing with redundancy of links or paths we are facing the polarization issue. This problem happens when one of the link is congested. Let’s take the example of LAGs, if the hashing algorithm is not efficient all the flows will pass through the same member of the LAG. In the MPLS World we are facing this especially for Layer 2 technologies: VPWS, VPLS.

Note that we can have the same issue for ECMP.

The hashing in the MPLS Core vary from vendors, let’s take the example of Cisco, for IP payloads the load-balancing is done on IP header (L3/L4 fields) and for non-IP payloads it is done on the last label, this is required to preserve the characteristics of the emulated service (VPWS).
The routers distinguish the IP from non-IP payloads by checking the version field of the paylods: 4 for IPv4 and 6 for IPv6. But unfortunately if you have some non-IP payloads that starts with 4 or 6, the load-balancing can be messy, and it already happens with MAC addresses starting with 4 ;). So Cisco highly recommend to use for non-IP payloads in the L2VPN the CW which starts with a 0.

For your information, the router will check the IP header up to 4 labels in the stack, if you have more labels it won’t go further for load-balancing!

Here are some illustrations:

Header stack for IP/VPN traffic, just after the bottom label we have the IP header:

Header stack for VPWS, just after the bottom label we have the CW:

Take this simple topology where we have a VPWS service between PE1 and PE2, I put two flows (different src/dest IP):

In this example we have only one VPWS, or AToM circuit so we will have for the two flows only one PW MPLS Label. So as stated before P1 will do the hash based on the bottom label which is the same and not efficient because all the flows will go on the bottom or top member of the LAG between the P routers.

The solutions
In the IETF, they worked on solutions in order to fix this issue:
– Flow-Aware Transport (FAT) Label, RFC 6391
– Entropy Label, RFC 6790

The main goal is to extract the appropriate keys from a given packet on the customer face side on the ingress PE (src/dst IP, MAC), input them to its load-balancing function, and place the result in an additional label in order to load-balance efficiently over an MPLS network.

FAT Label
It applies only for L2VPN service. The concept is to insert a flow label (an MPLS label like) on the ingress PE which encapsulates the layer 2 packets from the attachment circuit (customer) in MPLS packets. This will allow to have more entropy in the label stack. At the P, it is a traditional MPLS packet, load-balanced based on the bottom label which is the Flow label. Once arrived at the Egress PE, it will decapsulate the flow label and sends the layer 2 packets to the customer. The only routers that are aware of the flow label is the PE.
This is signaled by LDP with two flavors:
– Static, locally configured
– Dynamic, T/R bits (in the Flow Label sub-TLV ) capabilities exchanged through LDP

The FAT label is inserted between the CW and the PW MPLS Label.

Entropy Label
This is used for L2VPN and L3VPN. Here we have two labels inserted, the top one is a reserved label called ELI for Entropy Label indicator, if the value is 7 then we will have the presence of an Entropy label (EL) just after. This is required to distinguish unambiguously between entropy labels and service labels on the Egress PE.
Depending on the implementation, the LSRs (Label Switching Router or P) will use the Entropy Label for hashing or the entire stack label (except the ELI Label).
It is signaled by LDP, BGP or RSVP.

With Fat or Entropy label on our previous example, the flows will be well shared among the members of the bundle:

Pros and cons
Entropy allows to do have the same hashing methodology whatever vendor or payload/service (L2/3VPNs) present on your network. You can improve the performance by not doing fairly deep packet inspection (.ie 4 labels) in order to do the hashing for IP payloads.
Not only the PEs but the P must support also Entropy Label.
Flow Label is easy to implement: only on the PEs but applies only for L2VPN traffic.

Is the entropy stack inserted after the 1st label or before the last one? I’m just thinking extremes here. If you have some crazy CSC scenario and you are using FRR and happen to have a reroute right there and then, it’s possible your stack would have 2 extra labels so the readable entropy value could be pushed beyond the 4 readable stack. Presumably the entropy labels sit just beyond any transport labels in the end to end path? Does the P router need to read the entropy labels or just support the concept?

Good thought, okay with the CSC , assume CSC CE and CSC PEs use different labeling mechanism and failure happens for the protected LSP,then you can have maximum 5 labels in the label stack which some vendors even may not have support, now for same scenario , since backup LSP may or may not be used for long time , redundant links and load balancing for that LSP may not be a good reason. And second now put 6 or more label into the stack will not be supported by most of the vendors I think although I even did not mention about MTU issues.Also lets clarify here for the reader that load balancing is not always good for all MPLS applications and all traffic types because of latency and jitter caused by different quening characteristics of different paths, for example voice traffic wants predictable treatment, think also MPLS-TP we don’t want ECMP like behavior also for the same reason. Thanks for post Youssef, good job.

Good catch, from the RFC in section 4.3 you can read that if a P router does not support the EL then it SHOULD use as much of the whole label stack as feasible as keys for the load-balancing function (except the ELI). And then depending on the vendor implementation, one vendor can use only the EL or the whole stack or more deep inside the payload!

And for the label stack size, the IETF is working on the problem for Segment Routing because you can have easily more than one transport label and like you said it can be an issue for the performance: solutions like putting the EL before the transport labels or hop by hop the EL label is inserted just after one transport label (lot’s of swap operation) and the third solution is to advertise via IGP the position of the EL. For more details see draft-kini-mpls-entropy-label-src-stacked-tunnels-01

All of this because Juniper didn’t do L3 ECMP correctly in first place.
And thus try to resolve a L3 ECMP issue by addressing it from L2VPN and make it the newer default.

Good Juniper Marketing, bad for all OPERATORS/CUSTOMERS…

Why adding 2 labels in the stack for L3VPN (80% of deployments) ??
Is this not adding complexity (additional bugs) for weakest result, hashing on IP src/dst + Port src/dst isn’t going to provides best granularity for L3VPN ??
Of course Cisco will support Entropy, my question is what will be the CUSTOMER/OPERATOR gain ??