Doing Etherchannel Over 3, 5, 6, and 7 Link Bundles

As a follow-up to my previous article on Port Channels titled “4 Types of Port Channels and When They’re Used” I wanted to talk a bit about the long-standing rule that says you should always create your Etherchannel (EC) bundles with a number of links that works out to a power of two (ie, 2,4 or 8 links). That rule is less applicable today than it used to be.

Why Power of Two?

The short answer? In bundles that are not powers of two, when the hash algorithm is run, some links will always come up more than other links. This results in more traffic being put on some of the links and prevents an even spread.

The long answer? The switch hardware — and I’m talking specifically about the Catalyst 6500 in this post so that means the Policy Feature Card 3 (PFC3) found on the Sup720 — which is responsible for performing the EC hash uses a 3-bit hash. A 3-bit hash results in 8 possible values (called “buckets”). Now when an EC is formed, each of the links in the EC is assigned some number of buckets. In a two link EC, each link receives 4 buckets. In an eight link EC, each receives 1. When the PFC3 runs the hash on a packet, the output is a number that corresponds to the number of a bucket. Whichever link has that bucket is the one that will receive the packet.

In the case of a three link EC, since 8 is not divisible by 3, the buckets are not evenly distributed among the links.

Traffic Spread with 3-bit Hash

Bundle

Link 1

Link 2

Link 3

Link 4

Link 5

Link 6

Link 7

Link8

2 Links

50%

50%

3 Links

37.5%

37.5%

25%

4 Links

25%

25%

25%

25%

5 Links

25%

25%

25%

12.5%

12.5%

6 Links

25%

25%

12.5%

12.5%

12.5%

12.5%

7 Links

25%

12.5%

12.5%

12.5%

12.5%

12.5%

12.5%

8 Links

12.5%

12.5%

12.5%

12.5%

12.5%

12.5%

12.5%

12.5%

This chart shows the traffic distribution breakdown for all possible link counts. Continuing with the 3 link example, you can see that the hash algorithm would chose the 3rd link less often than links 1 and 2 resulting in uneven utilization.

Bundles of Any Size

Since the size of the hash is baked into the hardware, there’s no changing the “power of two” rule on the PFC3. However, with the PFC4 found on the Sup2T, the hash size has been changed to 8 bits which results in 256 buckets. This allows for a more fine-grained distribution of buckets across the available links. Again, not all links will receive an exactly even share of buckets (unless the link count is still a power of two) but the disparity between links is much lower.

Traffic Spread with 8-bit Hash

Bundle

Link 1

Link 2

Link 3

Link 4

Link 5

Link 6

Link 7

Link8

2 Links

50%

50%

3 Links

33.6%

33.2%

33.2%

4 Links

25%

25%

25%

25%

5 Links

20.4%

19.9%

19.9%

19.9%

19.9%

6 Links

16.8%

16.8%

16.8%

16.4%

16.4%

16.4%

7 Links

14.5%

14.5%

14.5%

14.5%

14%

14%

14%

8 Links

12.5%

12.5%

12.5%

12.5%

12.5%

12.5%

12.5%

12.5%

So Which Hardware Does What Again?

On the Catalyst 6500, the PFC3/4 are found on these supervisors:

PFC

Found on Supervisor

Hash Size

PFC3

Sup32, Sup720

3-bit

PFC4

Sup2T

8-bit

So on the 6500, the Sup2T is able to more evenly distribute traffic across any sized EC.

And What About non-6500 Switches?

I don’t have time to research all platforms, but I do know that the Nexus 7000 breaks down like this:

Linecard

Hash Size

M1

3-bits

F1

8-bit

Since the supervisor on the Nexus 7000 does not handle any of the data plane functions, the hash size is entirely dependent on the linecard. I don’t know specifically about the M2 or F2, but I would ASSuME they are the same as the M1/F1.

Disclaimer:
The opinions and information expressed in this blog article are my own and not necessarily those of Cisco Systems.

Hello Joel,
I would be grateful if you can explain the Hash algorithm and indexing used in Catalyst 3750 switches. Because i noticed that on Catalyst 3560,3750,index number is always 0 for all interfaces. Please kindly note me on this.

I would love to give an update on the Cat 3k and 2k switches but I haven’t been able to find any information on the hash size they use. I’m pretty sure it’s going to be 3-bit. If and when I finally get that information, I will post it here.

When you say the index number is 0 for all interfaces, where are you seeing that?

I see what you mean now. My understanding is that the index and also the load are always zeroes on the lower end switches. On the Cat 6k you’ll see numbers in both columns.

Have you ever used the “test etherchannel load-balance” command on the 3750? You give it a src and dst MAC or IP address and it will tell you which interface in the etherchannel the frames will be put on. That should help you get some more information as to where your flows are going.

The RBH is the output from the etherchannel hashing algorithm and determines which physical port to use for an egress packet. If you use the “show etherchannel load-balance” command, you can see what information from the packet is fed into the hash algorithm. More info here:

Thanks for your amazing writing, it really helped me understand the whole thing, but I think in 3bit hashing table the 7 Links Bundle entries are miswritten, only the 1st link is 25%, and the next 6 Links must have the value of 12.5%.

Thanks Joel, I was just asked this question and said without a doubt that uneven links would be uneven load. Came across your blog after a search and stand corrected. We do have Sup2Ts and therefore have a non-issue with uneven links. Thanks for the right up.