I am currently using an Intel PRO/1000 PT LP Quad Port Server NIC with Ubuntu 12.04 LTS Server 64 Bit. I installed Gnome so I have a GUI.

I downloaded bmon 2.0.1 to monitor the network, and right now according to /proc/net/bonding/bond0 the bond is up and working. I also believe the switch is configured correctly.

The network setup has two links from the Ubuntu host going to a switch - these links are teamed. Two separate links to the switch go to two hosts which copy a file from the Ubuntu teamed NIC host.

When I copy two files (one to each single-link host), I get near-wire-speed for only one of the bonded pair, and the other one does nothing! This is despite the fact that the switch is doing MAC hashing and the files are going to separate places. I am copying the files off of an SSD to two RAM disks, so I know those are not the bottleneck. I should easily be able to saturate a single 1 gig line.

I am using the latest (2.0.0) e1000e driver from Intel, and while I realize Ubuntu is not officially supported by Intel, I would like to get this working in Ubuntu.

1 Answer
1

A hash algorithm is in use to deterministically distribute traffic across the two links in your channel. This algorithm might be using src/dst MAC address or L2/L3/L4 information in various permutations. For purposes of this discussion let's assume that it's MAC based and that to achieve a 50/50 split we can simply take the final digit in the destination's MAC address and if it's an even number (0,2,4,6,8,A,C,E) it goes on link 1 and if it's odd (1,3,5,7,9,B,D,F) it goes on link 2. If the destination hosts in question have MAC addresses ending in 00 and 02 then they will both (correctly) be placed on the first link.

The actual algorithms in use aren't exactly like the pidgin version above, but the effect is the same - over a sufficiently large population of addresses the distribution will approach 50/50. On much smaller samples, however, there's also a 50% chance that all of your traffic will be driven into a single link.

The best suggestion I'd have is to try different hashing algorithms. In practical terms your odds get better if more information is figured into the hash - so L3 addresses, L4 port numbers, etc all improve the probability that your traffic will be more evenly distributed.

As an interim troubleshooting step you could also try changing the MAC address of one of the receiving hosts to observe if you can influence the hash appropriately.