#4 connection-based XOR load balancing

A less deterministic version of the balance-xor policy.
Balance-mpxor bases its slave choice on [(( source MAC
address
XOR'd with destination MAC address) XOR'd with (source port
XOR'd with destination port)) modula slave count]. The
resulting behavior is a connection-based slave choosing
process. This would be useful in a sitaution where static
source and destination addresses are shuttling large
amounts
of data between themselves over multiple TCP connections.
----

Basically, we have an HPSS system that would get stuck
on particular slaves while communicating with the same
hosts if using balance-xor. This additional slave
choosing heuristic allows the same to hosts to
communicate using multiple slaves, each dedicated to
somewhat unique TCP connections instead of particular
hosts. Multiple downloads/uploads between the HPSS
system and the same client no longer have to share the
same slave. Of course, groups of client MAC addresses
and port numbers can more than likely be combined to
produce the same slave number, but it is still less
deterministic than using only the MAC address.

"mpxor" stands for "MAC and port XOR"

NOTE: this patch is based upon a 2.4.23 kernel which
has already been patched with the "Intel Contributions
2.4" patch (bonding_cleanup_4-2.4.23.tar.bz2). So, to
use this patch you would need to have applied the
"Intel Contributions 2.4" patch yourself or use kernel
source which already incorporates it. Then apply the
balance-mpxor ontop of it.

Why include the source MAC? If it orignated locally the
source MAC should be the same. If we are routing the source
MAC should again be the same. The only place it should be
different is bridging, and I would think that we would have
enough entropy from the remaining XORs.

OK after rewriting my response 4 times, I am starting to
think you have a good point ;). Each time I try to justify
keeping the MAC in the mix, I realize that if the port is
supplying "enough entropy", adding the MAC has as equal
chance to make interface choosing worse or better. .

If I can't come up a decent reason in the next couple of
days, I'll resubmit.

Do you find that this is useful enough to eventually make
itself into the actual distribution? I certainly hope so!