Comments

This patch adds better IPv6 failover support for bonding devices,
especially when in active-backup mode and there are only IPv6 addresses
configured, as reported by Alex Sidorenko.
- Creates a new file, net/drivers/bonding/bond_ipv6.c, for the
IPv6-specific routines. Both regular bonds and VLANs over bonds
are supported.
- Adds a new tunable, num_unsol_na, to limit the number of unsolicited
IPv6 Neighbor Advertisements that are sent on a failover event.
Default is 1.
- Creates two new IPv6 neighbor discovery functions:
ndisc_build_skb()
ndisc_send_skb()
These were required to support VLANs since we have to be able to
add the VLAN id to the skb since ndisc_send_na() and friends
shouldn't be asked to do this. These two routines are basically
__ndisc_send() split into two pieces, in a slightly different order.
- Updates Documentation/networking/bonding.txt and bumps the rev of bond
support to 3.4.0.
On failover, this new code will generate one packet:
- An unsolicited IPv6 Neighbor Advertisement, which helps the switch
learn that the address has moved to the new slave.
Testing has shown that sending just the NA results in pretty good
behavior when in active-back mode, I saw no lost ping packets for example.
-Brian
Signed-off-by: Brian Haley <brian.haley@hp.com>
---

On Tue, Oct 07, 2008 at 09:13:01PM -0400, Brian Haley wrote:
> This patch adds better IPv6 failover support for bonding devices, > especially when in active-backup mode and there are only IPv6 addresses > configured, as reported by Alex Sidorenko.>> - Creates a new file, net/drivers/bonding/bond_ipv6.c, for the> IPv6-specific routines. Both regular bonds and VLANs over bonds> are supported.>> - Adds a new tunable, num_unsol_na, to limit the number of unsolicited> IPv6 Neighbor Advertisements that are sent on a failover event.> Default is 1.>> - Creates two new IPv6 neighbor discovery functions:>> ndisc_build_skb()> ndisc_send_skb()>> These were required to support VLANs since we have to be able to> add the VLAN id to the skb since ndisc_send_na() and friends> shouldn't be asked to do this. These two routines are basically> __ndisc_send() split into two pieces, in a slightly different order.>> - Updates Documentation/networking/bonding.txt and bumps the rev of bond> support to 3.4.0.>> On failover, this new code will generate one packet:>> - An unsolicited IPv6 Neighbor Advertisement, which helps the switch> learn that the address has moved to the new slave.>> Testing has shown that sending just the NA results in pretty good > behavior when in active-back mode, I saw no lost ping packets for > example.>> -Brian>> Signed-off-by: Brian Haley <brian.haley@hp.com>
The Kconfig / build portions of this look fine to me.

Brian,
I'll make the same comment I did in the earlier version,
which is I don't think we need a new control for the count. Instead,
it should use the count for unsolicited NA's we send when adding
an address. It is the same case, really, and I, FWIW, would rather
we didn't have any more sysctl's than we need.
+-DLS
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

David Stevens wrote:
> Brian,> I'll make the same comment I did in the earlier version,> which is I don't think we need a new control for the count. Instead,> it should use the count for unsolicited NA's we send when adding> an address. It is the same case, really, and I, FWIW, would rather> we didn't have any more sysctl's than we need.> > +-DLS>
Are you referring to 'dad_transmits' because I don't see any sysctls
that control the number of unsolicited NAs. In fact, in normal operations
we don't send any unsolicited NAs at all.
This functionality effectively duplicates the gratuitous arps controls on the
bond.
-vlad
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Well, actually, it looks like I'm suggesting you to re-use something that
doesn't
exist. :-)
MLD (and IGMP) has such a thing where unsolicited advertisements are sent
multiple times, with delays in between, to account for lossy networks
possibly
dropping the first one. There are configurable counts associated with
probes
and retransmit intervals for solicits, but I don't see the equivalent yet
for
unsolicited NA's.
So, instead, what I suggest is that you add (or find!) THAT knob, instead
of a
bonding-specific one. Because adding an address that wasn't there before
has identical issues with unsolicited NA's as bonding has with activating
a
new address. The default should probably be 1, but if you ever need to
send multiple unsolicited NA's for bonding, you probably also need it for
adding a normal address on the same network. dad_transmits is similar,
but not really the same thing.
+-DLS
netdev-owner@vger.kernel.org wrote on 10/08/2008 10:40:13 AM:
> Brian,> I'll make the same comment I did in the earlier version,> which is I don't think we need a new control for the count. Instead,> it should use the count for unsolicited NA's we send when adding> an address. It is the same case, really, and I, FWIW, would rather> we didn't have any more sysctl's than we need.> > +-DLS> > --> To unsubscribe from this list: send the line "unsubscribe netdev" in> the body of a message to majordomo@vger.kernel.org> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Vlad Yasevich <vladislav.yasevich@hp.com> wrote:
>> +>> + list_for_each_entry(bond, &bond_dev_list, bond_list) {>> + if (bond->dev == event_dev) {>> + switch (event) {>> + case NETDEV_UP:>> + ipv6_addr_copy(&bond->master_ipv6, &ifa->addr);>> + return NOTIFY_OK;>>I think you want to store the first address configured on the device (most>likely link-local), and not overwrite it every time a new address is>configured. Since new addresses can be configured rather often (think>temporary, new RAs, etc) we really want the most stable address we can have.>Also, since ND is a link protocol, link-local is sufficient.
That depends upon how the IPv6 unsolicited NAs are handled by
the switch. For IPv4, we issue a gratuitous ARP for one of the IP
addresses on the interface to update the switch's MAC table; for this
case, it doesn't matter which IP address is used.
If IPv6-smart switches snoop the same way, then it again doesn't
matter which IPv6 address is used; this is just to update the MAC table.
I'll agree that it's logically sensible to use a link-local, though.
If, on the other hand, IPv6 needs an update for each configured address,
then storing just one IPv6 address is insufficient (as we'd need an NA
for each address).
-J
---
-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

David Stevens wrote:
> Well, actually, it looks like I'm suggesting you to re-use something that > doesn't> exist. :-)> > MLD (and IGMP) has such a thing where unsolicited advertisements are sent> multiple times, with delays in between, to account for lossy networks > possibly> dropping the first one. There are configurable counts associated with > probes> and retransmit intervals for solicits, but I don't see the equivalent yet > for> unsolicited NA's.
I don't see an equivalent either, since the only unsolicited NA the
kernel sends is for DAD, which uses dad_transmits.
I left the MLD changes out of this patch so I could work on it
separately, when I get to it I'll make sure to look at the issues you
raised in your other email so it follows the RFC, or at least the Linux
behavior.
> So, instead, what I suggest is that you add (or find!) THAT knob, instead > of a> bonding-specific one. Because adding an address that wasn't there before> has identical issues with unsolicited NA's as bonding has with activating > a> new address. The default should probably be 1, but if you ever need to> send multiple unsolicited NA's for bonding, you probably also need it for> adding a normal address on the same network. dad_transmits is similar,> but not really the same thing.
The problem is that dad_transmits can be set to zero, although not
recommended, so if we used that value then bonding failover would be
just as broken. I think having this new tunable stay in the bonding
code is useful since that's the code that's actually doing the transmit.
-Brian
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jay Vosburgh wrote:
> Vlad Yasevich <vladislav.yasevich@hp.com> wrote:> >>> +>>> + list_for_each_entry(bond, &bond_dev_list, bond_list) {>>> + if (bond->dev == event_dev) {>>> + switch (event) {>>> + case NETDEV_UP:>>> + ipv6_addr_copy(&bond->master_ipv6, &ifa->addr);>>> + return NOTIFY_OK;>> I think you want to store the first address configured on the device (most>> likely link-local), and not overwrite it every time a new address is>> configured. Since new addresses can be configured rather often (think>> temporary, new RAs, etc) we really want the most stable address we can have.>> Also, since ND is a link protocol, link-local is sufficient.> > That depends upon how the IPv6 unsolicited NAs are handled by> the switch. For IPv4, we issue a gratuitous ARP for one of the IP> addresses on the interface to update the switch's MAC table; for this> case, it doesn't matter which IP address is used.> > If IPv6-smart switches snoop the same way, then it again doesn't> matter which IPv6 address is used; this is just to update the MAC table.> I'll agree that it's logically sensible to use a link-local, though.> If, on the other hand, IPv6 needs an update for each configured address,> then storing just one IPv6 address is insufficient (as we'd need an NA> for each address).
My testing has shown that it doesn't matter which address I send the NA
for, I can ping both the link-local and global without dropping a packet
simultaneously. I'm using a Procurve 5400 series switch for what it's
worth that has IPv6 support, not sure if something not as recent would
behave any different.
-Brian
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Jay Vosburgh wrote:
> Vlad Yasevich <vladislav.yasevich@hp.com> wrote:> >>> +>>> + list_for_each_entry(bond, &bond_dev_list, bond_list) {>>> + if (bond->dev == event_dev) {>>> + switch (event) {>>> + case NETDEV_UP:>>> + ipv6_addr_copy(&bond->master_ipv6, &ifa->addr);>>> + return NOTIFY_OK;>> I think you want to store the first address configured on the device (most>> likely link-local), and not overwrite it every time a new address is>> configured. Since new addresses can be configured rather often (think>> temporary, new RAs, etc) we really want the most stable address we can have.>> Also, since ND is a link protocol, link-local is sufficient.> > That depends upon how the IPv6 unsolicited NAs are handled by> the switch. For IPv4, we issue a gratuitous ARP for one of the IP> addresses on the interface to update the switch's MAC table; for this> case, it doesn't matter which IP address is used.> > If IPv6-smart switches snoop the same way, then it again doesn't> matter which IPv6 address is used; this is just to update the MAC table.> I'll agree that it's logically sensible to use a link-local, though.> If, on the other hand, IPv6 needs an update for each configured address,> then storing just one IPv6 address is insufficient (as we'd need an NA> for each address).>
Yes, but the unsolicited NA for the global address just looks rather strange
when the link local one is provide. Also, with temporaries that can come and
go, it's better to use a stable address.
We are simply using it to refresh the MAC tables and for a while I thought it
would be sufficient to do just one ARP or ND, but then I realized that in an
environment where 2 systems are connected back-to-back, you would potentially
need to do both. Need to play with this config...
-vlad
> -J> > ---> -Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hi David
David Stevens wrote:
> Well, actually, it looks like I'm suggesting you to re-use something that > doesn't> exist. :-)> > MLD (and IGMP) has such a thing where unsolicited advertisements are sent> multiple times, with delays in between, to account for lossy networks > possibly> dropping the first one. There are configurable counts associated with > probes> and retransmit intervals for solicits, but I don't see the equivalent yet > for> unsolicited NA's.> > So, instead, what I suggest is that you add (or find!) THAT knob, instead > of a> bonding-specific one. Because adding an address that wasn't there before> has identical issues with unsolicited NA's as bonding has with activating > a> new address.
Adding a new address triggers a DAD probe which is enough to let the switch
learn the MAC address. It's a different scenario for a link failover in
bonding. Also, adding a new address will automatically trigger and MLD response
if a corresponding solicited node multicast address is added.
So the bonding case is rather different from everything else we've had so far.
-vlad
> The default should probably be 1, but if you ever need to> send multiple unsolicited NA's for bonding, you probably also need it for> adding a normal address on the same network. dad_transmits is similar,> but not really the same thing.> > +-DLS
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Vlad Yasevich <vladislav.yasevich@hp.com> wrote:
>Jay Vosburgh wrote:>> Vlad Yasevich <vladislav.yasevich@hp.com> wrote:>> >>>> +>>>> + list_for_each_entry(bond, &bond_dev_list, bond_list) {>>>> + if (bond->dev == event_dev) {>>>> + switch (event) {>>>> + case NETDEV_UP:>>>> + ipv6_addr_copy(&bond->master_ipv6, &ifa->addr);>>>> + return NOTIFY_OK;>>> I think you want to store the first address configured on the device (most>>> likely link-local), and not overwrite it every time a new address is>>> configured. Since new addresses can be configured rather often (think>>> temporary, new RAs, etc) we really want the most stable address we can have.>>> Also, since ND is a link protocol, link-local is sufficient.>> >> That depends upon how the IPv6 unsolicited NAs are handled by>> the switch. For IPv4, we issue a gratuitous ARP for one of the IP>> addresses on the interface to update the switch's MAC table; for this>> case, it doesn't matter which IP address is used.>> >> If IPv6-smart switches snoop the same way, then it again doesn't>> matter which IPv6 address is used; this is just to update the MAC table.>> I'll agree that it's logically sensible to use a link-local, though.>> If, on the other hand, IPv6 needs an update for each configured address,>> then storing just one IPv6 address is insufficient (as we'd need an NA>> for each address).>> >>Yes, but the unsolicited NA for the global address just looks rather strange>when the link local one is provide. Also, with temporaries that can come and>go, it's better to use a stable address.
As I said, I'll agree that it's logically sensible to use a
link-local address. This appears to be just cosmetic, though, and
(apparently, from what Brian Haley says) doesn't affect the switch
response to the update. But, wait, there's more...
>We are simply using it to refresh the MAC tables and for a while I thought it>would be sufficient to do just one ARP or ND, but then I realized that in an>environment where 2 systems are connected back-to-back, you would potentially>need to do both. Need to play with this config...
Yah, I've been thinking about that in the background, too,
specifically for cases with devices that cannot change their MAC address
(bonding fail_over_mac enabled); in those cases, the MAC changes during
a failover, so the gratuitous update is particularly important. The
fail_over_mac is used for Infiniband (fixed MAC) and a few ethernet
multiport devices that are confused by having more than one of their
ports set to the same MAC.
If those devices (when run back to back without a switch) need a
gratutious for each address, they'll need it for IPv4 and IPv6, I
suspect. I've not heard of any problems of this sort with Infiniband,
but I'm not sure how common back to back is with Infiniband (not very, I
suspect).
I think the non-fail_over_mac back to back connect case is ok,
at least for linux, because ARP already connects the MAC address to the
bonding device, not the underlying slave.
As you say, something to play with (but not today, alas, as my
office space is being remodeled).
-J
---
-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Well, I think the reason to send mulitple of them is identical.
If one is dropped due to network load, it won't happen; sending
multiple increases the odds of success.
DAD itself should update caches for neighboring nodes, so I
guess it makes sense that it isn't sending unsolicited NA's, But
that makes me think that the DAD retransmit counter is the one
you want. At least, the part of the DAD retransmit counter that is
for updating other nodes' caches. :-)
For MLD and IGMP, they were explicit SHOULD's-- I need to have
a look at ND RFC's to again to see what it says about it.
I don't think that alone is a reason to block the patch, but I also
don't think that updating neighbor caches with a new MAC address
is a unique requirement of bonding. Moving an address manually
ought to be identical in needs and behavior, as well as very-quick
reboots where the hardware changed. Thus, I don't think the knob
ought to be specific to bonding. I guess that leads to the suggestion
that you re-use the DAD counter for that.
References to MLD now and before are just me looking for an
analog to what ND should be doing. No new knob is definitely
required for them, since they already have this support for
unsolicited reports.
+-DLS
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

David Stevens wrote:
> Well, I think the reason to send mulitple of them is identical.> If one is dropped due to network load, it won't happen; sending> multiple increases the odds of success.> > DAD itself should update caches for neighboring nodes, so I> guess it makes sense that it isn't sending unsolicited NA's, But> that makes me think that the DAD retransmit counter is the one> you want. At least, the part of the DAD retransmit counter that is> for updating other nodes' caches. :-)
Nope, DAD doesn't trigger a cache update.
> > For MLD and IGMP, they were explicit SHOULD's-- I need to have> a look at ND RFC's to again to see what it says about it.> > I don't think that alone is a reason to block the patch, but I also> don't think that updating neighbor caches with a new MAC address> is a unique requirement of bonding.
Well, the mac address is not new since the same address is replicated
across all slaves. Also, unsolicited NAs are not permitted to change
the neighbor cache entries other then state. An unsolicited NA will
cause an existing entry to go from REACHABLE to STALE, and nothing else.
So, it use in bonding is really the same as gratuitous ARP.
> Moving an address manually> ought to be identical in needs and behavior, as well as very-quick> reboots where the hardware changed. Thus, I don't think the knob> ought to be specific to bonding. I guess that leads to the suggestion> that you re-use the DAD counter for that.
Yes, a dad counter could be re-used for this, but in some scenarios
it's overkill. Frankly, NA itself is an overkill. There may be
some unintentional consequences to using it that I am looking at now.
> > References to MLD now and before are just me looking for an> analog to what ND should be doing. No new knob is definitely> required for them, since they already have this support for> unsolicited reports.>
The problem is MLDs are only triggered when you are adding a new IPv6
multicast address. However, in the bond failover case, we are simply
moving a hardware multicast address from one slave interface to
another while leaving the IPv6 multicast address on the master bond interface.
Thus there is not trigger to fire off an MLD report.
-vlad
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

On Wed, 2008-10-08 at 15:01 -0400, Brian Haley wrote:
> David Stevens wrote:> > Well, actually, it looks like I'm suggesting you to re-use something that > > doesn't> > exist. :-)> > > > MLD (and IGMP) has such a thing where unsolicited advertisements are sent> > multiple times, with delays in between, to account for lossy networks > > possibly> > dropping the first one. There are configurable counts associated with > > probes> > and retransmit intervals for solicits, but I don't see the equivalent yet > > for> > unsolicited NA's.> > I don't see an equivalent either, since the only unsolicited NA the > kernel sends is for DAD, which uses dad_transmits.
Doesn't DAD use neighbor solicitation rather than unsolicited NA?
Can we use NS in the bonding failover scenario too?
Thanks
Sridhar
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Sridhar Samudrala wrote:
> On Wed, 2008-10-08 at 15:01 -0400, Brian Haley wrote:>> David Stevens wrote:>>> Well, actually, it looks like I'm suggesting you to re-use something that >>> doesn't>>> exist. :-)>>>>>> MLD (and IGMP) has such a thing where unsolicited advertisements are sent>>> multiple times, with delays in between, to account for lossy networks >>> possibly>>> dropping the first one. There are configurable counts associated with >>> probes>>> and retransmit intervals for solicits, but I don't see the equivalent yet >>> for>>> unsolicited NA's.>> I don't see an equivalent either, since the only unsolicited NA the >> kernel sends is for DAD, which uses dad_transmits.> > Doesn't DAD use neighbor solicitation rather than unsolicited NA?
Yes. There is one case in the NS code that will respond with an
unsolicited NA if we get a NS doing DAD. I guess I should have made it
clearer that it's when we're defending our address during a DAD probe.
> Can we use NS in the bonding failover scenario too?
Both and NS and NA seemed to update the switch, so either one can be
sent on a failover event. It seemed to be the consensus that the NA was
more appropriate, especially since we can send it without the solicited
bit set.
-Brian
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html