I recently ran into a situation where I needed two IP addresses on the same subnet assigned to one Linux host so that we could run two SSL/TLS sites. My first approach was to use IP aliasing, e.g. using eth0:0, eth0:1, etc, but our network admins have some fairly strict settings in place for security that squashed this idea:

They use DHCP snooping and normally don't allow static IP addresses. Static addressing is accomplished by using static DHCP entries, so the same MAC address always gets the same IP assignment. This feature can be disabled per switchport if you ask and you have a reason for it (thankfully I have a good relationship with the network guys and this isn't hard to do).

With the DHCP snooping disabled on the switchport, they had to put in a rule on the switch that said MAC address X is allowed to have IP address Y. Unfortunately this had the side effect of also saying that MAC address X is ONLY allowed to have IP address Y. IP aliasing required that MAC address X was assigned two IP addresses, so this didn't work.

There may have been a way around these issues on the switch configuration, but in an attempt to preserve good relations with the network admins I tried to find another way. Having two network interfaces seemed like the next logical step. Thankfully this Linux system is a virtual machine, so I was able to easily add a second network interface (without rebooting, I might add - pretty cool). A few keystrokes later I had two network interfaces up and running and both pulled IP addresses from DHCP.

But then the problem came in: the network admins could see (on the switch) the ARP entry for both interfaces, but only the first network interface that I brought up would respond to pings or any sort of TCP or UDP traffic.

After lots of digging and poking, here's what I came up with. It seems to work, but it also seems to be a lot of work for something that seems like it should be simple. Any alternate ideas out there?

arp_filter - BOOLEAN
1 - Allows you to have multiple network interfaces on the same
subnet, and have the ARPs for each interface be answered
based on whether or not the kernel would route a packet from
the ARP'd IP out that interface (therefore you must use source
based routing for this to work). In other words it allows control
of which cards (usually 1) will respond to an arp request.
0 - (default) The kernel can respond to arp requests with addresses
from other interfaces. This may seem wrong but it usually makes
sense, because it increases the chance of successful communication.
IP addresses are owned by the complete host on Linux, not by
particular interfaces. Only for more complex setups like load-
balancing, does this behaviour cause problems.
arp_filter for the interface will be enabled if at least one of
conf/{all,interface}/arp_filter is set to TRUE,
it will be disabled otherwise

And voila! Everything seems to work just fine. Sending pings to both IP addresses works fine. Sending pings from this system to other systems and forcing the ping to use a specific interface works fine (ping -I eth0 10.0.0.1, ping -I eth1 10.0.0.1). And most importantly, all TCP and UDP traffic to/from either IP address works as expected.

So again, my question is: is there a better way to do this? This seems like a lot of work for a seemingly simple problem.

Update: The solution above ended up being incomplete. Things worked fine if traffic stayed on the same subnet, but communications to other subnets using the 2nd interface would not work properly. Rather than digging a bigger hole I ended up talking with the network admins and got them to allow multiple IP addresses for the one interface and used IP aliasing (e.g. eth0 and eth0:0).

The key distinction to remember is that the IP does not belong to the interface, it belongs to the machine. So it is correct to send out either interface for either IP in this setup, hence why it requires some trickery to not do that.
–
MikeyBNov 29 '11 at 23:24

I believe source routing isn't necessary in this case because the arp filtering should apply only when the switch is sending to an interface /has to find it among its ports. I could be wrong, but when the machine sends data towards the switch, the IP in the source field (IP header) isn't checked, just the arp sending the packet.
–
AndreasMDec 2 '11 at 14:09

MikeyB is correct, source routing is the only way. The machine can choose any outbound interface it wants to send traffic, as long as its on the same subnet. Doesnt matter if the IP isnt really located on that interface or not.
–
PatrickDec 2 '11 at 14:30

1

AndreasM: The incoming packets aren't the problem, it's the outgoing packets which are the problem. Without source routing, all outgoing packets will use the default route, which is bound to one interface or the other. Even though the outgoing packets will have the correct source IP address, the filters on the switch will not allow them to go out since that IP address does not belong to the MAC address of that interface. To the switch it just looks like a client is trying to spoof the IP of another client. (These are smart layer 3 switches).
–
Scott DuckworthDec 2 '11 at 14:30

IP aliases are obsolete and a hack, do not reflect reality, not to mention that taking down the primary will take down all aliases. Use ip from iproute2 to add more than one address to the same interface.
–
pilonaAug 22 '13 at 20:16