Hi,
I have a OpenBSD 4.9 with 3 NICs. For testing purposes pf is disabled. I can connect to services (eg. ssh) from internet via 1st ISP (shh 78.w.x.y, ping works) but i cant connect via 2nd ISP (ssh 178.w.x.y, ping dosent work). I would like to reach my server from two ISP at the same time. Here is my config:

I was trying to set multipath route with two default gateways. But then I cant reach ssh from both interfaces. When i try to connect, I always get connection to only one, the second one was unreachable. I was trying also route-to statement in pf.conf without success. And traffic from lan goes always from both interfaces.

So the next solution was only route-to with one default gateway (I want all traffic go via pppoe0 ($ext_if1) and pass in (on $ext_if2) only 3 services via em0 - one rdr-to rule, ssh and vpn). It will be best choice for my needs. After reading a lot of faqs and manuals i try to use tags in pf.conf:

I'd like to know just what you did, and, what results were seen, in as much detail as possible. I have to guess that you still had a non-multipathed default route. Ignoring PF entirely, and focusing on your routing environment, I have some questions:

Did you enable IPv4 multipath routing in sysctl.conf(5)? Confirm it is enabled with $ sysctl net.inet.ip.multipath

How are you creating the multipath routes? With a !route command in your applicable hostname.if(5) files? With an rc.local(8) script that issues route flush followed by the applicable route add -mpath commands?

How many default routes are in your routing table? Two? Three? If you have more than two, you have a problem, either caused by a mygate(5) setting or by dhclient(8) configuration accepting a default route, or by not flushing and reloading your routing table correctly.

Did you watch both interfaces with tcpdump(8) when pinging, or connecting with ssh? I'm guessing that packets coming in to IF2 were still being responded to via IF1.

While we await more information from you, I may be able to find time this weekend to run some tests. I've got a topology in mind, which tests a local "server" with external users. If the test were reversed; where the local system is the "user", it would be nearly the same; this topology just includes port forwarding along with NAT.
Please let me know if you would be interested in this type of problem recreation / resolution, before I invest the time and effort:

---

Five systems: An "internet user", two "ISPs", a "router", and a "server".

Four networks: an "Internet", between each "ISP" and the "router", and between the "router" and the "server."

Three tests: static provisioning, DHCP provisioning, and a NATted "server".

I'd like to know just what you did, and, what results were seen, in as much detail as possible. I have to guess that you still had a non-multipathed default route. Ignoring PF entirely, and focusing on your routing environment

Ok, i will focus on routing but i can do this in Monday. I will turn off my pf, because I was trying mpath with my pf enabled and try describe more details.

Quote:

Originally Posted by jggimi

[*]Did you enable IPv4 multipath routing in sysctl.conf(5)? Confirm it is enabled with $ sysctl net.inet.ip.multipath

Yes, it was enabled in sysctl.conf > net.inet.ip.multipath=1. Also i rebooted my OpenBSD box.

Quote:

Originally Posted by jggimi

[*]How are you creating the multipath routes? With a !route command in your applicable hostname.if(5) files? With an rc.local(8) script that issues route flush followed by the applicable route add -mpath commands?

Since I was working remotely i used /etc/hostname.if to make changes in gateways and then reboot.
hostname.em0
!/sbin/route add -mpath default 178.x.y.z
hostname.pppoe0
!/sbin/route add -mpath default 87.x.y.z

After that (working from my home) I can only ssh via em0, pppoe0 was unreachable. Although in my pf.conf i had:
pass in on em0 proto tcp from any to any port 22
pass in on pppoe0 proto tcp from any to any port 22

That was weird.

Quote:

Originally Posted by jggimi

[*]How many default routes are in your routing table? Two? Three? If you have more than two, you have a problem, either caused by a mygate(5) setting or by dhclient(8) configuration accepting a default route, or by not flushing and reloading your routing table correctly.

Im sure it was only two default routes. /etc/mygate is blank, also I had static ip (dont need dhclient)

Quote:

Originally Posted by jggimi

[*]Did you watch both interfaces with tcpdump(8) when pinging, or connecting with ssh? I'm guessing that packets coming in to IF2 were still being responded to via IF1.

No I didn't check it. I will do it in Monday.

But when I have only one default route to my if1 and Im trying ssh from outside via if2 I see incoming connection in tcpdump on that interface but nothing happen.

Btw. when mpath was enabled I can connect to outside services from OpenBSD box without problem (ex. www, ping etc). Some packets goes via em0 and some via pppoe0.
Ex. when I connect to my home ssh box it was always from em0.

Quote:

Originally Posted by jggimi

Please let me know if you would be interested in this type of problem recreation / resolution, before I invest the time and effort:

Im sure there is no need to do that, because I think that the problem is in my config/routing settings. I will get more details on Monday trying to set mpath from beginning.

I was able to connect ssh sessions from both ISPs, and from my "Internet" user machine, using either ISP as the gateway route. Here's an example of three ssh sessions: 10.0.1.1 and 10.0.2.2 are the ISPs, 10.0.0.3 is the internet user. In this instance, routed through ISP 2.

I was able to both initiate connections outbound, and port forward to the inbound "server" with the following pf.conf. The first line NATs all outbound traffic from the internal network according to it's appropriate trunk, however it gets routed. The second line used port forwarding to expose a service, in this case sshd(8), from the internal server.

Code:

match out from em2:network to any nat-to {em0,em1}
match in proto tcp from any to any port 2222 rdr-to 10.0.3.5 port 22
pass log all

To make my testing easier, I ended up setting up the "user" machine to also use multipathing. It could route through ISP1 or ISP2.

I discovered an error I'd made while setting up the lab environment. I'd neglected to add routes between the ISPs "customer" networks (10.0.1, 10.0.2) using the "internet" network (10.0.0). I discovered this by using tcpdump(8).

If you are unable to recreate the same success I've had, please consider using tcpdump and watching traffic flow (or not flow) across your NICs.

I happened across a linked article mentioned this week at the OpenBSD Journal, about using virtual routing domains -- and the article suggested the possibility of using them to connect with multiple ISPs, though it did not show a PF ruleset that might be applied in the solution.

This might be a way to circumvent your apparent pppoe restriction.

The article page provides a contact link for the author, as well as a comments section.

I seem to remember having a problem with a PPPOE adsl connection and it didn't work properly until I adjusted the MTU... because there was an MTU size problem a few packets actually would make it through if they were small enough, but most everything would be fragmented and then (for whatever reason) dropped.

If I get access to that host again in the next week or so I'll get the working configuration for it and my notes.