I've already posted this on the FreeBSD forums but figured I'd get some more eyes on it. This is more network-related than security related, but the answer may lie in ipfw. I'm just not sure yet.

The diagram to reference:

The "router" and "server" in this case are both running FreeBSD. The router is bridging interfaces em0 and re0 together, represented on the router as the black line (em0) and the red line (re0). Verizon uses a single /24 broadcast domain for their business class customers, and out of that, they allocate a set number of public IPs per customer. In my case, I have 13: .210-.222.

The router has an IP address on the bridge0 interface of .222. The server has a 2xGigE LACP-enabled lagg interface, with an IP of .210. The server also has a few jails on it, each with a public IP.

The basic problem is that the router receives ARP information from Verizon's upstream (a Juniper router) for all of the IPs on that server. So, for instance, the ARP entry on my router for .210 will be Verizon's MAC address, not my server's lagg0 interface. They're on the same (bridged) VLAN, but for some reason the router hears it from VZ and ignores it from the server.

From a machine on the private VLAN (note the green network in the diagram), I telnet to xx.yy.zz.215 port 80, because it's a jail running apache. The first one succeeds:

I'm kind of lost here. The work-around for now is a series of static ARP entries on the router pointing each of the public IP addresses on joker's lagg0 interface to the proper MAC. But that seems like a bad solution and I'm going to have to remember those are there in case I ever re-use one of those IPs somewhere else. It would be better to figure out how to block the ARP replies coming back from Verizon's router, but only for those IP addresses that belong to me. I can't just blindly block all ARPs from the VZ router; doing that will mean I'll never learn his MAC.

I understand you have multiple external IP addresses, so perhaps a bidirectional NAT (binat) configuration would enable you to provision em0 and re0 on separate Ethernets.

Allow me to apologize for the seemingly dismissive last reply. I've never really been a fan of the obfuscatia that comes along with NAT. It does create some interesting troubleshooting problems; I see that sort of thing in production environments all the time.

I am, however, warming up to the idea of doing it, because it'll allow the router to ... well... route. Sorta.

I'm proposing a series of changes to my pf.conf that will look like this:

Here's where it gets a bit confusing for me, but it probably is pretty easy. I've been avoiding NAT for my existing RFC1918 LAN when the destination is my public servers. In other words: I want the public servers to see SRC==192.168.xx.yy, not the NAT IP. If a public server has to start a conversation with the private LAN, I also want that to be native. I only want the private LAN to be NAT'd when it leaves the building.

See this post for explanation. I'm going to have to change the NAT lines, and I think this will accomplish what I'm after?

Code:

no nat on $ext_int from $local_ipv4_lan to $external_ipv4_lan
nat on $vz_int from $local_ipv4_lan to any -> XX.YY.ZZ.221

My gut tells me the first line is superfluous because the second line only says NAT on outgoing on the Verizon interface. Thoughts? This should allow the now BiNAT'd IPs to get to the RFC1918 space with their respective public IPs, right (assuming it they pass further rules in PF of course)?

In other words: I want the public servers to see SRC==192.168.xx.yy, not the NAT IP. If a public server has to start a conversation with the private LAN, I also want that to be native. I only want the private LAN to be NAT'd when it leaves the building.

Here I go replying to myself; this isn't going to work, is it? The bi-nat lines will only be applied for traffic going through the $vz_int. If I left the rules as written, the hosts in $int_int wouldn't even be able to talk to the public IP addresses, would they? Would the router know how to respond to packets source from $int_int and destined for the BiNAT'd IPs? Or would it just drop them since they never traverse the $vz_int? Would I need to change the interface in the binat-to rules to "any"?

Each public IP should be able to talk to other public IPs via the router.

Each private IP on $int_int should be able to talk to the public IPs via the router.

Each public IP should be able to talk to the private IPs on $int_int via the router.

Briefly, the only place I would suggest to use (bidirectional) NAT is on your external, public network. If the router is routing, rather than bridging, your external facing network is only attached to the router, and none of your servers would have an external address -- the mapping to Internet-facing addresses would be done by the router, and only the router.

---

J65nko reminds me that there are at least two "classic" DMZ topologies that might be considered, so I'll briefly describe them:

DMZ subnet sits between two firewalls, an "inner" and an "outer."

DMZ subnet resides in isolation via a single firewall.

While I personally prefer the dual firewall topology as I believe it offers a better defense in depth, either would provide a better defensive governance than you are currently deploying or considering among your solution-set.

What is the purpose of your setup? What are you trying to achieve?
Why did you choose this topology instead of a classic DMZ setup?

Per my previous posts: the basic problem is the way my ISP allocates IPs (static block of XX.YY.ZZ.210 - .222) and the network those IPs reside on. Specifically: I have to use their router on their premises as the default route (XX.YY.ZZ.1). They don't allocate me an actual CIDR block and route it towards me.

So, I have 3 basic choices:
1. DMZ where each server is on the same broadcast domain as the .1 address, meaning the servers are wide open and it's up to their local packet filters (or whatever) to protect themselves. I'm not interested in DMZs, so that's not a consideration.

2. L3 Bridged Interface such as what I started this thread with: Bridge the ISP side and broadcast domain that my servers are on, and use PF (iptables in the case of Linux) on the router to protect the network. The router isn't routing in this case, it's just a pass-through. The servers still set their default routes to XX.YY.ZZ.1.

3. A binat setup where the server's have an RFC1918 address that gets 1-to-1 NAT'd to the public IPs that I own. In this case, every packet is routed (sort of... a NAT isn't really a router) through the router as opposed to bridged/passed-through.

I'm doing choice #2 right now, but it's presenting me with all sorts of ARP problems on the FreeBSD router. The router is learning the ARPs for my IPs from the ISP's router, not from my servers. So the work-around for the time being is a collection of static ARPs on the router.

What's important to note is that my router is also the termination point for my existing RFC1918 subnet (wireless network, NAS, etc). So packets from that LAN to the world need to be NAT'd. Packets from it to the public side of my network should not be NAT'd. And packets from the public side of my network to the private side should also not be NAT'd.

If I go with choice #3, my concern is what will happen with the existing private LAN when it tries to talk to one of my public IPs? Will the right thing happen regarding the binat?

...If I go with choice #3, my concern is what will happen with the existing private LAN when it tries to talk to one of my public IPs? Will the right thing happen regarding the binat?

No. If you use NAT on your Internet-exposed subnet, you will have to route traffic to local servers (whether in a DMZ or not) via their local, untranslated network addresses.

See the Redirection and Reflection section of the PF User's Guide. The chapter hasn't changed much since the fork, other than using different PF syntax. It does discuss DMZs, but not as a direct solution.

If the router is routing, rather than bridging, your external facing network is only attached to the router, and none of your servers would have an external address -- the mapping to Internet-facing addresses would be done by the router, and only the router.

Yep, I understand that. I've been a network engineer and architect for over 2 decades. What I don't understand (yet) is how FreeBSD handles all of these things. To be honest, Linux does this part really well and I'm actually considering converting just the router back to it. But, before I throw in the towel:

In a binat scenario, the router's interfaces:

em0: XX.YY.ZZ.222/24 # Public side, Verizon IPs. Of those, I have 13.

re0: 172.16.0.0/24 # RFC1918 side of "public" servers. Binat sources.

re1: 192.168.100.0/24 # Private LAN for wireless, NFS server, etc.

So, packets coming in em0 destined for one of the public IPs will get binat'd out towards re0. Conversely, packets from the 172.16.0.0/24 LAN outbound on any of the interfaces also need to be binat'd via the same 1-to-1 mapping.

Packets coming in re1 and destined to the Internet should be NAT'd via the last NAT rule I listed. The many-to-1. Packets coming in re1 and destined to XX.YY.ZZ.210 - .220 should not be many-to-1 NAT'd, but should trigger the 1-to-1 binat.

This is where it gets confusing. Theoretically, if the router gets a packet destined for XX.YY.ZZ.210-.220 (the binat IPs) from any interface, it should know that it owns those 11 IPs, and do the binat appropriately. Correct?

For instance, let's say I have the NAT'ing rules applied as previously posted. And...

source: 192.168.100.10
destination: XX.YY.ZZ.210

The packet from 192.168.100.10 should hit interface re1, and then... what? Will the binat, which is sitting on interface em0, get triggered? Will the router know to send the packet, translated, to 172.16.0.210? Or will it just get dropped?

Most commonly, we use a split-horizon DNS. As an example, your authoritative nameserver would provide resolution for Internet-facing clients, and your caching nameserver would provide local resolution to local clients, resolving to your RFC1918 addresses.

Most commonly, we use a split-horizon DNS. As an example, your authoritative nameserver would provide resolution for Internet-facing clients, and your caching nameserver would provide local resolution to local clients, resolving to your RFC1918 addresses.

Yep, I figured that was going to be the next suggestion, and I'm not game for that either. I'm going to drop this binat idea on the floor and stick with the bridging. If it continues giving me problems, I'm going to have to revert back to Linux for the router.

Good luck, no matter what decisions you eventually make, or find yourself driven to by an ISP that is unwilling or unable to deploy TCP/IP as designed.

I'm almost ashamed to admit that: proxy arping solves the issue and allows the router to act as a router. As a network architect/engineer, I despise the idea of proxy arping. But in a small, well-defined and contained network such as my premises, it's not (that much of) a problem.

The basic steps:
1. Address the Verizon-facing interface on the router out of my static IP allotment, with the default route to .1
2. Address the public server facing interface on the router out of a completely phony IP block (I used 10.0.0.1/8).
3. Enable proxy arping on the router. In /etc/rc.conf:

Code:

# Set Proxy ARP
arpproxy_all="YES"

And then from the CLI:

Code:

# service routing restart

4. Set each of the public servers' default route to the IP address on the router, not Verizon's .1.

Voila. It works. It's sad that I have to deploy a broken network technology to fix a broken network. Yay Verizon!