How to make PF redirect traffic to localhost

Here in the company I work for, I also perform as a network administrator. Router, firewall, office VPNs, network segments, switches, cabling, that sort of thing.

The difficult part is that we have two separate internet connections through different providers, the perimeter router hence has two external IP addresses and three routing tables.

And we obviously want to

1. Spread outgoing traffic evenly through each provider.
2. Pass each provider's private networks through the owner.
3. Whenever either provider is down, pass the traffic through the other.
4. Redirect some incoming traffic to selected servers in the DMZ.
5. Redirect some other incoming traffic to the router itself.

I've had it all set up once four years ago using FreeBSD, ipfw and natd and it's been working ever since. The problem is maintenance. Having changed over time, ipfw ruleset got huge and unreadable, natd is troublesome to reconfigure and switches to userland consume CPU.

Now as the company is growing, we have new offices, new services, new hardware, and so I thought that rather than trying to shoehorn the changes, I'd do it over again using something else instead of ipfw/natd. For example, pf.

After reading man and a couple books it all seemed clear. Some things I indeed worked around easily. But not this one.

Say I want to redirect incoming internet traffic from either provider to a service running at the router itself listening at localhost. This should be something like

nat on ext -> (ext)
rdr on ext from any to (ext) tag FOO -> localhost
pass in on ext tagged FOO

Right ? Well, if ext is your only internet connection, yes. Otherwise you have ext1 and ext2, each having its own default gateway and it bites you. See, if ext1 is 1.1.1.1, ext2 is 2.2.2.2, a SYN packet arrives over ext1

ext1: 1.2.3.4 -> 1.1.1.1

it goes through the rdr rule first and becomes

ext1: 1.2.3.4 -> 127.0.0.1 tagged FOO

before it is seen by the pass rule. The routing information has been lost. Now even if the service responds with SYN/ACK, where should it be sent to ? Ideally we would need something like

rdr on ext1 from any to (ext) rtable 1 tag FOO -> localhost
rdr on ext2 from any to (ext) rtable 2 tag FOO -> localhost

to attach routing to a state as early as possible. Unfortunately, using rtable with rdr is not supported in FreeBSD 8. You can use rtable with pass, but it is too late a point.

My first reaction was to tag each provider's incoming traffic differently and play from there:

rdr on ext1 from any to (ext1) tag EXT1 -> localhost
rdr on ext2 from any to (ext2) tag EXT2 -> localhost
pass in on ext1 tagged EXT1
pass in on ext2 tagged EXT2
pass in on dmz route-to (ext1 gw1) tagged EXT1
pass in on dmz route-to (ext2 gw2) tagged EXT2

The trick here is that we intercept the packets from the service response at re-entry, they are automatically attached to an already tagged state and therefore we can determine where to route them. And it works, but not with loopback (note that I've used dmz for interface name). Specifically for the loopback interface the response packets avoid filtering, the

pass in on lo0 route-to (ext1 gw1) tagged EXT1

rule never fires and the matching packets go directly to routing. It doesn't work.

After some obligatory hair pulling, I've come up with the following working ruleset:

set skip on lo0
nat on ext1 -> (ext1)
nat on ext2 -> (ext2)
rdr on ext1 from any to (ext1) tag FOO -> localhost
rdr on ext2 from any to (ext2) tag FOO -> localhost
pass in on ext1 reply-to (ext1 gw1) tagged FOO
pass in on ext2 reply-to (ext2 gw2) tagged FOO

And an ever important addition to this:

route add default -iface lo0

Now, I have no idea why localhost needs to be specifically routed to lo0, but you can google up "route-to lo0" for a bunch of posts similar to mine. Without it the redirected packets somehow can't reach localhost. But as we already have reply-to clause on a pass rule, we cannot also include route-to (which is yet another obstacle). There appears no other way but to set a default route to lo0. It even makes sense, because on this machine I treat localhost as publicly accessible.

As a bonus this ruleset also works for redirected connections that reach out to DMZ, which is enough to keep me happy for the moment.