The difficult part is that we have two separate internet connections through different providers, the perimeter router hence has two external IP addresses and three routing tables.
And we obviously want to
- Spread outgoing traffic evenly through each provider.
- Pass each provider's private networks through the owner.
- Whenever either provider is down, pass the traffic through the other.
- Redirect some incoming traffic to selected servers in the DMZ.
- Redirect some other incoming traffic to the router itself.
Now as the company is growing, we have new offices, new services, new hardware, and so I thought that rather than trying to shoehorn the changes, I'd do it over again using something else instead of ipfw/natd. For example, pf.
After reading man and a couple books it all seemed clear. Some things I indeed worked around easily. But not this one.
Say I want to redirect incoming internet traffic from either provider to a service running at the router itself listening at localhost. This should be something like
nat on ext -> (ext) rdr on ext from any to (ext) tag FOO -> localhost pass in on ext tagged FOORight ? Well, if ext is your only internet connection, yes. Otherwise you have ext1 and ext2, each having its own default gateway and it bites you. See, if ext1 is 18.104.22.168, ext2 is 22.214.171.124, a SYN packet arrives over ext1
ext1: 126.96.36.199 -> 188.8.131.52it goes through the rdr rule first and becomes
ext1: 184.108.40.206 -> 127.0.0.1 tagged FOObefore it is seen by the pass rule. The routing information has been lost. Now even if the service responds with SYN/ACK, where should it be sent to ? Ideally we would need something like
rdr on ext1 from any to (ext) rtable 1 tag FOO -> localhost rdr on ext2 from any to (ext) rtable 2 tag FOO -> localhostto attach routing to a state as early as possible. Unfortunately, using rtable with rdr is not supported in FreeBSD 8. You can use rtable with pass, but it is too late a point.
My first reaction was to tag each provider's incoming traffic differently and play from there:
rdr on ext1 from any to (ext1) tag EXT1 -> localhost rdr on ext2 from any to (ext2) tag EXT2 -> localhost pass in on ext1 tagged EXT1 pass in on ext2 tagged EXT2 pass in on dmz route-to (ext1 gw1) tagged EXT1 pass in on dmz route-to (ext2 gw2) tagged EXT2The trick here is that we intercept the packets from the service response at re-entry, they are automatically attached to an already tagged state and therefore we can determine where to route them. And it works, but not with loopback (note that I've used dmz for interface name). Specifically for the loopback interface the response packets avoid filtering, the
pass in on lo0 route-to (ext1 gw1) tagged EXT1rule never fires and the matching packets go directly to routing. It doesn't work.
After some obligatory hair pulling, I've come up with the following working ruleset:
set skip on lo0 nat on ext1 -> (ext1) nat on ext2 -> (ext2) rdr on ext1 from any to (ext1) tag FOO -> localhost rdr on ext2 from any to (ext2) tag FOO -> localhost pass in on ext1 reply-to (ext1 gw1) tagged FOO pass in on ext2 reply-to (ext2 gw2) tagged FOOAnd an ever important addition to this:
route add default -iface lo0Now, I have no idea why localhost needs to be specifically routed to lo0, but you can google up "route-to lo0" for a bunch of posts similar to mine. Without it the redirected packets somehow can't reach localhost. But as we already have reply-to clause on a pass rule, we cannot also include route-to (which is yet another obstacle). There appears no other way but to set a default route to lo0. It even makes sense, because on this machine I treat localhost as publicly accessible.
As a bonus this ruleset also works for redirected connections that reach out to DMZ, which is enough to keep me happy for the moment.