As I describe in an earlier blog post, I am running an OpenBSD packet filter firewall which has three network interfaces connected to the same ISP. Everything worked so well until the ISP changed something in their configuration and two of the interfaces started to get the same next hop router (gateway) through DHCP configuration. This obviously causes problems with e.g. ARP and routing in general. The solution to this was to start using the “routing domain” feature of OpenBSD.
All interfaces are in routing domain “0” by default. I then set the two “extra” outgoing interfaces to routing domains “1” and “2”. That way each outgoing network interface has its own routing table and ARP table, and routing/ARP problems with the “duplicate next hop” were fixed. However, by definition, routing doesn’t work between routing domains. Luckily I found a way around this by tweaking pf.conf. The solution was to
a) split the “binat” rules and use “rtable” keyword for the rule used for incoming packets, and
b) add the “rtable” keyword for outgoing packets.
That way the route lookup is done on the correct routing tables for both incoming and outgoing packets.
Here are the modified sections in /etc/pf.conf:
# binat on em2 for host "ps3"
match out on $if_ext3 inet from $ps3 to any nat-to $if_ext3 static-port
match in on $if_ext3 inet from any to $if_ext3 rdr-to $ps3 rtable 0
# binat on em1 for host "core7"
match out on $if_ext2 inet from $core7 to any nat-to $if_ext2 static-port
match in on $if_ext2 inet from any to $if_ext2 rdr-to $core7 rtable 0
# NAT on em0 for the rest of the hosts
match out on $if_ext1 from $home_net_v4 nat-to ($if_ext1)
...
pass out quick on $if_ext1 inet from ($if_ext1) modulate state
pass out quick on $if_ext2 inet from ($if_ext2) modulate state rtable 1
pass out quick on $if_ext3 inet from ($if_ext3) modulate state rtable 2
