[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

stalled connections between pf servers



I am new to using PF, long time user of IPFilter. I switched to OpenbSD/PF last
week to setup a system to provide me with redundancy for my outbound
connections. The setup consists of 2 machines, each connected to a different
internet connection, and serving two internal subnets. (See ASCII diagram below)
Each machine provides redundancy for the other. Either machine can be master for
a subnet, but in normal operation, only one is master for a given subnet.  
The setup works great, my traffic from both subnets makes it to the internet
even if one machine is down. 
My problem is with traffic between the two subnets.
For example, I will start a ssh connection on one subnet, connecting to a
machine on the other subnet and after a few sec, or it seems whenever a large
amount of data goes through, the connection will stall/hang. 
I can do this will FTP, SSH, scp etc, even my mail hangs if I have more than say
5 messages waiting. I've tried to debug this myself with tcpdump, and I see the
connections going through but when the app hangs, I do not really see anything
out of the ordinary. 
My problem is exacerbated by the number of interfaces on each box, two for each
subnet in this case, which makes watching traffic difficult. I've seen some
postings about using route-to but I am not clear how that would help me here (or
even what my true problem is). 
One test I was able to do was to take one machine offline, and test my resultant
connectivity between the two subnets. I took one machine at a time offline, and
found running with 1 server (regardless of which) I was able to transfer a large
file with no problem at all between subnets. So I think I can rule out hardware
as my issue. 
I have one running theory at this time, but I'm not exactly certain how I could
go about testing it out. (viewing the ASCII diagram for this will help). Traffic
entering on fxp0 on HOBBES (from 192.168.20.0/24) and destined for
192.168.30.0/24 has two possible routes. It could be routed to CALVIN ( the
master for 192.168.30.0/24 ), or simply leave out fxp1 which is on the 30.0
network. Further, I have noticed through some testing that if I ping
192.168.30.1 from the 20.0 network, HOBBES responds, even though it is the
backup for that carp interface. So packets never make it to the other server I
believe. 
Now, the return traffic on the 30.0 network thinks (and rightly so) that it's
default gateway is CALVIN, and if my above test is true, then instead of passing
the packet over to HOBBES for routing to the 20.0 network, the packet leaves
CALVIN on the x12 interface and comes back on the 20.0 network from a different
location. 
So is their indeed some route-to work I need to perform here? 
I'm sure I'm not describing this completely, glad to provide more information.
A diagram of my redundant network:
http://webpages.charter.net/cycler/network.txt
# CALVIN
calvin# cat /etc/hostname.carp0
inet 192.168.30.1 255.255.255.0 192.168.30.255 vhid 1 advskew 1 pass foo
calvin# cat /etc/hostname.carp3
inet 192.168.20.1 255.255.255.0 192.168.20.255 vhid 2 advskew 100 pass bar
# HOBBES
hobbes# cat /etc/hostname.carp0
inet 192.168.30.1 255.255.255.0 192.168.30.255 vhid 1 advskew 100 pass foo
hobbes# cat /etc/hostname.carp3
inet 192.168.20.1 255.255.255.0 192.168.20.255 vhid 2 advskew 1 pass bar
** Both routers have an identical pf.conf. Ony the top 4 macros are different. 
calvin# cat /etc/pf.conf
#####################################################
# Macros
#####################################################
ext_if          = "xl0"
heartbeat_if    = "dc0"
server_if       = "xl1"
workstation_if  = "xl2"
web_proxy       = "192.168.30.x"
ssl_server      = "192.168.30.x"
ftp_server      = "192.168.30.x"
mail_server     = "192.168.30.x"
foo           = "192.168.20.x"
bar           = "192.168.20.x"
allow_ssh = "{xxx.xxx.xxx.xxx/32}"
internal = "{ 192.168.20.0/24, 192.168.30.0/24, 10.0.0.0/24 }"
#####################################################
#Options
#####################################################
set loginterface $ext_if
#####################################################
# NAT
#####################################################
nat on $ext_if from $server_if:network to any -> ($ext_if)
nat on $ext_if from $workstation_if:network to any -> ($ext_if)
#####################################################
# Redirection
#####################################################
# FTP
rdr on $ext_if proto tcp from any to any port 21 -> $ftp_server port 21
rdr on $ext_if proto tcp from any to any port 49152:65535 -> $ftp_server port
49152:65535
#Web
rdr on $ext_if proto tcp from any to any port 80 -> $web_proxy port 80
rdr on $ext_if proto tcp from any to any port 443 -> $ssl_server port 443
#Mail Traffic
rdr on $ext_if proto tcp from any to any port 25 -> $mail_server port 25
rdr on $ext_if proto tcp from any to any port 110 -> $mail_server port 110
#####################################################
# setup a default deny policy
#####################################################
block log all
#####################################################
# pass traffic on the loopback interface in either direction
#####################################################
pass quick on lo0 all
#####################################################
# CARP / pfsync
#####################################################
pass log on $server_if proto carp keep state
pass log on $workstation_if proto carp keep state
pass log on $heartbeat_if proto carp keep state
pass log on $heartbeat_if proto pfsync keep state
#####################################################
# Heartbeat Traffic
#####################################################
pass in quick on $heartbeat_if proto { tcp, icmp } from any to any keep state
#####################################################
# pass all traffic to and from the local network
#####################################################
pass in quick on $server_if from $internal to any keep state flags S/SA
pass out quick on $server_if from any to $internal keep state flags S/SA
pass in quick on $workstation_if from $internal to any keep state flags S/SA
pass out quick on $workstation_if from any to $internal keep state flags S/SA
#####################################################
# Outbound
#####################################################
pass out log quick on $ext_if all keep state
#####################################################
# SSH Traffic
#####################################################
pass in log quick on $ext_if proto tcp from $allow_ssh to any port ssh keep
state flags S/SA
#####################################################
# Web Traffic
#####################################################
pass in log quick on $ext_if proto tcp from any to any port 80 flags S/SA keep
state
pass in log quick on $ext_if proto tcp from any to any port 443 flags S/SA keep
state
#####################################################
# Mail Traffic
#####################################################
pass in log quick on $ext_if proto tcp from any to any port 25 keep state flags
S/SA
pass in log quick on $ext_if proto tcp from any to any port 110 keep state flags
S/SA
#####################################################
# Passport VPN
#####################################################
pass in quick on $ext_if proto gre from xxx.xxx.xxx.xxx/32 to any keep state
#####################################################
# Stupid FTP Traffic
#####################################################
# in on $ext_if
pass in quick on $ext_if proto tcp from any to $ftp_server port 21 keep state
flags S/SA
pass in quick on $ext_if proto tcp from any to $ftp_server port > 49151 keep
state flags S/SA
# out on $server_if
pass out quick on $server_if proto tcp from any to $ftp_server port 21 keep
state flags S/SA
pass out quick on $server_if proto tcp from any to $ftp_server port > 49151 keep
state flags S/SA
# out on $workstation_if
pass out quick on $workstation_if proto tcp from any to $ftp_server port 21 keep
state flags S/SA
pass out quick on $workstation_if proto tcp from any to $ftp_server port > 49151
keep state flags S/SA