[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

load-balancing outgoing connections opened by the firewall



Hi,
I decided to start a new thread because I realized that the problem I am 
facing is related to TCP connections as a whole, and not specifically with 
transparent squid behaviour.
On message http://www.benzedrine.cx/pf/msg06133.html, Daniel recommends:
"Instead of using route-to on $int_if, you can let connections go out
through the one interface to the default gateway, and use route-to on a
'pass out on $ext_if1' rule to re-route the outgoing connection to
another interface. Packets will 'try' to get out on the default
interface, but re-routing occurs before they are actually sent out
through the interface.
  pass out on $ext_if1 route-to { ($ext_if1 $gwy_if1), \
	($ext_if2 $gwy_if2) } round-robin ... keep state
Where $ext_if1 is the interface to your default gateway, where all
connections will go out through by default. Half of them will be
re-routed out on $ext_if2, and half will go out throuh $ext_if1.
You'd use the same construct if you wanted to load-balance outgoing
connections opened by the firewall itself (say, a DNS server there),
which don't arrive in on any interface at all.
"
After that, on message http://www.benzedrine.cx/pf/msg06298.html, Daniel 
warned me that I should change my NAT rules so that packets re-routed 
could have the source address replaced by the new interfaces IP address:
"Try these NAT rules instead
  nat on $ext_if1 from !($ext_if1) to any -> ($ext_if1)
  nat on $ext_if2 from !($ext_if2) to any -> ($ext_if2)
"
This is my setup:
ext_if1:rl0 (default)
IP=200.177.74.209
GW=200.177.74.1
ext_if2:tun0 (vr0)
IP=201.9.168.157
GW=200.164.195.8
int_if1:rl1
IP=192.168.1.254
I tried these new rules. To simplify the tests I tried to establish an SSH
connection with an Internet host. What I noticed is that, when PF 
round-robin decides to re-route the packets, somehow there are
still packets trying to go out on the second interface with the source
address of the default interface. 
Also, I noticed that in some of the tests (not all of them!!!), PF is 
logging packets trying to go out on the default interface with the 
internal network address (192.168.1.21 in the logs below, for example) as 
if they had not been translated by the NAT rules.
Finally, I noticed that some packets with IP address of the second 
interface are being sent out on the default interface.
I would very much appreciate any help, hints, suggestions of new tests, 
etc.
Regards,
Emilio
----------------
Below is the log obtained from the pflog0 interface:
Feb 23 10:41:55.197509 rule 25/0(match): pass in on rl1: 192.168.1.21.1545 > 
192.168.1.254.53:  40360+[|domain]
Feb 23 10:41:55.197925 rule 25/0(match): pass out on rl1: 192.168.1.254.53 > 
192.168.1.21.1545:  40360[|domain]
Feb 23 10:41:55.202344 rule 25/0(match): pass in on rl1: 192.168.1.21.1546 > 
217.22.55.50.22: S 1633046058:1633046058(0) win 16384 
<mss 1460,nop,nop,sackOK> (DF)
Feb 23 10:41:55.202412 rule 26/0(match): pass out on rl0: 200.177.74.92.51744 >
217.22.55.50.22: S 1633046058:1633046058(0) win 16384 
<mss 1460,nop,nop,sackOK> (DF)
Feb 23 10:41:58.212738 rule 25/0(match): pass in on rl1: 192.168.1.21.1546 > 
217.22.55.50.22: S 1633046058:1633046058(0) win 16384 
<mss 1460,nop,nop,sackOK> (DF)
(here is the packet with an address from the internal network)
Feb 23 10:41:58.212763 rule 26/0(match): pass out on rl0: 192.168.1.21.1546 > 
217.22.55.50.22: S 1633046058:1633046058(0) win 16384 
<mss 1460,nop,nop,sackOK> (DF)
(here is the packet with source address of the default interface trying to 
go out on the second interface)
Feb 23 10:41:58.212782 rule 26/0(match): pass out on tun0: 200.177.74.92.51744 >
217.22.55.50.22: S 1633046058:1633046058(0) win 16384 
<mss 1460,nop,nop,sackOK> (DF)
Below is the log from the vr0 interface (second interface):
Feb 23 10:42:04.248605 0:d:87:a3:99:94 0:3:e3:5d:d3:7 8864 70: PPPoE-Session
        code Session, version 1, type 1, id 0xa319, length 50
        IP: 200.177.74.92.51744 > 217.22.55.50.22: 
	S 1633046058:1633046058(0) win 16384 
        <mss 1452,nop,nop,sackOK> (DF
Below is the log from the rl0 (default) interface:
Feb 23 10:41:55.480153 0:e0:7d:fe:cd:d3 0:5:9a:d2:34:54 0800 60: 
201.9.166.23.58478 > 217.22.55.50.22: R 1633046059:1633046059(0) win 0 (DF)
Feb 23 10:41:58.476358 0:e0:7d:fe:cd:d3 0:5:9a:d2:34:54 0800 60: 
201.9.166.23.58478 > 217.22.55.50.22: R 1633046059:1633046059(0) win 0 (DF)
Feb 23 10:42:04.475790 0:e0:7d:fe:cd:d3 0:5:9a:d2:34:54 0800 60: 
201.9.166.23.58478 > 217.22.55.50.22: R 1633046059:1633046059(0) win 0 (DF)
Below is the firewall's route table:
netstat -rnf inet
Routing tables
Internet:
Destination        Gateway            Flags     Refs     Use    Mtu  
Interface
default            200.177.74.1       UGS         3      297      -   rl0
127/8              127.0.0.1          UGRS        0        0  33224   lo0
127.0.0.1          127.0.0.1          UH          2        0  33224   lo0
192.168.1/24       link#2             UC          1        0      -   rl1
192.168.1.21       0:e0:7d:da:8f:48   UHLc        2       33      -   rl1
200.164.195.8      201.9.166.23       UH          1        0   1492   tun0
200.165.104.14     200.164.195.8      UGHS        0        6   1492   tun0
200.177.74/24      link#1             UC          1        0      -   rl0
200.177.74.1       0:5:9a:d2:34:54    UHLc        1        0      -   rl0
200.177.74.92      127.0.0.1          UGHS        0        0  33224   lo0
224/4              127.0.0.1          URS         0        0  33224   lo0
The ruleset follows:
ext_if1="rl0"
gw_if1="200.177.74.1"
gw_if2="200.164.195.8"
ext_if2="tun0"
int_if="rl1"
lan_net=$int_if:network
table <gws_if1> { $gw_if1 }
table <gws_if2> { $gw_if2 }
tcp_INservices = "{ 22 }"
icmp_types = "echoreq"
priv_nets =  "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"
# scrub
scrub in all random-id fragment reassemble
# nat/rdr
nat on $ext_if1 from !($ext_if1) to any -> ($ext_if1)
nat on $ext_if2 from !($ext_if2) to any -> ($ext_if2)
# default deny
block in log from any to any
block out log from any to any
# loopback
pass quick on lo0 all
# antispoof
block drop in log quick on $ext_if1 from $priv_nets to any
block drop out log quick on $ext_if1 from any to $priv_nets
block drop in log quick on $ext_if2 from $priv_nets to any
block drop out log quick on $ext_if2 from any to $priv_nets
# THIS IS RULE 25 mentioned in the logs 
pass in quick log-all on $int_if from $int_if:network to any keep state
# THIS IS RULE 26 mentioned in the logs above
pass out quick log-all on $ext_if1 route-to \
    { ($ext_if1 <gws_if1>) , ($ext_if2 <gws_if2>) } round-robin \
    inet proto tcp from any to any keep state
# firewall services
pass in log-all quick on $ext_if1 reply-to ($ext_if1 $gw_if1) \
   inet proto tcp from any to ($ext_if1) port $tcp_INservices \
   flags S/SA keep state 
pass in log-all quick on $ext_if2 reply-to ($ext_if2 $gw_if2) \
   inet proto tcp from any to ($ext_if2) port $tcp_INservices \
   flags S/SA keep state 
#icmpB
pass in log-all quick on $ext_if1 reply-to ($ext_if1 $gw_if1) \
   inet proto icmp all icmp-type $icmp_types keep state 
pass in log-all quick on $ext_if2 reply-to ($ext_if2 $gw_if2) \
   inet proto icmp all icmp-type $icmp_types keep state 
# outside services
pass out on $ext_if1 proto tcp all flags S/SA keep state
pass out on $ext_if2 proto tcp all flags S/SA keep state
pass out on $ext_if1 proto {udp icmp} all keep state
pass out on $ext_if2 proto {udp icmp} all keep state