[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: load balancing outgoing traffic: 1st TCP connection RESET



--- Emilio Lucena <[email protected]> wrote:
> My guess is that this has something to do with TCP
> port allocation. At 
> least, this seems to be the case when I try to use a
> similar rule for 
> load-balancing ICMP traffic, like:
> 
> pass out quick log-all on $ext_if1 route-to \
>    { ($ext_if1 <gws_if1>) , ($ext_if2 <gws_if2>) } \
>    inet proto icmp keep state
> 
> ICMP traffic originated from the firewall itself is
> load-balanced without 
> a problem. However, ICMP traffic originated at my
> internal network does 
> not go through if PF chooses to re-route it through
> the second external 
> interface. And I see error messages of the type:
> 
> Apr 15 11:43:07 blt-ha /bsd: pf: NAT proxy port
> allocation (50001-65535) 
> failed
> 
> 
> Thanks in advance for any help.
> 
> Regards,
> 
> Emilio
> 
Hi
I have almost the same situation.
>So my questions are:
>
>1. Why the first TCP connection is RST'ed by PF ?
>
>2. Why the second TCP connection is established?
I think that state which is reset is from route-to
rule.In my config below you
can see that connection from rule 20.This rule cause
re-routing. The re-routed
packet then pass out through second external
interface, which also create state.This
is the answer for your 2-nd question.
The SYN_SENT:CLOSED connetion always open time window
for 120 seconds.During this interval
all packets are passed to re-routed interface.
rl0 tcp 192.168.0.11:4898 -> 213.137.58.74:57364 ->
193.110.223.9:6667       SYN_SENT:CLOSED
   [789377754 + 2]  [0 + 1]
   age 00:01:24, expires in 00:00:36, 10:0 pkts, 499:0
bytes, rule 20
   id: 4264ea7100010166 creatorid: 95f6c7b3
After those 120 second route-to rule 20 is checked
again.If connection have
enough luck 50/50 ,it will be route to the same
interface for another 120 seconds.
If not match sorry.
During my test also notice that route-to rule creates
10-15x more connection.I forgot make
copy ot this but it looks like
all states on ext_if1 25
all states on ext_if2 31
all state on rule @20 550
pfctl -vvss don't print those fantom states, but pftop
and pfctl -vvsr confirm that states
Every thing work if i use load-balancing incoming
connection on $int_if
Test PC is i386 running OpenBSD 3.6-stable
:pf.conf
ext_if2="rl1"
ext_gw2="192.168.5.1"
ext_ip2="192.168.5.199"
int_if1="xl0"
int_ip1="192.168.0.199"
ext_if1="rl0"
ext_ip1="213.137.58.74"
ext_ip1_2="10.17.2.1"
ext_gw1="213.137.58.100"
set state-policy if-bound
scrub in all
##################### Translation
####################################
# Note: Unlike filter rules, translation rules are
first match
nat on $ext_if1 from !($ext_if1) to 10.0.0.0/8 ->
$ext_ip1_2
nat on $ext_if1 from !($ext_if1) to 213.137.48.0/24 ->
$ext_ip1
nat on $ext_if1 from !($ext_if1) to any -> $ext_ip1
nat on $ext_if2 from !($ext_if2) to any -> $ext_ip2
block in log all label block_in
block out log all label block_out_all
@16	pass out on $ext_if2 proto tcp from any to any
flags S/SA modulate state queue (q_def2,q_pri2)
@17,@18	pass out on $ext_if2 proto { udp, icmp } from
any to any keep state
@19	pass in quick on $ext_if1 reply-to ($ext_if1
$ext_gw1) \
        proto icmp from any to any label
"frodo_icmp_in" keep state
@20	pass out quick on $ext_if1 route-to ($ext_if2
$ext_gw2) from any to any keep state probability 50%
@21	pass out on $ext_if1 proto tcp from $ext_ip1 to
any flags S/SA modulate state queue (q_def,q_pri)
@22,@23 pass out on $ext_if1 proto { udp, icmp } from
$ext_ip1 to any keep state
@24	pass out on $ext_if1 proto tcp from $ext_ip1 to
<freenet> flags S/SA keep state queue free-out
@25	pass out on $ext_if1 inet proto tcp from
$ext_ip1_2 to any modulate state queue free-out label
"10.17.2.1_tcp"
@26,@27	pass out on $ext_if1 proto { icmp, udp } from
$ext_ip1_2 to any keep state queue free-out label
"10.17.2.1_udp"
@28	pass out on $ext_if1 proto udp from $ext_ip1_2 to
10.10.10.11 keep state queue free2speed label
"10.17.2.1_speed"
#######################
@29	pass out on $ext_if1 route-to ($ext_if2 $ext_gw2)
from $ext_if2 to any
@30,@31 pass out on $ext_if2 route-to ($ext_if1
$ext_gw1) from $ext_if1 to any
@50	pass in quick on $int_if from <ip0-11> to any
flags S/SA keep state \
        queue (ip1-2, speed) label "redeye"
#output from pftop
tcp  In  192.168.0.11:4898     193.110.223.9:6667     
                     4:4     14 86395     18  3891  
640   640   277 50
tcp  Out 192.168.0.199:22      192.168.0.11:4757      
                     4:4     20 86395     61  6576  
254  1208   328 15
icmp In  213.137.58.101:29946  213.137.58.74:29946    
                     0:0      1     9     12   448    
          448 19
tcp  Out 192.168.0.11:4898     193.110.223.9:6667   
213.137.58.74:57364    2:0     14   106     10   499  
 75    75    35 20
tcp  Out 192.168.0.11:4898     193.110.223.9:6667   
192.168.5.199:63877    4:4      6 86395     17  3843  
 75    75   640 16
tcp  Out 213.137.58.74:57364   193.110.223.9:6667   
192.168.5.199:54699   10:10    14    76      3   136  
  0     0     9 16
# pfctl -vvss
xl0 tcp 193.110.223.9:6667 <- 192.168.0.11:4898      
ESTABLISHED:ESTABLISHED
   [2891168587 + 1403864244]  [789377838 + 5840]
   age 00:01:24, expires in 23:58:45, 10:8 pkts,
499:3392 bytes, rule 50
   id: 4264ea7100010165 creatorid: 95f6c7b3
xl0 tcp 192.168.0.199:22 -> 192.168.0.11:4757      
ESTABLISHED:ESTABLISHED
   [3325655420 + 64631]  [296635713 + 17468]
   age 00:01:30, expires in 24:00:00, 91:80 pkts,
9580:7256 bytes, rule 15
   id: 4264ea7100010164 creatorid: 95f6c7b3
rl0 tcp 192.168.0.11:4898 -> 213.137.58.74:57364 ->
193.110.223.9:6667       SYN_SENT:CLOSED
   [789377754 + 2]  [0 + 1]
   age 00:01:24, expires in 00:00:36, 10:0 pkts, 499:0
bytes, rule 20
   id: 4264ea7100010166 creatorid: 95f6c7b3
rl1 tcp 192.168.0.11:4898 -> 192.168.5.199:63877 ->
193.110.223.9:6667       ESTABLISHED:ESTABLISHED
   [789377838 + 5840](+133126236)  [939928866 +
65437](+1951239721)
   age 00:01:16, expires in 23:58:45, 9:8 pkts,
451:3392 bytes, rule 16
   id: 4264ea7100010168 creatorid: 95f6c7b3
rl1 tcp 213.137.58.74:57364 -> 192.168.5.199:54699 ->
193.110.223.9:6667       TIME_WAIT:TIME_WAIT
   [789377754 + 5841](+1233729573)  [936714380 +
65535](+3894880133)
   age 00:01:24, expires in 00:00:06, 2:1 pkts, 88:48
bytes, rule 16
   id: 4264ea7100010167 creatorid: 95f6c7b3
#
#pfctl -xm
#cat /var/log/messages
Apr 20 00:38:12 frodo /bsd: pf: state insert failed:
tree_ext_gwy lan: 213.137.58.74:61820 gwy:
213.137.58.74:61820 ext: 66.246.231.101:80
Apr 20 00:38:18 frodo /bsd: pf: state insert failed:
tree_ext_gwy lan: 213.137.58.74:58603 gwy:
213.137.58.74:58603 ext: 66.246.231.101:80
Apr 20 04:10:35 frodo /bsd: pf: state insert failed:
tree_ext_gwy lan: 213.137.58.74:63202 gwy:
213.137.58.74:63202 ext: 193.110.223.9:6667
Unix is very simple, but it takes a genius to understand the simplicity.
(Dennis Ritchie)
Key fingerprint=2499 DE87 82ED 23A8 FD20 3078 04FE 610E 300D 6655
__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com