[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: rdr not redirecting when target is localhost



I haven't found a solution to my problem (below) but I have more
data and a short-term workaround.
The problem seems to be that redirection with rdr does not
work in true transparent bridge mode, when the target of the
rdr is on the localhost interface of the bridge.  Although
reading all the docs (I've got through a pile of printouts
about an inch thick) generally gives the impression that
this ought to work, no-where does it ever say so explicitly.
All the examples I have found anywhere of redirection have
either been for machines configured with IP addresses on
the interfaces and running single-IP NAT, or they have been
redirecting to IP addresses on the internal interface network.
I've also researched old mailing list postings and have found
a handful of people asking about redirection under a transparent
bridge/firewall but not one of them received an answer or ever
followed up to say they'd done it successfully.
I'm slowly coming to the conclusion that it is not my ignorance
of pf which is the problem but that what I want to do may not
actually be possible.
There is actually a hint of what the underlying problem may be in
the section "Redirection and Reflection" of the rdr.html section
of the pf FAQ.  It's not quite the same case but it is similar:
	"Redirection and Reflection
	Often, redirection rules are used to forward incoming
	connections from the Internet to a local server with
	a private address in the internal network or LAN, as in: 
		server = 192.168.1.40
		rdr on $ext_if proto tcp from any to $ext_if port 80 -> $server \
		   port 80
	But when the redirection rule is tested from a client 
	on the LAN, it doesn't work. The reason is that redirection
	rules apply only to packets that pass through the specified
	interface ($ext_if, the external interface, in the example).
	Connecting to the external address of the firewall from a
	host on the LAN, however, does not mean the packets will
	actually pass through its external interface. The TCP/IP stack
	on the firewall compares the destination address of incoming
	packets with its own addresses and aliases and detects
	connections to itself as soon as they have passed the internal
	interface. Such packets do not physically pass through the
	external interface, and the stack does not simulate such a
	passage in any way. Thus, PF never sees these packets on the
	external interface, and the redirection rule, specifying the
	external interface, does not apply. 
	Adding a second redirection rule for the internal interface
	does not have the desired effect either. When the local client
	connects to the external address of the firewall, the initial
	packet of the TCP handshake reaches the firewall through the
	internal interface. The redirection rule does apply and the
	destination address gets replaced with that of the internal
	server. The packet gets forwarded back through the internal
	interface and reaches the internal server. But the source address
	has not been translated, and still contains the local client's
	address, so the server sends its replies directly to the
	client. The firewall never sees the reply and has no chance
	to properly reverse the translation. The client receives a
	reply from a source it never expected and drops it. The TCP
	handshake then fails and no connection can be established. "
I don't know if this is the same problem but maybe the quote
above might give someone an idea of where to start?
The workaround I have applied until a pf guru can tell me the
proper way to fix this, is to renumber my internal mail server
with a private address, put a private address on the internal
interface of my greylisting box, give the mailserver's real address
to the public interface of my greylisting box, and turn on
NAT.  This will undoubtedly have unforseen consequences and
is a management nightmare as far as the real mailserver itself
is concerned, but it does work and means I can get my server
into production for now.
My test dev environment now looks like this:
 +------------------+          +-------------+          +---------------+
 |                  |          |             |          |               | |                  |          |192.168.1.1  |          |   (bogus IP)  | |      192.168.1.6 >---------/      NAT      /---------< 129.113.1.99  | |  (mail.)utpa.edu |          |  129.113.1.6|          |(outside.)     | |underlying server |          |             |          | somewhere.com | +------------------+          +-------------+          +---------------+
 Dummy server                  Greylister               Dummy client
                               impersonating
                               mail.utpa.edu
If this group isn't where the pf developers hang out, please tell
me where I should go to ask my question.  I think I need guru-level
help on this one :-/
It may be that pf lacks a feature, which is that on a redirection,
it needs to be told not only the rewritten IP and port, but also the
interface to which to send the packet?
Thanks
Graham Toal <[email protected]>
--------------------------------------------------------
> From [email protected] Sat Oct  8 09:17:45 2005
> Date: Sat, 08 Oct 2005 09:00:45 -0500
> From: Graham Toal <[email protected]>
> To: [email protected]
> Subject: rdr not redirecting when target is localhost
>
> This is my second adventure with pf.  The first was setting
> up spamd on a system with an IP - not the preferred solution
> but one that was forced on me at the time because I didn't
> have physical access to the mailserver in a way that would
> allow me to put a transparent bridge in front of it.  Now I do,
> and so I'm upgrading our greylister to use transparent bridging
> - but I've hit a problem which has driven me crazy for 2 days
> now but which I'm hoping will be obvious to the folks here...
>
>
> The short summary first: I'm pretty sure I've set up pf
> properly - I've tried the out-of-the-box config, a simple
> 3-line very basic config, many example configs lifted from
> web pages - they all behave the same way.  Which is that when
> I redirect port 25 connections to port 8025 on localhost, the
> rewritten packets go out via $int_if rather than lo0.  I've
> confirmed this carefully with the log option, pflog0, and
> tcpdumping of int_if and ext_if and lo0.
>
>
> My guess at the moment is that there is something external to pf.conf
> that I have missed.  I've turned on ip forwarding as per the docs,
> but not ip6 forwarding because we don't use it here.  I've set
> up the bridge interfaces, and I've enabled pf.
>
> At the moment I'm not even looking at the use of tables - I'm
> trying to redirect *all* smtp connections while testing, just to
> keep it simple.  If I set up the rdr to redirect -> $server_ip 25
> rather than -> 127.0.0.1 8025, that actually works.  So the problem
> is not in the redirect rule matching the incoming packets, but in
> the fact that the redirection is to localhost.
>
>
> If that wasn't enough for someone to spot the problem, well,
> here's the long version:
>
>
> 1) I've set up three machines in a test lab, not connected to
>    the internet; one pretends to be our central mailer, another
>    is a bogus dotcom; between them sits the OpenBSD configured
>    as a transparent bridge.  The BSD has a third interface with
>    an IP so I can work on it from home over the weekend, but I'm
>    fairly sure that this is not involved in the issues I'm seeing.
>    fxp1 and xl0 are nowhere to be found in netstat -nr (below)
>    and fxp0 is never referenced at all in pf.conf.
>
>
>                              129.113.28.220(real IP)
>                                    |> +----------------+          +------^------+          +---------------+
> |                |          |     \_/     |          |               |> |     (bogus IP) |          |             |          |   (bogus IP)  |> |    129.113.1.6 >----------<.............>----------< 129.113.1.99  |> |(mail.)utpa.edu |     no ip|   BRIDGE    |no ip     |(outside.)     |> |                |          |             |          | somewhere.com |> +----------------+          +-------------+          +---------------+
> Dummy server                Greylister               Dummy client
>
>
> I have set up the fake server and the fake client with fake DNS that
> allows them to send mail backward and forward from utpa.edu and
> somewhere.com.  I've tested this and monitored the data flowing through
> the bridge interface on the OpenBSD system, with tcpdump.
>
>
> Here are some sketchy notes of what I did on the OpenBSD which is
> freshly installed from the current release over the net (3.7), just
> to check that I did all the major configs:
>
> ========================================================================
>
> network interfaces are xl0 (card) fxp0 (onboard, left as viewed from front
> above), fxp1
> - select fxp0 for internet (this lets us distinguish IN and OUT of the bridge
> more easily)
>
>
> edit /etc/sysctl.conf to add
> net.inet.ip.forwarding=1
> kern.emul.linux=1
>
> set ntpd_flags="" in /etc/rc.conf (takes effect on reboot; meanwhile, issue
> /usr/sbin/ntpd)
>
> set up these files in /etc:
>
> echo up>hostname.xl0
> echo up>hostname.fxp1
>
> (hostname.fxp0 should already be set up)
>
> cat > bridgename.bridge0 <<EOD
> add xl0
> add fxp1
> up
> EOD
>
> Test with "brconfig bridge0 add xl0 add fxp1 up". Rebooting makes that
> permanent when pf is enabled:
>
> edit /etc/rc.conf to set pf=YES
>
> /etc/pf.conf:
>
> uncomment all lines
>
> remove "block in" and replace with "pass in keep state" - don't
> want m/c configured as firewall, just as open transparent bridge
> which intercepts only port 25.
>
> edit /etc/rc.conf.local:
>
> spamd_flags="-v -G 1:24:864"
> spamd_grey=YES          # use spamd greylisting if YES
>
> reboot
> ======================================================================
>
> I've tried many pf.conf files including the stock on; I've tried them
> with the default firewall config enabled and with firewalling completely
> turned off.  The other ports work fine, it's just the port 25 redirection
> that is problematic.  The simplest config was something like this:
>
> ext_if="xl0"
> rdr pass on $ext_if proto tcp from any to any port smtp -> 127.0.0.1 port spamd
>
> and the current one is this:
>
> ext_if="xl0"
> int_if="fxp1"
>
> scrub in
>
> nat on $ext_if from !($ext_if) -> ($ext_if:0)
> rdr pass on $ext_if proto tcp from any to any port smtp -> 127.0.0.1 port spamd
>
> #block in
> pass out log keep state
>
> pass log quick on { lo $int_if }
> antispoof log quick for { lo $int_if }
>
> pass in log on $ext_if proto tcp to ($ext_if) port smtp keep state
> pass out log on $ext_if proto tcp from ($ext_if) to port smtp keep state
>
> ======================================================================
>
> this is what I'm seeing in the log for pflog0:
>
> ## 07:11:09.445062 129.113.1.99.46981 > 127.0.0.1.8025: S 961513652:961513652(0) win 5840 <mss 1460,sackOK,timestamp[|tcp]> (DF) [tos0x10]
> ## 07:11:33.438319 129.113.1.99.46981 > 127.0.0.1.8025: S 961513652:961513652(0) win 5840 <mss 1460,sackOK,timestamp[|tcp]>(DF) [tos 0x10]
>
> and for xl0: (ext_if)
>
> ## 07:11:09.445019 129.113.1.99.46981 > 129.113.1.6.25: S 961513652:961513652(0) win 5840 <mss 1460,sackOK,timestamp 6943078 70,nop,wscale 2> (DF) [tos 0x10]
> ## 07:11:33.438286 129.113.1.99.46981 > 129.113.1.6.25: S 961513652:961513652(0) win 5840 <mss 1460,sackOK,timestamp 69454787 0,nop,wscale 2> (DF) [tos 0x10]
>
> and for fxp1: (int_if)
>
> ## 07:15:30.076828 129.113.1.99.41109 > 127.0.0.1.8025: S 1258620181:1258620181(0) win 5840 <mss 1460,sackOK,timestamp 69691492 0,nop,wscale 2> (DF) [tos 0x10]
> ## 07:15:36.075129 129.113.1.99.41109 > 127.0.0.1.8025: S 1258620181:1258620181(0) win 5840 <mss 1460,sackOK,timestamp 69697492 0,nop,wscale 2> (DF) [tos 0x10]
>
> nothing shows up on lo0.
>
> ======================================================================
> greylist1# netstat -nr
> Routing tables
>
> Internet:
> Destination Gateway Flags Refs Use Mtu
> Interface
> default            129.113.28.1       UGS        16    45806      -   fxp0
> 127/8              127.0.0.1          UGRS        0        0  33224   lo0
> 127.0.0.1          127.0.0.1          UH          2      167  33224   lo0
> 129.113.28/24      link#2             UC          3        0      -   fxp0
> 129.113.28.1       aa:0:4:0:ff:f      UHLc        1        0      -   fxp0
> 129.113.28.75      0:10:5a:29:62:b3   UHLc        2     1697      -   fxp0
> 129.113.28.220     127.0.0.1          UGHS        0       11  33224   lo0
> 129.113.28.221     0:4:76:cf:bb:cb    UHLc        1     1370      -   fxp0
> 224/4              127.0.0.1          URS         0        0  33224   lo0
>
> Internet6:
> Destination Gateway Flags Refs     Use    Mtu  Interface
> ::/104 ::1 UGRS 0        0      -   lo0 =>
> ::/96 ::1 UGRS 0        0      -   lo0
> ::1 ::1 UH 12       76  33224   lo0
> ::127.0.0.0/104 ::1 UGRS 0        0      -   lo0
> ::224.0.0.0/100 ::1 UGRS 0        0      -   lo0
> ::255.0.0.0/104 ::1 UGRS 0        0      -   lo0
> ::ffff:0.0.0.0/96 ::1 UGRS 0        0      -   lo0
> 2002::/24 ::1 UGRS 0        0      -   lo0
> 2002:7f00::/24 ::1 UGRS 0        0      -   lo0
> 2002:e000::/20 ::1 UGRS 0        0      -   lo0
> 2002:ff00::/24 ::1 UGRS 0        0      -   lo0
> fe80::/10 ::1 UGRS 0        0      -   lo0
> fe80::%xl0/64 link#1 UC 0        0      -   xl0
> fe80::204:76ff:fecf:bd5d%xl0 0:4:76:cf:bd:5d UHL 0        0      -   lo0
> fe80::%fxp0/64 link#2 UC 0        0      -   fxp0
> fe80::2c0:9fff:fe04:7a19%fxp0 0:c0:9f:4:7a:19 UHL 0        0      -   lo0
> fe80::%fxp1/64 link#3 UC 0        0      -   fxp1
> fe80::2c0:9fff:fe04:7a18%fxp1 0:c0:9f:4:7a:18 UHL 0        0      -   lo0
> fe80::%lo0/64 fe80::1%lo0 U 0        0      -   lo0
> fe80::1%lo0 link#7 UHL 0        0      -   lo0
> fec0::/10 ::1 UGRS 0        0      -   lo0
> ff01::/32 ::1 UC 0        0      -   lo0
> ff02::%xl0/32 link#1 UC 0        0      -   xl0
> ff02::%fxp0/32 link#2 UC 0        0      -   fxp0
> ff02::%fxp1/32 link#3 UC 0        0      -   fxp1
> ff02::%lo0/32 ::1 UC 0        0      -   lo0
>
> Encap:
> Source Port Destination Port Proto SA(Address/Proto/Type/Direction)
>
> ======================================================================
>
> I'd much appreciate any pointers anyone can give me.  I'ld like to
> start with a guaranteed minimal pf file that does only one thing:
> reroutes all connections in on 25 to localhost 8025.  If I can get
> that working, I can take care of all of the rest myself.
>
> If you can give me a known working pf file for the above configuration
> and it *doesn't* work, I need pointers on where to look elsewhere
> for the problem, because I'm all out of ideas.  Let's just say
> Google is no longer my best friend :-)  I think I've read every
> pf page on the net...
>
>
> Thanks in advance,
>
>
> Graham <[email protected]>
>