[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

State searches sky rocket / Firewall dies



I've sent a very similar message to [email protected] on the 5th, and didn't find 
any help.
Today my client asked me to unplug my OpenBSD firewall because of the 
following problem. Any help, ideas, thoughts, etc. on this will be most 
appreciated as currently neither OpenBSD or I are looking very good in 
the eyes of this client. Prior to this problem a OBSD 3.5 firewall 
using the exact same ruleset (on slower hardware) was in, and didn't 
have any such problems.
The message I sent to [email protected]:
Have a 3.6 firewall/bridge that every once in a while (3 times so far 
today) will drop numerous packets for about 5 to 10 minutes, only to 
come back fine again.
I turned PF debugging to 'misc', and what I saw in the logs just prior 
to one of these episodes was:
Dec 15 13:02:32 baracus /bsd: pf: State failure on: ? ? ? ? |Dec 15 13:02:32 baracus /bsd: pf: BAD state: TCP 216.194.85.40:80 
216.194.85.40:80 216.79.119.184:1998 [lo=1975532775 high=1975549336 
win=65535 modulator=0]
?[lo=1392693791 high=1392759326 win=16560 modulator=0] 4:2 RA 
seq=1975532775 ack=1392693791 len=0 ackskew=0 pkts=1:2 dir=in,fwd
Dec 15 13:02:32 baracus /bsd: pf: State failure on: ? ? ? ? |Dec 15 13:02:32 baracus /bsd: pf: BAD state: TCP 216.194.85.40:80 
216.194.85.40:80 216.79.119.184:2013 [lo=2974235582 high=2974252143 
win=65535 modulator=0]
?[lo=151789620 high=151855155 win=16560 modulator=0] 4:2 RA 
seq=2974235582 ack=151789620 len=0 ackskew=0 pkts=1:2 dir=in,fwd
Dec 15 13:02:32 baracus /bsd: pf: State failure on: ? ? ? ? |Dec 15 13:02:32 baracus /bsd: pf: BAD state: TCP 216.194.85.40:80 
216.194.85.40:80 216.79.119.184:2011 [lo=2230903065 high=2230919626 
win=65535 modulator=0]
?[lo=1017748228 high=1017813763 win=16560 modulator=0] 4:2 RA 
seq=2230903065 ack=1017748228 len=0 ackskew=0 pkts=1:2 dir=in,fwd
Which is just a very very small piece of all the errors that happened at 
13:02, give or take a couple of seconds.
The other thing I noticed was the state table stayed relatively empty, 
however number of searches/second reported skyrocketed. It may just be 
that I wasn't able to catch it at the right time.
The two things I tried were 'set limit states 40000' and
'set optimization aggressive'. Which for all I know may have put off 
this happening, but certainly didn't prevent it.
Any direction on this would be appreciated. This server is 100% a 
firewall, with nothing else other than sshd running.
dmesg (from messages, sorry):
Dec 10 14:47:38 baracus /bsd: OpenBSD 3.6 (GENERIC) #59: Fri Sep 17 
12:32:57 MDT 2004
Dec 10 14:47:38 baracus /bsd: ? ? 
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC
Dec 10 14:47:38 baracus /bsd: cpu0: Intel Pentium III ("GenuineIntel" 
686-class, 128KB L2 cache) 952 MHz
Dec 10 14:47:38 baracus /bsd: cpu0: 
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
Dec 10 14:47:38 baracus /bsd: real mem ?= 132685824 (129576K)
Dec 10 14:47:38 baracus /bsd: avail mem = 114442240 (111760K)
Dec 10 14:47:38 baracus /bsd: using 1645 buffers containing 6737920 
bytes (6580K) of memory
Dec 10 14:47:38 baracus /bsd: mainbus0 (root)
Dec 10 14:47:38 baracus /bsd: bios0 at mainbus0: AT/286+(00) BIOS, date 
10/09/01, BIOS32 rev. 0 @ 0xfdb70
Dec 10 14:47:38 baracus /bsd: pcibios0 at bios0: rev 2.1 @ 
0xf0000/0x10000
Dec 10 14:47:38 baracus /bsd: pcibios0: PCI IRQ Routing Table rev 1.0 @ 
0xf6a20/80 (3 entries)
Dec 10 14:47:38 baracus /bsd: pcibios0: PCI Interrupt Router at 000:31:0 
("Intel 82801AA LPC" rev 0x00)
Dec 10 14:47:38 baracus /bsd: pcibios0: PCI bus #2 is the last bus
Dec 10 14:47:38 baracus /bsd: bios0: ROM list: 0xc0000/0x8000
Dec 10 14:47:38 baracus /bsd: cpu0 at mainbus0
Dec 10 14:47:38 baracus /bsd: pci0 at mainbus0 bus 0: configuration mode 
1 (no bios)
Dec 10 14:47:38 baracus /bsd: pchb0 at pci0 dev 0 function 0 "Intel 
82810" rev 0x03: rng active, 9Kb/sec
Dec 10 14:47:38 baracus /bsd: vga1 at pci0 dev 1 function 0 "Intel 82810 
Graphics" rev 0x03: aperture at 0xdc000000, size 0x4000000
Dec 10 14:47:38 baracus /bsd: wsdisplay0 at vga1: console (80x25, vt100 
emulation)
Dec 10 14:47:38 baracus /bsd: wsdisplay0: screen 1-5 added (80x25, vt100 
emulation)
Dec 10 14:47:38 baracus /bsd: ppb0 at pci0 dev 30 function 0 "Intel 
82801AA Hub-to-PCI" rev 0x02
Dec 10 14:47:38 baracus /bsd: pci1 at ppb0 bus 1
Dec 10 14:47:38 baracus /bsd: ppb1 at pci1 dev 3 function 0 "DEC 21152 
PCI-PCI" rev 0x03
Dec 10 14:47:38 baracus /bsd: pci2 at ppb1 bus 2
Dec 10 14:47:38 baracus /bsd: fxp0 at pci2 dev 4 function 0 "Intel 
82557" rev 0x05: irq 11, address 00:03:47:08:e2:61
Dec 10 14:47:38 baracus /bsd: inphy0 at fxp0 phy 1: i82555 10/100 media 
interface, rev. 0
Dec 10 14:47:38 baracus /bsd: fxp1 at pci2 dev 5 function 0 "Intel 
82557" rev 0x05: irq 10, address 00:03:47:08:e2:62
Dec 10 14:47:38 baracus /bsd: inphy1 at fxp1 phy 1: i82555 10/100 media 
interface, rev. 0
Dec 10 14:47:38 baracus /bsd: ichpcib0 at pci0 dev 31 function 0 "Intel 
82801AA LPC" rev 0x02
Dec 10 14:47:38 baracus /bsd: pciide0 at pci0 dev 31 function 1 "Intel 
82801AA IDE" rev 0x02: DMA, channel 0 wired to compatibility, channel 1 
wired to comp
atibility
Dec 10 14:47:38 baracus /bsd: wd0 at pciide0 channel 0 drive 0: <Maxtor 
2B010H1>
Dec 10 14:47:38 baracus /bsd: wd0: 16-sector PIO, LBA, 9771MB, 20012832 
sectors
Dec 10 14:47:38 baracus /bsd: wd0(pciide0:0:0): using PIO mode 4, 
Ultra-DMA mode 4
Dec 10 14:47:38 baracus /bsd: pciide0: channel 1 disabled (no drives)
Dec 10 14:47:38 baracus /bsd: uhci0 at pci0 dev 31 function 2 "Intel 
82801AA USB" rev 0x02: irq 12
Dec 10 14:47:38 baracus /bsd: usb0 at uhci0: USB revision 1.0
Dec 10 14:47:38 baracus /bsd: uhub0 at usb0
Dec 10 14:47:38 baracus /bsd: uhub0: Intel UHCI root hub, class 9/0, rev 
1.00/1.00, addr 1
Dec 10 14:47:38 baracus /bsd: uhub0: 2 ports with 2 removable, self 
powered
Dec 10 14:47:38 baracus /bsd: "Intel 82801AA SMBus" rev 0x02 at pci0 dev 
31 function 3 not configured
Dec 10 14:47:38 baracus /bsd: auich0 at pci0 dev 31 function 5 "Intel 
82801AA AC97" rev 0x02: irq 10, ICH AC97
Dec 10 14:47:38 baracus /bsd: ac97: codec id 0x83847609 (SigmaTel 
STAC9721/23)
Dec 10 14:47:38 baracus /bsd: ac97: codec features 18 bit DAC, 18 bit 
ADC, SigmaTel 3D
Dec 10 14:47:38 baracus /bsd: audio0 at auich0
Dec 10 14:47:38 baracus /bsd: isa0 at ichpcib0
Dec 10 14:47:38 baracus /bsd: isadma0 at isa0
Dec 10 14:47:38 baracus /bsd: pckbc0 at isa0 port 0x60/5
Dec 10 14:47:38 baracus /bsd: pckbd0 at pckbc0 (kbd slot)
Dec 10 14:47:38 baracus /bsd: pckbc0: using irq 1 for kbd slot
Dec 10 14:47:38 baracus /bsd: wskbd0 at pckbd0: console keyboard, using 
wsdisplay0
Dec 10 14:47:38 baracus /bsd: pcppi0 at isa0 port 0x61
Dec 10 14:47:38 baracus /bsd: midi0 at pcppi0: <PC speaker>
Dec 10 14:47:38 baracus /bsd: sysbeep0 at pcppi0
Dec 10 14:47:38 baracus /bsd: lpt0 at isa0 port 0x378/4 irq 7
Dec 10 14:47:38 baracus /bsd: lm0 at isa0 port 0x290/8: W83627HF
Dec 10 14:47:38 baracus /bsd: npx0 at isa0 port 0xf0/16: using exception 
16
Dec 10 14:47:38 baracus /bsd: pccom0 at isa0 port 0x3f8/8 irq 4: 
ns16550a, 16 byte fifo
Dec 10 14:47:38 baracus /bsd: fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
Dec 10 14:47:38 baracus /bsd: fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 
head, 18 sec
Dec 10 14:47:38 baracus /bsd: biomask f36d netmask ff6d ttymask ffef
Dec 10 14:47:38 baracus /bsd: pctr: 686-class user-level performance 
counters enabled
Dec 10 14:47:38 baracus /bsd: mtrr: Pentium Pro MTRR support
Dec 10 14:47:38 baracus /bsd: dkcsum: wd0 matched BIOS disk 80
Dec 10 14:47:38 baracus /bsd: root on wd0a
Dec 10 14:47:38 baracus /bsd: rootdev=0x0 rrootdev=0x300 rawdev=0x302
Dec 10 14:47:38 baracus savecore: no core dump
pf.conf, slightly sanitized:
$ sudo cat /etc/pf.conf
###################################################
## MACROS
###################################################
## Settings
###########
set limit states 40000
set optimization aggressive
set debug misc
## Interfaces
#############
# External to internet (bridge0)
ext_if = "fxp0"
# Internal to lan (bridge0)
int_if = "fxp1"
# Loopback interface
lpb_if = "lo0"
## Servers
##########
# baracus
# OpenBSD 3.6
baracus = "216.194.85.46"
# mightythor.advantcomp.com
mightythor1 = "216.194.85.43"
mightythor2 = "216.194.85.44"
mightythor = "{" $mightythor1 $mightythor2 "}"
# Windows 2000 Server
netdisciple1 = "216.194.85.34"
netdisciple2 = "216.194.85.35"
netdisciple = "{" $netdisciple1 $netdisciple2 "}"
# Windows 2003 Server
bizminerdb = "216.194.85.45"
# FreeBSD 4.8
boudica1 = "216.194.85.48"
boudica2 = "216.194.85.50"
boudica = "{" $boudica1 $boudica2 "}"
# Windows 2003 Server
ardvark = "216.194.85.40"
# Redhat Linux 9.0
locnet1 = "216.194.85.37"
locnet2 = "216.194.85.39"
# OpenBSD 3.5
gak1 = "216.194.85.41"
gak2 = "216.194.85.42"
# Crap
devadv = "216.194.85.51"
## Private IPs
##############
private = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, 10.0.0.0/8 }"
###################################################
## OPTIONS
###################################################
set loginterface fxp0
set optimization normal
set block-policy drop
###################################################
## NORMALIZATION
###################################################
# scrub incoming packets
scrub in on $ext_if all
###################################################
## TRANSLATION
###################################################
# redirect connections to boudica from any on port 26 to
# port 25 on the server
rdr on $ext_if proto tcp from any to $boudica1 port 26 -> $boudica1 port 
25
###################################################
## FILTER RULES
###################################################
## Pass in/out on loopback interface
####################################
pass in quick on $lpb_if all
pass out quick on $lpb_if all
## Filter on external interface
###############################
# default deny in and log
block in log on $ext_if all
#pass in quick on $ext_if all
# stop spoofing attempts and log them
block in quick log on $ext_if from $private to any
block out quick log on $ext_if from any to $private
# let everything out that isn't spoofed
pass out quick on $ext_if from any to any keep state
# baracus
pass in quick on $ext_if proto tcp from any to $baracus port 22 keep 
state label BARACUS
# 
pass in quick on $ext_if proto tcp from any to $netdisciple port { 21 25 
80 110 366 443 993 995 3389 9999 } flags S/SA keep state label 
NETDISCIPLE
# 
pass in quick on $ext_if proto tcp from any to $mightythor port { 21 80 
443 3389 } flags S/SA keep state label MIGHTYTHOR
# 
# no external access except through authpf rules
block in quick on $ext_if from 64.41.168.243 to any
pass in quick on $ext_if proto tcp from any to $bizminerdb port 3389 
flags S/SA keep state
# 
pass in quick on $ext_if proto tcp from any to $boudica port { 21 22 25 
69 80 110 143 993 } flags S/SA keep state label BOUDICA
pass in quick on $ext_if proto udp from any to $boudica port 69 keep 
state label BOUDICA
# 
pass in quick on $ext_if proto tcp from any to $ardvark port { 21 80 
1433 3389 } flags S/SA keep state label ARDVARK
# 
pass in quick on $ext_if proto tcp from any to $locnet2 port { 21 22 
80 } flags S/SA keep state label LOCNET
pass in quick on $ext_if proto udp from any to $locnet2 port 53 keep 
state label LOCNET
# 
# - Explicitly block people to Gak
block in quick on $ext_if from 208.166.208.214 to $gak1 label GAK
# - Allow standard services
pass in quick on $ext_if proto tcp from any to $gak1 port { 21 22 25 80 
110 143 443 993 } flags S/SA keep state label GAK
pass in quick on $ext_if proto tcp from any to $gak1 port 49152 >< 65535 
flags S/SA keep state
pass in quick on $ext_if proto udp from any to $gak1 port 53 keep state 
label GAK
pass in quick on $ext_if proto tcp from any to $gak2 port { 80 443 } 
flags S/SA keep state label GAK
# - Special People allowed in
pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 5432 
flags S/SA keep state label GAK
pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 5432 
flags S/SA keep state label GAK
pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 5432 
flags S/SA keep state label GAK
pass in quick on $ext_if proto tcp from xx.xx.xx.xx to $gak1 port 3306 
flags S/SA keep state label GAK
#
pass in quick on $ext_if proto tcp from any to $devadv port { 21 80 443 
1433 3389 } flags S/SA keep state label DEVADV
# allow useful ICMP packets for ping
pass in quick on $ext_if inet proto icmp all icmp-type echoreq keep 
state
# anchor for authpf to allow certain users special access
anchor authpf in on $ext_if
block in log on $ext_if all
## Pass in/out on internal interface
####################################
pass in quick on $int_if all
pass out quick on $int_if all
General PF stats:
INFO:
Status: Enabled for 0 days 00:00:11 ? ? ? ? ? ?Debug: Misc
Hostid: 0xd40ec4ab
Interface Stats for fxp0 ? ? ? ? ? ? ?IPv4 ? ? ? ? ? ? IPv6
? Bytes In ? ? ? ? ? ? ? ? ? ? ?1779014344 ? ? ? ? ? ? ? ?0
? Bytes Out ? ? ? ? ? ? ? ? ? ?12435348156 ? ? ? ? ? ? ?352
? Packets In
? ? Passed ? ? ? ? ? ? ? ? ? ? ? ?12134747 ? ? ? ? ? ? ? ?0
? ? Blocked ? ? ? ? ? ? ? ? ? ? ? ? 464176 ? ? ? ? ? ? ? ?0
? Packets Out
? ? Passed ? ? ? ? ? ? ? ? ? ? ? ?16979477 ? ? ? ? ? ? ? ?0
? ? Blocked ? ? ? ? ? ? ? ? ? ? ? ? 297329 ? ? ? ? ? ? ? ?5
State Table ? ? ? ? ? ? ? ? ? ? ? ? ?Total ? ? ? ? ? ? Rate
? current entries ? ? ? ? ? ? ? ? ? ? ?162
? searches ? ? ? ? ? ? ? ? ? ? ? ?59072275 ? ? ? ?57800.7/s
? inserts ? ? ? ? ? ? ? ? ? ? ? ? ?2416059 ? ? ? ? 2364.0/s
? removals ? ? ? ? ? ? ? ? ? ? ? ? 2415897 ? ? ? ? 2363.9/s
Counters
? match ? ? ? ? ? ? ? ? ? ? ? ? ? 32351974 ? ? ? ?31655.6/s
? bad-offset ? ? ? ? ? ? ? ? ? ? ? ? ? ? 0 ? ? ? ? ? ?0.0/s
? fragment ? ? ? ? ? ? ? ? ? ? ? ? ? ? 126 ? ? ? ? ? ?0.1/s
? short ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 23 ? ? ? ? ? ?0.0/s
? normalize ? ? ? ? ? ? ? ? ? ? ? ? ? ?196 ? ? ? ? ? ?0.2/s
? memory ? ? ? ? ? ? ? ? ? ? ? ? ? ?295524 ? ? ? ? ?289.2/s
? bad-timestamp ? ? ? ? ? ? ? ? ? ? ? ? ?0 ? ? ? ? ? ?0.0/s
TIMEOUTS:
tcp.first ? ? ? ? ? ? ? ? ? 120s
tcp.opening ? ? ? ? ? ? ? ? ?30s
tcp.established ? ? ? ? ? 86400s
tcp.closing ? ? ? ? ? ? ? ? 900s
tcp.finwait ? ? ? ? ? ? ? ? ?45s
tcp.closed ? ? ? ? ? ? ? ? ? 90s
tcp.tsdiff ? ? ? ? ? ? ? ? ? 30s
udp.first ? ? ? ? ? ? ? ? ? ?60s
udp.single ? ? ? ? ? ? ? ? ? 30s
udp.multiple ? ? ? ? ? ? ? ? 60s
icmp.first ? ? ? ? ? ? ? ? ? 20s
icmp.error ? ? ? ? ? ? ? ? ? 10s
other.first ? ? ? ? ? ? ? ? ?60s
other.single ? ? ? ? ? ? ? ? 30s
other.multiple ? ? ? ? ? ? ? 60s
frag ? ? ? ? ? ? ? ? ? ? ? ? 30s
interval ? ? ? ? ? ? ? ? ? ? 10s
adaptive.start ? ? ? ? ? ? ? ?0 states
adaptive.end ? ? ? ? ? ? ? ? ?0 states
src.track ? ? ? ? ? ? ? ? ? ? 0s
LIMITS:
states ? ? hard limit ?40000
src-nodes ?hard limit ?10000
frags ? ? ?hard limit ? 5000