[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

pfsync flooding and carp



I posted this on [email protected] but didn't get a response (posted it very early last Saturday morning. probably lost in the Monday rush). I'm about to file a bug, but thought I'd post here first.

I have two firewalls with nearly identical hardware: FirewallA has 1G
RAM and a 2.4 P4 cpu and FirewallB has 512M of RAM and a 1.8ghz P4.
Remaining parts are the same: both are Supermicro 5012B-6 servers,
Seagate SCSI hard drives, and Intel PRO/1000MT dual port cards. OS wise,
they're nearly identical configurations running 3.6-beta (Firewall B is
a Sept. 17th snapshot and FirewallA is an Aug. 18th snapshot). They
service two different DSL lines as well as have different WAN IPs and
NAT mappings (for each respective DSL line). Both firewalls are running
CARP for 2 interfaces (LAN connections) and are running pfsync across a
directly connected crossover cable. These are for a small office, so there aren't many state table entries.


The Problem:
If I demote the primary firewall, FirewallA, to backup, shortly after,
the FirewallB, now acting as the master, beguns flooding the pfsync
interface with thousands of state updates which all appear to fail:

# /home/sean> netstat -s -p pfsync
pfsync:
        3984145 packets received (IPv4)
        0 packets received (IPv6)
                0 packets discarded for bad interface
                0 packets discarded for bad ttl
                0 packets shorter than header
                0 packets discarded for bad version
                0 packets discarded for bad HMAC
                0 packets discarded for bad action
                0 packets discarded for short packet
                0 states discarded for bad values
                0 stale states
                7767614 failed state lookup/inserts
        3969354 packets sent (IPv4)
        0 packets sent (IPv6)
                0 send failed due to mbuf memory error
                80 send error

Load predictably spikes at this point as well. Eventually the machine
becomes unreachable. Switching CARP status doesn't help; I have to
filter pfsync messages for a second on the backup firewall, which stops the flood from the new master.


Anyone have idea what's going? Attached inline are dmesgs from both
machines.

thanks,
Sean

OpenBSD 3.6-beta (GENERIC) #23: Wed Aug 18 11:39:39 MDT 2004
    [email protected]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.40 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM
real mem  = 1073324032 (1048168K)
avail mem = 987521024 (964376K)
using 4278 buffers containing 53768192 bytes (52508K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(f2) BIOS, date 03/31/03, BIOS32 rev. 0 @ 0xfb330
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
pcibios0 at bios0: rev 2.1 @ 0xf0000/0xdf84
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfde90/240 (13 entries)
pcibios0: PCI Exclusive IRQs: 5 9 10 11 12
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371SB ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x5600
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82845 Host" rev 0x04
ppb0 at pci0 dev 1 function 0 "Intel 82845 AGP" rev 0x04
pci1 at ppb0 bus 1
ppb1 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x05
pci2 at ppb1 bus 2
em0 at pci2 dev 1 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03:
irq 5, address: 00:04:23:a7:b0:a4
em1 at pci2 dev 1 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03:
irq 12, address: 00:04:23:a7:b0:a5
ahc1 at pci2 dev 5 function 0 "Adaptec AIC-7899 U160" rev 0x01: irq 10
scsibus0 at ahc1: 16 targets
sd0 at scsibus0 targ 0 lun 0: <FUJITSU, MAP3367NC, 0106> SCSI3 0/direct
fixed
sd0: 35046MB, 48122 cyl, 2 head, 745 sec, 512 bytes/sec, 71775284 sec total
ahc2 at pci2 dev 5 function 1 "Adaptec AIC-7899 U160" rev 0x01: irq 11
scsibus1 at ahc2: 16 targets
fxp0 at pci2 dev 6 function 0 "Intel 82557" rev 0x08: irq 11, address
00:30:48:52:14:d8
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
fxp1 at pci2 dev 7 function 0 "Intel 82557" rev 0x08: irq 9, address
00:30:48:52:14:d9
inphy1 at fxp1 phy 1: i82555 10/100 media interface, rev. 4
vga1 at pci2 dev 8 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801BA LPC" rev 0x05
pciide0 at pci0 dev 31 function 1 "Intel 82801BA IDE" rev 0x05: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, CD-224E, 1.9A> SCSI0 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
uhci0 at pci0 dev 31 function 2 "Intel 82801BA USB" rev 0x05: irq 10
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
"Intel 82801BA SMBus" rev 0x05 at pci0 dev 31 function 3 not configured
uhci1 at pci0 dev 31 function 4 "Intel 82801BA USB2" rev 0x05: irq 11
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
lm0 at isa0 port 0x290/8: W83627HF
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ed45 netmask ff65 ttymask ffe7
pctr: user-level cycle counter enabled
ahc1: target 0 using 16bit transfers
ahc1: target 0 synchronous at 80.0MHz DT, offset = 0x7f
dkcsum: sd0 matched BIOS disk 80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02

OpenBSD 3.6 (GENERIC) #59: Fri Sep 17 12:32:57 MDT 2004
    [email protected]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 1.80GHz ("GenuineIntel" 686-class) 1.80 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM
real mem  = 536453120 (523880K)
avail mem = 482594816 (471284K)
using 4278 buffers containing 26927104 bytes (26296K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(de) BIOS, date 07/19/03, BIOS32 rev. 0 @ 0xfb330
apm0 at bios0: Power Management spec V1.2
apm0: AC on, battery charge unknown
pcibios0 at bios0: rev 2.1 @ 0xf0000/0xdf84
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfde90/240 (13 entries)
pcibios0: PCI Exclusive IRQs: 5 9 10 11 12
pcibios0: PCI Interrupt Router at 000:31:0 ("Intel 82371SB ISA" rev 0x00)
pcibios0: PCI bus #2 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x5600
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel 82845 Host" rev 0x04
ppb0 at pci0 dev 1 function 0 "Intel 82845 AGP" rev 0x04
pci1 at ppb0 bus 1
ppb1 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x05
pci2 at ppb1 bus 2
em0 at pci2 dev 1 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03:
irq 5, address: 00:04:23:a7:ad:c4
em1 at pci2 dev 1 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03:
irq 12, address: 00:04:23:a7:ad:c5
ahc1 at pci2 dev 5 function 0 "Adaptec AIC-7899 U160" rev 0x01: irq 10
scsibus0 at ahc1: 16 targets
sd0 at scsibus0 targ 0 lun 0: <FUJITSU, MAP3367NC, 0108> SCSI3 0/direct
fixed
sd0: 35046MB, 48122 cyl, 2 head, 745 sec, 512 bytes/sec, 71775284 sec total
ahc2 at pci2 dev 5 function 1 "Adaptec AIC-7899 U160" rev 0x01: irq 11
scsibus1 at ahc2: 16 targets
fxp0 at pci2 dev 6 function 0 "Intel 82557" rev 0x08: irq 11, address
00:30:48:52:94:b6
inphy0 at fxp0 phy 1: i82555 10/100 media interface, rev. 4
fxp1 at pci2 dev 7 function 0 "Intel 82557" rev 0x08: irq 9, address
00:30:48:52:94:b7
inphy1 at fxp1 phy 1: i82555 10/100 media interface, rev. 4
vga1 at pci2 dev 8 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801BA LPC" rev 0x05
pciide0 at pci0 dev 31 function 1 "Intel 82801BA IDE" rev 0x05: DMA,
channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus2 at atapiscsi0: 2 targets
cd0 at scsibus2 targ 0 lun 0: <TEAC, CD-232E, 1.0A> SCSI0 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
uhci0 at pci0 dev 31 function 2 "Intel 82801BA USB" rev 0x05: irq 10
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
"Intel 82801BA SMBus" rev 0x05 at pci0 dev 31 function 3 not configured
uhci1 at pci0 dev 31 function 4 "Intel 82801BA USB2" rev 0x05: irq 11
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
lm0 at isa0 port 0x290/8: W83627HF
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ed45 netmask ff65 ttymask ffe7
pctr: user-level cycle counter enabled
ahc1: target 0 using 16bit transfers
ahc1: target 0 synchronous at 80.0MHz DT, offset = 0x7f
dkcsum: sd0 matched BIOS disk 80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02