[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Max table size... The 'pfctl Cannot allocate memory' issue



On 10/11, Daniel Hartmeier wrote:
> On Mon, Oct 11, 2004 at 07:01:45PM +0200, Csillag Tamas wrote:
> 
> > The problem is if I want to load NJABL RBL list to <spamd> I get
> > pfctl Cannot allocate memory.
> 
> That list is well over 2'000'000 entries large, if I understand their
> web page correctly.
Hmm, you are right I have just realized that!
In fact it is:
/tmp# wc -l njabl.ip 
 1982219 njabl.ip
 
> 
> On i386, each table entry is 156 bytes. That means the entire table
> would need over 297MB of kernel memory. Even with the changes that went
> in, that's too much.
If I remember correctly Cedric said that his patch allow to use all the
(768Mb) kernel memory. In that case I only need ~300Mb and that leave
~~500Mb free.
Right now top tells:
Memory: 98M/358M act/tot  Free: 646M  Swap: 0K/1248M used/tot
(With dynamic-njabl and cbl loaded 615'491 + 1'060'340)
I would like to use that 646Mb if it is possible.
It is only a pure (statefull) packet filter firewall (only ftp-proxy at user level)
So that memory won't be missing.
> On macppc, I can load 1'000'000 entries of 160
> bytes each, costing 152MB of kernel memory.
> 
> You could start shaving off some fat from struct pfr_kentry:
> 
> struct pfr_kentry {
> 	...
>         u_int64_t        pfrts_packets[PFR_DIR_MAX][PFR_OP_TABLE_MAX];
>         u_int64_t        pfrts_bytes[PFR_DIR_MAX][PFR_OP_TABLE_MAX];
> 	...
> };
> 
> That's 2*2*3*8 == 96 bytes just for the packet/bytes counters (shown by
> pfctl -t -vvTs). If you don't care for those, remove the counters (and
> adjust pfctl so it doesn't rely on them being present). Maybe make it
> #ifdef PF_LEAN_TABLES or such :)
Hmm, thanks. The only problem that I do not know much about the OpenBSD
internals (yet). This firewall is in production, so I cannot play on it
right now. I hope I will get a second one to build a redundant carp
cluster (In that case I can experiment on the slave).
So it would be better to increase the number of allocable pages (if it's
possible).
I am already set in pf_table.c: pfr_initialize(void)
pool_init(&pfr_ktable_pl, sizeof(struct pfr_ktable), 0, 0, 0,
            "pfrktable", &pool_allocator_oldnointr);
	...pfr_kentry_pl...
>From pool_allocator_nointr to pool_allocator_oldnointr
(I do not know if it counts)
> Daniel
Thanks.
PS: I will try removing packet/bytes counters if it is the only way.
-- 
cstamas