suse 9.2: ip_conntrack: table full, dropping packet
Hi, I recently upgraded some servers to suse linux 9.2 pro from 8.1 pro. Two of the servers are getting quite a lot of traffic and they quickly run out of space in the ip conntrack hashtable, giving the following message: Jan 28 08:43:25 nuspelen kernel: ip_conntrack: table full, dropping packet. I now increased the maximum number of tracked connections to about 128000, but I'm not sure yet the system will run stable with this size. I've googled a bit on the error message; it seems that some timeouts in the conntracking code are way too high. Is there a way to lower these limits? Are there patches available? I really like to run on stock kernels (for the security updates and the QA), but rolling my own is no real problem. -- Erik Hensema (erik@hensema.net)
erik@hensema.netErik Hensema wrote:
Hi,
I recently upgraded some servers to suse linux 9.2 pro from 8.1 pro. Two of the servers are getting quite a lot of traffic and they quickly run out of space in the ip conntrack hashtable, giving the following message:
Jan 28 08:43:25 nuspelen kernel: ip_conntrack: table full, dropping packet.
I now increased the maximum number of tracked connections to about 128000, but I'm not sure yet the system will run stable with this size.
I've googled a bit on the error message; it seems that some timeouts in the conntracking code are way too high. Is there a way to lower these limits? Are there patches available? I really like to run on stock kernels (for the security updates and the QA), but rolling my own is no real problem.
I've tried 4 times now to send a detailed reply, but Mozilla keeps crashing on me, so I give up. Time to try an update.. later. You would do well to check out the netfilter home site, www.netfilter.org. I found a very good connection tracking tutorial there, which leads me to believe that the timeout values really aren't your problem. The following are specifically recommended in any iptables configuration: iptables -A INPUT -p tcp !--syn -m state --state NEW -j DROP which will dump any new connection that does not have the SYN bit set. (A stray ACK packet can establish a NEW connection, for some arcane reasons I do not claim to understand.) iptables -A INPUT -m state --state INVALID -j DROP which will dump anything that is not a valid new connection, and is not part of, or related to, an established connection. These two together might reduce your traffic somewhat. As for the timeout values themselves, they are in /usr/src/linux/net/ipv4/netfilter/ip_conntrack_proto_tcp.c and similar files exist for UDP and ICMP. There is also a netfilter extension, it's named in one of the tutorials, that will allow these and other values to be set on the fly, if you really do conclude you need this. However, the conntrack tutorial does give a detailed discussion on the life of a connection table entry. Of interest here is how an ESTABLISHED entry disappears, because that is the largest timeout value. As soon as your firewall sees a FIN packet from the remote client, the corresponding table entry will be marked CLOSED and will disappear 10 seconds later. If a remote client crashes and never sends a FIN, then possibly the remote system will emit a RST packet, or else after a suitable value established in your server configuration, the server will issue a RST packet. This is a "reset" packet, and it causes the table entry to be marked CLOSED. No need to do anything else, it's automatic (or that is what all that stuff leads me to believe). Setting a suitable timeout in your server is a matter of experience, I suppose. Your system should be stable with that many conntrack entries; they only take "350" bytes each -- and note that the conntrack table is NEVER swapped out. The default works out to 32K entries on a system with 512MB of installed RAM. One thing is recommended at the same time as increasing the size of the table, and that is increasing the size of the hash table. This can be done when the conntrack module is loaded, eg. modprobe ip_conntrack hashsize=1023 This is done in the /sbin/SuSEfirewall2 script, if you are using that. One important note, it is specifically recommended that the hashsize NOT be an even number, even numbers are evil and powers of two are the most evil of all. Something to do with the hashing algorithm. Now to vegetate in front of the TV, and munch on some nutritious junk food, before I tackle Mozilla. :-) I hope all this is of some use to you.
Darryl Gregorash wrote:
The following are specifically recommended in any iptables configuration:
iptables -A INPUT -p tcp !--syn -m state --state NEW -j DROP
which will dump any new connection that does not have the SYN bit set. (A stray ACK packet can establish a NEW connection, for some arcane reasons I do not claim to understand.)
iptables -A INPUT -m state --state INVALID -j DROP
I was poking around the Shorewall website, and came across this item: "1. Recent 2.6 kernels include code that evaluates TCP packets based on TCP Window analysis. This can cause packets that were previously classified as NEW or ESTABLISHED to be classified as INVALID. The new kernel code can be disabled by including this command in your /etc/shorewall/init file: echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal Additional kernel logging about INVALID TCP packets may be obtained by adding this command to /etc/shorewall/init: echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" It should also be possible to include either or both of those commands in /etc/init.d/boot.local
On Wed, Feb 02, 2005 at 06:32:33PM -0600, Darryl Gregorash wrote:
Darryl Gregorash wrote:
The following are specifically recommended in any iptables configuration:
iptables -A INPUT -p tcp !--syn -m state --state NEW -j DROP
which will dump any new connection that does not have the SYN bit set. (A stray ACK packet can establish a NEW connection, for some arcane reasons I do not claim to understand.)
iptables -A INPUT -m state --state INVALID -j DROP
I was poking around the Shorewall website, and came across this item:
"1. Recent 2.6 kernels include code that evaluates TCP packets based on TCP Window analysis. This can cause packets that were previously classified as NEW or ESTABLISHED to be classified as INVALID.
The new kernel code can be disabled by including this command in your /etc/shorewall/init file:
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal
The server held up for 11 days before starting to drop packets again. I've now gone back to a stateless firewall (ipchains style). Is a hashsize of about 128000 for the ip_conntrack database too low for a webserver handling about 100 hits/sec? -- Erik Hensema (erik@hensema.net)
participants (2)
-
Darryl Gregorash
-
Erik Hensema