still have problems with "kernel: ip_conntrack: table full, dropping packet."
Hi everyone, I still get the error message "kernel: ip_conntrack: table full, dropping packet." on SuSE 9.0 with SuSEfirewall2 (all updates installed) about every 2-4 weeks (there was a discussion about this some month ago, but no solution...). The only thing that seems to help when this occurs is rebooting the machine (as far as I could figure). The machine does have some servers behind it and filters about 500MB of traffic average per day. Below are some informations about the error-messages and the configuration and the state of the server when the problem occurs: linux:~ # cat /var/log/messages | grep ip_ Feb 24 12:13:05 linux kernel: ip_conntrack: table full, dropping packet. Feb 24 12:13:09 linux kernel: ip_conntrack: table full, dropping packet. Feb 24 12:13:15 linux kernel: ip_conntrack: table full, dropping packet. Feb 24 12:13:20 linux kernel: ip_conntrack: table full, dropping packet. Feb 24 12:13:24 linux kernel: ip_conntrack: table full, dropping packet. Feb 24 12:13:38 linux kernel: ip_conntrack: table full, dropping packet. Feb 24 12:13:39 linux kernel: ip_conntrack: table full, dropping packet. Feb 24 12:13:56 linux kernel: ip_conntrack: table full, dropping packet. linux:~ # cat /proc/net/ip_conntrack | wc -l 1913 linux:~ # cat /proc/sys/net/ipv4/ip_conntrack_max 32760 linux:~ # iptables-save | wc -l 922 Does anybody have the same problem and hopefully a solution? Any ideas how I can at least find out, which packets are dropped? Greetings, Ralf
On Thursday 24 February 2005 06:27, Ralf Ronneburger wrote: <snip>
Does anybody have the same problem and hopefully a solution? Any ideas how I can at least find out, which packets are dropped?
Hi Ralf, This looked like an interesting problem to me last time it came up. I don't have an answer for you and since you haven't found one yet and it's been a month, I tried a little Google research. My search string was just the kernel error message, as you'll see, and the results: "Results 1 - 10 of about 6,550 for linux kernel: ip_conntrack: table full, dropping packet. (0.88 seconds)" Briefly scanning several pages of the results leads me to believe this problem isn't at all unique to SuSE and is somehow related to the 2.4 kernel. Maybe it's time for you to become more aggressive in your research or to upgrade that system? ... to SLES9, perhaps, or even 9.2 with the current kernel? ... given the server's important job and the volume of traffic it's handling? Just my two cents to throw in the hat. Good luck & regards, - Carl -- _______________________________________________________________________ C. E. Hartung Business Development & Support Services http://www.cehartung.com/ carlh@cehartung.com Dover Foxcroft, Maine, USA Public Keys 68396713 & F8207216 Reg. Linux User #350527 http://counter.li.org/
Hi Carl, Carl E. Hartung wrote:
"Results 1 - 10 of about 6,550 for linux kernel: ip_conntrack: table full,
dropping packet. (0.88 seconds)"
Briefly scanning several pages of the results leads me to believe this problem isn't at all unique to SuSE and is somehow related to the 2.4 kernel.
I've found these results, too, but only questions and no answers. Besides - there are also users that have this problem in 2.6 and some claimed this to be a problem introduced with 2.6.10 which leads me to believe it's not unique for 2.4 and it's still not fixed in 2.6... Greetings, Ralf
Upgrading to SuSE 9.2 will not solve the problem in any way. I had the same problem, and it was solved by removing the ip_conntrack module from that server. I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied. lr2-tit1:~ # cat /etc/SuSE-release SuSE Linux 9.2 (i586) VERSION = 9.2 lr2-tit1:~ # uname -a Linux lr2-tit1 2.6.8-24.11-default #1 Fri Jan 14 13:01:26 UTC 2005 i686 athlon i386 GNU/Linux lr2-tit1:~ # lr2-tit1:~ # lsmod Module Size Used by af_packet 20872 0 evdev 8960 0 joydev 9664 0 sg 35872 0 st 37404 0 sd_mod 16912 0 sr_mod 16292 0 ide_cd 38176 0 cdrom 36508 2 sr_mod,ide_cd nvram 8328 0 edd 10012 0 ipt_REJECT 6784 15 cls_u32 8452 247 ipt_TOS 2560 2 sch_htb 23808 2 iptable_mangle 2944 1 iptable_filter 3072 1 ip_tables 17664 4 ipt_REJECT,ipt_TOS,iptable_mangle,iptable_filter ipv6 237312 33 via_agp 8960 1 agpgart 32168 1 via_agp sata_via 7428 0 libata 41860 1 sata_via scsi_mod 111052 5 sg,st,sd_mod,sr_mod,libata subfs 7552 1 r8169 18184 0 dm_mod 54524 0 usbcore 106724 1 sk98lin 173676 1 ext3 115688 2 jbd 61348 1 ext3 lr2-tit1:~ # cat /etc/sysctl.conf # Disable response to broadcasts. # You don't want yourself becoming a Smurf amplifier. net.ipv4.icmp_echo_ignore_broadcasts = 1 # enable route verification on all interfaces net.ipv4.conf.all.rp_filter = 1 # enable ipV6 forwarding #net.ipv6.conf.all.forwarding = 1 net.ipv4.netfilter.ip_conntrack_max = 65535 Best regards, Sandu Mihai Carl E. Hartung wrote:
On Thursday 24 February 2005 06:27, Ralf Ronneburger wrote: <snip>
Does anybody have the same problem and hopefully a solution? Any ideas how I can at least find out, which packets are dropped?
Hi Ralf,
This looked like an interesting problem to me last time it came up. I don't have an answer for you and since you haven't found one yet and it's been a month, I tried a little Google research. My search string was just the kernel error message, as you'll see, and the results:
"Results 1 - 10 of about 6,550 for linux kernel: ip_conntrack: table full, dropping packet. (0.88 seconds)"
Briefly scanning several pages of the results leads me to believe this problem isn't at all unique to SuSE and is somehow related to the 2.4 kernel.
Maybe it's time for you to become more aggressive in your research or to upgrade that system? ... to SLES9, perhaps, or even 9.2 with the current kernel? ... given the server's important job and the volume of traffic it's handling?
Just my two cents to throw in the hat. Good luck & regards,
- Carl
On 24 Feb 2005 at 17:10, Sandu Mihai wrote:
I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied.
My be you should try to cat /proc/net/ip_conntrack and investigate why it gets filled. Perhaps by a virus, a port scan or like from internal. mfg Andreas Kunberger ITV Denkendorf
The firewall is filtering a quite busy network The problem is, that in my firewall rules I do not use anything that might trigger a connection tracking (that is why I was able to remove it, and live after that :) ) The problem is, that the modif of conntrack table size does not seems to work :( Best regards, Sandu Mihai Andreas Kunberger wrote:
On 24 Feb 2005 at 17:10, Sandu Mihai wrote:
I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied.
My be you should try to cat /proc/net/ip_conntrack and investigate why it gets filled. Perhaps by a virus, a port scan or like from internal.
mfg Andreas Kunberger ITV Denkendorf
On 25 Feb 2005 at 9:44, Sandu Mihai wrote:
The firewall is filtering a quite busy network The problem is, that in my firewall rules I do not use anything that might trigger a connection tracking (that is why I was able to remove it, and live after that :) ) The problem is, that the modif of conntrack table size does not seems to work :(
The kernel seems to keep an entry in this table for every connection form this server, for every connection he is routing (even without nat) and if there is rule for a transparent proxy on the server there seem to be an entry for _every_ address that can possibly use this gateway! Try conntrack-viewer from P.Lagace http://cv.intellos.net/ mfg Andreas Kunberger ITV Denkendorf
Maybe you could set up a rule to log all IP packets coming into your machine - then check the log file for any suspicious looking IP addresses - eg loads of duff packets from the same IP. Or, are you under a DDoS attack? HTH - Keith http://www.karsites.net/ SPDTool - an idea for a structured open source development CASE tool. Find out more at the above link! On Fri, 25 Feb 2005, Andreas Kunberger wrote:
To: suse-security@suse.com From: Andreas Kunberger <lise@itv-denkendorf.de> Subject: Re: [suse-security] still have problems with "kernel: ip_conntrack: table full, dropping packet."
On 24 Feb 2005 at 17:10, Sandu Mihai wrote:
I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied.
My be you should try to cat /proc/net/ip_conntrack and investigate why it gets filled. Perhaps by a virus, a port scan or like from internal.
mfg Andreas Kunberger ITV Denkendorf
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
Sandu Mihai wrote:
Upgrading to SuSE 9.2 will not solve the problem in any way. I had the same problem, and it was solved by removing the ip_conntrack module from that server. I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied.
The problem is in our bug tracking system but it's hard to reproduce. Can you please post the content of /proc/net/ip_conntrack and /proc/net/ip_conntrack_expect when the problem occurs? cu Ludwig -- (o_ Ludwig Nussel //\ SUSE LINUX Products GmbH, Development V_/_ http://www.suse.de/
Ludwig Nussel wrote:
Sandu Mihai wrote:
Upgrading to SuSE 9.2 will not solve the problem in any way. I had the same problem, and it was solved by removing the ip_conntrack module from that server. I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied.
The problem is in our bug tracking system but it's hard to reproduce. Can you please post the content of /proc/net/ip_conntrack and /proc/net/ip_conntrack_expect when the problem occurs?
To those seeing the problem on SUSE LINUX 9.2: Can you please try these settings and see if the problem occurs again? echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid This will change the way TCP window tracking works and makes the kernel log pakets that look suspicious to conntrack. Thanks, cu Ludwig -- (o_ Ludwig Nussel //\ SUSE LINUX Products GmbH, Development V_/_ http://www.suse.de/
On Mon, Mar 07, 2005 at 05:29:30PM +0100, Ludwig Nussel wrote:
Ludwig Nussel wrote:
Sandu Mihai wrote:
Upgrading to SuSE 9.2 will not solve the problem in any way. I had the same problem, and it was solved by removing the ip_conntrack module from that server. I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied.
I hope it's OK if I'll jump in this thread. I have the same problem with a SuSE 9.0 Gateway. For your Information: behind the Gateway theres a proxy Server (192.168.100.2) that connects to a Trendmicro Viruswall on the Gateway (192.168.100.1:8080)
The problem is in our bug tracking system but it's hard to reproduce. Can you please post the content of /proc/net/ip_conntrack and /proc/net/ip_conntrack_expect when the problem occurs?
output is attached
To those seeing the problem on SUSE LINUX 9.2: Can you please try these settings and see if the problem occurs again?
echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
Any tips on what to do with my SuSE 9.0 box?
This will change the way TCP window tracking works and makes the kernel log pakets that look suspicious to conntrack.
Thanks, cu Ludwig
TIA marc
Hi Marc, Marc Samendinger wrote:
<snip>
Any tips on what to do with my SuSE 9.0 box? </snip>
do you have an ftp-server behind the box? What I found out for SuSE 9.0 is, that ftp-connections through the firewall boost up the connection-usage. Besides you can find out, how close you are to the "kernel: ip_conntrack: table full, dropping packet." messages, when you check the following: linux:~ # cat /proc/slabinfo | grep ip_conntrack ip_conntrack 32566 32772 320 2729 2731 1 linux:~ # cat /proc/sys/net/ipv4/ip_conntrack_max 32760 Once the the number of currently active objects (in this case 32566) gets up to the number configured in ip_conntrack_max, then you'll get the "dropping packet"-message in /var/log/messages and then afaik all you can do is reboot. Greetings, Ralf
On Wed, Mar 09, 2005 at 10:28:46AM +0100, Ralf Ronneburger wrote:
Hi Marc,
Marc Samendinger wrote:
<snip>
Any tips on what to do with my SuSE 9.0 box? </snip>
do you have an ftp-server behind the box? What I found out for SuSE 9.0 is, that ftp-connections through the firewall boost up the connection-usage. Besides you can find out, how close you are to the
theres no ftp server behind the gateway. And
"kernel: ip_conntrack: table full, dropping packet." messages, when you check the following:
linux:~ # cat /proc/slabinfo | grep ip_conntrack ip_conntrack 32566 32772 320 2729 2731 1
I haven't known that one thanks. But the machine is allready rebooted, I'll check this next time.
linux:~ # cat /proc/sys/net/ipv4/ip_conntrack_max 32760
Once the the number of currently active objects (in this case 32566) gets up to the number configured in ip_conntrack_max, then you'll get the "dropping packet"-message in /var/log/messages and then afaik all you can do is reboot.
Greetings,
Ralf
Thanks for your efforts marc
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, Ralf Ronneburger wrote: | do you have an ftp-server behind the box? What I found out for SuSE 9.0 | is, that ftp-connections through the firewall boost up the | connection-usage. Besides you can find out, how close you are to the | "kernel: ip_conntrack: table full, dropping packet." messages, when you | check the following: | | linux:~ # cat /proc/slabinfo | grep ip_conntrack | ip_conntrack 32566 32772 320 2729 2731 1 | linux:~ # cat /proc/sys/net/ipv4/ip_conntrack_max | 32760 | | Once the the number of currently active objects (in this case 32566) | gets up to the number configured in ip_conntrack_max, then you'll get | the "dropping packet"-message in /var/log/messages and then afaik all | you can do is reboot. nope, you can raise the number of possible conntrack entries. It depends on how much ram your box have but usually doubleing the value is no problem. Simply do: echo 65520 > /proc/sys/net/ipv4/ip_conntrack_max (or if unsure about ram usage, make it just 1.5 or so) This fixes this issue temporarly cause after reboot the default value depending on your system memory is calculated and used. So after reboot you need to do the echo again. Regards, Sven -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQFCLsoDQoCguWUBzBwRAjsvAKCZC1LZfxDtw0oHW4cEF/31smh9VwCfQpw7 8DZJnxPmiLNKB3YfwQ4FyAE= =AnkC -----END PGP SIGNATURE-----
Hi Sven, Sven 'Darkman' Michels wrote:
nope, you can raise the number of possible conntrack entries. It depends on how much ram your box have but usually doubleing the value is no problem. Simply do: echo 65520 > /proc/sys/net/ipv4/ip_conntrack_max (or if unsure about ram usage, make it just 1.5 or so)
yes, agreed. But to know how close you are to your limit (RAM-limit or whatever limit you've set) you can still use slabinfo. And eventually (because of some overflow) you'll reach every limit. Greetings, Ralf
Or you can update your /etc/sysctl.conf file so the change is permenant ----- Original Message ----- From: "Sven 'Darkman' Michels" <sven@darkman.de> To: <suse-security@suse.com> Sent: Wednesday, March 09, 2005 4:03 AM Subject: Re: [suse-security] still have problems with "kernel: ip_conntrack: table full, dropping packet."
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi there,
Ralf Ronneburger wrote:
| do you have an ftp-server behind the box? What I found out for SuSE 9.0 | is, that ftp-connections through the firewall boost up the | connection-usage. Besides you can find out, how close you are to the | "kernel: ip_conntrack: table full, dropping packet." messages, when you | check the following: | | linux:~ # cat /proc/slabinfo | grep ip_conntrack | ip_conntrack 32566 32772 320 2729 2731 1 | linux:~ # cat /proc/sys/net/ipv4/ip_conntrack_max | 32760 | | Once the the number of currently active objects (in this case 32566) | gets up to the number configured in ip_conntrack_max, then you'll get | the "dropping packet"-message in /var/log/messages and then afaik all | you can do is reboot.
nope, you can raise the number of possible conntrack entries. It depends on how much ram your box have but usually doubleing the value is no problem. Simply do: echo 65520 > /proc/sys/net/ipv4/ip_conntrack_max (or if unsure about ram usage, make it just 1.5 or so)
This fixes this issue temporarly cause after reboot the default value depending on your system memory is calculated and used. So after reboot you need to do the echo again.
Regards, Sven -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux)
iD8DBQFCLsoDQoCguWUBzBwRAjsvAKCZC1LZfxDtw0oHW4cEF/31smh9VwCfQpw7 8DZJnxPmiLNKB3YfwQ4FyAE= =AnkC -----END PGP SIGNATURE-----
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
-- No virus found in this incoming message. Checked by AVG Anti-Virus. Version: 7.0.308 / Virus Database: 266.7.0 - Release Date: 3/8/2005
Hi Sandu, I'm picking up this thread again, as the latest kernel patch for 9.2 is supposed to have a fix for that problem ("A dst leak problem in the ip_conntrack module of the iptables firewall was fixed. Only SUSE Linux versions using the 2.6 kernels are affected."). Does this fix it for you or for anybody else with the same problem on 9.2? The reason for my question - if it does not fix it, then there's no reason for me to update from 9.0 to 9.2, otherwise this would be a very strong reason to do so. Thanks and greetings, Ralf Sandu Mihai wrote:
Upgrading to SuSE 9.2 will not solve the problem in any way. I had the same problem, and it was solved by removing the ip_conntrack module from that server. I have tryied to bump up the conntrack table size using /etc/sysctl.conf and boot.sysctl, it had no effect whatsoever. The system in question is a SuSE 9.2 Proffesional with the latest patches applied.
participants (9)
-
Andreas Kunberger -
Carl E. Hartung -
Harry Crowder -
Ludwig Nussel -
Marc Samendinger -
Ralf Ronneburger -
Sandu Mihai -
suse＠karsites.net -
Sven 'Darkman' Michels