DOS problem with SuSE 8.1 kernel 2.4.19- (neighbour table overflow)
Hello all, Sorry for the long post, but I really have a no idea anymore ... I'm still having problems with "neighbour table overflows" when we use iptables 1.2.7a and kernel 2.4.19 form SuSE 8.1 original distro. The PC is working as firewall/proxy between a LAN (10.104.0.0/16), an intranet (10.101.0.0 - 10.107.0.0/16) and the internet, which we can only acces through cascaded proxies at 149.206.x.y. I thought last week that we solved it, when our cisco found "incomplete" TCP connections and we switched network cards, but the problem keeps coming back. We even switched the complete machine in the meantime. Today we have the situation as follows: One of the other participants in the intranet got the blaster worm in their net, they are still struggling with this. So we get bombed with tons of connections on port 135 from their destination, when the worm(s) scan our net. This fills our arp table / ip connection table with some 17.000-20.000(!) connections in "half open" state, the kernel then throws thousnads of messages like "Neighbour table overflow" and "neighbour table flood" at high rates. When memory is filled, the network services on this box simply stop working. I would call this a classical DOS attack, but what can I do against it ? I already drop all relevant packets from that source, I would have thought that the iptables / kernel code could manage this traffic (sitting behind a 2 MBit link with a PIII-500 / 256 MB RAM) Here is what I do against the most common "attacks" # NETBIOS # run_iptables -A common -p udp --dport 137:139 -j reject run_iptables -A common -p udp --dport 445 -j reject run_iptables -A common -p tcp --dport 139 -j reject run_iptables -A common -p tcp --dport 445 -j reject run_iptables -A common -p tcp --dport 135 -j reject ############################################################################ # UPnP # run_iptables -A common -p udp --dport 1900 -j DROP ############################################################################ # BROADCASTS # run_iptables -A common -d 255.255.255.255 -j DROP run_iptables -A common -d 224.0.0.0/4 -j DROP ############################################################################ # AUTH -- Silently reject it so that connections don't get delayed. # run_iptables -A common -p tcp --dport 113 -j reject ############################################################################ # DNS -- Silenty drop late replies run_iptables -A common -p udp --sport 53 -mstate --state NEW -j DROP Any helping tip is welcome, Philipp
Philipp Rusch wrote:
Today we have the situation as follows: One of the other participants in the intranet got the blaster worm in their net, they are still struggling with this. So we get bombed with tons of connections on port 135 from their destination, when the worm(s) scan our net. This fills our arp table / ip connection table with some 17.000-20.000(!) connections in "half open" state, the kernel then throws thousnads of messages like "Neighbour table overflow" and "neighbour table flood" at high rates. When memory is filled, the network services on this box simply stop working. I would call this a classical DOS attack, but what can I do against it ? I already drop all relevant packets from that source, I would have thought that the iptables / kernel code could manage this traffic (sitting behind a 2 MBit link with a PIII-500 / 256 MB RAM)
Here is what I do against the most common "attacks" [snipped firewalling]
iptables won't help in this case cause your problem is not the 20000 half open connections, it's the 20000 (spoofed) ARP's. Did you try to extend your arp table like i suggested you last time? The only thing which will work is the kernel tuning imho. Another Problem is that you only have 256MB ram. If you said you've 20.000 Arp requests, i would say you've a big network behind it. So if you use connection tracking, you use a lot of memory just for that. So try to resize your arp table to the suggested values and take a look if it helps. Watch the arp table and the count with a script like: while true; do DSTRING=$(date +%s) arp -an > arp-${DSTRING}.log sleep 1 done so you can check after a half hour or so how many arp entries you really had and set the table values to a better size. HTH, Sven
Hi Sven, ok, I did all the things you told me last time and I have this script running now, let's see and watch it later on ... There is no huge network - its about 70 users or so, the other network is about a 100 users, I thought that if there is no reply to a request, the garbage collection should do its job and clear the arp table "pretty soon". Additionally I changed the gc_interval from "30" to "10" and the gc_stale_time from "60" down to "30", I hope this speeds up the thing. I still think that a PC of this class should *easily* handle a situation like this, because that's what a firewall / proxy is for ! I am quite astonished that I have ever to restart otherwise rock solid Linux- machines like this :-( If you say that this is no problem of iptables, then let's agree that this is at least a problem how unwanted packets are handled, yes ? Regards, Philipp Sven 'Darkman' Michels schrieb:
Philipp Rusch wrote:
Today we have the situation as follows: One of the other participants in the intranet got the blaster worm in their net, they are still struggling with this. So we get bombed with tons of connections on port 135 from their destination, when the worm(s) scan our net. This fills our arp table / ip connection table with some 17.000-20.000(!) connections in "half open" state, the kernel then throws thousnads of messages like "Neighbour table overflow" and "neighbour table flood" at high rates. When memory is filled, the network services on this box simply stop working. I would call this a classical DOS attack, but what can I do against it ? I already drop all relevant packets from that source, I would have thought that the iptables / kernel code could manage this traffic (sitting behind a 2 MBit link with a PIII-500 / 256 MB RAM)
Here is what I do against the most common "attacks" [snipped firewalling]
iptables won't help in this case cause your problem is not the 20000 half open connections, it's the 20000 (spoofed) ARP's. Did you try to extend your arp table like i suggested you last time? The only thing which will work is the kernel tuning imho. Another Problem is that you only have 256MB ram. If you said you've 20.000 Arp requests, i would say you've a big network behind it. So if you use connection tracking, you use a lot of memory just for that. So try to resize your arp table to the suggested values and take a look if it helps. Watch the arp table and the count with a script like: while true; do DSTRING=$(date +%s) arp -an > arp-${DSTRING}.log sleep 1 done
so you can check after a half hour or so how many arp entries you really had and set the table values to a better size.
HTH, Sven
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
Another question is, what *exactly* happens when ARP requests are spoofed ? Why are there so many connections showing up in seconds and then sitting there and using up resources ? (I watched about 5.000 new connections in 1 minute (!) in iptraf, when after ~ 17.500 connections that scnning worm decided to use another network address than ours. Still our machine had to be rebooted, rcnetwork restart did not help cleaning it up. I'd like to throw away these "half open" ip connections after seconds or at least be able to control how many of them are allowed in a certain time interval. But how ? When after scanning the other ip ranges the worm comes back to us, it's half an hour (even with my "enlarged" arp tables) and we have to reboot again. There are only several machines infected, so it is enough to have 2 or3 PCs with this nice "blaster" to completely disable a 2 MBit link. Bad thing ! Regards, Philipp Sven 'Darkman' Michels schrieb:
Philipp Rusch wrote:
Today we have the situation as follows: One of the other participants in the intranet got the blaster worm in their net, they are still struggling with this. So we get bombed with tons of connections on port 135 from their destination, when the worm(s) scan our net. This fills our arp table / ip connection table with some 17.000-20.000(!) connections in "half open" state, the kernel then throws thousnads of messages like "Neighbour table overflow" and "neighbour table flood" at high rates. When memory is filled, the network services on this box simply stop working. I would call this a classical DOS attack, but what can I do against it ? I already drop all relevant packets from that source, I would have thought that the iptables / kernel code could manage this traffic (sitting behind a 2 MBit link with a PIII-500 / 256 MB RAM)
Here is what I do against the most common "attacks" [snipped firewalling]
iptables won't help in this case cause your problem is not the 20000 half open connections, it's the 20000 (spoofed) ARP's. Did you try to extend your arp table like i suggested you last time? The only thing which will work is the kernel tuning imho. Another Problem is that you only have 256MB ram. If you said you've 20.000 Arp requests, i would say you've a big network behind it. So if you use connection tracking, you use a lot of memory just for that. So try to resize your arp table to the suggested values and take a look if it helps. Watch the arp table and the count with a script like: while true; do DSTRING=$(date +%s) arp -an > arp-${DSTRING}.log sleep 1 done
so you can check after a half hour or so how many arp entries you really had and set the table values to a better size.
HTH, Sven
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
participants (2)
-
Philipp Rusch -
Sven 'Darkman' Michels