Re: [suse-security] Under DDoS Attack
Another suggestion... "I don't think that works out. Whenever I might send a FIN - what prevents my Apache from being attacked from the same bot after seconds again?" You mentioned that it happening from the same bot(s) again and again... Am I wrong? If you are able to produce a list using netstat and output it into a text file, you may then be able to narrow down networks from which the attack is originating. Afterwards, you can contact your upstream ISP and they will be more than happy to block the rogue traffic from reaching your network. They are quite happy to work with folks on things such as this as very often the traffic also effects others that they host services for by simply 'busying' things up with useless traffic... good luck!!! >>> Syv Ritch10/27/05 12:51 PM >>> media Formel4 wrote: > - Is it possible with spoofed IP numbers to establish connections to > port 80? As far as I know you should get stuck after "SYN". > I'm asking that, because tracing back the IPs in question I find very > often unrouted areas and non-reachable (but maybe firewalled) IPs. > > Also I found a group of 300 IPs coming from an american company network. > I contacted them and they stated too, that those IPs were not in use and > not routed right now... > > > > - How can I secure this server and/or stop this attack? I think that you are looking at wrong point. Preventing a DDOS is not the job of the web server, but the job of the router/firewall. "Real routers/firewalls" will deal easily with these problems. - No spoofing of IPs through validation where the packet comes from... - No fragmented packets - Limit the number of open/unfinished connections... Cisco Pix 501, 515... depending on size and volumes Cisco 1811... Not cheap but when configured properly, guaranteed to work. -- Thanks http://www.911networks.com When the network has to work Cisco/Microsoft -- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
Timothy Hall schrieb:
Another suggestion...
If you are able to produce a list using netstat and output it into a text file, you may then be able to narrow down networks from which the attack is originating. Afterwards, you can contact your upstream ISP and they will be more than happy to block the rogue traffic from reaching your network. They are quite happy to work with folks on things such as this as very often the traffic also effects others that they host services for by simply 'busying' things up with useless traffic...
Looking at the sorted list I'm lucky finding 3 or 4 IPs coming from the same class-B network... Blocking out those whole ranges would mean "blocking the whole internet". Pretty secure, but not really useable. Just to give you an image of what I'm talking, here is the end of the sorted block list: 86.141.169.190 86.192.209.103 86.192.228.171 86.193.197.64 86.195.240.239 86.195.241.164 86.197.89.113 86.199.116.107 86.200.119.203 86.39.49.163 86.39.49.209 86.40.11.182 86.42.46.152 86.42.6.96 86.52.121.189 86.56.128.235 87.116.186.194 87.207.57.195 87.248.16.153 87.49.46.196 87.74.14.181 87.74.44.193 87.81.180.177 87.89.129.200 88.104.169.188 88.105.188.79 88.105.203.170 88.106.14.128 88.107.172.176 88.107.37.73 88.109.124.30 88.110.131.17 88.110.150.174 88.110.37.239 88.110.67.130 88.110.68.53 88.111.71.32 88.111.75.8 As you can see: They've got not much in common... I'm still not sure that they aren't spoofed. During the last hours I blocked more than 6000 IPs and per minute the count raises by 30 - 40...
media Formel4 wrote:
88.111.75.8
As you can see: They've got not much in common...
I'm still not sure that they aren't spoofed. During the last hours I blocked more than 6000 IPs and per minute the count raises by 30 - 40...
What firewall is in front of that host? I'd try to setup a reverse-proxy infront of it, together with an OpenBSD packetfilter. The key to fending-off a DDoS-attacks is to have more resources than the attacker - both bandwidth and raw processing power. If you don't have these resources, you can also just go home and wait till it's over because with so many zombies, the attacker can just flood you "conventionally" until your upstream provider is fed-up enough with it so that he just disconnects your system.... cheers, Rainer
Rainer Duffner schrieb:
media Formel4 wrote:
88.111.75.8
As you can see: They've got not much in common...
I'm still not sure that they aren't spoofed. During the last hours I blocked more than 6000 IPs and per minute the count raises by 30 - 40...
What firewall is in front of that host? I'd try to setup a reverse-proxy infront of it, together with an OpenBSD packetfilter.
As I said - its a root server. Nothing in front but the pure internet...
The key to fending-off a DDoS-attacks is to have more resources than the attacker - both bandwidth and raw processing power.
Neither bandwidth nor processing power is the problem. The server stays below a load of 1 and the traffic is almost invisible compared to the standard traffic to the otther servers, because the requests are empty and block others to get content from my server...
If you don't have these resources, you can also just go home and wait till it's over because with so many zombies, the attacker can just flood you "conventionally" until your upstream provider is fed-up enough with it so that he just disconnects your system....
I do have the ressources - but I'm running out of options how to use them to fight back the attackers. The list of blocked IPs reached 10.000 in the meantime...
/ 2005-10-27 23:59:54 +0200 \ b@rry:
As I said - its a root server. Nothing in front but the pure internet...
Why not have a firewall in front of it? Root server or no, something that can manage the connections to the box with relatively low connection timeouts?
as mentioned before, you could try to proxy... somewhen, Apache TimeOut directive will be able to configure initial request timeout, inter-packet timeout and total timeout independently... until then, try attached script. quick hack in the last half our, so beware. adjust the timeouts. consider security implications. maybe you want to do more verbose, or near to no logging. or to feed "timed out" ips into some iptables script (after sanitizing) feel free to reimplement in C, use select loops instead of many processes, add command line parameters to adjust the various timeouts, or just ignore it altogether. or course, you want to change your apache to listen on the loopback only, and some different port, and change $apache_port accordingly. then change $P_port to where your apache typically listens. maybe you want to bind() more specifically, not to ANY. run it like "perl -wT myproxy.pl > myproxy.log" if this script breaks terribly in the real world, so what: anyways, has been a nice excercise... cheers, Lars Ellenberg
On Thu, Oct 27, 2005 at 11:59:54PM +0200, b@rry wrote:
As I said - its a root server. Nothing in front but the pure internet...
Why not have a firewall in front of it? Root server or no, something that can manage the connections to the box with relatively low connection
timeouts?
Maybe just maybe, because a firewall isn't going to do a THING against a DDOS attack? And for the other person who said call the ISP so they can "set the router to block the packets"..... Lol, if it was hat easy Yahoo, Microsoft and SCO wouldn't have been taken down.
Hi! On 27 Oct 2005, at 21:42, media Formel4 wrote:
I do have the ressources - but I'm running out of options how to use them to fight back the attackers.
The list of blocked IPs reached 10.000 in the meantime...
I'd recommend writing a small connection proxy program which listens on port 80, takes the connections and forwards only the requests which come in to the apache (running on a different port). You'd run into the 1024 filedescriptor limit, but then you can always reap the oldest 'empty' connection as soon as you reach 1000. Should work as long as the rate of these empty connection openings is not in the kHz range ;-) But, alas, no time to code it up right now. :-( Ciao, Roland -- TU Muenchen, Physik-Department E18, James-Franck-Str. 85747 Garching Telefon 089/289-12592; Telefax 089/289-12570 -- A mouse is a device used to point at the xterm you want to type in. Kim Alm on a.s.r. -----BEGIN GEEK CODE BLOCK----- Version: 3.12 GS/CS/M/MU d-(++) s:+ a-> C+++ UL++++ P-(+) L+++ E(+) W+ !N K- w--- M + !V Y+ PGP++ t+(++) 5 R+ tv-- b+ DI++ e+++>++++ h---- y+++ ------END GEEK CODE BLOCK------
Hi all, first of all thanks for all your suggestions. Today (day 3 of the Attack) we decided to switch IPs - which was not the best choice at all. Right after changing the nameservers too, we had to find out that the DDos now splitted up. We faced now two DDoS Attacks. One was still whacking the old IP (on which we set up a honeypot for further investigations), while the second one attacked the new IP, simply following the DNS change. Due to customer complains we decided to ask our upstream provider to close every IP traffic to the new IP number but the one coming from local peerings, which decreased the attack to a value the server can handle easy. Sure, it is no longer available for the whole internet, but due to the fact that the content is mainly interesting for local surfer it is not such a big loss. The attack is still going on and we counted >18.000 IP numbers in the meantime, coming from all over the world. Right now scanners are tracing all that kind of traffic, which is an interesting new step in DDoS attacks because it is beyond SYN-flooding. The attacker uses definetly spoofed IP numbers because many of them are not routable - but it doesn't seem to be a compromised router, because in that case it must have been the one closest to our network. And that one is still in service while the attack was reduced (not stopped) by closing a lot of transit traffic. So still the one question is left open: How can the attacker instantiate an ESTABLISHED connection while using spoofed IPs? When I get some more informations out of our trackings, I will inform you... Thanks, Ralf Koch Roland Kuhn schrieb:
Hi!
On 27 Oct 2005, at 21:42, media Formel4 wrote:
I do have the ressources - but I'm running out of options how to use them to fight back the attackers.
The list of blocked IPs reached 10.000 in the meantime...
I'd recommend writing a small connection proxy program which listens on port 80, takes the connections and forwards only the requests which come in to the apache (running on a different port). You'd run into the 1024 filedescriptor limit, but then you can always reap the oldest 'empty' connection as soon as you reach 1000. Should work as long as the rate of these empty connection openings is not in the kHz range ;-)
But, alas, no time to code it up right now. :-(
Ciao, Roland
-- TU Muenchen, Physik-Department E18, James-Franck-Str. 85747 Garching Telefon 089/289-12592; Telefax 089/289-12570 -- A mouse is a device used to point at the xterm you want to type in. Kim Alm on a.s.r. -----BEGIN GEEK CODE BLOCK----- Version: 3.12 GS/CS/M/MU d-(++) s:+ a-> C+++ UL++++ P-(+) L+++ E(+) W+ !N K- w--- M + !V Y+ PGP++ t+(++) 5 R+ tv-- b+ DI++ e+++>++++ h---- y+++ ------END GEEK CODE BLOCK------
Don't you wish you could make stupidity a crime? The DDOS attack bots which are rooted machines and Windows boxes that are totally screwed, and all from people who don't update or open every attachment sent.....
On Saturday 29 October 2005 02:02, media Formel4 wrote:
So still the one question is left open: How can the attacker instantiate an ESTABLISHED connection while using spoofed IPs?
Essentially, you can't. However, as someone has already mentioned, you could brute-force the sequence number expected for a SYNACK "cookie" by sending them blindly after the first SYN. There is a way to stop these from creating an established connection, but you'll need to write a program that actually detects these attempts and deletes the connection. This can be done in linux, but it may take a day or two to develop, if you're familiar with netfilter. The idea is that if you receive a SYN, send a SYNACK, and then wait for the reply and you actually receive a reply from that IP that is somehow invalid before receiving the valid one, you just delete the conntrack entry as if the first SYN packet was never received. This will result in sending a single RST after other packets coming in for the same connection (which you may want to rate limit) and it won't bug apache about an open tcp socket, which is exactly what you need. However, your machine will still get loaded because of all the traffic causing all the state changes in IP stack, and there is a very real possibility that these IP addresses are not spoofed, but actually just machines that have been compromised a while ago and were just waiting to start flooding some IP with junk requests. Check with tcpdump if you are actually receiving lots of ack packets that you should not be seeing. -- Jure Koren, n.i.
participants (8)
-
Allen
-
b@rry
-
Jure Koren
-
Lars Ellenberg
-
media Formel4
-
Rainer Duffner
-
Roland Kuhn
-
Timothy Hall