Hi list, sorry for the double-post but my thread opener was in reply to another post which confuses the mailinglist structure... #################################################################### Hi list, right now we're experiencing a (for me) very uncommon DDoS attack against one of our webservers. Looking with netstat we find hundreds of established connections to our Apache webserver, but nothing in the logs - which means the attacker opens up a connection (not only a SYN request as in SYN flood attacks) and then blocks the Apache child until it hits timeout. This attack comes from thousands of IP numbers (bots?) all over the world. Question is: - Is it possible with spoofed IP numbers to establish connections to port 80? As far as I know you should get stuck after "SYN". I'm asking that, because tracing back the IPs in question I find very often unrouted areas and non-reachable (but maybe firewalled) IPs. Also I found a group of 300 IPs coming from an american company network. I contacted them and they stated too, that those IPs were not in use and not routed right now... - How can I secure this server and/or stop this attack? Thanks, Ralf Koch
media Formel4 wrote:
Question is:
- Is it possible with spoofed IP numbers to establish connections to port 80? As far as I know you should get stuck after "SYN". I'm asking that, because tracing back the IPs in question I find very often unrouted areas and non-reachable (but maybe firewalled) IPs.
i would say no (else the school was pretty useless ;-)
- How can I secure this server and/or stop this attack?
this attack is very mean and it succeeds almost always (even if you just do it from a single attacking machine). i would do a search on google, there are definitively others who were under the same sort of attack. just some thoughts about how it could be possible to protect (at least a bit). maybe it's possible to let netfilters connection tracking do the work for you. if you got it installed on your machine just enable it (by writing a simple rule, something like "iptables -A INPUT -p tcp --dport 80 --state NEW,ESTABLISHED -j ACCEPT") and then set the size of the connection table to some small number (check google how to do it). the idea behind it is, that i assume (i didn't try it or investigated in it!!) that the connection tracking will always drop the connection that was the longest non active and so the connections that send something should be kept alive and the "just open" sessions would be dropped. if you set the number to 100 or something, the backend httpd process should be protected (maybe). but take care that connection tracking doesn't lock you out as it is used on all connection (not just the one you write a rule for) good luck markus
Thanks,
Ralf Koch
Markus Roth schrieb:
media Formel4 wrote:
Question is:
- Is it possible with spoofed IP numbers to establish connections to port 80? As far as I know you should get stuck after "SYN". I'm asking that, because tracing back the IPs in question I find very often unrouted areas and non-reachable (but maybe firewalled) IPs.
i would say no (else the school was pretty useless ;-)
"There are more things in heaven and earth, Horatio, then are drempt of in your philosophy." Maybe you can send a spoofed SYN followed by a (or a dozen?) spoofed ACK where you "guess" the correct seq_num/ack_num? I'm not sure if this is not possible...
- How can I secure this server and/or stop this attack?
this attack is very mean and it succeeds almost always (even if you just do it from a single attacking machine). i would do a search on google, there are definitively others who were under the same sort of attack.
All Ideas they produce is something like "Change the IP" - which is IMHO not a good solution, because not everything on that server is hostname driven...
just some thoughts about how it could be possible to protect (at least a bit). maybe it's possible to let netfilters connection tracking do the work for you. if you got it installed on your machine just enable it (by writing a simple rule, something like "iptables -A INPUT -p tcp --dport 80 --state NEW,ESTABLISHED -j ACCEPT") and then set the size of the connection table to some small number (check google how to do it). the idea behind it is, that i assume (i didn't try it or investigated in it!!) that the connection tracking will always drop the connection that was the longest non active and so the connections that send something should be kept alive and the "just open" sessions would be dropped. if you set the number to 100 or something, the backend httpd process should be protected (maybe). but take care that connection tracking doesn't lock you out as it is used on all connection (not just the one you write a rule for)
That might be worth a thought. Right now I've got a script running checking the web server and when MaxClients is reached for more then 20 seconds, all IPs are collected and every IP that was more then 5 times in that collection get blocked. I've got now a list of more then 4700 IPs blocked and the attack is still going on...
good luck
Thanks, guess I need it... Ralf Koch
Ralf, On Thursday 27 October 2005 08:24, media Formel4 wrote:
...
All Ideas they produce is something like "Change the IP" - which is IMHO not a good solution, because not everything on that server is hostname driven...
Perhaps a first step is to correct this problem. If this is a production system, then you absolutely need the flexibility of redeploying and reconfiguring, so embedded IP addresses are almost certainly a mistake.
...
...
Ralf Koch
Randall Schulz -- P.S. Please don't send mail to my personal mailbox! The list is the place (the only place) for this dialog.
My idea might be a a beginners idea. But may be... How is it about the construction: - current apache: very short timeout + automatic forward of the page to another subdomain. - subdomain: to host the "regular" pages with regular timeout. That should work without large differents for the clients and should block the attack. Best regards, Christian
What about if you could modify your script to tell apache via localhost that those connections are finished. So as the bad packets attack apache with half-opened connections, as your script identifies those rouge connections, your script spoofs some packets locally on your machine, and sends them to apache, telling it those connections are FINished and no longer needed? Would that work? Regards - Keith Roberts On Thu, 27 Oct 2005, media Formel4 wrote:
That might be worth a thought. Right now I've got a script running checking the web server and when MaxClients is reached for more then 20 seconds, all IPs are collected and every IP that was more then 5 times in that collection get blocked. I've got now a list of more then 4700 IPs blocked and the attack is still going on...
I don't think that works out. Whenever I might send a FIN - what prevents my Apache from being attacked from the same bot after seconds again? This is the same with reducing the timeout. Whenever I do this, I only raise the load of the server - but the same IP numbers keep bullying my Apache again and again... Thanks, Ralf suse@karsites.net schrieb:
What about if you could modify your script to tell apache via localhost that those connections are finished.
So as the bad packets attack apache with half-opened connections, as your script identifies those rouge connections, your script spoofs some packets locally on your machine, and sends them to apache, telling it those connections are FINished and no longer needed?
Would that work?
Regards - Keith Roberts
On Thu, 27 Oct 2005, media Formel4 wrote:
That might be worth a thought. Right now I've got a script running checking the web server and when MaxClients is reached for more then 20 seconds, all IPs are collected and every IP that was more then 5 times in that collection get blocked. I've got now a list of more then 4700 IPs blocked and the attack is still going on...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Thursday 2005-10-27 at 18:09 +0200, media Formel4 wrote:
I don't think that works out. Whenever I might send a FIN - what prevents my Apache from being attacked from the same bot after seconds again?
The script would have to do both things, close the connection in apache and lock the incoming IP. But, if those IPs are spoofed, as you think, chances are some will seem to come from your real clients sometime. Best thing would probably be a module in apache for ignoring empty requests. Is it doable? What about the MACs, can they be traced? Any matches there? Forgive me if that's a novice like question. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFDZLmztTMYHG2NR9URAuinAJ4rmdmf58Aa7QAx6RjuYs944Q58qQCdG5wP 8Ge19SbRy4DaVBB2M/jjfDo= =fbKO -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
The Thursday 2005-10-27 at 18:09 +0200, media Formel4 wrote:
I don't think that works out. Whenever I might send a FIN - what
Mitigating DDoS attacks is mostly contingent on the type of attack going on. Back in the older days, simple SYN cookies and proper load ballancing could mitigate most of the attacks going on. Attacks are much more complicated these days. I've seen DDoS attacks in the form of DNS requests. You get 20k machines requesting queries from your DNS server, GOOD LUCK! It's also worth noting that changing IP addresses works about as often as the other criticized suggestions. A large portion of the attacks going on these days, reflect what the underground hackers are calling "DRDoS" attacks. These attacks involve dropping uplink providers by overwhelming border-gate routers and the likes. Changing your ip address will have absolutely no effect in these cases. It's hard to tell when these types of attacks are going on because the gate router, doing its job, simply submits the traffic to the entire subnet. I've disassembled drone nets that exceeded 20k infected machines. Some of them were dial-up accounts, most of them were cable/dsl accounts. Attacks don't need to be "professional" in any capacity. 20k dialup connections is enough to do some sort of damage. In most cases, packet throttling with QoS and Syn cookies, is a viable means of mitigating attacks. Of course it doesn't always work. I'm Joe Schmo sitting at home with $50 pseudo router. But if you're running a business, on the internet, you need to have some of these "best practices" ironed out. Also, colocations (i think it was mentioned in an earlier posting) seems to be quite productive in mitigating several forms of DDoS attacks out there. Tim Rainier Information Services, Kalsec, INC trainier@kalsec.com "Carlos E. R." <robin1.listas@tiscali.es> wrote on 10/30/2005 07:16:40 AM: prevents my
Apache from being attacked from the same bot after seconds again?
The script would have to do both things, close the connection in apache and lock the incoming IP. But, if those IPs are spoofed, as you think, chances are some will seem to come from your real clients sometime. Best
thing would probably be a module in apache for ignoring empty requests. Is it doable?
What about the MACs, can they be traced? Any matches there? Forgive me if that's a novice like question.
- -- Cheers, Carlos Robinson
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76
iD8DBQFDZLmztTMYHG2NR9URAuinAJ4rmdmf58Aa7QAx6RjuYs944Q58qQCdG5wP 8Ge19SbRy4DaVBB2M/jjfDo= =fbKO -----END PGP SIGNATURE-----
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
media Formel4 wrote:
- Is it possible with spoofed IP numbers to establish connections to port 80? As far as I know you should get stuck after "SYN". I'm asking that, because tracing back the IPs in question I find very often unrouted areas and non-reachable (but maybe firewalled) IPs.
Also I found a group of 300 IPs coming from an american company network. I contacted them and they stated too, that those IPs were not in use and not routed right now...
- How can I secure this server and/or stop this attack?
I think that you are looking at wrong point. Preventing a DDOS is not the job of the web server, but the job of the router/firewall. "Real routers/firewalls" will deal easily with these problems. - No spoofing of IPs through validation where the packet comes from... - No fragmented packets - Limit the number of open/unfinished connections... Cisco Pix 501, 515... depending on size and volumes Cisco 1811... Not cheap but when configured properly, guaranteed to work. -- Thanks http://www.911networks.com When the network has to work Cisco/Microsoft
Syv Ritch schrieb:
media Formel4 wrote:
- How can I secure this server and/or stop this attack?
I think that you are looking at wrong point. Preventing a DDOS is not the job of the web server, but the job of the router/firewall. "Real routers/firewalls" will deal easily with these problems.
Sure - but therefore you need your own network environment. I'm talking about a root server. No chance (right now) to place a hardware firewall in front.
- No spoofing of IPs through validation where the packet comes from... - No fragmented packets - Limit the number of open/unfinished connections...
Is there a way to set this up with iptables? I know there are things like --limit and --iplimit inside - but that won't help me with the attack which is set by full established HTTP connections which simply don't carry a request...
media Formel4 wrote:
Sure - but therefore you need your own network environment. I'm talking about a root server. No chance (right now) to place a hardware firewall in front.
Then, you have to wait until it's over. The zombies die after a few days or they are re-rented as spam-bots and get RBLed (and the ISP they belong to cuts them off). Or go to incidents@securityfocus.com and ask there if someone else has a solution. cheers, Rainer
On Thu, Oct 27, 2005 at 04:23:27PM +0200, media Formel4 wrote:
Question is:
- Is it possible with spoofed IP numbers to establish connections to port 80? As far as I know you should get stuck after "SYN".
If Syn cookies are enabled it could be done by blind connection forgery. Sending lots of pakets containing random data instead of real cookie content could result in very few connections. Not an appropriate method to start some DoS attack.
I'm asking that, because tracing back the IPs in question I find very often unrouted areas and non-reachable (but maybe firewalled) IPs.
What if someone managed to compromise a router nearby? -- Stefan Tichy ( s.list at pi4tel dot de )
Hi all, Please bear with me as I have just started to gather entropy on this thread. As a first step I find useful to organize thoughts to bring in the gaps between each helping mind around this issue. So... 1) The attack As far as I can tell the attack is characterized by invalid connections to TCP/IP port 80 with difusely distributed origin (DDoS). Those connections are eating up computational resources (read file descriptors) rendering the server unusable by legitimate users. Connections eat up resources for a fixed time when a new connection is estabilised by the attacker. 2) Defense The pratical defensive strategy is blocking the attacker before the bad guy hits the server process (on the webserver or ISP). Reducing time out is considered unpractical since dynamic pages which need interaction with databases would be impacted. The attacker is also using distributed computational resources, which lead to the belief that reducing the timeout would only shorten the attack loop. Hypothetically the origin IPs can be being spoofed (unlikelly if the 3 way handshake is being completed). As the attack is not sending any data, making a signature is very hard. Attack has been mitigated by blocking IPs that have more than 5 connections whenever resources reach maximum usage (MaxClients). 3) Weapons Attacker is potentially using compromised machines over the internet where zombies are constantly firing attacks toward the victimin. Defensive weaponary includes firewalling, proxying or adding software layers (mods) in front of the server process in order to identify traffic as offending or legitimate with the smallest false-positive and false-negative ratio. Few questions that would be wise asking the the victim: Q1: Do you suspect of someone that might be interested in being that evil, and unfairlly taking you out of business? Q2: Can you provide attack sample (namelly a tcpdump file) for investigative purposes by this community (it is advisable that you tamper on any sensible information) Q3: Can you spot some kind of pattern of the offending traffic. Could this raise the possibility of making a signature for the attack? Some initial research rose these onto surface, which might be of some use: http://lists.suse.com/archive/suse-security/2003-Aug/0222.html http://dominia.org/djao/limitipconn.html http://lists.suse.com/archive/suse-security/2003-Aug/0230.html http://www.gotroot.com/tiki-index.php?page=Which+mod_security+rules http://www.modsecurity.org/documentation/overview.html Peace, HLM On Thursday 27 October 2005 12:23, media Formel4 wrote:
Hi list,
sorry for the double-post but my thread opener was in reply to another post which confuses the mailinglist structure...
####################################################################
Hi list,
right now we're experiencing a (for me) very uncommon DDoS attack against one of our webservers. Looking with netstat we find hundreds of established connections to our Apache webserver, but nothing in the logs - which means the attacker opens up a connection (not only a SYN request as in SYN flood attacks) and then blocks the Apache child until it hits timeout. This attack comes from thousands of IP numbers (bots?) all over the world.
Question is:
- Is it possible with spoofed IP numbers to establish connections to port 80? As far as I know you should get stuck after "SYN". I'm asking that, because tracing back the IPs in question I find very often unrouted areas and non-reachable (but maybe firewalled) IPs.
Also I found a group of 300 IPs coming from an american company network. I contacted them and they stated too, that those IPs were not in use and not routed right now...
- How can I secure this server and/or stop this attack?
Thanks,
Ralf Koch
participants (11)
-
Carlos E. R. -
Christian Volkmann -
Helio Leonardo Mota -
Markus Roth -
media Formel4 -
Rainer Duffner -
Randall R Schulz -
Stefan Tichy -
suse＠karsites.net -
Syv Ritch -
trainier＠kalsec.com