Detection of DoS Attacks on Webserver
Hi! I'm currently doing my diploma work. its about detecting HTTP-DoS Attacks on web servers. it tries to detect this attacks based on statistical anomalies of the requests. it looks at the request rate (with an exponential moving average), the period in which a client is active, the time distribution of the requests (chi square), the return status of the web server and the distribution of the URI's hit. till now, it looks pretty promising but one very important factor is unaccounted, the coast of a single request! what i like to do is calculating a coast for each request, so that request that charge the CPU more or demand lots of bandwidth, have a higher suspicious level. the hole analysis process is done on the logfile of a web server. to train the system old logfiles are feed. after that the system switches to live mode and watches the access logfile (like tail -f) for changes and tries to detect attacks in "real time" (with the possibility to block them via an apache module or the firewall (time based rule)). some companies in switzerland sponsored their logfiles to develop the system. the bad thing, non of this logfiles contains the time to handle the requests! now my question, is somebody willing to give me some log data in combined format but with the time to handle a request. apache2 would log this with the "%D" parameter in microseconds. logfiles with about 100'000 - x'000'000 requests would be optimal. you can anonimize your logdata with http://sourceforge.net/projects/anonlog/ if you like or have to. an other thing which would be very interesting is if somebody has logfiles where script kiddies tried to bring down a web server with lots of requests to cpu intensive dynamic pages or got attacked in an other way. if you think i'm running in the wrong direction with this project or have other things on your mind which i should consider, please let my know! regards markus roth
Try www.grc.com for DOS attack story. On Sat, 13 Nov 2004 17:49:51 +0100, Markus Roth <mroth@netlabs.ch> wrote:
Hi!
I'm currently doing my diploma work. its about detecting HTTP-DoS Attacks on web servers. it tries to detect this attacks based on statistical anomalies of the requests. it looks at the request rate (with an exponential moving average), the period in which a client is active, the time distribution of the requests (chi square), the return status of the web server and the distribution of the URI's hit. till now, it looks pretty promising but one very important factor is unaccounted, the coast of a single request! what i like to do is calculating a coast for each request, so that request that charge the CPU more or demand lots of bandwidth, have a higher suspicious level.
the hole analysis process is done on the logfile of a web server. to train the system old logfiles are feed. after that the system switches to live mode and watches the access logfile (like tail -f) for changes and tries to detect attacks in "real time" (with the possibility to block them via an apache module or the firewall (time based rule)).
some companies in switzerland sponsored their logfiles to develop the system. the bad thing, non of this logfiles contains the time to handle the requests! now my question, is somebody willing to give me some log data in combined format but with the time to handle a request. apache2 would log this with the "%D" parameter in microseconds. logfiles with about 100'000 - x'000'000 requests would be optimal. you can anonimize your logdata with http://sourceforge.net/projects/anonlog/ if you like or have to.
an other thing which would be very interesting is if somebody has logfiles where script kiddies tried to bring down a web server with lots of requests to cpu intensive dynamic pages or got attacked in an other way.
if you think i'm running in the wrong direction with this project or have other things on your mind which i should consider, please let my know!
regards markus roth
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
Markus You could try posting your request to the Log Analysis mailing list http://lists.shmoo.com/mailman/listinfo/loganalysis If they cant help directly, your work would be of interest to them. The companion loganalysis.org web site is also worth a look http://www.loganalysis.org/ On Sat, 13 Nov 2004 17:49:51 +0100, Markus Roth <mroth@netlabs.ch> wrote:
Hi!
I'm currently doing my diploma work. its about detecting HTTP-DoS Attacks on web servers. ...SNIP... some companies in switzerland sponsored their logfiles to develop the system. the bad thing, non of this logfiles contains the time to handle the requests! now my question, is somebody willing to give me some log data in combined format but with the time to handle a request. apache2 would log this with the "%D" parameter in microseconds. logfiles with about 100'000 - x'000'000 requests would be optimal.
All the best Richard Westlake School of Crystallography, Birkbeck College, Malet Street, London WC1E 7HX Tel: 020-7631-6859 ---------------------------------------------------------------------- Truth endures but spelling changes -- Anon. ----------------------------------------------------------------------
Hi Markus. Interesting topic. Your idea seems very handy for doing forensic analysis, after a HTTP-DoS/DDoS attack. I think that IPTables firewall could be used to help limit or prevent such attacks from occuring. There is a development library for the IPTables packet filter, that allows a user to write loadable modules for the packet filter. I think it should be possible to write a module that will que incoming packets in userland memory. The packets can then be inspected for certain clues that would be indicative of a HTTP-DoS attack. DDoS may be a bit more trickier to detect, as the source IP's will be varied, but even so, there may still be a very high number of new connection requests coming, in a very short time, from the same source IP, which would indicate a possible DoS or DDoS attack underway. The user written module should then be able to generate and add new rules to the IPTables firewall, to block such DoS/DDoS attacks. After a certain amount of time, the user written module should then be able to remove those added rules from the firewall packet filter. I suppose you would call this adaptive or intelligent firewalling, as the firewall adapts itself in response to what it sees in the INPUT chain. I need to write a white paper on this, and make it available for all to read, and hopefully someone will take up the idea and develop it into something functional! Kind Regards - Keith Roberts http://www.karsites.net/
On Saturday 13 November 2004 17:35, suse@karsites.net wrote:
Hi Markus.
Interesting topic.
Agreed and I've done some work in this area. http://www.networkengineer.biz/ExecutiveADS/index.htm
Your idea seems very handy for doing forensic analysis, after a HTTP-DoS/DDoS attack.
actually one can nip such in the bud and tell others.
I think that IPTables firewall could be used to help limit or prevent such attacks from occuring.
Alas such solution is quite linux-specific. This problem is of far wider scope. Even if you argue to leave Microsoft users of IIS to their fate apache runs on far more than Linux.
There is a development library for the IPTables packet filter, that allows a user to write loadable modules for the packet filter.
yes that's a reasonable approach on Linux but you have to construct solutions in a modular fashion. Certainly "firewall rule" is an option (but you can't just stick it in there and leave it forever, it has to be aged out at some point)
I think it should be possible to write a module that will que incoming packets in userland memory. The packets can then be inspected for certain clues that would be indicative of a HTTP-DoS attack.
very apache-specific. Furthermore, the API may well change -- indeed can same module work on v1 and v2 apache? no.
DDoS may be a bit more trickier to detect, as the source IP's will be varied, but even so, there may still be a very high number of new connection requests coming, in a very short time, from the same source IP, which would indicate a possible DoS or DDoS attack underway.
Ah, but in this case you see they are open proxies and if you but detect them with my Perl module for same...
I need to write a white paper on this, and make it available for all to read, and hopefully someone will take up the idea and develop it into something functional!
Actually I need to -publish- a paper on this at a conference this spring. Whitepaper is already up and far more comprehensive in its vision. funding would help.
Your idea seems very handy for doing forensic analysis, after a HTTP-DoS/DDoS attack.
actually one can nip such in the bud and tell others.
did not realise the method Markus was using was almost in real time.
I think that IPTables firewall could be used to help limit or prevent such attacks from occuring.
Alas such solution is quite linux-specific. This problem is of far wider scope. Even if you argue to leave Microsoft users of IIS to their fate apache runs on far more than Linux.
Yes - agreed. Markus idea of using the web server logs is alot more practical - and also should work with any webserver logs, and firewall that supports dymamic loading/unloading of firewall rules! I had ideas on implementing something at a packet filtering level, that would also be possible to implement in main IP backbone routers - by doing some sort of intelligent filtering as close to the source of the DoS attack as possible.
I think it should be possible to write a module that will que incoming packets in userland memory. The packets can then be inspected for certain clues that would be indicative of a HTTP-DoS attack.
very apache-specific. Furthermore, the API may well change -- indeed can same module work on v1 and v2 apache? no.
ditto as above Regards - Keith Roberts Will read & study and reply to Markus's email soon! Starting to formulate some ideas for his version of DoS attack prevention - this could even turn into something of practical use, when Markus has finished his thesis!
On Sunday 14 November 2004 02:03, Dana Hudes wrote:
yes that's a reasonable approach on Linux but you have to construct solutions in a modular fashion. Certainly "firewall rule" is an option (but you can't just stick it in there and leave it forever, it has to be aged out at some point)
Just ignoring (firewalling) incoming traffic is not going to keep your webserver on the net, when bandwidth is depleted. A firewall rule on your side is not going to stop a DDoS attack if it is saturating your connection (a coordinated attack from a few hundred zombies probably will be sufficient). Now how is such an automated tool supposed to contact your uplink provider and filter out this traffic, before it can clog your connection? Arjen
On Tue, 16 Nov 2004, Arjen de Korte wrote:
To: suse-security@suse.com From: Arjen de Korte <suse+security@de-korte.org> Subject: Re: [suse-security] Detection of DoS Attacks on Webserver
Just ignoring (firewalling) incoming traffic is not going to keep your webserver on the net, when bandwidth is depleted. A firewall rule on your side is not going to stop a DDoS attack if it is saturating your connection (a coordinated attack from a few hundred zombies probably will be sufficient). Now how is such an automated tool supposed to contact your uplink provider and filter out this traffic, before it can clog your connection?
Arjen
--snip-- I think there is a way to detect DDoS attacks as well. IIRC, a DDoS attack is done by many machines sending TCP connection requests to different http servers, with the source IP address being spoofed with the IP address of the http server to target in the attack. As the different servers receive the TCP connect requests, they respond by sending an SYN/ACK packet back to the spoofed address of the server under attack. Why would a http server send out an SYN connection request to another http server? AFAIK It's only the browsers that normally send TCP connection request packets to http servers. The way I see it is like this: If a http web server is receiving loads of SYN/ACK packets then this is NOT normal or expected behaviour, as it's the servers job to send out these packets to a client's browser, in response to the original SYN new connection request sent by that client. I think it may be possible to set up some sort of firewall packet level monitoring, that would be implemented in the main routers on the internet. This packet monitoring would then look for packets with the following characteristics: 1> the destination IP address is the same (it has to be the same, otherwise the DDoS attack would not work!) 2> the packet is a SYN/ACK packet (which should not really under normal circumstances be being sent to an http server - not to sure about proxies or http forwarding requests though) these suspect SYN/ACK response packets are all targeting the same IP destination address. This is definately NOT normal behaviour. (A client will send out a few SYN new connection request packets to a web server, then wait for the SYN/ACK response from that server, returning to the client.) Under a heavy DDoS attack, there will be a great amount of the above type of packets, all sent to the same IP address. This is usually enough to block all access to the server, by virtue of the sheer number of packets being sent. The main internet routers should then be able to build the appropriate dynamic firewall rules to block all these suspect packets. This would stop the DDoS attack from even reaching it's intended target. The web server should then be able to function as normal, without even being aware that it under a DDoS attack! --snip-- Kind Regards - Keith Roberts http://www.karsites.net
A denial of service attack is one thing if its smurf but ISPs often rate-limit ICMP as a matter of policy to perhaps 64kbit/second. this may be triggerd only after certain volume etc. A web site is normally attacked with tcp open and perhaps even more it is attacked by sending various buffer overrun and of course brute force password attacks. Blocking Ip packets from the source address of such attackers at even the local host of the web server itself still prevents it from getting up the stack to the application layer where it causes all sorts of httpd processing. Instead you just throw the SYN on the floor. Far more effective than a .htaccess rule at reducing the load on your server. If you can clip it at the WAN gateway router so much the better. Of course one would like to have ISP filter it but no system exists now to distribute in an authenticated manner the IP address of zombies and other attackers. My work includes plans for such mechanisms.
suse@karsites.net wrote:
Hi Markus.
Hi Keith!
Interesting topic.
yes it is. do your remember that you lead me 10 months ago in this direction? i asked on the list for an interesting security topic for my diploma work and you told me about an iptables module to try to limit (D)DoS attacks. i invested 4 months to learn about DDoS attacks and tried to do a concept to block them. at the beginning everything looked quite simple, count the number of ip packets for each ip and so on. but after a while i had to admit that it wasn't at all as simple as i thought. if somebody is interested in why i think it is not possible with this attempt i can write more on it. ok, so i had to change my project a little bit. cause the (D)DoS topic is very interesting i decided to stay on it and do it for web servers. the main advantage over the previous work is that it is on the application layer and so i don't have to cope with spoofed packets were every packet is coming from a different ip address and there is no real chance to block such an attack.
Your idea seems very handy for doing forensic analysis, after a HTTP-DoS/DDoS attack.
I think that IPTables firewall could be used to help limit or prevent such attacks from occuring.
yes, this is already done. the system works like this. after the system is trained, it is reading continuously from the apache access logfile (or whatever web server can output combined or common format). The analysis happens with a delay of about 1 second. if a very implausibly or suspicious behavior of a client is detected, this ip address gets temporarily blacklisted. This blacklist is implemented via an apache module. if this client makes its next request it gets an HTTP error, explaining that he violated our policy and should now go to drink a coffee. when he does it (waits a little bit), he will be allowed to continue surfing on our server. if more requests knock on our door form this ip, we conclude that there's no human being behind this requests and block this ip on the firewall. i wrote a simple script which allows to do time based rules for iptables, so rules get removed automatically after some time.
There is a development library for the IPTables packet filter, that allows a user to write loadable modules for the packet filter.
I think it should be possible to write a module that will que incoming packets in userland memory. The packets can then be inspected for certain clues that would be indicative of a HTTP-DoS attack.
at the beginning of the project i wanted to write exactly such a module (cause netfilter hacking is very cool . the problem is that you are in the wrong place (my opinion) on the firewall. i found out that i would have to write a hole proxy that i can analyse the requests. its by far not enough to just look at the ip packets with the GET, POST or whatever request type in it. you have to consider fragmented ip packets and you have also to look at the response from the web server. the next problem is, that if your server supports SSL connections, your out of business (or you have to do man in the middle). at the end it was to much work to write a very fast proxy that runs on the firewall and is able to implement all HTTP standards without being an own security weakness. the next idea was to write an apache module which does the analysis. a company offered my ssh access to their web server. the problem was that the web server had to run at all times and so to restart the server all the time i compiled my apache module wasn't an option. the third idea, is now the one that gets implemented. it just works on the logfiles. this is not the fastest solution but the one with the cleanest separation. one very big advantage of this layout is, that the system can be trained with old logdata from many days in just a few minutes.
DDoS may be a bit more trickier to detect, as the source IP's will be varied, but even so, there may still be a very high number of new connection requests coming, in a very short time, from the same source IP, which would indicate a possible DoS or DDoS attack underway.
until now DDoS attacks aren't handled explicitly. the process implemented to detect attacks looks at the behavior of the request. it tries to find inhuman behavior. for example it is uncommon that a user makes every second 10 requests to always the same URL over a minute on the other side it is common that a user makes 100 requests to 100 URL s in 2 seconds (and then waits for some time) because he downloads a page with many images on it. in the training phase the program tries to find out what the common behavior for a web server is. i won't have time to implement something for DDoS attacks in this diploma work (cause it ends in a few weeks) but the link from martin mcleod brought me to an idea. the problem with DDoS attacks is that a single attack source isn't harmfully but a lot of them are. so i could try to classify the behavior of all clients. if a lot of clients have the same classification simultaneously and are using together a lot of resources, all of them get looked out....
The user written module should then be able to generate and add new rules to the IPTables firewall, to block such DoS/DDoS attacks.
After a certain amount of time, the user written module should then be able to remove those added rules from the firewall packet filter.
I suppose you would call this adaptive or intelligent firewalling, as the firewall adapts itself in response to what it sees in the INPUT chain.
well this sounds good -> adaptive or intelligent, will use this in my documentation
I need to write a white paper on this, and make it available for all to read, and hopefully someone will take up the idea and develop it into something functional!
i'm always having an open ear for your ideas!
Kind Regards - Keith Roberts
cu markus
* Markus Roth; <mroth@netlabs.ch> on 13 Nov, 2004 wrote:
Hi!
an other thing which would be very interesting is if somebody has logfiles where script kiddies tried to bring down a web server with lots of requests to cpu intensive dynamic pages or got attacked in an other way.
if you think i'm running in the wrong direction with this project or have other things on your mind which i should consider, please let my know!
Have you looked at modsecurity http://www.modsecurity.org -- Togan Muftuoglu | Unofficial SuSE FAQ Maintainer | Please reply to the list; http://susefaq.sf.net | Please don't put me in TO/CC. Nisi defectum, haud refiecendum
Togan Muftuoglu wrote:
* Markus Roth; <mroth@netlabs.ch> on 13 Nov, 2004 wrote:
Hi!
an other thing which would be very interesting is if somebody has logfiles where script kiddies tried to bring down a web server with lots of requests to cpu intensive dynamic pages or got attacked in an other way.
if you think i'm running in the wrong direction with this project or have other things on your mind which i should consider, please let my know!
Have you looked at modsecurity http://www.modsecurity.org
thanks for the hint! i didn't found modsecurity when i was looking for existing work in this area. modsecurity looks very good (and i had almost a cardiac arrest when i saw the pretty page and thougt all my work is for /dev/null, sombody else did it before ;-). now i'm quite glad that i saw that modsecurity does something diffrent than i do. modsecurity looks for all kind of injection or other misuse attacks using known patterns, my system should detect (D)DoS attacks that target the machines cpu, memory, bandwidth and such stuff unsing anomaly detection. regards markus
* Markus Roth; <mroth@netlabs.ch> on 16 Nov, 2004 wrote:
thanks for the hint!
I am not so sure if you are going to like this second link though http://www.nuclearelephant.com/projects/dosevasive/ -- Togan Muftuoglu | Unofficial SuSE FAQ Maintainer | Please reply to the list; http://susefaq.sf.net | Please don't put me in TO/CC. Nisi defectum, haud refiecendum
thanks again this one i like very much! my project is actually based on the idea of mod_dosevasive. when i started my work, i reviewed the code of mod_dosevasive. i found some things that i didn't liked very much and tried to contact the author. i never got an answer so i decided to start an own project. mod_dosevasive is pretty basic. it is absolutly static, it just looks if more then X requests are comming from the same ip in Y seconds (or if Z requests hit the same URL in T seconds). this module is quite good to defeat attacks but you will get a huge amount of false positives. it is not able to recognize a users behaviour it just counts requests. regards markus Togan Muftuoglu wrote:
* Markus Roth; <mroth@netlabs.ch> on 16 Nov, 2004 wrote:
thanks for the hint!
I am not so sure if you are going to like this second link though http://www.nuclearelephant.com/projects/dosevasive/
Hello everybody I allow my to send you a link to the results of my diploma work: http://www.solutix.ch/scrutinizer The last 10 weeks my partner and I worked very hard to do this project. Not everything worked out like expected and our project management was a mess. We weren't able to finish everything and it's not ready for productive use. However we archived very good results during testing. If you're interested in this topic plase have a look! Some cut backs had to be made on documentation, on how to train the system. if you have a question please ask, I'm willingly to answer. Thanks to all the people how already gave me feedback on this project! Have a nice christmas time Markus
participants (7)
-
Arjen de Korte -
Dana Hudes -
Markus Roth -
martin mcleod -
Richard Westlake -
suse＠karsites.net -
Togan Muftuoglu