On Monday 07 November 2005 07:52 pm, Dan Abernathy wrote:

Hello all

I notice several automated break-in attempts appearing in /var/log/messages regarding sshd. A small sample:

Nov 7 14:34:10 d8400 sshd[18607]: Invalid user a from 71.129.198.189 Nov 7 14:34:11 d8400 sshd[18609]: Invalid user aaron from 71.129.198.189

Hundreds of entries like the above, working their way through every English letter using common first names, also names of services like Apache.

I ran across this Novell Cool Solutions article: http://www.novell.com/coolsolutions/trench/16341.html

It describes the use of a shell script, run once per minute using a cron job, that parses information from /var/log/messages and adds offending IP addresses to /etc/hosts.deny.

I saved the script as ssh-block.sh in my home folder, in a subdirectory named cronjobs. I used su to become root and ran crontab -e, and added the script to run once per minute.

Entries in /var/log/messages confirm that the shell script is indeed being run every minute, but the contents of /etc/hosts.deny hasn't changed at all, despite the presence of failed sshd log-ins (also written to /var/log/messages).

Can someone knowledgeable about shell scripts take a look at the URL above and see if there is a problem with the script? If not, what did I do incorrectly? What's the proper way to run this job regularly? I assume it must be run as root, both to read /var/log/messages and to write to /etc/hosts.deny.

Dan

Since you didn't give us your cron entry, I can only assume you used the full path to the script in the cron line.... Also, what are the permissions on the script? Did you chmod it for execution?

On Monday 07 November 2005 4:52 pm, Dan Abernathy wrote:

Entries in /var/log/messages confirm that the shell script is indeed being run every minute, but the contents of /etc/hosts.deny hasn't changed at all, despite the presence of failed sshd log-ins (also written to /var/log/messages).

So, what happens when you su to root and run the script manually? Does /etc/hosts.deny get updated? Scott -- POPFile, the OpenSource EMail Classifier http://popfile.sourceforge.net/ Linux 2.6.11.4-21.9-default x86_64 SuSE Linux 9.3 (x86-64)

* Dan Abernathy [11-07-05 19:55]:

I notice several automated break-in attempts appearing in /var/log/messages regarding sshd. A small sample:

Nov 7 14:34:10 d8400 sshd[18607]: Invalid user a from 71.129.198.189 Nov 7 14:34:11 d8400 sshd[18609]: Invalid user aaron from 71.129.198.189

Hundreds of entries like the above, working their way through every English letter using common first names, also names of services like Apache.

I ran across this Novell Cool Solutions article: http://www.novell.com/coolsolutions/trench/16341.html

It describes the use of a shell script, run once per minute using a cron job, that parses information from /var/log/messages and adds offending IP addresses to /etc/hosts.deny.

Look closely at http://sf.net/projects/denyhosts I have been using it for about a month and it appears to work very well. added to root's crontab */6 * * * * /usr/bin/denyhosts.py -c /etc/denyhosts.cfg has it's own annotated config file -- Patrick Shanahan Registered Linux User #207535 http://wahoo.no-ip.org @ http://counter.li.org HOG # US1244711 Photo Album: http://wahoo.no-ip.org/gallery2

Patrick Shanahan wrote:

* Dan Abernathy [11-07-05 19:55]:

...

I notice several automated break-in attempts appearing in /var/log/messages regarding sshd. A small sample:

Nov 7 14:34:10 d8400 sshd[18607]: Invalid user a from 71.129.198.189 Nov 7 14:34:11 d8400 sshd[18609]: Invalid user aaron from 71.129.198.189

Hundreds of entries like the above, working their way through every English letter using common first names, also names of services like Apache.

I ran across this Novell Cool Solutions article: http://www.novell.com/coolsolutions/trench/16341.html

It describes the use of a shell script, run once per minute using a cron job, that parses information from /var/log/messages and adds offending IP addresses to /etc/hosts.deny.

Look closely at http://sf.net/projects/denyhosts

I have been using it for about a month and it appears to work very well.

I created my own using Perl and fileschanged (http://fileschanged.sourceforge.net/). The fileschanged monitors the secure log and when the log changes calls my Perl script. My script determines if new entries are like Nov 7 13:37:15 hostname sshd[9847]: Did not receive identification string from 221.253.105.173 or Nov 7 13:49:35 hostname sshd[9992]: Illegal user james from 61.144.56.34 If I get a match an entry is added using iptables to reject the host. The script traps the dictionary attacks within a few seconds. A side effect of using iptables is the incoming connection hangs waiting for a timeout. When I mentioned this script to friend at work, his response was to block everyone except IPs or domains where you expect connection to come from instead of being reactive. Tom

On Tue, 8 Nov 2005 11:52 am, Dan Abernathy wrote:

I notice several automated break-in attempts appearing in /var/log/messages regarding sshd. A small sample:

Just to follow up on my last post, to implement rate limiting on SSH attacks I pasted Andrew Pollock's rules into /etc/sysconfig/Susefirewall2-custom Edit /etc/sysconfig/Susefirewall2 and uncomment FW_CUSTOMRULES="/etc/sysconfig/scripts/SuSEfirewall2-custom" /etc/init.d/Susefirewall2_setup reload See how it goes! Here is the /etc/sysconfig/Susefirewall2-custom file. Any line that begins with "--" has been wrapped and needs to be appended to the line before. # # /etc/sysconfig/Susefirewall2-custom # ###################################################### # Rate limit brute force SSH attacks, rules by Andrew Pollock # # # # http://blog.andrew.net.au/2005/02/17#ipt_recent_and_ssh_attacks # #-----------------------------------------------------------------------# # First whitelist a few hosts iptables -N SSH_WHITELIST iptables -A SSH_WHITELIST -s trusted.host -m recent --remove --name SSH -j ACCEPT iptables -A SSH_WHITELIST -s also.reliable -m recent --remove --name SSH -j ACCEPT iptables -A SSH_WHITELIST -s alianet.alia.org.au -m recent --remove --name SSH -j ACCEPT iptables -A SSH_WHITELIST -s flat.alia.org.au -m recent --remove --name SSH -j ACCEPT # Then implement the "recent" based filter iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set --name SSH iptables -A INPUT -p tcp --dport 22 -m state --state NEW -j SSH_WHITELIST iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j ULOG --ulog-prefix SSH_brute_force iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl --name SSH -j DROP

On Tuesday 08 November 2005 01:52, Dan Abernathy wrote:

Hello all

I notice several automated break-in attempts appearing in /var/log/messages regarding sshd. A small sample:

Nov 7 14:34:10 d8400 sshd[18607]: Invalid user a from 71.129.198.189 Nov 7 14:34:11 d8400 sshd[18609]: Invalid user aaron from 71.129.198.189

You could use public key authentication only and disable password logins. That way you could sleep better and all the other methods described would serve that these attacks don't fill up your logs. Ulf

I would recommend DenyHosts project (http://denyhosts.sourceforge.net/) Seems to work pretty well in my case... Dmitry

Thomas A. Lowery wrote:

When I mentioned this script to friend at work, his response was to block everyone except IPs or domains where you expect connection to come from instead of being reactive. Just recently I noticed on my SuSE 9.0 computer a similar problem with vsftpd attacks. I remember having read this thread so I searched for it to give me some ideas on how to respond. I mentioned my problem to an admin at work and he also thought the above approach was a good idea especially since I only run that for a few family and friends.

With a little Googling and man page reading I figured out how to configure the hosts.allow and deny. In hosts.deny I have ALL : ALL and in hosts.allow I have something like ALL : somedomain.com. The problem is that the connecting client times out trying to connect. I remember reading somewhere that I had to put the host in hosts or the DNS lookup would timeout so I suppose that is what is happening. If somedomain.com was a fixed address host, then putting it in hosts would be no problem. Since my family and friends all have dynamic addresses, this is a problem. We all use the free name from www.noip.com but I can't put those in hosts since the address is not fixed. How can I get the tcp_wrappers stuff to work with DNS so I can use the dynamic addresses? So far, all the Googling I have done has led me to a few pages that give some basic info about setting up the hosts.allow and hosts.deny files but I haven't found anything about this DNS problem. Damon Register

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Saturday 2006-03-25 at 12:24 -0500, Damon Register wrote:

Just recently I noticed on my SuSE 9.0 computer a similar problem with vsftpd attacks. I remember having read this thread so I searched for it to give me some ideas on how to respond. I mentioned my problem to an admin at work and he also thought the above approach was a good idea especially since I only run that for a few family and friends.

When possible, yes.

With a little Googling and man page reading I figured out how to configure the hosts.allow and deny. In hosts.deny I have ALL : ALL and in hosts.allow I have something like ALL : somedomain.com. The problem is that the connecting client times out trying to connect. I remember reading somewhere that I had to put the host in hosts or the DNS lookup would timeout so I suppose that is what is happening. If somedomain.com was a fixed address host, then putting it in hosts would be no problem. Since my family and friends all have dynamic addresses, this is a problem. We all use the free name from www.noip.com but I can't put those in hosts since the address is not fixed.

How can I get the tcp_wrappers stuff to work with DNS so I can use the dynamic addresses? So far, all the Googling I have done has led me to a few pages that give some basic info about setting up the hosts.allow and hosts.deny files but I haven't found anything about this DNS problem.

I don't think it is a timeout problem. The problem is that you need reverse name solving to work. I mean, you need to resolve the IP your friends are comming in to the name you allowed in the hosts.allow. At the time of the connection, the only thing known is the IP. From the IP the system finds out a name... that will not be that of noip.com, but one from their ISP, or none at all. Bad luck. You could setup a complicated mail robot. Your friend sends a gpg authentificated email saying the IP that he is comming from. The robot, fired by procmail, interprets that, adds the IP to the hosts.allow file, and then your friend can connect. The robot can also program an "at" command to remove the IP after an hour, for instance. But no, don't ask me for such a robot, I don't have it! Another idea: use the "recent" module of iptables to automatically block repeated "polls". There was a brief thread in the security list time ago. I I use this in SuSEfirewall2-custom, function fw_custom_before_antispoofing(): iptables -A INPUT -p tcp --syn --dport 22 -m recent --name sshattack --set iptables -A INPUT -p tcp --dport 22 --syn -m recent --name sshattack --update --seconds 60 --hitcount 6 -j LOG --log-prefix SSH attack: ' iptables -A INPUT -p tcp --dport 22 --syn -m recent --name sshattack --update --seconds 60 --hitcount 6 -j REJECT On the sixth attempt to connect to port 22 within 60 seconds, that IP is rejected. It is simple to modify. Don't ask me much about it, I'm no expert on iptables ;-) - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFEJamAtTMYHG2NR9URAgQOAKCKjqGAiCE6ovoiNlkydHJkTEJuQQCggJSl XO88qQRab/d6B1SZeyl/Y4g= =XbpO -----END PGP SIGNATURE-----

Damon Register wrote:

...

Another idea: use the "recent" module of iptables to automatically block repeated "polls". There was a brief thread in the security list time ago. I think I stumbled on that in googling around today. That sounds interesting but I suppose that I would have to use something newer than SuSE 9.0.

I've just now set it up on my gateway to see how it works - I haven't really been too bothered by the ssh dictionary-attacks, but what Carlos describes looks interesting and quite elegant. Oh, and this gateway is running SuSE 7.1 with kernel 2.4.27. So your SUSE 9.0 should be perfectly fine - maybe with a kernel and/or iptables update. /Per Jessen, Zürich

Per Jessen wrote:

Damon Register wrote:

...

...

Another idea: use the "recent" module of iptables to automatically block repeated "polls". There was a brief thread in the security list time ago. I think I stumbled on that in googling around today. That sounds interesting but I suppose that I would have to use something newer than SuSE 9.0.

I've just now set it up on my gateway to see how it works - I haven't really been too bothered by the ssh dictionary-attacks, but what Carlos describes looks interesting and quite elegant.

There is a problem with the recent module and its interpretation of Jifies. ssh login does not work when one has just booted, until jifie gets 0 and starts incrementing, then it works. (That's roughly 5 minutes.) There is a bug report in Debian about this. (A friend of mine stumbled over that problem, we used that technique for a while for the German CTAN node.) If this is really of interest, I can look up the details of the Jifie problem and post them here. We have now abandoned that approach, also for other reasons. Let me see, maybe the following is of interest... ----- snip snap -------------------------------------------- THE PROBLEM: We need open ssh connections from the outside. We can not turn off password authentication for all users; it must be possible to log in from a customer or a friend's system when one doesn't have one's private ssh key. Lots of password attacks are run against open ssh ports. We want to defend against these attacks in a reasonable way. THE REQUIREMENTS: It would be best to react only on attacks, and not on arbitrary ssh connections. Alternatively, reacting on lots of ssh connections from the same IP address in a short time frame is possible and can be used as an approximation for an attack situation. Manually mantained configuration files should not be changed permanently by automatic procedures. This makes those file hard to maintain and makes them differ from their committed version. (Most configuration files are under version control.) If the protection mechanism needs to keep state, it shall do so in its own file. The ssh server is not necessarily run on the firewall. I.e., the firewall may forward ssh connection to a system in the DMZ. The solution must be integrated into the operations environment. I.e., proper integration into boot procedures, monitoring, log rotation, and other operation processes is mandatory. False positives may happen, i.e., categorization of ssh requests as attacks that aren't. It must be possible to manually correct false positives. Observation has detected that attacks from the same IP address are rare for a longer duration. Using all IP addresses where any attack has ever happened for ssh request rejection is therefore overshoot. It reduces performance and is not good for manual inspection in case of connection problems or false positives. As risk mitigation strategy, it is sufficient to keep connection reject lists for the duration of server uptime, i.e., the list can and should be discarded at boot time. SOLUTION APPROACHES: There are several scripts available that parse log files for failed password attempts and modify /etc/hosts.deny after an attack has been detected. These scripts modify a manually maintained configuration file. The deny rules in this file grow without bounds, no purging is ever done. Integration in boot and log rotation processes does not exist. Therefore we have chosen to skip this approach. The ipt_recent module for iptables allow to specify thresholds for amount of connections in a given time, specific for IP addresses and protocols. That solution would be a very good choice -- if it would work. ipt_recent doesn't work correctly when Jifies in the Linux kernel overflow. Then it blocks every request, even though they didn't pass the threshold. Therefore we have chosen to skip this approach. CHOSEN SOLUTION: We combine the log parsing approach with transient firewall rules that exist only until the next boot. For that, -- An iptables chain inet_ssh is established that is used for all incoming ssh traffic. -- logsurfer is used to parse syslog message files for sshd failed password entries. -- If more than three failed passwords from one IP address appear within 10 minutes, a drop rule for that IP address is added to the inet_ssh chain. -- Failed password attempts from trusted hosts are ignored. Technically, logsurfer is used because I don't know a better log watching and event creation system. I would prefer to have a better one; it is not really suited for the task, as explained below. The logsurfer configuration ignores failed password messages for IP addresses that are known to belong to the Intranet. (If somebody broke in there already, we have other problems than ssh attacks.) Then the logsurfer configuration triggers execution of the command check-ssh-attack if more than three other failed password messages occurs, or if more than 10 minutes passed since such a message occured. This is a serious deficiency of logsurfer that it does only support timeouts and message limits, but no threshold specifications. I.e., I would need `do action <a> after $n$ events in timespan $t$' but logsurfer does not support such typical threshold-triggered event creation. As a work-around, the messages are passed to check-ssh-attack on stdin. That script can now count the messages and can add the iptables drop rule if three messages are received, or ignore them at will. Adding the iptables drop rule is done by an ssh call. This supports the requirement that the ssh daemon and thus the log file can be on a different host than the firewall. The script sshdeny that does the drop rule addition is careful that each IP address is listed only once in that chain (for maintainability) -- multiple invocations of that script can be run in parallel for the same IP address if multiple hosts are attacked at the same time. For that, shlock from the inn distribution is utilized. This has also the advantage that logsurfer may not be run as root. We introduce the new user logsurf:root (group rights appropriate to read /var/log/messages), the ssh key of that user is placed in root's authorized key files. Access with that ssh key is restricted to IP addresses where logsurfer-ssh-depend runs on. (Too bad that one can only specify a command for specific ssh keys, but is not allowed to add an argument.) A better solution would be that sshdeny reads the IP address from stdin and that ssh key is bound to always execute sshdeny. Two management commands are supplied: -- sshdeny-status lists the current content of the inet_ssh chain. -- sshdeny-reset empties the chain. PROBLEMS: I had the case that attack connections are opened and quite some time is waited before the authentication is tried. When we notice the third authentication failure, there are already other attack connections established. These connections are not blocked, of course. So, in practice, an attacker might have more than three tries to guess an account and a password. -------------------- snip snap -------------------------------- (That's from the README of software that I wrote to handle the case.) I hope that some of the discussions and thoughts above are interesting for some of you. Cheers, Joachim -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod Email: jschrod@acm.org Roedermark, Germany

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Monday 2006-03-27 at 16:40 +0200, Joachim Schrod wrote:

We have now abandoned that approach, also for other reasons. Let me see, maybe the following is of interest...

Really interesting, thank you :-) ... ...

logsurfer is used because I don't know a better log watching and event creation system. I would prefer to have a better one; it is not really suited for the task, as explained below.

If what you need is simply isolating certain messages, that can be easily done with syslog-ng: it has regexp filters, so that you can send the isolated messages to a file, socket or network destination. What I wouldn't know right now is how to trigger execution of an script, though, I'd have to think about it; I'm not a linux programmer. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFEKcectTMYHG2NR9URArODAJ9gUHjtcR3NbgNBDQSvItAPRUa/wgCfXH+i FDsjKwrfQKQNF7L3A9veU7U= =SE4H -----END PGP SIGNATURE-----

Per Jessen wrote:

...

That solution would be a very good choice -- if it would work. ipt_recent doesn't work correctly when Jifies in the Linux kernel overflow. Then it blocks every request, even though they didn't pass the threshold.

Umm, that's a showstopper alright. I found these:

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=415 http://blog.blackdown.de/2005/05/09/fixing-the-ipt_recent-netfilter-module/

Thanks for pointing them out. I haven't followed that lead for some time now, and it's good to know that this opportunity exists as well.

I sort of like your alternative solution, although it is a little too complicated/over-engineered for my own needs.

I know that it's too complex for most installations, that's why I don't spend the effort to release the software. Joachim -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod Email: jschrod@acm.org Roedermark, Germany

Carlos E. R. wrote:

Another idea: use the "recent" module of iptables to automatically block repeated "polls". There was a brief thread in the security list time ago. I just re-read your post and decided it looks interesting. I want to give it a try

I I use this in SuSEfirewall2-custom, function fw_custom_before_antispoofing(): before your post, I didn't even know of the existence of this file.

On the sixth attempt to connect to port 22 within 60 seconds, that IP is rejected. It is simple to modify. Don't ask me much about it, I'm no expert on iptables ;-) That's ok, I barely understand it. I am looking at the SuSEfirewall2-custom file and am a little confused. The text at the top says there is no help and I can accept that. I suppose I just plug in custom rules in the places where I want them to happen. The last item in each section is "true". Is this always there regardless whether I have custom rules in that section? should it always be true?

Damon Register