Hi to all, it seems that I have a serious problem, although I updated my SuSE 8.1 server quite recently (all security updates were applied). I have two questions: 1) What to do right now to prevent any misconduct of my server? 2) How to clean up the server? Description of the problem: one of my users (mvasilic) noticed that someone from IP 81.196.122.7 logged to his account (that IP originates from Romania, and we are in Serbia). Close inspection shows that indeed someone was logged to our server from that IP, and obviously was running some kind of a rootkit: ------------------------------------------------------------------------ octopus:~ # ps aux | grep mvasilic mvasilic 6620 0.0 0.2 1644 580 ? S Nov22 0:00 bash mvasilic 3241 0.0 0.0 1380 4 ? S Nov22 0:00 ./root root 11279 0.0 0.2 1624 600 pts/1 S 16:10 0:00 grep mvasilic ---------------------------------------------------------------------- In the /tmp directory there are several interesting files: ----------------------------------------------------------------------- octopus:/tmp # ls -l | grep mvasilic drwxr-xr-x 3 mvasilic users 72 2003-11-22 20:26 -rw-r--r-- 1 mvasilic users 0 2003-11-22 13:30 982235016-gtkrc-429249277 -rw-r--r-- 1 mvasilic users 4215 2003-11-22 21:08 lstermcap -rwxr-xr-x 1 mvasilic users 5410 2003-11-22 21:11 own.so -rw-r--r-- 1 mvasilic users 453 2003-11-22 21:10 report -rw-r--r-- 1 mvasilic users 58 2003-11-22 21:10 suidprogs ---------------------------------------------------------------------- File report says: ---------------------------------------------------------------------- octopus:/tmp # cat report RwX Super Linux Xploit report : ======================================== .o. The scanner found /usr/bin/lpr could be xploitable. .o. The scanner found /usr/X11R6/bin/X could be exploitable. .o. The scanner found /usr/bin/crontab could be xploitable. .o. The scanner found /bin/mount could be exploitable. This script was originally scripted by so1o@insecurity.org Modifications to Linux by Kbyte@biogate.com 2 bugs found. --------------------------------------------------------------------- There is also the directory ' ', which contains directory s: --------------------------------------------------------------------- octopus:/tmp/ /s # ls -l total 48 drwxr-xr-x 3 mvasilic users 144 2003-11-22 20:42 . drwxr-xr-x 3 mvasilic users 72 2003-11-22 20:26 .. drwxr-xr-x 9 mvasilic users 376 2003-01-14 14:34 .haos -rwxr-xr-x 1 mvasilic users 37162 2003-07-19 22:37 c -rwxr-xr-x 1 mvasilic users 35 2003-05-19 12:40 s -rwxr-xr-x 1 mvasilic users 29 2003-05-17 04:46 t -------------------------------------------------------------------- Directory .haos is a very rich one: ------------------------------------------------------------------- octopus:/tmp/ /s/.haos # ls -l total 146 drwxr-xr-x 9 mvasilic users 376 2003-01-14 14:34 . drwxr-xr-x 3 mvasilic users 144 2003-11-22 20:42 .. drwxr-xr-x 2 mvasilic users 120 2002-05-12 05:16 FTP -rwxr-xr-x 1 mvasilic users 15633 2002-02-01 04:18 dat1 -rwxr-xr-x 1 mvasilic users 21794 2002-02-01 04:18 dat2 drwxr-xr-x 3 mvasilic users 96 2002-05-11 13:43 haos1 drwxr-xr-x 3 mvasilic users 96 2002-05-11 11:00 haos2 -rwxr-xr-x 1 mvasilic users 14380 2002-02-01 05:24 haosp -rwxr-xr-x 1 mvasilic users 16500 2002-02-01 05:17 haosv -rwxr-xr-x 1 mvasilic users 1560 2002-02-01 04:18 haosx drwxr-xr-x 6 mvasilic users 1920 2002-03-30 05:30 libpcap-0.6.2 drwxr-xr-x 11 mvasilic users 392 2003-01-07 00:26 massrooter drwxr-xr-x 2 mvasilic users 272 2002-10-02 15:31 nebunu drwxr-xr-x 2 mvasilic users 408 2002-01-12 18:00 strobe -rwx------ 1 mvasilic users 64652 2002-04-01 19:45 superwu ---------------------------------------------------------------- And so on. Please, help! Best regards, Antun Balaz Institute of Physics, Belgrade Serbia and Montenegro
* Antun Balaz;
Hi to all, it seems that I have a serious problem, although I updated my SuSE 8.1 server quite recently (all security updates were applied).
I have two questions:
1) What to do right now to prevent any misconduct of my server?
Immediately disconnect the machine from the Internet
2) How to clean up the server?
Make a clean install from scratch and make sure before putting it back to the net all patches are applied either use fou4suse or YOU to automate the process. HTH -- Togan Muftuoglu Unofficial SuSE FAQ Maintainer http://susefaq.sf.net
Unfortunately, I have 100+ users, my server is mail and web server, so it is impossible to just disconnect it and install from scratch. Besides, all security pathes were allready applied, so this will not help me, or I am mistaken? Any other suggestions? Thanks, Antun ------------------------------------------- Antun Balaz http://www.phy.bg.ac.yu/~antun/ Institute of Physics, Belgrade Serbia and Montenegro http://www.phy.bg.ac.yu/ ------------------------------------------- On Mon, 24 Nov 2003, Togan Muftuoglu wrote:
* Antun Balaz;
on 24 Nov, 2003 wrote: Hi to all, it seems that I have a serious problem, although I updated my SuSE 8.1 server quite recently (all security updates were applied).
I have two questions:
1) What to do right now to prevent any misconduct of my server?
Immediately disconnect the machine from the Internet
2) How to clean up the server?
Make a clean install from scratch and make sure before putting it back to the net all patches are applied either use fou4suse or YOU to automate the process.
HTH --
Togan Muftuoglu Unofficial SuSE FAQ Maintainer http://susefaq.sf.net
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
* Antun Balaz;
Unfortunately, I have 100+ users, my server is mail and web server, so it is impossible to just disconnect it and install from scratch. Besides, all security pathes were allready applied, so this will not help me, or I am mistaken?
Do you routinely install security patches or did you recently applied the patches
Any other suggestions?
1) Backup your data ( unless you were not doing that before) 2) carefully check your data 3) check the machine with rootkit pls. send replies to the mailinglist do not put me in TO or CC or BCC -- Togan Muftuoglu Unofficial SuSE FAQ Maintainer http://susefaq.sf.net
Dear Togan, I install security paches routinely every week. Can you give me more details how to check if my server is afftected in any way (I deleted content of /tmp directory and applied all measures suggested by Bjorn - thanks a lot), i.e. if some files are replaced etc. Thanks, Antun Balaz Institute of Physics, Belgrade Serbia and Montenegro
Do you routinely install security patches or did you recently applied the patches
Any other suggestions?
1) Backup your data ( unless you were not doing that before) 2) carefully check your data 3) check the machine with rootkit
pls. send replies to the mailinglist do not put me in TO or CC or BCC
--
Togan Muftuoglu Unofficial SuSE FAQ Maintainer http://susefaq.sf.net
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
On Nov 24, Antun Balaz
I install security paches routinely every week. I think you could save more time, if you did an upgrade to a newer distribution. Harddisks are not so expensive any more, I usually buy a new harddisk if I update a system, install everything there, copy configuration, check everything and swap harddisk.
Using a stone-old distribution will cost 10x more effort than updating every 2 years to the latest SuSE (especially because 9.0 is on ftp servers now, you can get it for free ;-) Markus -- _____________________________ /"\ Markus Gaugusch ICQ 11374583 \ / ASCII Ribbon Campaign markus@gaugusch.at X Against HTML Mail / \ Linux 2.4.21-99-athlon
Thanks to all. By the way, searching the Internet (by Google) looking for rootkit that was used on my machine, I found this site http://www.hackemate.com.ar/ which is full of useful material for hackers. Can we do something about it? Best regards, Antun Balaz Institute of Physics, Belgrade Serbia and Montenegro On Mon, 24 Nov 2003, Markus Gaugusch wrote:
On Nov 24, Antun Balaz
wrote: I install security paches routinely every week. I think you could save more time, if you did an upgrade to a newer distribution. Harddisks are not so expensive any more, I usually buy a new harddisk if I update a system, install everything there, copy configuration, check everything and swap harddisk.
Using a stone-old distribution will cost 10x more effort than updating every 2 years to the latest SuSE (especially because 9.0 is on ftp servers now, you can get it for free ;-)
Markus
-- _____________________________ /"\ Markus Gaugusch ICQ 11374583 \ / ASCII Ribbon Campaign markus@gaugusch.at X Against HTML Mail / \ Linux 2.4.21-99-athlon
-- Check the headers for your unsubscription address For additional commands, e-mail: suse-security-help@suse.com Security-related bug reports go to security@suse.de, not here
On Monday 24 November 2003 16:37, Antun Balaz wrote:
Thanks to all. By the way, searching the Internet (by Google) looking for rootkit that was used on my machine, I found this site
which is full of useful material for hackers. Can we do something about it?
You could email the site's ISP, to try to get it closed down. Running traceroute on that address goes back to a machine called "ns1.powered-hosting.com". It may be that powered-hosting.com's server has been cracked, and is now serving rootkits, or they may be doing it intentionally. The last hop goes through a router in "prima.net.ar". You might be able to get them to do something about it. You could choose to inform the FBI, or the equivalent agency in your country. However, whether you can/should do is another question. They are not necessarily the people actually cracking your machine, and the issues of "free speech" may come in to play here. A lot of the parts of Linux could be used as cracking tools; should all Linux download sites be taken offline?
On Monday 24 November 2003 15:16, David Smith wrote:
On Monday 24 November 2003 16:37, Antun Balaz wrote:
Thanks to all. By the way, searching the Internet (by Google) looking for rootkit that was used on my machine, I found this site
which is full of useful material for hackers. Can we do something about it?
However, whether you can/should do is another question. They are not necessarily the people actually cracking your machine, and the issues of "free speech" may come in to play here. A lot of the parts of Linux could be used as cracking tools; should all Linux download sites be taken offline?
There was a good article on that site about rootkits: http://www.hackemate.com.ar/textos/papers/BUANZO-Detecting_and_Understanding... That the original poster might benefit from, if inclined to utilize the site returned from google as being pertinent. In the security community in general, there is a general reliance on an open security model, as opposed to "security through obscurity". I don't think it is healthy to take down the sites you can so easily find (like this one) -- they aren't the problem; there's a million sites out there you can't find so easily. --r dorothy@oz:~> ls scarecrow tinman lion dorothy@oz:~> find . -name home There's no place like home.
* Antun Balaz;
Dear Togan, I install security paches routinely every week. Can you give me more details how to check if my server is afftected in any way (I deleted content of /tmp directory and applied all measures suggested by Bjorn - thanks a lot), i.e. if some files are replaced etc.
If you have deleted the /tmp that it is difficult to do analysis but not imposible. Have you tried to check your system with chkrootkit ? Doing update every week for me is a long time YMMV. Either use YOU or fou4suse and let them do the updating patches for you everyday. When there is a Security anouncement from SuSE immediately apply the patch. If I were you to sleep a little bir comfortable I would have reinstalled the whole machine. HTH -- Togan Muftuoglu Unofficial SuSE FAQ Maintainer http://susefaq.sf.net
On Mon, 24 Nov 2003, Antun Balaz wrote:
Hi to all, it seems that I have a serious problem, although I updated my SuSE 8.1 server quite recently (all security updates were applied).
I have two questions:
1) What to do right now to prevent any misconduct of my server? 2) How to clean up the server?
Description of the problem: one of my users (mvasilic) noticed that someone from IP 81.196.122.7 logged to his account (that IP originates from Romania, and we are in Serbia). Close inspection shows that indeed someone was logged to our server from that IP, and obviously was running some kind of a rootkit:
1. Get the machine offline. Now. 2. No, don't plug it back online. 3. Verify how they got in to the user's account: - 'xhost +' and no firewall on port 6000? - On this machine. - On machine with X server (beware of MS X-servers!) - Passwords typed on insecure machines. - Same password on multiple systems, where another system may be taken. - Username/password borrowed by others. This is a script-kiddie. It's highly unlikely that they've cracked their way in through a service the way things look here. Then they'd own the account owning the service. 4. They've owned a user. Is there any indication that they've gotten a root user. Does the user in question _have_ root access? Check carefully. They obviously haven't had time to clean up thoroughly, check /var/log/messages etc. 5. If you're 100% sure it's only the user, clean up for that user: - New password - Remove crontab - Remove ~/.ssh, ~/.shosts, ~/.rhosts, etc. - Remove at jobs 6. If you're not 100% sure, reinstall and configure from scratch is your one and only answer, with new passwords for all users, etc, etc, etc. 7. Plug back online. BTDT, -Bjørn -- Bjørn Tore Sund Phone: (+47) 555-84894 Stupidity is like a System administrator Fax: (+47) 555-89672 fractal; universal and Math. Department Mobile: (+47) 918 68075 infinitely repetitive. University of Bergen VIP: 81724 teknisk@mi.uib.no Email: bjornts@mi.uib.no http://www.mi.uib.no/
Hi, Following procedure might work without reinstall: - Take a second machine, install a fresh Linux with the same install media used for the infected - apply same updates as for infected machine - use tripwire to generate a new DB on the second machine for all executable dirs (/bin /usr/bin /sbin /usr/sbin ...) or better all dirs except /tmp and spool dirs - cp tripwire and db to infected machine - tripwire check - replace infected binarys - get chkrootkit - mount / from second machine with (ro,no_root_squash) - check infected machine with chkrootkit, bin mounted via nfs, do not use bins from infected (install nfsd temporary if not installed yet, or burn on CD and use this) - replace infected binarys again - reinstall kernel package on infected - check init scripts - reboot - second run all checks U might use "lsattr / |less" first, look for files with flag "a u i" set (these are 99% infected). Use chattr to remove the undeletable flags (typical script kiddies use these flags to prevent root from removing infected files) and replace the bins. Repeat the checks several times. After first run you should look into all dirs, including /dev /tmp and spool dirs. Download and use kstat to check the kernel. It helps to use a static kernel without module support, as the this reduces the number of working rootkits :-) This worked for me on a remote controlled machine. It is recomended to sniff on the network to check for connections attempts during the procedure. And keep an eye on the fixed machine for some month. Ciao, Dieter --------------------------------------------------------------- Dieter Kirchner Systemadministration BUPNET +49 551 54707 62 D-Goettingen http://www.bupnet.de ---------------------------------------------------------------
Dear Linuxfriends, Am Montag, 24. November 2003 17:42 schrieb Dieter Kirchner:
U might use "lsattr / |less" first, look for files with flag "a u i" set (these are 99% infected). I studied man chattr: #When a file with the `u' attribute set is deleted, its contents are saved. But where are they saved? #This allows the user to ask for its undeletion. Where should I ask? ;-)
chattr is for ext2 I use ext3, the file protected with u can be deleted. Does chattr work only with ext2, it should work with ext3? Or is nor still implemented? Thank you ... -- # MfG Michael Maldener + Das beste Linux ist die Pluralitaet aller Linuxica ;)
participants (8)
-
Antun Balaz
-
Bjorn Tore Sund
-
David Smith
-
Dieter Kirchner
-
dim owner
-
MALDENER.de@t-online.de
-
Markus Gaugusch
-
Togan Muftuoglu