Re: [suse-security] Strange server crash
I use backup scripts but none is doing such big files ~ around 300 MB. I looked over the log files in the last 3 crashes ( 12 oct, 9 sept, 1 sept) and I couldn't asociate it with any cron or script or process.... only if one of my script hangs... that is again strange, because I do only copy, database dumps... I also do rsync with another share from network. Regards, Andy. ----- Original Message ----- From: b@rry To: 'Andy' ; suse-security@suse.com Sent: Thursday, October 13, 2005 12:03 PM Subject: RE: [suse-security] Strange server crash check your backup processes. I had kernel panics happening to me when using tar to archive to a single file over 20G. Seems a combination of tar, gzip and reiserfs did not like the large archives and would cuase a memory condition that sent the machines into kernel panic. During kernel panic, nothing gets written to disk at all so your logs just stop dead. I found that shortly before this dead ste, I was unable to get onto the machine but it would respond, and then it would just die completely with no icmp responses even. what scripts do you run? ------------------------------------------------------------------------------ From: Andy [mailto:frum@ar-sd.net] Sent: 12 October 2005 05:57 PM To: suse-security@suse.com Subject: [suse-security] Strange server crash Hi to all, We have a Fujitsu Siemens server, Dual Xeon 2.8, 160GB sata HDD configured in software raid. On the server there is Apache, Php, vsftp and postgresql db running. Suse 9.1 64 Prof is installed. About once a month the server just freezes. Sometimes I can ping and it respondes, sometimes not. I cannot connect to the server anymore froma any point of view. None of the services are running anymore, http, ftp, samba etc... I cannot know if the console works because I'm not near the server. I looked over and over the log files and couldn't see anything wrong. Or maybe my log settings are not right? Any clues? Thanks in advance. Andy
I looked over the log files in the last 3 crashes ( 12 oct, 9 sept, 1 sept) and I couldn't asociate it with any cron or script or process.... only if one of my script hangs... that is again strange, because I do only copy, database dumps...
I also do rsync with another share from network.
I have tried tar, rsyn, and rdiff backup all of which have had similar memory leak problems causing kernel panics. Tar caused the problem when writing many files to a single large archive. Rsync and rdiff-backup caused problems with the high volume of files. I have now resorted to "batching" my backups with date stamps inserted into log files so that I can trace any further errors to a direct script and time. When backing up using rsynch, try inserting date stamps one dir at a time, that way at least you will know if a perticular dir/process is causing the problem. Bit of a pain in the ass, but less of a pain than a panic. The dir structure I backup is basically: /top/of/dirstructure/0/0/0/0/0 To /top/of.dir/structure/9/9/9/9/9 With each dir containing 0/0/0/0/0 0/0/0/0/1 0/0/0/0/2 0/0/0/0/...9 0/0/0/1/0 0/0/0/1/1 0/0/0/1/2 Etc So large number of dirs. I have scripted #!/bin/bash rm -rf /var/log/backup.log echo "Start Dir Struc 0" `date` /var/log/backup.log Rdiff/rsync /top/of/dirstructure/0/ echo "End Dir Struc 0" `date` /var/log/backup.log sleep 2 echo "Start Dir Struc 1" `date` /var/log/backup.log Rdiff/rsync /top/of/dirstructure/1/ echo "End Dir Struc 1" `date` /var/log/backup.log sleep 2 Etc... This way, I was able to ensure that if anything went wrong I would always know very closely where and could check my dir contents for file allocations that were haywire. Also, breaking it up into many processes while it makes the backup process take much longer, I have not had a single issue since doing this...
I have tried tar, rsyn, and rdiff backup all of which have had similar memory leak problems causing kernel panics. Tar caused the problem when writing many files to a single large archive. Rsync and rdiff-backup caused problems with the high volume of files.
I looked over the log files in the last 3 crashes ( 12 oct, 9 sept, 1 sept) and I couldn't asociate it with any cron or script or process.... only if one of my script hangs... that is again strange, because I do only copy, database dumps...
I also do rsync with another share from network.
I have tried tar, rsyn, and rdiff backup all of which have had similar memory leak problems causing kernel panics. Tar caused the problem when writing many files to a single large archive. Rsync and rdiff-backup caused problems with the high volume of files.
I have now resorted to "batching" my backups with date stamps inserted into log files so that I can trace any further errors to a direct script and time. When backing up using rsynch, try inserting date stamps one dir at a time, that way at least you will know if a perticular dir/process is causing
problem.
Bit of a pain in the ass, but less of a pain than a panic.
The dir structure I backup is basically: /top/of/dirstructure/0/0/0/0/0 To /top/of.dir/structure/9/9/9/9/9
With each dir containing 0/0/0/0/0 0/0/0/0/1 0/0/0/0/2 0/0/0/0/...9 0/0/0/1/0 0/0/0/1/1 0/0/0/1/2 Etc
So large number of dirs.
I have scripted
#!/bin/bash
rm -rf /var/log/backup.log echo "Start Dir Struc 0" `date` /var/log/backup.log Rdiff/rsync /top/of/dirstructure/0/ echo "End Dir Struc 0" `date` /var/log/backup.log sleep 2 echo "Start Dir Struc 1" `date` /var/log/backup.log Rdiff/rsync /top/of/dirstructure/1/ echo "End Dir Struc 1" `date` /var/log/backup.log sleep 2
Etc...
This way, I was able to ensure that if anything went wrong I would always know very closely where and could check my dir contents for file allocations that were haywire.
Also, breaking it up into many processes while it makes the backup
High volume of files with rsync?
That doesn't make any sense. Rsync is designed to compare the two systems
to decide what files have
changed, as opposed to everything.
Also, does reiserfs not have a filesystem dumping utility like the
`dump/restore` utilities for ext2/3 ?
That is really what you should be using, if one exists. If not, I would
get ahold of the developer folks
at Novell and ask them why they haven't developed one yet. :-)
Unfortunately, I fear as though we've wandered off the path of on-topic
discussions for this list.
I certainly don't mind it, but others might. What other suse mailing
lists are out there?
Tim Rainier
Information Services, Kalsec, INC
trainier@kalsec.com
"b@rry"
take much longer, I have not had a single issue since doing this...
Unfortunately, I fear as though we've wandered off the path of on-topic discussions for this list. I certainly don't mind it, but others might. What other suse mailing lists are out there?
This is still a security spin on this. I use rdiff-backup to backup between two sevrers securely. I experience this same problem using that tool Anyone else have any ideas how to securely transfer large volumes of ifles between machines for backup purposes?
You can tunnel most any linux utility through an ssh connection.
There is also Secure Copy (scp).
Also, here's a document that explains how to use rsync through an ssh
tunnel. (ignore the macosx stuff, the general information still applies).
If rsync won't do, use the same tunneling concepts for the utility you do
want to use.
I'm still bent on there being a "dump filesystem" utility that does full
and incremental levels of backups on filesystems or disks.
I'm not sure of any for reiserfs, but I don't use reiserfs. Might be
worth looking into.
Tim Rainier
Information Services, Kalsec, INC
trainier@kalsec.com
"b@rry"
Unfortunately, I fear as though we've wandered off the path of on-topic discussions for this list. I certainly don't mind it, but others might. What other suse mailing lists are out there?
This is still a security spin on this. I use rdiff-backup to backup between two sevrers securely. I experience this same problem using that tool Anyone else have any ideas how to securely transfer large volumes of ifles between machines for backup purposes?
participants (3)
-
Andy -
b＠rry -
trainier＠kalsec.com