Keep in mind, if someone broke into the machine and can read the script he also has access to the sensitive Information on this machine. He do not need to break into the Backupserver because he is in the Server where the Backup was made.

one additional thing would be to make sure the files are encrypted already. Such that he not only has to break into the machine he has to break the encryption too. You also have to make sure your .ssh directory where the private key is stored is not world readable. Otherwise you'll have defeated the purpose of the whole exercise infact probably fared worse than with passowrd authentication. Noah.

-----BEGIN PGP SIGNED MESSAGE----- Hash: RIPEMD160 Have you looked into using amanda? It supports kerberos. Or, you can use something like stunnel, or ssh to tunnel the traffic from amanda. http://www.amanda.org BTW: The orielly book has a chapter devoted to amanda. Robert Simmons Systems Administrator http://www.wlcg.com/ On Mon, 6 Aug 2001, Maarten J H van den Berg wrote:

On Tuesday 31 July 2001 14:35, Lukas Feiler wrote:

[sorry for my late reply]

...

I want to do the following: backup all my sensitive date from my main server, pack it into one file and then get it transfered to my backup server.

That's fine but my problem is that those two machines aren't in the same local network. So if I do not encrypt my data it would be (more or less) visible to everybody on the net (who has some hacking knowledge). But as I said this data is sensible (passwords, creditcards, ...)! So I thought of ssh or scp BUT how to automate this process of backing up? I would have to specify user AND password in my backup-script. How do specify a password for ssh / scp in a script??

Instead, the best (and almost completely secure in every aspect) is to use an RSA certificate, and put the command, client-IP etc. which the client uses inside the authorized_keys file on the server: That will make sure that when using that specific certificate, the client is FORCED to run EXACTLY the command specified. Thus, even if the clientsystem gets fully compromised, the backupserver remains safe from the attacker. You can choose to use ssh-agent, or even leave the passphrase blank, as little harm can be done anyway. Worst case would be overwriting the backup with an empty / corrupt one...

There is documentation with ssh how this enforcing works exactly, read it well because it isn't trivial to setup; you have to have the commands exactly right. Once it works however you have a secure backup connection, without establishing an (unwanted) trust- relationship. I've done this myself. Just follow the docs, run sshd in debug level to find the necessary commandstring, and you're fine.

I lost the bookmark to the site where I initially read those docs... :-( But google will help you. The O' Reilly book has some info too.

Good luck, Maarten

-- brick (brik) n. (4) pl. Another item that can be used to crash windows.

Maarten J. H. van den Berg ~~//~~ network administrator van Boetzelaer van Bemmel - Amsterdam - The Netherlands http://vbvb.nl T+31204233288 F+31204233286 G+31651994273

-- To unsubscribe, e-mail: suse-security-unsubscribe@suse.com For additional commands, e-mail: suse-security-help@suse.com

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (FreeBSD) Comment: For info see http://www.gnupg.org iD8DBQE7btJ0v8Bofna59hYRAxbMAKCUYKB2ybrDJ4YJc3N0f1yn9LWzOwCgoglX 2pNvlup5q9b4HA2eIRXhciA= =fA5y -----END PGP SIGNATURE-----

On Mon, 6 Aug 2001 20:32:27 +0200 Maarten J H van den Berg wrote:

On Monday 06 August 2001 18:29, Maarten J H van den Berg wrote:

[followup to self, with additional info...]

Let me explain this setup a bit more:

Let's say, on the client named foobarclient.com you wish to run this next command in a backupscript:

rsync -e ssh -ogLv /var/foobar.tar.gz root@foobarserver.com:/var/backup/

--snip-- As a followup to this thread that went on a few weeks ago, I thought I'd alert anyone who's not reading bugtraq to the following problem.. http://packetderm.cotse.com/mailing-lists/bugtraq/2001/Sep/0189.html -- Viel Spaß Nix - nix@susesecurity.com http://www.susesecurity.com

You could use amanda with kerberos enabled. Just compile it with the --with-krb4-security= configure flag and a couple others if you want. Once its kerberized (version 4 only tho) you will get two new dumptype options, krb4-auth and kencrypt. kencrypt will encrypt the entire connection with the krb4 session key. You can add another measure of protection by using tcpwrappers for the amandad client by running it from inetd. amanda supports the native "dump" program for dumping entire filesystems, and it supports tar. This setup is much safer then putting a plaintext password in some script, or using S/Key ssh type stuff. Amanda project's homepage: http://www.amanda.org/ Robert Simmons Systems Administrator http://www.wlcg.com/ On Fri, 31 Aug 2001, Peter Nixon wrote:

On Mon, 6 Aug 2001 18:29:27 +0200 Maarten J H van den Berg wrote:

...

On Tuesday 31 July 2001 14:35, Lukas Feiler wrote:

[sorry for my late reply]

...

I want to do the following: backup all my sensitive date from my main server, pack it into one file and then get it transfered to my backup server.

That's fine but my problem is that those two machines aren't in the same local network. So if I do not encrypt my data it would be (more or less) visible to everybody on the net (who has some hacking knowledge). But as I said this data is sensible (passwords, creditcards, ...)! So I thought of ssh or scp BUT how to automate this process of backing up? I would have to specify user AND password in my backup-script. How do specify a password for ssh / scp in a script??

Instead, the best (and almost completely secure in every aspect) is to use an RSA certificate, and put the command, client-IP etc. which the client uses inside the authorized_keys file on the server: That will make sure that when using that specific certificate, the client is FORCED to run EXACTLY the command specified. Thus, even if the clientsystem gets fully compromised, the backupserver remains safe from the attacker. You can choose to use ssh-agent, or even leave the passphrase blank, as little harm can be done anyway. Worst case would be overwriting the backup with an empty / corrupt one...

There is documentation with ssh how this enforcing works exactly, read it well because it isn't trivial to setup; you have to have the commands exactly right. Once it works however you have a secure backup connection, without establishing an (unwanted) trust- relationship. I've done this myself. Just follow the docs, run sshd in debug level to find the necessary commandstring, and you're fine.

I lost the bookmark to the site where I initially read those docs... :-( But google will help you. The O' Reilly book has some info too.

Good luck, Maarten

-- brick (brik) n. (4) pl. Another item that can be used to crash windows.

Maarten J. H. van den Berg ~~//~~ network administrator van Boetzelaer van Bemmel - Amsterdam - The Netherlands http://vbvb.nl T+31204233288 F+31204233286 G+31651994273

-- To unsubscribe, e-mail: suse-security-unsubscribe@suse.com For additional commands, e-mail: suse-security-help@suse.com

I highly recomend that if you are doing any sort of remote file copies that you take a look at rsync instead of scp.

--

Viel Spaß

Nix - nix@susesecurity.com http://www.susesecurity.com

-- To unsubscribe, e-mail: suse-security-unsubscribe@suse.com For additional commands, e-mail: suse-security-help@suse.com

Depends on how complicated your backups are. For example the client box tarball's it all up, one file to move, things are suddenly a lot different then maintaining a multi gigabyte file tree. Let's assume for a moment we're talking file trees with lots of different owners and perms, and no tarballs. Yes rsync needs to run as root on the server, to set file perms/etc, this can be somewhat mitigated by chroot'ing it (probably will be ok, but chroot can be broken out of by root, so some buffer overflow in rsync with a hostile client might be bad news). Basically any backup software will have to run as root to set file perms, setuid/setgid bits, yadayada (kernel capabilities and whatnot aside). Hopefully that software was built with this in mind and supports some nice controls (like only write/read files in /foo/backups/*). Kurt

* Kurt Seifried wrote on Fri, Aug 31, 2001 at 15:36 -0600:

Yes rsync needs to run as root on the server, to set file perms/etc, this can be somewhat mitigated by chroot'ing it (probably will be ok, but chroot can be broken out of by root, so some buffer overflow in rsync with a hostile client might be bad news).

Yep. And in fact root permissions of the whole service are _not_ required.

Basically any backup software will have to run as root to set file perms, setuid/setgid bits, yadayada (kernel capabilities and whatnot aside).

It would be possible to have a small permission correction process, which would be more simple code - because of that not that risky. Second, it would be possible to put the permissions in some special file (TRANS.TBL or the one used when doing umsdos). This file could be used on restore operations or used by some correction process. More safe.

Hopefully that software was built with this in mind and supports some nice controls (like only write/read files in /foo/backups/*).

Well, once I overwrote local /etc/ with a buggy rsync-backup script. Well, luckyly the local backup of /etc/ was done first and correct :) But this shows the dangers. When useing tar cf - | ssh > cat backup.tar only on one side (and this is the read-only side) are required. No network-root-connections and nothing. Even a buggy ssh wrapper has only backup-user permissions. And hacker could read/steal the backups (which is very bad, too, but) not compromise systems or manipulate data. oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.

* teo@gecadsoftware.com wrote on Sat, Sep 01, 2001 at 01:21 +0300:

I am using rsync -e ssh w/ no problem so far. what do you mean about keeping ownership?

Well, if you rsync i.e. /home, you need to preserve the attributes on restore. That means, /home/steffen has still to be owned by steffen after restore operation. If you have serveral hundert users, it's not easy to fix if not done automatically. The permissions are important, too, imagine a world-writeable ~/.ssh or a not world-readble ~/public_html. oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.

* teo@gecadsoftware.com wrote on Sat, Sep 01, 2001 at 18:43 +0300:

...

The permissions are important, too, imagine a world-writeable ~/.ssh or a not world-readble ~/public_html.

but this is easily achived with --archive option. as I understand it, the issue is when restoring from backups, and given that that's a root task, I see no problems with it.

Yep, that's it. root task on both sides!

when you backup with rsync, use -a to preserve ownership and rights, recreate links

BTW, I get often errors when rsync tries to create symlinks: file already exists... oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.

[Carlos said:]

i am thinking for using rsync over ssh. but i have a problem, the sys ask me for writting the ssh password. there is any method using rsync and ssh for sending the password whitout writte it? my plan is to use rsync over ssh in a cron. thanks

You have to set up the client side with ssh-keygen with no passphrase, and use ssh with "-oBatchMode=yes". So to use rsync you would: rsync -a -e 'ssh -oBatchMode=yes' ... Hope this helps. -- Bill Duncan, VE3IED | BeachNet --> http://www.beachnet.org bduncan@BeachNet.org | - Network/System Administration +1 416 693-5960 | - System Analysis/Design/Programming

* Maarten J H van den Berg wrote on Fri, Sep 07, 2001 at 12:44 +0200:

On Friday 31 August 2001 23:21, you wrote:

Again a late reply... here goes

No problem, I'm glad that you found time for this article!

...

* Maarten J H van den Berg wrote on Mon, Aug 06, 2001 at 18:29 +0200: [..] But back to the issue; you say rsync should filter paths and so on, I beg to differ (Caveat: If I'm not overlooking something) Look at my config. This is what the client commandline looks like:

rsync -e ssh -ogptLrcv --stats --partial --exclude=logs/ --safe-links /etc/foobar root@www.xxx.yyy.zzz:/home/backups/Host-AAA/

BTW, why not "-a" == "-rlptg". Differs on links to your flags. If I understand correctly, -L "follows" the links and dumps the files. Bad on /etc/rc.d/rc2.d and friends.

And this is the server-side exerpt from authorized_keys:

command="rsync --server -vLogtprc --partial --safe-links . /home/backups/Host-AAA/",no-port-forwarding,no-X11-forwarding,no-agent-forwarding 1024 35 123456789....................123456789 root@aaa.bbb.ccc.ddd

Okay, now try to run a doctored backupcommand like:

rsync -e ssh -ogptLrcv --stats --partial --exclude=logs/ --safe-links /etc/foobar root@www.xxx.yyy.zzz:/home/backups/Host-AAA/../../../

[...looks safe...]

So you see, as I wrote earlier, the SERVER IS ENFORCING the command which the client tries to run. (Again, unless I made a mistake that is.)

Well, looks nice. I don't know why, but I though to write some wrapper which allows /home/backups/whatever/*, I mean that allows "deeper" definitions - but now I think you're right and this shouldn't be neccesary. If you specify /../ at the end, SSH shouldn't even start rsync, so this should be safe. But the question is, if the rsync protocol is internally safe. scp hat such a bug, it created the file requested by the client even if outside it's destination IIRC. So it may be possible to create a modified rsync client, which delivers ordinary files plus a new /root/.ssh/authorized_keys or whatever. But I think it would take some time to analyse the rsync sources.

If you can show me a commandline that does indeed break out of the target dir, I'd be VERY interested to know, needless to say.

Well, I think it's required to patch the rsync client. I don't know if the server checks the path of the files, I never looked at the rsync protocol. Maybe it's possible to specify a list like /etc/foobar --> /home/backups/Host-AAA/foobar /etc/foobar2 --> /home/backups/Host-AAA/foobar2 /etc/foobar3 --> /etc/foobar3 /etc/foobar4 --> /home/backups/Host-AAA/foobar4 In such a case, the server should reject foobar3 since it's outside the destination. Or imagine: /etc/foobar2 --> /home/backups/Host-AAA/foobar2 /etc/foobar3 --> /home/backups/Host-AAA/../../../etc/foobar3 or: /etc/foobar2 --> /home/backups/Host-AAA/foobar2 /../../../etc/foobar3 --> /home/backups/Host-AAA/../../../etc/foobar3 usually rsync wouldn't generate such lists of course, but it may be possible to patch it in such a way (on client side). But it's difficult to check it in a few minutes, since the patch isn't trivial I think. Are there any experiences or audits?

I commented on the use of tarballs above. It wasn't useable for me at the time. However, If you CAN, this is indeed a safer option.

You're right, rsync has nice advantages, it's fast and a restore of a single file (i.e. deleted by accident) is much more easy. A second point: you decided that the clients "pushes" it's files to the backupserver. I decided that the backupserver "polls" the files from the client. I though, since it better since: - the load of the backupserver is constant on backups, since all clients can be polled after the other (no "concurrent" backups) - A single point for "logging" (I see if a host was down on backup time. A host which is down won't mail - you won't see any error message). - root access via key needed. I deciced to put the client keys on a key disc, told ssh to use a key which is a symlink to automounted /misc/floppy. Once a week, i.e. saturday night, an operator inserts the disk with the keys into the backup server, and not a key with a single key to each backupclient What are your reasons for doing client "push"? oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.

* Maarten J H van den Berg wrote on Fri, Sep 07, 2001 at 15:44 +0200:

On Friday 07 September 2001 14:30, you wrote:

...

* Maarten J H van den Berg wrote on Fri, Sep 07, 2001 at 12:44 +0200:

...

...

rsync -e ssh -ogptLrcv --stats --partial --exclude=logs/ --safe-links /etc/foobar root@www.xxx.yyy.zzz:/home/backups/Host-AAA/

via

...

...

command="rsync --server -vLogtprc --partial --safe-links . /home/backups/Host-AAA/",no-port-forwarding,no-X11-forwarding,no-agen t-forwarding 1024 35 123456789....................123456789 root@aaa.bbb.ccc.ddd

I'm not even sure if this safe behaviour is enforced by rsync, or by sshd. I think, by sshd. My guess is, the server 'parses' "/home/backups/../", which obviously gives "/home/", compares that to its "enforced" command inside authorized_keys, and silently disregards the requested path.

Well, actually it's like that. SSHD ignores the parameters of the client command completely and uses the parameter list from command="". sshd delivers the parameters in SSH_ORIGINAL_COMMAND (or similar), openssh seems not to have that nice feature.

...

But the question is, if the rsync protocol is internally safe.

Of course there could be a problem somewhere. That is why you must try to make it even more restricted, for instance by adding the source IP adress to the authorized_keys command etc.

Yep, but an attacker who steals a root key from a host can connect from that host too, so I don't see a big improvement currently.

Maybe one could (I did not try this though) also look into time-restrictions "only between 01:00 hours and 01:30 hours", and / or using ssh-agent instead of passphrase-less keys, and whatnot...

Well, if the attacker isn't silly, she connects like the cronjob. Same parameters but trojaned rsync client. She would just exchange the rsync binary, but not touch any cron jobs or whatever I guess. But in normal life I would think that this would be a theoretical attack only, since this would require some research of you setups and so on. But who knows, maybe someone is really interested in your backups :)

In any case, if a client gets compromised without me noticing it, it's only a matter of time (ttysnoop et al, you know what I mean)

Exactly, I think that is the point where the security of that particular job is sufficient. In case of compromising by a good attacker with a lot of time you're lost anyways :)

Good point. As with everything, security should indeed ALWAYS be enforced on the server-end

Yep, of course. [...]

...

usually rsync wouldn't generate such lists of course, but it may be possible to patch it in such a way (on client side). But it's difficult to check it in a few minutes, since the patch isn't trivial I think. Are there any experiences or audits?

Not that I did look into. Besides, I sadly lack the c / c++ skills needed for auditing it myself... :-\

Well, and I don't think customers would pay for it :) Would become an expensive backup :)

...

- A single point for "logging" (I see if a host was down on backup time. A host which is down won't mail - you won't see any error message).

Indeed. But mailing the timestamps of /home/backups/* gives at least an indication whether the backup still runs. Not that that is foolproof...

Ohh, really? I just look from time to time :)

These servers are colocated, not easily accessed. For us, any user-intervention was unacceptable.

Yes, the old problem. Anythink w/o user interaction can be easier compromised as when done with user interaction. After all, I don't think it makes a big difference if an attacker has to wait until the disc get's inserted. Even with some smartcards or whatever this won't help, since it's not possible to be sure that the smart card reading host is safe and not compromised. For backups like that rsync job, it would be nice to have some mode in which the permissions get saved in some ordinary file. In that case the backup job wouldn't need root permissions. But restore operations would be more complex.

...

What are your reasons for doing client "push"?

Ehm... Let me think back... One thing I can mention, if one of the hosts is compromised, it stays confined to that host (unless the rsync setup has securityflaws like you describe above, that is). With server-pull, IF the backupserver should get compromised, you lose everything, as that server has access to ALL those clients at some level (root or unpriv).

Yep, good point...

Another point is that we need to backup files from many different filesystems, like /var, /usr, /home, /etc. That is difficult to control.

As long as adminstrated by a single person I don't see a difference, but at least if they admin of the backup client has no shell access to the baskupserver client pull is configurable by him. So far it was very nice and productive to discuss it with you. Thank you. oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.

* Maarten J H van den Berg wrote on Fri, Sep 07, 2001 at 19:46 +0200:

On Friday 07 September 2001 17:50, you wrote:

...

* Maarten J H van den Berg wrote on Fri, Sep 07, 2001 at 15:44 +0200:

[...rsync via ssh wrapper...] [...command="rsync"...]

...

Well, actually it's like that. SSHD ignores the parameters of the client command completely and uses the parameter list from command="". sshd delivers the parameters in SSH_ORIGINAL_COMMAND (or similar), openssh seems not to have that nice feature.

Thanks for clearing that up. However, this IS in fact openssh... Or do you just mean that openssh does the same thing but does not provide the original command in SSH_ORIGINAL_COMMAND ?

Yep, this environment variable gets not set by OpenSSH. I cannot imagine that this is a big task to supply argv through the environment, I should make a patch, but time is missing as always. I have some wrapper scripts which evalute SSH_ORIGINAL_COMMAND. By this, it's possible to allow a well defined set of parameters. I used perl to filter that. If and only if the parameters are ok, the action is taken. I thought about some generic wrapper, which can be configured to allow a defined command set with some parameters, i.e. the fourth parameter can be arbitary as long as it's a number or whatever. Due to this, I still have some machines running non-open SSH servers...

("Here is your server. To be totally sure, we just audited all of the kernelsources 2.4.9. We just thought it was a nice addition to the service-level you usually get from us. And ehm... here is your bill.")

:) hihi, or even: "Here is your server. We just finished the audits of kernel 2.0.11 and printed a book with open issues which are not easy to fix. We estimated to additional years to be able to offer you a audited backup solution, but for now it's cheaper to use a DAT for each host." :)

...

Ohh, really? I just look from time to time :)

Hehehe. No, I myself insist on 200+ cron messages in my inbox daily ;-)) (NOT)

Well, I think about a netsaint plugin for such purposes. Each server could write a timestamp in a special file, i.e. /root/current_timestamp (each 5 minutes or so), and the plugin verifies the file inside the backup. It alerts if the timestamp is older as i.e. 24 hours or so. Well, I think this would take a few hours to implement...

Yes... But if you think not in absolute terms ("totally secure") but relative terms ("one level more difficult to break than before"), every little bit can help. But like I said, someone determined...

You're right of course. After all it shows up in pracitise that security through obscurity helps often against script kiddies.

You know, that is *exactly* what my first approach to this problem was. Like taking a ls-lR list, feed that to some nice script that replays all chmod/chown commands neccessary to put things back to how they were.

I think this would be difficult to parse. I made a example perl script which I attach to this mail. It gets a list of parameters of filenames (i.e. find /path | xargs ./perm.pl). It processes all files via lstat(2), puts some values into a hash. After that, it stores that hash on disk (just example code). Then it deletes the filelist and informations from memory. This is like on backup operations. After that, the scripts opens the disk file and load the hash back (meant for restore operations). It prints out chmod ; chown commands to restore the old state. Well, this isn't for production since it does eval on the permissions file ("filelist.dump"). This is dangerous if that file contains other perl statements, and by this the permission file is very security relevant :). But it's just an example. What to do with it? If someone would use such a thing, I would modify the script to process filenames from stdin (or to use find internally). The call could be "find /path -print0 | perl.pl -o filelist.dump". If started with "-o" it stops after writing filelist to disk. On restore, call "perl.pl -s filelist.dump" to set the saved permissions. Maybe some control on which files that should apply and so on. In this case, the script startes with reading filelist.dump from disk and does not print the chown/chmod commands but executes them. Well, and I think I would add support for the times (mtime...) which shouldn't be a problem, and some error handling of course. This is not much code in such a case, I think 2 hours programming (one of them for considerations) but 10 hours testing :) Maybe I'll get some statements; if someone is interested in such a script and if we find some tester I could implement it. Well, drop me a note.

In the end I could not find anything that does this, and I already told you about my limited programming skills so I dropped that idea. However, this is really something I'd like to see as a tool. Maybe someday, someone will write it... In that case, no more need for elevated priviledges for backups.

I'm not really sure if someone would use it, since if it's not integrated into a tool like rsync it's additional work to put it into the cronjobs. What do you think?

...

As long as adminstrated by a single person I don't see a difference, but at least if they admin of the backup client has no shell access to the baskupserver client pull is configurable by him.

I meant "client push" in the last sentence of course.

If it's root who does the backup, there is no real problem (but that may not be advisable). If it's an ordinary user who does the backups, it could create havoc on all the different permissions one has to set up.

What do you mean? I don't think it's possible to let an ordinary user doing backups. I meant, there is rootA on backupserver which is different from rootB on client B. In your case (client push) rootA could configure access to some path, and rootB on client B could manage the exclude lists or similar.

One could also run scripts on the client and the server simultaneously, on the client side a root-cronjob copies & chowns some root-owned files to a special backup dir which is only readable by the backup daemon, and the server pulls them with a non-priviledged account. But it is doubtful that gives you any more security (or even, less!) The whole point of -for instance- /etc/shadow is that it is NOT readable by anyone...

Well, the backupuser don't need a shell account or password, so it's not a bad security risk I think. But it's much efford to copy all files local first, loseing all permissions. I think it's the same as client push to backup user on backup host directly.

Besides, a lot of things can break this setup. For instance, the machines have to be time-sync'ed for this to work well.

Well, ntp should run on every host all the time, my netsait checks that :)

Same here, It was very nice to exchange views on this subject. Speak to you later, I hope.

Thank you. Maybe we'll find the ideal solution? Hum. :) Someone with time to patch rsync here? Shouln't be too much work: instead of _doing_ "chmod mode, file" printf mode, file to some permission file - or similar. But on restore? Just `permission file` command substition? What with error handling... Well...

Oh, and: Have a nice weekend !

Same to you and the list! oki, Steffen -- Dieses Schreiben wurde maschinell erstellt, es trägt daher weder Unterschrift noch Siegel.