[opensuse] rsync question--link-dest= question
Hi, Suppose I do this rsync run: OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINATION=001 rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINATION This creates a backup run to machine Isengard wich has an rsyncd daemon running. We can check on it this way: Telcontar:~ # rsync --password-file /root/cripta/keys/rsync.secrets root@Isengard:: mirrors clonar directorio a disco externo de archivado fusion clonar directorio a disco externo de archivado Videos clonar directorio a disco externo de archivado MyBook_Telcontar copia de seguridad de Telcontar Telcontar:~ # We can see the directories there: Telcontar:~ # rsync --password-file /root/cripta/keys/rsync.secrets root@Isengard::MyBook_Telcontar drwxr-xr-x 28 2019/08/05 16:09:19 . drwxr-xr-x 0 2019/08/01 14:55:28 000 drwxr-xr-x 384 2018/11/24 11:04:40 001 drwxr-xr-x 364 2018/11/24 11:04:40 002 drwxr-xr-x 22 2019/08/05 16:14:45 cosas Telcontar:~ # The backup is created. Now, when I do a second backup, the intention is to do it to another directory (002), making all unchanged files hard links to the previous backup. rsync is informed of the existence of a previous backup using "--link-dest". I have done this before on local directories: rsync $OPTIONS --link-dest=$PREVIOUS/boot /boot /$DESTINATION Now, this does not work: OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINATION=002 PREVIOUS=001 rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets --link-dest=root@Isengard::MyBook_Telcontar/$PREVIOUS/ \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINATION I get this error: --link-dest arg does not exist: root@Isengard::MyBook_Telcontar/001 But it does, AFAIK: Telcontar:~ # rsync --password-file /root/cripta/keys/rsync.secrets root@Isengard::MyBook_Telcontar/001 drwxr-xr-x 384 2018/11/24 11:04:40 001 Telcontar:~ # Telcontar:~ # rsync --password-file /root/cripta/keys/rsync.secrets root@Isengard::MyBook_Telcontar/001/ drwxr-xr-x 384 2018/11/24 11:04:40 . -rw-r--r-- 52,184 2016/12/24 00:59:15 .readahead -rw------- 19,127 2017/09/25 02:03:08 DEADJOE -rw------- 4,288,512 2015/02/24 14:28:22 core drwxr-xr-x 24 2016/03/14 14:40:16 %{_rundir} ... I also tried: rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets --link-dest=$PREVIO \ ... --link-dest arg does not exist: 001 What am I doing wrong? Or is --link-dest not supported on rsyncd daemon (remote)? Both $DESTINATION and $PREVIOUS are on the same partition (btrfs), so hard links are possible. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On Tue, Aug 6, 2019 at 1:18 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
--link-dest arg does not exist: root@Isengard::MyBook_Telcontar/001
--link-dest by definition refers to directory on target system. ...
I also tried:
rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets --link-dest=$PREVIO \
Definition of $PREVIO was never shown before.
...
--link-dest arg does not exist: 001
Any reason you do not read manuals? "If DIR is a relative path, it is relative to the destination directory." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/08/2019 12.39, Andrei Borzenkov wrote:
On Tue, Aug 6, 2019 at 1:18 PM Carlos E. R. <robin.listas@telefonica.net> wrote:
--link-dest arg does not exist: root@Isengard::MyBook_Telcontar/001
--link-dest by definition refers to directory on target system.
...
I also tried:
rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets --link-dest=$PREVIO \
Definition of $PREVIO was never shown before.
In this post, yes. PREVIO is PREVIOUS in Spanish. I tried to translate the script for your ease, maybe one escaped my view in the post. The script only uses PREVIO. The actual command that is running has: root 3113 12.8 0.6 146508 56900 pts/7 S+ 12:14 3:52 | | \_ rsync --archive --acls --xattrs --hard-links --sparse --stats --human-readable --password-file /root/cripta/keys/rsync.secrets --link-dest=root@Isengard::MyBook_Telcontar/001 --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ / root@Isengard::MyBook_Telcontar/002 See? --link-dest=root@Isengard::MyBook_Telcontar/001
...
--link-dest arg does not exist: 001
Any reason you do not read manuals?
I did, several times. That does not mean that I understand every detail. The manual lacks examples.
"If DIR is a relative path, it is relative to the destination directory."
I do not understand that. The paths I give as source or destination are absolute, or so I understand. So, maybe you say that I should type: --link-dest=../$PREVIOUS Well, it seems to work, at least it doesn't complain. Thanks. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On 08/06/2019 05:18 AM, Carlos E. R. wrote:
Hi,
Suppose I do this rsync run:
OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINATION=001
rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINATION
<snip> Carlos, With all the trouble you are having with various includes/excludes, why not just setup to run with '--no-R --files-from=yourlist' You can use find to create 'yourlist' file containing the directories to sync (one-per-line) and then instead of the chain of --include= --exclude= or --filter= .... you simply have a single clean rsync of rsync -uai -r --no-R --files-from=yourlist / user@host:/destdir (-r is needed if you want recursive as -a does not imply -r when using --files-from. adjust the informational and update control options as needed) If you make yourlist contain absolute paths, then your destdir is simply / as well. The first / (srcdir, after --files-from=yourlist) just provides the base for the source files in yourlist and work if they are absolute when run from anywehre or in the form ./dirname when run from that directory. yourlist can be accessed from either the source host or on the dest host. If you have a fixed set of directories to include/exclude, this may be a solution as it will just sync the directories named in yourlist. The only think you need be careful of is recursive control. But you can easily sync /somedir/data/sub/source While not syncing anything over 'source'... and sync /somedir/data/sub/source /somedir/data/sub/other /somedir/data/sub4 While not syncing anything from /somedir/data other than what you have named. rsync man page is pretty good, it just takes a bit to wrap your head around providing yourlist, the base / and then then destination. Once you figure out / is just the base for whats in yourlist, then it is no different at all from normal usage. -- David C. Rankin, J.D.,P.E.
On Tue, 6 Aug 2019 08:44:40 -0500 "David C. Rankin" <drankinatty@suddenlinkmail.com> wrote:
On 08/06/2019 05:18 AM, Carlos E. R. wrote:
Hi,
Suppose I do this rsync run:
OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINATION=001
rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINATION <snip>
Carlos,
With all the trouble you are having with various includes/excludes, why not just setup to run with '--no-R --files-from=yourlist'
You can use find to create 'yourlist' file containing the directories to sync (one-per-line) and then instead of the chain of --include= --exclude= or --filter= .... you simply have a single clean rsync of
rsync -uai -r --no-R --files-from=yourlist / user@host:/destdir
(-r is needed if you want recursive as -a does not imply -r when using --files-from. adjust the informational and update control options as needed)
If you make yourlist contain absolute paths, then your destdir is simply / as well. The first / (srcdir, after --files-from=yourlist) just provides the base for the source files in yourlist and work if they are absolute when run from anywehre or in the form ./dirname when run from that directory. yourlist can be accessed from either the source host or on the dest host.
If you have a fixed set of directories to include/exclude, this may be a solution as it will just sync the directories named in yourlist. The only think you need be careful of is recursive control. But you can easily sync
/somedir/data/sub/source
While not syncing anything over 'source'... and sync
/somedir/data/sub/source /somedir/data/sub/other /somedir/data/sub4
While not syncing anything from /somedir/data other than what you have named.
rsync man page is pretty good, it just takes a bit to wrap your head around providing yourlist, the base / and then then destination. Once you figure out / is just the base for whats in yourlist, then it is no different at all from normal usage.
That's quite nice. I was going to suggest some other points: (1) Run rsync on the backup host and have a read-only module on the source host - I believe that's more secure (2) Start simple with one or two directories and then add more once you're sure how it all works (3) Build a small test source tree so you can check how rsync works without having to wait ages for it to copy your entire filesystem (4) Ditch all the ** args. You don't need them; I've never used them. (5) Do use some relative patterns to exclude unwanted backup files etc and to include everything else -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/08/2019 16.01, Dave Howorth wrote:
On Tue, 6 Aug 2019 08:44:40 -0500 "David C. Rankin" <> wrote:
Notice that this thread was only about --link-dest=. Andrei B. made the suggestion that made that part work (and many of the others), so this thread can be considered solved. ...
That's quite nice. I was going to suggest some other points:
(1) Run rsync on the backup host and have a read-only module on the source host - I believe that's more secure
Well, that is what I'm doing :-) Only that I don't know if the source host module is read only or not. The words "read-only" or "read only" do not appear on the manual.
(2) Start simple with one or two directories and then add more once you're sure how it all works
I started by simply copying from my notes and several backup scripts that I have :-) Only that till now, I called it several times, instead of fighting include/excludes. This time I wanted to solve that issue.
(3) Build a small test source tree so you can check how rsync works without having to wait ages for it to copy your entire filesystem
Yes, that could be. But then the source would not be "/".
(4) Ditch all the ** args. You don't need them; I've never used them.
Well, it does not work without them...
(5) Do use some relative patterns to exclude unwanted backup files etc and to include everything else
It is using "--relative", but I do not fully understand why. The final, working, concoction, is: OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINO=002 PREVIO=001 rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets --link-dest=../$PREVIO \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINO This run, making hard links to the previous one, runs very fast. If you want to suggest changes, I can try them on DEST=003 - if it goes slow, it is not working right ;-) -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos E. R. schrieb am 07.08.19 um 12:42:
On 06/08/2019 16.01, Dave Howorth wrote:
On Tue, 6 Aug 2019 08:44:40 -0500 "David C. Rankin" <> wrote:
Notice that this thread was only about --link-dest=. Andrei B. made the suggestion that made that part work (and many of the others), so this thread can be considered solved.
...
That's quite nice. I was going to suggest some other points:
(1) Run rsync on the backup host and have a read-only module on the source host - I believe that's more secure
Well, that is what I'm doing :-)
Only that I don't know if the source host module is read only or not. The words "read-only" or "read only" do not appear on the manual.
This you configure in /etc/rsyncd.conf (on the source host in this case, since you run the command "rsync ... $SOURCEHOST::modulename /local/dir" on the host where the backup will be placed) [modulename] path = /some/where comment = foo read only = true list = yes ... See man 5 rsyncd.conf Werner -- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 07/08/2019 12.49, Werner Flamme wrote:
Carlos E. R. schrieb am 07.08.19 um 12:42:
Only that I don't know if the source host module is read only or not. The words "read-only" or "read only" do not appear on the manual.
This you configure in /etc/rsyncd.conf (on the source host in this case, since you run the command "rsync ... $SOURCEHOST::modulename /local/dir" on the host where the backup will be placed)
[modulename] path = /some/where comment = foo read only = true list = yes ...
See man 5 rsyncd.conf
Ah, I see. Another man page. Can't be done, the daemon runs on the destination machine. The source machine is using direct access on the CLI. In fact, I use that setting by default on my daemons, I had forgotten. Then I set "read only = false" on specific modules. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On Wed, 7 Aug 2019 12:42:55 +0200 "Carlos E. R." <robin.listas@telefonica.net> wrote:
On 06/08/2019 16.01, Dave Howorth wrote:
On Tue, 6 Aug 2019 08:44:40 -0500 "David C. Rankin" <> wrote:
Notice that this thread was only about --link-dest=. Andrei B. made the suggestion that made that part work (and many of the others), so this thread can be considered solved.
...
That's quite nice. I was going to suggest some other points:
(1) Run rsync on the backup host and have a read-only module on the source host - I believe that's more secure
Well, that is what I'm doing :-)
No, it isn't! In your very next message you explain to Werner that you're doing it the other way around. And looking at the command you use, it is clearly intended to run on the source machine. So please start again and read what I wrote with a clear head.
Only that I don't know if the source host module is read only or not. The words "read-only" or "read only" do not appear on the manual.
As Werner points out, you're looking at the wrong man page for rsyncd configuration.
(2) Start simple with one or two directories and then add more once you're sure how it all works
I started by simply copying from my notes and several backup scripts that I have :-)
Only that till now, I called it several times, instead of fighting include/excludes. This time I wanted to solve that issue.
(3) Build a small test source tree so you can check how rsync works without having to wait ages for it to copy your entire filesystem
Yes, that could be. But then the source would not be "/".
If you set your include/exclude patterns sensibly, that will not be a major concern.
(4) Ditch all the ** args. You don't need them; I've never used them.
Well, it does not work without them...
Because of #5
(5) Do use some relative patterns to exclude unwanted backup files etc and to include everything else
It is using "--relative", but I do not fully understand why.
That's not what I said. I said relative patterns, i.e. patterns that don't start with a slash.
The final, working, concoction, is:
OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINO=002 PREVIO=001 rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets --link-dest=../$PREVIO \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINO
This run, making hard links to the previous one, runs very fast. If you want to suggest changes, I can try them on DEST=003 - if it goes slow, it is not working right ;-)
Well, I'm pleased you got something to work. If you're interested in learning about rsync and how people normally use it, I suggest following my steps. They will have no effect on your working system unless and until you choose to do so. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2019-08-07 at 14:25 +0100, Dave Howorth wrote:
On Wed, 7 Aug 2019 12:42:55 +0200 "Carlos E. R." <> wrote:
On 06/08/2019 16.01, Dave Howorth wrote:
On Tue, 6 Aug 2019 08:44:40 -0500 "David C. Rankin" <> wrote:
Notice that this thread was only about --link-dest=. Andrei B. made the suggestion that made that part work (and many of the others), so this thread can be considered solved.
...
That's quite nice. I was going to suggest some other points:
(1) Run rsync on the backup host and have a read-only module on the source host - I believe that's more secure
Well, that is what I'm doing :-)
No, it isn't! In your very next message you explain to Werner that you're doing it the other way around. And looking at the command you use, it is clearly intended to run on the source machine. So please start again and read what I wrote with a clear head.
Sorry, language issue here.
(2) Start simple with one or two directories and then add more once you're sure how it all works
I started by simply copying from my notes and several backup scripts that I have :-)
Only that till now, I called it several times, instead of fighting include/excludes. This time I wanted to solve that issue.
(3) Build a small test source tree so you can check how rsync works without having to wait ages for it to copy your entire filesystem
Yes, that could be. But then the source would not be "/".
If you set your include/exclude patterns sensibly, that will not be a major concern.
? To me they were obvious and sensible.
(4) Ditch all the ** args. You don't need them; I've never used them.
Well, it does not work without them...
Because of #5
(5) Do use some relative patterns to exclude unwanted backup files etc and to include everything else
It is using "--relative", but I do not fully understand why.
That's not what I said. I said relative patterns, i.e. patterns that don't start with a slash.
Relative to what? If I want to backup the entire machine, the obvious thing is to say "/".
The final, working, concoction, is:
OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINO=002 PREVIO=001 rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets --link-dest=../$PREVIO \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINO
This run, making hard links to the previous one, runs very fast. If you want to suggest changes, I can try them on DEST=003 - if it goes slow, it is not working right ;-)
Well, I'm pleased you got something to work. If you're interested in learning about rsync and how people normally use it, I suggest following my steps. They will have no effect on your working system unless and until you choose to do so.
Start over the design running the CLI command on the backup machine, which has much less CPU power? Running backup processes on the "backup server", maybe a cron job, makes sense if the patterns when the client machines are up and available is known or predicable. That is not the case. It is the person on the client (me) who knows when to do the backup. And knowing that it takes about thirty hours, I see best to have that process running on the client machine, where I see and control it easily. I run it like I do in order to distribute the resources I have: much more power on the client machine than in the server. The server is in fact overloaded by compression and encryption, it overheats and has to be throttled. The reason I put the disk in the server and not directly in the client is simply because the client doesn't have an USB3 socket... On an office, it would be the backup server machine which would do the job, so to have it all in a single machine, easier to configure. But this is not an office. - -- Cheers, Carlos E. R. (from openSUSE 15.0 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCXUrVWxwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfVgbkAn2k15FhPeUUyvqacvXyQ lLYEBKjoAJoDQfUNGK53PrVxBnuqayFCLnjw8w== =C/7f -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/08/2019 15.44, David C. Rankin wrote:
On 08/06/2019 05:18 AM, Carlos E. R. wrote:
Hi,
Suppose I do this rsync run:
OPTIONS="--archive --acls --xattrs --hard-links --sparse --stats --human-readable " DESTINATION=001
rsync $OPTIONS --password-file /root/cripta/keys/rsync.secrets \ --relative --include=/data/vmware** --include=/data/storage_** --include=/data/cripta** --include=/data/xtr** --include=/data/raid** \ --exclude=/lost+found --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/run \ --exclude=/data/** --exclude=/mnt --exclude=/other --exclude=/media --exclude=/var/lib/samba/lock/ --exclude=/var/run/ \ / root@Isengard::MyBook_Telcontar/$DESTINATION
<snip>
Carlos,
With all the trouble you are having with various includes/excludes, why not just setup to run with '--no-R --files-from=yourlist'
Notice that I had the backup made from the start (since hour one) simply by calling rsync several times to get it done (in this case, twice). The problem was making it using a single call - and that I finally got, with help :-) ...
rsync man page is pretty good, it just takes a bit to wrap your head around
Oh, absolutely :-) It lacks examples. Yes, there are some, but they don't cover many cases. I have read many threads of people complaining of rsync complexity around the world.
providing yourlist, the base / and then then destination. Once you figure out / is just the base for whats in yourlist, then it is no different at all from normal usage.
-- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
participants (5)
-
Andrei Borzenkov -
Carlos E. R. -
Dave Howorth -
David C. Rankin -
Werner Flamme