[opensuse] strange samba rsync problem
I am having a lot of trouble trying to get a SuSE pc to connect to my samba server in such a way that rsync works. Here is the setup. I have a Samba server running on Solaris 10. It has been running for 2 years and the Windows pc connects just fine and I have even used cygwin rsync in the past to backup up to that mapped drive Now I am trying to do something similar with the SuSE pc. On the SuSE pc I can mount the cifs share and can read and write files on that mounted share. This is what I used to mount mount -t cifs -o uid=rosa,gid=users,nounix,iocharset=iso8859-15,credentials=/etc/zdrive //192.168.15.100/data /windows/data The strange problem I have is with rsync on SuSE. I try to backup a small folder in the /home/rosa directory to a place on the server rsync -avz --delete /home/rosa/and2/ /windows/data/rosatest It runs, the files are copied and a diff shows they are the same. Now I run the rsync again and since nothing is changed, it should not have to copy any files again but it copies the entire source all over again as if they all changed. If I change the destination to some place on the local hard drive, then it behaves just as I expect. Files are copied the first time but not again because they are now the same. Got any idea why this is happening? Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
----- Original Message -----
From: "Damon Register"
I am having a lot of trouble trying to get a SuSE pc to connect to my samba server in such a way that rsync works. Here is the setup. I have a Samba server running on Solaris 10. It has been running for 2 years and the Windows pc connects just fine and I have even used cygwin rsync in the past to backup up to that mapped drive
Now I am trying to do something similar with the SuSE pc. On the SuSE pc I can mount the cifs share and can read and write files on that mounted share. This is what I used to mount
mount -t cifs -o uid=rosa,gid=users,nounix,iocharset=iso8859-15,credentials=/etc/zdrive //192.168.15.100/data /windows/data
The strange problem I have is with rsync on SuSE. I try to backup a small folder in the /home/rosa directory to a place on the server
rsync -avz --delete /home/rosa/and2/ /windows/data/rosatest
It runs, the files are copied and a diff shows they are the same. Now I run the rsync again and since nothing is changed, it should not have to copy any files again but it copies the entire source all over again as if they all changed. If I change the destination to some place on the local hard drive, then it behaves just as I expect. Files are copied the first time but not again because they are now the same. Got any idea why this is happening?
step one is rsync -vvvvaz --del --progress source dest Maybe add -n and/or -i too. Then based on that will probably be some tweak to rsync options to tell it to ignore some metadata aspect that it currently looks at. Or possibly a tweak to the mount command or possibly even the samba options on the server, or all of the above. Remember samba is ultimately trying to emulate a non-unix filesystem, and not all unix fs features necessarily have any place or equivalent in ntfs or cifs, and so the remote file looks different to rsync because of some difference in the date/time stamps, metadata like uid/gid mapping, owndership, perms, acls, xattrs. Even though in this case the ultimate remote filesystem is also unix and all features probably translate fine. In this case no way would I use samba to do this sync. Keep the samba server and the share there for the ske of the pc's, sure, but to sync data fom the suse box I'd just use direct rsync. As in, run an rsync daemon on the solaris box and use rsync -avz --del /local/path user@sunbox::module/path on the suse box. -- Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote: > step one is rsync -vvvvaz --del --progress source dest > > Maybe add -n and/or -i too. I just tried that as you suggested with the -i but left out the -n dry run. > Then based on that will probably be some tweak to rsync options to tell it to ignore some metadata aspect that it currently looks at. Or possibly a tweak to the mount command or possibly even the samba options on the server, or all of the above. > Remember samba is ultimately trying to emulate a non-unix filesystem, and not all unix fs features necessarily have any place or equivalent in ntfs or cifs, and so the remote file looks different to rsync because of some difference in the date/time stamps, metadata like uid/gid mapping, owndership, perms, acls, xattrs. Even though in this case the ultimate remote filesystem is also unix and all features probably translate fine. I looked at the output but am not sure how to interpret it. I don't know what tweaks I would have to make. > In this case no way would I use samba to do this sync. > Keep the samba server and the share there for the ske of the pc's, sure, but to sync data fom the suse box I'd just use direct rsync. As in, run an rsync daemon on the solaris box and use rsync -avz --del /local/path user@sunbox::module/path > on the suse box. I am not sure I still remember the details but I did it this way at the suggestion of a sysadmin at work. If I still remember correctly, I think there were two reasons why I let samba do the net part and use rsync as if both source and dest were local. 1. I don't have to deal with another authentication since samba already did it 2. it is faster than through the remote rsync Of course I could be wrong but that is what I think I remember. I cut my test folder down to 1 file so there is less to look at with the rsync output. I emptied the destination and ran the rsync twice so that I expect the first to copy the file and the second should not copy the file. Now the problem is that I don't know how to interpret what I see. There isn't anything that stands out to me. cmd=machine= user= path=/windows/data/rosatest cmd[0]=. cmd[1]=/windows/data/rosatest note: iconv_open("UTF-8", "UTF-8") succeeded. (Client) Protocol versions: remote=30, negotiated=30 (Server) Protocol versions: remote=30, negotiated=30 sending incremental file list [sender] make_file(.,*,2) [sender] make_file(Andrw1AD.cpp,*,2) [sender] flist start=1, used=2, low=0, high=1 [sender] i=1 /home/rosa/and3 ./ mode=040755 len=80 uid=1000 gid=100 flags=5 [sender] i=2 /home/rosa/and3 Andrw1AD.cpp mode=0100644 len=5279 uid=1000 gid=100 flags=0 send_file_list done file list sent send_files starting server_recv(2) starting pid=4953 uid 1000(rosa) maps to 1000 process has 4 gids: 14 16 33 100 gid 100(users) maps to 100 recv_file_name(.) recv_file_name(Andrw1AD.cpp) received 2 names [receiver] flist start=1, used=2, low=0, high=1 [receiver] i=1 0 ./ mode=040755 len=80 gid=100 flags=5 [receiver] i=2 1 Andrw1AD.cpp mode=0100644 len=5279 gid=100 flags=0 recv_file_list done get_local_name count=2 /windows/data/rosatest generator starting pid=4953 delta-transmission disabled for local transfer or --whole-file recv_generator(.,0) recv_files(2) starting send_files(0, /home/rosa/and3/.) .d..tp..... ./ set modtime of . to (1220270221) Mon Sep 1 07:57:01 2008 delete_in_dir(.) [generator] make_file(Andrw1AD.cpp,*,2) recv_files(.) [generator] flist start=0, used=1, low=0, high=0 [generator] i=0 0 Andrw1AD.cpp mode=0100644 len=5279 gid=100 flags=0 recv_generator(.,1) recv_generator(Andrw1AD.cpp,2) send_files(2, /home/rosa/and3/Andrw1AD.cpp) > >f..t...... Andrw1AD.cpp generate_files phase=1 send_files phase=1 recv_files(Andrw1AD.cpp) recv_files phase=1 generate_files phase=2 send_files phase=2 send files finished total: matches=0 hash_hits=0 false_alarms=0 data=0 recv_files phase=2 generate_files phase=3 recv_files finished generate_files finished client_run waiting on 4953 sent 70 bytes received 18 bytes 176.00 bytes/sec total size is 5279 speedup is 59.99 (DRY RUN) _exit_cleanup(code=0, file=main.c, line=1031): entered _exit_cleanup(code=0, file=main.c, line=1031): about to call exit(0) --------------------------------------------------------------------- cmd= machine= user= path=/windows/data/rosatest cmd[0]=. cmd[1]=/windows/data/rosatest note: iconv_open("UTF-8", "UTF-8") succeeded. (Server) Protocol versions: remote=30, negotiated=30 (Client) Protocol versions: remote=30, negotiated=30 sending incremental file list [sender] make_file(.,*,2) [sender] make_file(Andrw1AD.cpp,*,2) [sender] flist start=1, used=2, low=0, high=1 [sender] i=1 /home/rosa/and3 ./ mode=040755 len=80 uid=1000 gid=100 flags=5 [sender] i=2 /home/rosa/and3 Andrw1AD.cpp mode=0100644 len=5279 uid=1000 gid=100 flags=0 send_file_list done file list sent send_files starting server_recv(2) starting pid=5002 uid 1000(rosa) maps to 1000 process has 4 gids: 14 16 33 100 gid 100(users) maps to 100 recv_file_name(.) recv_file_name(Andrw1AD.cpp) received 2 names [receiver] flist start=1, used=2, low=0, high=1 [receiver] i=1 0 ./ mode=040755 len=80 gid=100 flags=5 [receiver] i=2 1 Andrw1AD.cpp mode=0100644 len=5279 gid=100 flags=0 recv_file_list done get_local_name count=2 /windows/data/rosatest generator starting pid=5002 delta-transmission disabled for local transfer or --whole-file recv_generator(.,0) send_files(0, /home/rosa/and3/.) .d..tp..... ./ set modtime of . to (1220270221) Mon Sep 1 07:57:01 2008 delete_in_dir(.) [generator] make_file(Andrw1AD.cpp,*,2) [generator] flist start=0, used=1, low=0, high=0 [generator] i=0 0 Andrw1AD.cpp mode=0100644 len=5279 gid=100 flags=0 recv_generator(.,1) recv_generator(Andrw1AD.cpp,2) send_files(2, /home/rosa/and3/Andrw1AD.cpp) > >f..t...... Andrw1AD.cpp recv_files(2) starting generate_files phase=1 send_files phase=1 recv_files(.) recv_files(Andrw1AD.cpp) recv_files phase=1 generate_files phase=2 send_files phase=2 send files finished total: matches=0 hash_hits=0 false_alarms=0 data=0 recv_files phase=2 generate_files phase=3 recv_files finished generate_files finished client_run waiting on 5002 sent 70 bytes received 18 bytes 176.00 bytes/sec total size is 5279 speedup is 59.99 (DRY RUN) _exit_cleanup(code=0, file=main.c, line=1031): entered _exit_cleanup(code=0, file=main.c, line=1031): about to call exit(0) Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
step one is rsync -vvvvaz --del --progress source dest
Maybe add -n and/or -i too. That has proved most useful. I tried using the --iconv option but
Brian K. White wrote: that failed. After looking at the output I saw that it said the option was refused by the server. On further investigation I found that I had installed the latest server in the wrong location so that it was the old version that was running. Even with that correction it didn't work and I found it was because the build process couldn't find libiconv. I fixed CFLAGS and LDFLAGS but it seems I still have to find the correct settings so that it finds the shared lib at runtime. Fortunately there are quite a few programmers her that might be able to help me with that. At lease now I know that the server wasn't able to do any conversion and I think that once I fix that problem, I will have the correct file names. Thanks for your help. Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
----- Original Message -----
From: "Damon Register"
step one is rsync -vvvvaz --del --progress source dest
Maybe add -n and/or -i too. That has proved most useful. I tried using the --iconv option but
Brian K. White wrote: that failed. After looking at the output I saw that it said the option was refused by the server. On further investigation I found that I had installed the latest server in the wrong location so that it was the old version that was running. Even with that correction it didn't work and I found it was because the build process couldn't find libiconv. I fixed CFLAGS and LDFLAGS but it seems I still have to find the correct settings so that it finds the shared lib at runtime. Fortunately there are quite a few programmers her that might be able to help me with that.
At lease now I know that the server wasn't able to do any conversion and I think that once I fix that problem, I will have the correct file names. Thanks for your help.
Glad to help. Sounds like you're on track. However, I would (I do in fact) just install the prebuilt rsync from sunfreeware. Sample install recipe of a package on a sparc solaris 9 box: Browse here for packages, http://www.sunfreeware.com/indexsparc9.html wget ftp://ftp.sunfreeware.com/pub/freeware/sparc/9/bash-3.2.17-sol9-sparc-local.gz gunzip bash-3.2.17-sol9-sparc-local.gz pkgadd -d bash-3.2.17-sol9-sparc-local For any given package, first follow the dependancy links to other packages and perform the same instructions as above for those packages first. You said you have solaris 10, I don't remember what platform, so, assuming it's sparc, for you to install rsync, you could cut & paste this recipe verbtim, paste the whole block from sfi () { to } right into a telnet/ssh session. Then run the sfi <pkg> commands manually one at a time, in the order shown. sfi () { FTP="ftp://ftp.sunfreeware.com/pub/freeware/sparc/10/" SUF="-sol10-sparc-local" wget ${FTP}${1}${SUF}.gz || return gunzip ${1}${SUF}.gz pkgadd -d ${1}$SUF rm ${1}$SUF } sfi libgcc-3.4.6 sfi libiconv-1.11 sfi libintl-3.4.0 sfi popt-1.14 sfi rsync-3.0.3 If your platform is x86 then just go to www.sunfreeware.com and follow the link for your version and platform on th top-right. Then fetch the equivalent tar.gz's as above and run the same commands. If you don't have wget or curl just get the files any way you want of course and run the gunzip & pkgadd commands manually like the first example. -- Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
Sounds like you're on track. I thought I was but it is far from over.
However, I would (I do in fact) just install the prebuilt rsync from sunfreeware. I just did. I tried building it from source but that didn't get very far so I just got the prebuilt packages from SFW. I tested with that and still get the same old
You said you have solaris 10, I don't remember what platform, so, assuming it's sparc, It is an Ultra20 with an AMD64.
for you to install rsync, you could cut & paste this recipe verbtim, paste the whole block from sfi () { to } right into a telnet/ssh session. Then run the sfi <pkg> commands manually one at a time, in the order shown. That's a cool script. Thanks.
If you don't have wget or curl just get the files any way you want of course and run the gunzip & pkgadd commands manually like the first example. I had to get those a while ago for some reason I don't remember any more.
So far everything I tried gives me the same error. rsync: The server is configured to refuse --iconv rsync error: requested action not supported (code 4) at clientserver.c(840) [receiver=3.0.3] rsync: connection unexpectedly closed (5 bytes received so far) [sender] _exit_cleanup(code=12, file=io.c, line=635): entered rsync error: error in rsync protocol data stream (code 12) at io.c(635) [sender=3.0.3] _exit_cleanup(code=12, file=io.c, line=635): about to call exit(12) No matter what options I try, I still get the "refuse --iconv" message. I got the impression from the man page that if I didn't specify a charset that the iconv would be refused unless I added no-iconv to the refuse options. Since I wasn't getting anywhere with the Solaris setup, I installed SuSE 11 on an old spare PC. After getting the rsync daemon setup there, I tried it and got the same message. It seems that the rsync daemon on the SuSE system has the same problem. So, since the SuSE setup has the same problem I will focus on that first since I might have a better chance of getting help. Perhaps if I get the SuSE one working, I will have found the solution for the Solaris one as well. Can anyone help with rsync on SuSE? Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
----- Original Message -----
From: "Damon Register"
Brian K. White wrote:
Sounds like you're on track. I thought I was but it is far from over.
However, I would (I do in fact) just install the prebuilt rsync from sunfreeware. I just did. I tried building it from source but that didn't get very far so I just got the prebuilt packages from SFW. I tested with that and still get the same old
You said you have solaris 10, I don't remember what platform, so, assuming it's sparc, It is an Ultra20 with an AMD64.
for you to install rsync, you could cut & paste this recipe verbtim, paste the whole block from sfi () { to } right into a telnet/ssh session. Then run the sfi <pkg> commands manually one at a time, in the order shown. That's a cool script. Thanks.
If you don't have wget or curl just get the files any way you want of course and run the gunzip & pkgadd commands manually like the first example. I had to get those a while ago for some reason I don't remember any more.
So far everything I tried gives me the same error.
rsync: The server is configured to refuse --iconv rsync error: requested action not supported (code 4) at clientserver.c(840) [receiver=3.0.3] rsync: connection unexpectedly closed (5 bytes received so far) [sender] _exit_cleanup(code=12, file=io.c, line=635): entered rsync error: error in rsync protocol data stream (code 12) at io.c(635) [sender=3.0.3] _exit_cleanup(code=12, file=io.c, line=635): about to call exit(12)
No matter what options I try, I still get the "refuse --iconv" message. I got the impression from the man page that if I didn't specify a charset that the iconv would be refused unless I added no-iconv to the refuse options.
Since I wasn't getting anywhere with the Solaris setup, I installed SuSE 11 on an old spare PC. After getting the rsync daemon setup there, I tried it and got the same message. It seems that the rsync daemon on the SuSE system has the same problem. So, since the SuSE setup has the same problem I will focus on that first since I might have a better chance of getting help. Perhaps if I get the SuSE one working, I will have found the solution for the Solaris one as well.
Can anyone help with rsync on SuSE?
"man rsync", under --iconv=CONVERT_SPEC says nothing will happen unless there is a charset option in the daemon config file I put "charset = cp437" into /etc/rsyncd.conf on a remote machine and restarted rsyncd. and it says to use --iconv=local_charset[,remote_charset] and that valid values for charsets come from "iconv -l" first I tried a few tests with iconv itself, not involving rsyc, just to get so I know how to use iconv itself. After a lot of tests I'm not so sure this option is going to be very useful. It fails way too easily, saying that various characters are invalid either for the input or output character set. Even when both character sets do have the same character, iconv fails to translate them many times. However, eventually I arrive at a comand that converts something to something else, correctly. windows cp1252 has an e-umlaut at hex eb http://en.wikipedia.org/wiki/Windows-1252 cp437 has e-umlaut at hex 89 http://en.wikipedia.org/wiki/Code_page_437 Lets try to get iconv to translate the e-umlaut from windows to vga Rather than look at the character as displayed by my terminal, this email client, your email client, etc... lets just look at the output of "od" to remove the bezillion ambiguities. nc7:~ $ echo -e "\xeb\c" |od -h 0000000 00eb 0000001 (echo \xeb , no translation, and od -h shows 00eb as expected, now add iconv without changing anything else) nc7:~ $ echo -e "\xeb\c" |iconv -f cp1252 -t cp437 |od -h 0000000 0089 0000001 nc7:~ $ So now we have sample data that we know iconv can translate, and we know the valid iconv charset specifiers. _now_ try applying them to rsync. On the remote machine I create a file with a hex eb in it's name. nc7:~ $ mkdir i nc7:~ $ cd i nc7:~/i $ touch `echo -e "h\xebllo"` nc7:~/i $ ls --show-control-chars hëllo nc7:~/i $ Based on the iconv section of the man page, because my local & remote machines have the same default LANG and LC_* settings, it appears --iconv=. will not do anything, and so I need to specify at least a local character set. On the local machine, try to download the directory containing the file and convert from windows-1252 to cp437 (I don't want to have to specify that funky character on the command line, and I don't want to figure out should I use hex 89 on the local end in order to request a file with hex eb on the remote end? etc.. just get the whole directory. get fancier later if necessary.) nj2:~ $ rsync -avz --iconv=cp437,cp1252 nc7::bkw/i . receiving incremental file list rsync: The server is configured to refuse --iconv rsync error: requested action not supported (code 4) at clientserver.c(842) [sender=3.0.2] rsync: read error: Connection reset by peer (104) rsync error: error in rsync protocol data stream (code 12) at io.c(759) [receiver=3.0.0] nj2:~ $ Well the remote server has no charset option in it's rsyncd.conf, so now I add "charset = iso8859-1" to the remote servers /etc/rsyncd.conf, because although I'm translating from cp1252 to cp437, the remote server really isn't cp1252 in general and I don't want the remote rsync converting anyone elses files. Then restart rsyncd on the remote server. And try the same download again. Download with no conversion: nj2:~ $ rsync -avz --del nc7::bkw/i . receiving incremental file list i/ i/h?llo sent 77 bytes received 159 bytes 157.33 bytes/sec total size is 0 speedup is 0.00 nj2:~ $ Now with conversion: nj2:~ $ rsync -avz --del --iconv=cp437,cp1252 nc7::bkw/i . receiving incremental file list deleting i/h?llo i/hëllo sent 74 bytes received 157 bytes 462.00 bytes/sec total size is 0 speedup is 0.00 nj2:~ $ ls --show-control-chars i hëllo nj2:~ $ I think this is still full of problems. Almost all of my manual iconv test conversions failed. Too many times it said that a character was invald in either the input or output character sets. For instance, it wouldn't turn o-umlaut into a plain o if no o-umlaut existed in the output charset, nor would it pass the byte through without changing it, instead it output nothing except an error message. Adding -cs would hide the error messages and simply strip any bad characters, but that's not very useful for filenames. I don't know what your final answer should be other than forcing everything everywhere (samba and rsync) to use utf8 or utf16 so they all speak the same character set. -- Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
"man rsync", under --iconv=CONVERT_SPEC
says nothing will happen unless there is a charset option in the daemon config file I put "charset = cp437" into /etc/rsyncd.conf on a remote machine and restarted rsyncd. Do you have to restart? I thought I read somewhere that it reads
Brian K. White wrote: the config everytime it runs
and it says to use --iconv=local_charset[,remote_charset] Now I am really confused. I am looking at the rsync man page right now from http://www.samba.org/ftp/rsync/rsync.html I don't find what you said about the charset requirement. Am I missing something?
first I tried a few tests with iconv itself, not involving rsyc, just to get so I know how to use iconv itself. After a lot of tests I'm not so sure this option is going to be very useful. Thanks for all that work. I am beginning to thing the same about its usefulness.
not very useful for filenames. I don't know what your final answer should be other than forcing everything everywhere (samba and rsync) to use utf8 or utf16 so they all speak the same character set. I think that is the conclusion I reached too. Based on what I saw from the rsync'd file on the server, I conlcuded it was getting converted to utf8 so I tampered with the cifs mount to the samba server until I got it to match. Now when I use the samba connection to view the files that have been rsync'd to the server, the characters match. So, I may never figure out those iconv problems but I guess it isn't so important now.
Thanks again for all your help Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
----- Original Message -----
From: "Damon Register"
"man rsync", under --iconv=CONVERT_SPEC
says nothing will happen unless there is a charset option in the daemon config file I put "charset = cp437" into /etc/rsyncd.conf on a remote machine and restarted rsyncd. Do you have to restart? I thought I read somewhere that it reads
Brian K. White wrote: the config everytime it runs
I'm running a standalone rsyncd. It only runs once. Running rsync via rsh or ssh is different. (or inetd/xinetd)
and it says to use --iconv=local_charset[,remote_charset] Now I am really confused. I am looking at the rsync man page right now from http://www.samba.org/ftp/rsync/rsync.html I don't find what you said about the charset requirement. Am I missing something?
---quote--- --iconv=CONVERT_SPEC Rsync can convert filenames between character sets using this option. Using a CONVERT_SPEC of "." tells rsync to look up the default character-set via the locale setting. Alternately, you can fully specify what conversion to do by giving a local and a remote charset separated by a comma in the order --iconv=LOCAL,REMOTE, e.g. --iconv=utf8,iso88591. This order ensures that the option will stay the same whether you're pushing or pulling files. Finally, you can specify either --no-iconv or a CONVERT_SPEC of "-" to turn off any conversion. The default setting of this option is site-specific, and can also be affected via the RSYNC_ICONV environment variable. For a list of what charset names your local iconv library supports, you can run "iconv --list". [...] When you pass an --iconv option to an rsync daemon that allows it, the daemon uses the charset specified in its "charset" configuration parameter regardless of the remote charset you actually pass. Thus, you may feel free to specify just the local charset for a daemon transfer (e.g. --iconv=utf8). ---quote--- It looks like my previous example had a part that had no effect. According to the man-page, the remote charset part of the --iconv=local,remote has no effect ever when talking to a daemon, it -only- uses the charset option. That must be why it also doesn't work at all if there is no charset option. When NOT using the daemon, going to the same machine, which now has no charset in it's config any more btw: Notice the single ":" this time and different paths... Without conversion: bkw@nj2:~> rsync -avz --del nc7:i . bkw@nc7's password: receiving incremental file list i/ i/h?llo sent 34 bytes received 106 bytes 31.11 bytes/sec total size is 0 speedup is 0.00 With conversion: bkw@nj2:~> rsync -avz --del --iconv cp437,iso8859-1 nc7:i . bkw@nc7's password: receiving incremental file list deleting i/h?llo i/hëllo sent 31 bytes received 104 bytes 24.55 bytes/sec total size is 0 speedup is 0.00 bkw@nj2:~> These both used ssh, which means "i" is a directory in bkw's home directory on nc7. The rsyncd.conf file on nc7 has no charset option anywhere, but these transfers did not use the daemon that read that config file so it doesn't matter. However I would not do most transpers this way unless they are small and you can afford the cpu on the encryption and don't care that they will transfer -much- slower than native rsync. Mucking around with this is dangerous too... I almost deleted my entire home directory when iconv didn't like the characters and the charsets specified. bkw@nj2:~> rsync -avz --del --iconv=iso8859-1,cp437 -n nc7:i . bkw@nc7's password: receiving incremental file list [receiver] cannot convert filename: i/h??llo (Invalid or incomplete multibyte or wide character) IO error encountered -- skipping file deletion deleting ./public_html/jra/sig.pcx deleting ./public_html/jra/sig.pcl deleting ./public_html/jra/sig-u.pcx deleting ./public_html/jra/sig-u.pcl [...] deleting ./.dvipsrc deleting ./.bashrc deleting ./.bash_history . sent 15 bytes received 71 bytes 19.11 bytes/sec total size is 0 speedup is 0.00 (DRY RUN) rsync error: some files could not be transferred (code 23) at main.c(1538) [generator=3.0.0] bkw@nj2:~> -- Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
I'm running a standalone rsyncd. It only runs once. Running rsync via rsh or ssh is different. (or inetd/xinetd) That might explain the differences I encounter
It looks like my previous example had a part that had no effect. According to the man-page, the remote charset part of the --iconv=local,remote has no effect Now I am beginning to see that but it wasn't so obvious with the rather indirect man page explanation.
When NOT using the daemon, going to the same machine, which now has no charset in it's config any more btw: Notice the single ":" this time and different paths... I see. Maybe just for fun I might experiment with that later
Mucking around with this is dangerous too... I almost deleted my entire I learned that too. That is why I made a second manual backup somewhere else in case I trashed the rsync backup while experimenting. Thanks again for your help.
Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Damon Register wrote:
I am having a lot of trouble trying to get a SuSE pc to connect to my samba server in such a way that rsync works. Here is the setup. I have a Samba server running on Solaris 10. It has been running for 2 years and the Windows pc connects just fine and I have even used cygwin rsync in the past to backup up to that mapped drive
Have you considered using rsyncd (the rsync daemon) on the host server? This is largely FS independent, you do not need mount anything, and it is not that hard to set up. - -- ============================================================================== I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone. Bjarne Stroustrup ============================================================================== -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iEYEARECAAYFAki7/oMACgkQasN0sSnLmgJCRgCeKydp0sd8E9LzapRy0SVwgdvu CIAAn207oDzQvc4mNNUSUClHrP8TX7WS =IADQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Have you considered using rsyncd (the rsync daemon) on the host server? This is largely FS independent, you do not need mount anything, and it is not that hard to set up. With all the trouble I was having, I decided to give rsyncd a try. It turned out that it wasn't so hard to get running even on Solaris. The only trouble I had there is that I forgot to add to /etc/services but figured it out quickly. I had to update my rsnycd because I was having trouble but after some work with different rsync options on
G T Smith wrote: the client, I finally got it to work. Now there seems to be only one flaw. After the rsync operation I ran kdiff to see how the local compared with the remote samba mounted where the rsyncd had stored it. All but one file were exactly the same but that one file had an accented a that didn't get copied the same by the remote rsyncd. Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Damon Register wrote:
Have you considered using rsyncd (the rsync daemon) on the host server? This is largely FS independent, you do not need mount anything, and it is not that hard to set up. With all the trouble I was having, I decided to give rsyncd a try. It turned out that it wasn't so hard to get running even on Solaris. The only trouble I had there is that I forgot to add to /etc/services but figured it out quickly. I had to update my rsnycd because I was having trouble but after some work with different rsync options on
G T Smith wrote: the client, I finally got it to work.
Now there seems to be only one flaw. After the rsync operation I ran kdiff to see how the local compared with the remote samba mounted where the rsyncd had stored it. All but one file were exactly the same but that one file had an accented a that didn't get copied the same by the remote rsyncd.
Damon Register
Odd, and worrying, I assume you have checked the consistency of the result i.e whether this happens to the same file in the same place, is detectable regularly and random locations, or is a one off. The first indicates a possible character translation issue, the second could be related to your initial problem and indicates data corruption of some sort is occurring at unacceptable level and the cause may not be rsync but hardware related (and is potentially serious), the third means Murphy is on overtime. - -- ============================================================================== I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone. Bjarne Stroustrup ============================================================================== -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iEYEARECAAYFAki9MssACgkQasN0sSnLmgLvTwCg6e/g/ywxZ3hbEOqgb4AEVaNE k00AoIEL5Kcdrhp7DfC/m22B9JQbIT8V =4egQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Odd, and worrying, I assume you have checked the consistency of the result i.e whether this happens to the same file in the same place, is I think it is quite repeatable. As far as I know, that one file was
G T Smith wrote: the only one with non-ascii letters in the name. Just for fun I tried making a test here at work where we have a solaris hosted samba server and a drive mapped to it on our PCs. I created a plain text file with an accented a in the name. I copied it to a folder on the mapped drive. I logged into the Solaris system and did ls -l on that folder. The accented a was mangled. Stranger yet, on that Solaris system I ran the nautilus file manager where it showed the same file with the correct accent on the a.
indicates a possible character translation issue, the second could be I am fairly certain that is the case. I thought I remembered a recent discussion about the character translation issue in this list but I can't find that thread. Does anyone remember that thread? Does anyone know how to get the remote rsyncd daemon to adjust the character translation?
Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
----- Original Message -----
From: "Damon Register"
Odd, and worrying, I assume you have checked the consistency of the result i.e whether this happens to the same file in the same place, is I think it is quite repeatable. As far as I know, that one file was
G T Smith wrote: the only one with non-ascii letters in the name. Just for fun I tried making a test here at work where we have a solaris hosted samba server and a drive mapped to it on our PCs. I created a plain text file with an accented a in the name. I copied it to a folder on the mapped drive. I logged into the Solaris system and did ls -l on that folder. The accented a was mangled. Stranger yet, on that Solaris system I ran the nautilus file manager where it showed the same file with the correct accent on the a.
That's not strange at all. That's just the natural consequence of using different character sets in different interfaces to display the same string of bytes. What if any measures have you taken to ensure, or at last assure, that all things which touch the file are either all using the same character set and encoding, or failing that, that all parts are accurately and fully configured to know what character sets and encodings all other parts are using so that they may correctly translate in those cases where they might do so? If you have no idea, then you will regularly see what _looks_ like errors like this, unless you simply avoid using any characters outside of the traditional low-ascii alpha-numeric values where most character sets happen to use the same glyphs for that subset of ascii byte values. If you speak the word "see" into a tape recorder, And then play it back to a blindfolded, english-speaking, optometrist, they probably hear the word "see". Play the same tape back to a blindfolded, english-speaking, sailor, and they probably hear "sea". ... spanish-speaker, probably hears "si". ... elglish speaking software developer probably hears "C". etc etc etc... So it is with computers and character sets, character encoding schemes, fonts. There are some mechanisms for for putting data in context, so that when one program "speaks" a string of bytes, it also indicates what "language" those bytes are intended to be interpreted with. But those mechanisms are mostly recent developments and not fully implimented and not fully backwards compatible with older systems which had no such ability. So in windows you create a filename with an lower-case-a-acute, on an windows pc, in the USA, for an english speaking user, the file-save dialog UI is _probably_ using utf16, utf8, or codepage 1252. In all of those cases the a-acute just _happens_ to be the same integer value 0xe1 (0x00e1 for utf16). So, at last on the windows pc local disk the filenme probably has the byte 0xe1 in it. Now you copy that file to the samba share. Lets assume for the moment that samba does no magic translation of the filename at save-time, and so it just copies the e1 without caring what glyph might be associated with that value. Now you log in from the console or telnet in from a terminal emulator that is configured to accurately mimick the console. The console may or may not be configured to load a software font over the the vga hardware. In the USA, if the console is loading a font, it's probaly latin-1, aka iso-8859-1. I guess this was a bad example haracter, beacause in that charactr set too, once again 0xe1 just happens to be lower-case-a-acute. But, the character set that is built-in to most vga hardware is not any of the character sets mentioned so far, it's codepage 437. The glyph associated with the integer value 0xe1 in the codepage 437 character set is the alpha symbol. If the console is configured not to do any character translating or software font loading and is thus using codepage 437, and if samba is not doing any translating, then when you ls that file name at the console, instead of lower-case-a-acute, you'll see an alpha. But, almost any program in X on the same box is probably using latin-1. So, in nautilus file manager, you'll probably see the lower-case-a-acute. The character hasn't changed and isnt "wrong" any where. Merely if you are going to use _any_ characters outside of the least common denominator set of plain lower-ascii (aka 7-bit us-ascii) then you need to understand all about character sets and display contexts. If you don't or can't do that, then stick to the plain characters. Most character sets have the same glyphs for that subset of byte values. (0x20 to 0x7e or decimal 31 to 126) (and less tangibly, the same sorting rules for those characters) In fact, regardless of how much you might learn about this, and how well & completely you might configure all the software on all the devices in your organization, so that maybe they all speak utf8, so everything is consistent everywhere, you should probaly _still_ avoid special characters because you can never know what character set someone else is using who may need to handle files (or even merely their names) created within your organization. Brian K. White brian@aljex.com http://www.myspace.com/KEYofR +++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++. filePro BBx Linux SCO FreeBSD #callahans Satriani Filk! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Brian K. White wrote:
That's not strange at all. That's just the natural consequence of using different character sets in different interfaces to display the same string of bytes. Thanks for the detailed explanation. It makes a little more sense to me now.
What if any measures have you taken to ensure, or at last assure, that all things which touch the file are either all using the same character set and encoding, or failing that, that all parts are accurately and fully configured to know what character sets and encodings all other parts are using so that they may correctly translate in those cases where they might do so? Probably not enough yet :-( I am still trying to learn just what I need to do and how to do it
If you have no idea, then you will regularly see what _looks_ like errors like this, unless you simply avoid using any characters outside of the traditional low-ascii alpha-numeric values where most character sets happen to use the same glyphs for that subset of ascii byte values. That might be unavoidable since my wife is from Peru :-)
If you speak the word "see" into a tape recorder, And then play it back to a blindfolded, english-speaking, optometrist, they probably hear the word "see". Play the same tape back to a blindfolded, english-speaking, sailor, and they probably hear "sea". ... spanish-speaker, probably hears "si". ... elglish speaking software developer probably hears "C". sounds like something we heard in our cultural diversity class
If the console is configured not to do any character translating or software font loading and is thus using codepage 437, and if samba is not doing any translating, then when you ls that file name at the console, instead of lower-case-a-acute, you'll see an alpha. thanks for the good detail. I suspected it was a translation issue but didn't understand enough to analyze
Since I really only care about what the client PCs (windows and SuSE) see and use, what the Solaris server sees isn't so important to me. Perhaps if I make sure that the rsync operation and any later retrieval through the samba mount use the same character translation or maybe none at all then I might be ok. I finally found the new (since rsync 3) iconv option, I may be able to adjust what rsync does to match what samba does. If I remember correctly my fstab entry for the samba mount on the client specifies ISO-8859-15 so maybe I should do the same with rsync. I will test tonight Damon Register -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (4)
-
Brian K. White -
Damon Register -
Damon Register -
G T Smith