[opensuse] rsync question - NTFS - timestamps - possibly too esoteric
All, This may be off-topic in that I am creating files in opensuse, then running rsync in a cygwin environment to make a second copy. I'm wondering if this is a timestamp issue between how openSUSE and cygwin/windows do things. == details I have an external USB drive formatted NTFS with a directory full of roughly 500 GB worth of 1.5 GB files. For that drive: I wiped it in openSUSE (via dc3dd wipe=/dev/sdb) I partitioned it interactively with fdisk /dev/sdb I created the NTFS filesystem in openSUSE mkfs.ntfs -Q /dev/sdb1 I created the directory and the 500GB of files via openSUSE 13.1 I started rsync on it a couple days ago to make a second copy on a different disk. I don't remember if I ran rsync the first time from cygwin or openSUSE, but that first attempt got interrupted in the middle of the copy. I just re-ran rsync from inside cygwin: rsync -r /cygdrive/f/project_dir/data_dir /cygdrive/g/project_dir I'm watching what is happening with both "file explorer" and the "windows task manager - disk resource monitor". I'm seeing numerous of the 1.5GB data files being re-transferred, but not all. I have no reason to think they were faulty and rsync should just be checking the size and datestamp. It looks like a rsync bug to me. I killed rsync as soon as I saw this and have now restarted it with strace to capture the system calls. That straced version of the rsync invocation is still ongoing. The strace output is almost 5GB, so it will be a huge file to scan through at the end of the rsync process. Anyway, my suspicion is it may be an incompatibility between how openSUSE and cygwin store datestamps. Thus a date recorded by openSUSE may not exactly match a date recorded via cygwin and rsync thinks it needs to re-transfer the file. Alternatively, it could be an issue with NTFS. I haven't done a lot of rsync testing with NTFS as both the source and destination. Thanks Greg -- Greg Freemyer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-06-26 18:03, Greg Freemyer wrote:
All,
This may be off-topic in that I am creating files in opensuse, then running rsync in a cygwin environment to make a second copy.
I'm wondering if this is a timestamp issue between how openSUSE and cygwin/windows do things.
Yes, known thing. There is a two second grainity (? no, granularity) on ntfs partitions. rsync has a setting to account for it. I don't remember the option name, should be in the manual. I have not read your entire post - did I guess right? ;-) -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Thu, Jun 26, 2014 at 1:43 PM, Carlos E. R. <robin.listas@telefonica.net> wrote:
On 2014-06-26 18:03, Greg Freemyer wrote:
All,
This may be off-topic in that I am creating files in opensuse, then running rsync in a cygwin environment to make a second copy.
I'm wondering if this is a timestamp issue between how openSUSE and cygwin/windows do things.
Yes, known thing. There is a two second grainity (? no, granularity) on ntfs partitions. rsync has a setting to account for it. I don't remember the option name, should be in the manual.
I have not read your entire post - did I guess right? ;-)
You at least sent me looking for generic NTFS / Windows issues with rsync and I found several. You caused me to restart my rsync :( Turns out I was running a 32-bit version of cygwin/rsync so it was taking over a minute per file to copy. The new command is using 64-bit rsync (from cygwin)0 : rsync -r --modify-window=5 --progress --stats .... I have my files numbered, so I can see which ones are transferring. With this command it is doing: file01,02, 35, 38, 39, 40, 41, 42 43, ,,,,, It is taking about 30 seconds per file since they are 1.5 GB. I don't know why is skipped the files from 03 to 34 and 36/37, My guess at the moment is this is a Windows/Cygwin bug and if I used rsync exclusively from openSUSE I would be fine, even if I stick to NTFS on the external. (I'm off to test that theory). Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Jun 26, 2014 at 1:43 PM, Carlos E. R. <robin.listas@telefonica.net> wrote:
On 2014-06-26 18:03, Greg Freemyer wrote:
All,
This may be off-topic in that I am creating files in opensuse, then running rsync in a cygwin environment to make a second copy.
I'm wondering if this is a timestamp issue between how openSUSE and cygwin/windows do things.
Yes, known thing. There is a two second grainity (? no, granularity) on ntfs partitions. rsync has a setting to account for it. I don't remember the option name, should be in the manual.
I have not read your entire post - did I guess right? ;-)
I forgot to say: NTFS stores timestamps with 100 nsec accuracy, so the 2-second issue must be a Windows API issue, not an underlying NTFS format issue. (I did find discussion of the 2-second bug, but I'm not sure that is my problem.) http://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).as... If rsync from openSUSE has this problem too, then it really is a bug, but likely an upstream rsync or NTFS-3g bug. For now I've only seen it in cygwin/windows world. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-06-26 20:33, Greg Freemyer wrote:
On Thu, Jun 26, 2014 at 1:43 PM, Carlos E. R.
I forgot to say:
NTFS stores timestamps with 100 nsec accuracy, so the 2-second issue must be a Windows API issue, not an underlying NTFS format issue. (I did find discussion of the 2-second bug, but I'm not sure that is my problem.)
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).as...
If rsync from openSUSE has this problem too, then it really is a bug, but likely an upstream rsync or NTFS-3g bug. For now I've only seen it in cygwin/windows world.
AFAIK it is only an issue when dealing with mixed environments. Rsync with Linux uses... I don't know, milliseconds perhaps. But when dealing with fat, ntfs, or samba, it sees the timestamps with two second granularity. I did hit this issue, and I had to use a certain switch in rsync to patch it. Ah, found it: --modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Win- dows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second). Ah, maybe ntfs is not affected. But perhaps the windows api is, for compatibility, or depending on the particular api call that the windows rsync version uses. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Thu, Jun 26, 2014 at 3:10 PM, Carlos E. R. <robin.listas@telefonica.net> wrote:
On 2014-06-26 20:33, Greg Freemyer wrote:
On Thu, Jun 26, 2014 at 1:43 PM, Carlos E. R.
I forgot to say:
NTFS stores timestamps with 100 nsec accuracy, so the 2-second issue must be a Windows API issue, not an underlying NTFS format issue. (I did find discussion of the 2-second bug, but I'm not sure that is my problem.)
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).as...
If rsync from openSUSE has this problem too, then it really is a bug, but likely an upstream rsync or NTFS-3g bug. For now I've only seen it in cygwin/windows world.
AFAIK it is only an issue when dealing with mixed environments. Rsync with Linux uses... I don't know, milliseconds perhaps. But when dealing with fat, ntfs, or samba, it sees the timestamps with two second granularity. I did hit this issue, and I had to use a certain switch in rsync to patch it.
Ah, found it:
--modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Win- dows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second).
Ah, maybe ntfs is not affected. But perhaps the windows api is, for compatibility, or depending on the particular api call that the windows rsync version uses.
I'm not very happy at the moment and this is all pure openSUSE 13.1 I can't even get proper results with rsync from ext4 to ext4. I must be getting the basics wrong. Shouldn't this work: - create a directory on a ext4 filesystem. - but 3 1.5 GB files in it. - rsync -r --stats --progress <source_dir> /tmp/junk_dir - wait for it to complete - rerun the same rsync command My expectation is the last rsync command should be instantaneous. It is not, the 3 1.5GB files are being copied again. Thinking it might be the -r, I tried again without it. (rsync --stats --progress <source_dir>/* /tmp/junk_dir) same issue. I'm having a hard time believing rsync is broken, so what am I doing wrong? Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Greg Freemyer <greg.freemyer@gmail.com> [2014-06-26 16:56]:
I'm not very happy at the moment and this is all pure openSUSE 13.1
I can't even get proper results with rsync from ext4 to ext4. I must be getting the basics wrong.
Shouldn't this work:
- create a directory on a ext4 filesystem. - but 3 1.5 GB files in it. - rsync -r --stats --progress <source_dir> /tmp/junk_dir - wait for it to complete - rerun the same rsync command
My expectation is the last rsync command should be instantaneous. It is not, the 3 1.5GB files are being copied again.
Thinking it might be the -r, I tried again without it. (rsync --stats --progress <source_dir>/* /tmp/junk_dir)
same issue.
I'm having a hard time believing rsync is broken, so what am I doing wrong?
Greg
Try using the "-a" ("--archive") option to rsync instead of the "-r" option (which "-a" implies). "-a" turns on quite a few other things, including perms, timestamps, and owner and group matching. That's what I use most frequently for similar situations. --Phil -- Philip Amadeo Saeli psaeli@zorodyne.com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On June 26, 2014 7:46:58 PM EDT, Philip Amadeo Saeli <psaeli@zorodyne.com> wrote:
* Greg Freemyer <greg.freemyer@gmail.com> [2014-06-26 16:56]:
I'm not very happy at the moment and this is all pure openSUSE 13.1
I can't even get proper results with rsync from ext4 to ext4. I must be getting the basics wrong.
Shouldn't this work:
- create a directory on a ext4 filesystem. - but 3 1.5 GB files in it. - rsync -r --stats --progress <source_dir> /tmp/junk_dir - wait for it to complete - rerun the same rsync command
My expectation is the last rsync command should be instantaneous. It is not, the 3 1.5GB files are being copied again.
Thinking it might be the -r, I tried again without it. (rsync --stats --progress <source_dir>/* /tmp/junk_dir)
same issue.
I'm having a hard time believing rsync is broken, so what am I doing wrong?
Greg
Try using the "-a" ("--archive") option to rsync instead of the "-r" option (which "-a" implies). "-a" turns on quite a few other things, including perms, timestamps, and owner and group matching. That's what I use most frequently for similar situations.
--Phil
-a Seems to be working as I expect with opensuse for both ext4 and NTFS. Shouldn't -r work too? Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, 26 Jun 2014 21:08:46 -0400 Greg Freemyer wrote: 8< snipped >8
-a Seems to be working as I expect with opensuse for both ext4 and NTFS.
Shouldn't -r work too?
Greg
Since '-a' preserves modification times (among other things,) on subsequent passes, rsync will skip copying items where the modification times match. With '-r', copies written at the target receive new (contemporaneous) modification times. rsync can only guarantee reliable copies by overwriting items when the modification times don't match. If you use the '-a' option, it will only copy those items that have actually changed (or appeared) since the last pass. hth & regards, Carl -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2014-06-26 20:33, Greg Freemyer wrote:
On Thu, Jun 26, 2014 at 1:43 PM, Carlos E. R.
I forgot to say:
NTFS stores timestamps with 100 nsec accuracy, so the 2-second issue must be a Windows API issue, not an underlying NTFS format issue. (I did find discussion of the 2-second bug, but I'm not sure that is my problem.)
http://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).as...
If rsync from openSUSE has this problem too, then it really is a bug, but likely an upstream rsync or NTFS-3g bug. For now I've only seen it in cygwin/windows world.
AFAIK it is only an issue when dealing with mixed environments. Rsync with Linux uses... I don't know, milliseconds perhaps. But when dealing with fat, ntfs, or samba, it sees the timestamps with two second granularity. I did hit this issue, and I had to use a certain switch in rsync to patch it.
Ah, found it:
--modify-window When comparing two timestamps, rsync treats the timestamps as being equal if they differ by no more than the modify-window value. This is normally 0 (for an exact match), but you may find it useful to set this to a larger value in some situations. In particular, when transferring to or from an MS Win- dows FAT filesystem (which represents times with a 2-second resolution), --modify-window=1 is useful (allowing times to differ by up to 1 second).
Ah, maybe ntfs is not affected. But perhaps the windows api is, for compatibility, or depending on the particular api call that the windows rsync version uses.
Remember when Windows NT servers had to be rebooted after 48 days or they would crash... here's why: Unix time: linux-86ja:~> units You have: 2147483648 sec You want: year * 68.051104 / 0.014694839 You have: 4294967296 msec You want: day * 49.71027 / 0.020116568 You have: 2147483648 sec You want: day * 24855.135 You have: ^C Note that in Windows NT kept time in milliseconds with a 32-bit unsigned int, whereas Unix kept time in seconds using 32-bit signed int (which effectively makes for a 31-bit signed int). Maximum Windows NT uptime: 49 days Maximum Unix uptime: 68 YEARS = 24855 days. It's obvious which was the better design decision. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/26/2014 07:44 PM, Dirk Gently wrote:
Note that in Windows NT kept time in milliseconds with a 32-bit unsigned int, whereas Unix kept time in seconds using 32-bit signed int (which effectively makes for a 31-bit signed int).
Maximum Windows NT uptime: 49 days Maximum Unix uptime: 68 YEARS = 24855 days.
It's obvious which was the better design decision.
And now we have Linux running on 64 bit machines. Just let me see what my calculator says. 2^64 - ERROR No, really, its somewhere around 2x10^19. Compare at http://en.wikipedia.org/wiki/Future_of_an_expanding_universe#Timeline However as far as most loop counters go an 8-bit machine is adequate :-) -- Good teaching is one-fourth preparation and three-fourths theatre. Gail Godwin -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/27/2014 09:41 AM, Anton Aylward wrote:
And now we have Linux running on 64 bit machines. Just let me see what my calculator says.
2^64 - ERROR
Hmmm... KCalc on my computer returns 1.84467440737e+19 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 06/27/2014 10:05 AM, James Knott wrote:
On 06/27/2014 09:41 AM, Anton Aylward wrote:
And now we have Linux running on 64 bit machines. Just let me see what my calculator says.
2^64 - ERROR Hmmm... KCalc on my computer returns 1.84467440737e+19
As does "Realcalc" on my Android phone. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (7)
-
Anton Aylward -
Carl Hartung -
Carlos E. R. -
Dirk Gently -
Greg Freemyer -
James Knott -
Philip Amadeo Saeli