[opensuse] Is rsync broken?
# blkid /dev/sda30 # mkfs.ext3 -I128 -b1024 -L ostwEXT3k3 /dev/sda30 # tune2fs -c0 -i0 /dev/sda30 # mount -t ext3 -o noatime /dev/sda30 /mnt # cd /mnt # mount /dev/sda10 # df . (99% free, 1% used) # rsync -av --stats --progress --exclude 'lost+found' /disks/s133/ . # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda8 7281115 5048620 1859709 74% / (13.1) /dev/sda10 5655815 4759270 605692 89% /disks/s133 (TW on ext4) /dev/sda30 5655815 5281956 87101 99% /mnt Given this sequence of events, what besides broken rsync could be responsible for source used being so much less than target used? This is not the first time I've seen this happen. Last time, used reached 100% and rsync aborted. I tried again, except booted to 42.1: Before: # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda19 5655815 2928686 2436276 55% / (42.1 on ext4) /dev/sda10 5655815 4759270 605692 89% /disks/s133 (TW on ext4) /dev/sda30 5655815 3727 5365330 1% /mnt # rsync -av --stats --progress --exclude 'lost+found' /disks/s133/ . # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda19 5655815 2928686 2436276 55% / (42.1 on ext4) /dev/sda10 5655815 4759270 605692 89% /disks/s133 (TW on ext4) /dev/sda30 5655815 5289173 87084 99% /mnt :-( -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/31/2016 10:12 AM, Felix Miata wrote:
what besides broken rsync could be responsible for source used being so much less than target used?
sparse files? Check with "du -shxc ..." vs. "du --app -shxc ...". Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-03-31 09:54, Bernhard Voelker wrote:
On 03/31/2016 10:12 AM, Felix Miata wrote:
what besides broken rsync could be responsible for source used being so much less than target used?
sparse files? Check with "du -shxc ..." vs. "du --app -shxc ...".
I don't remember what rsync's defaults are but are delinked hard links another possibility? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Content-ID: <alpine.LSU.2.20.1603311148540.6209@Grypbagne.inyvabe> On Thursday, 2016-03-31 at 10:45 +0100, Dave Howorth wrote: El 2016-03-31 a las 10:45 +0100, Dave Howorth escribió:
sparse files? Check with "du -shxc ..." vs. "du --app -shxc ...".
I don't remember what rsync's defaults are but are delinked hard links another possibility?
Other filesystems mounted on the source, perhaps. Like /proc, /sys, /dev... - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlb88pAACgkQtTMYHG2NR9VxMwCaArKirJ5t4LHxri7gGEewwMCr gtMAn0hdR++mRnewoqvRV/qwWlvugJ5U =7ol2 -----END PGP SIGNATURE-----
On 03/31/2016 11:45 AM, Dave Howorth wrote:
On 2016-03-31 09:54, Bernhard Voelker wrote:
On 03/31/2016 10:12 AM, Felix Miata wrote:
what besides broken rsync could be responsible for source used being so much less than target used?
sparse files? Check with "du -shxc ..." vs. "du --app -shxc ...".
I don't remember what rsync's defaults are but are delinked hard links another possibility?
Good point. Therefore maybe 'rsync -SHAXaxi ' would be better in the OPs use case. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Bernhard Voelker composed on 2016-03-31 05:53 (UTC-0400):
Dave Howorth wrote:
Bernhard Voelker wrote:
Felix Miata wrote:
what besides broken rsync could be responsible for source used being so much less than target used?
sparse files? Check with "du -shxc ..." vs. "du --app -shxc ...".
I don't remember what rsync's defaults are but are delinked hard links another possibility?
Good point. Therefore maybe 'rsync -SHAXaxi ' would be better in the OPs use case.
Very interesting. Looks like H was a missing key ingredient, but using Bernhard's option set inverted the differential: I tried once again booted to 42.1: Before: # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda19 5655815 2928686 2436276 55% / (42.1 on ext4) /dev/sda10 5655815 4759270 605692 89% /disks/s133 (TW on ext4) /dev/sda30 5655815 3727 5365330 1% /mnt # rsync -SHAXaxi --stats --progress --exclude 'lost+found' /disks/s133/ . # df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda19 5655815 2928686 2436276 55% / (42.1 on ext4) /dev/sda10 5655815 4759270 605692 89% /disks/s133 (TW on ext4) /dev/sda30 5655815 4231922 1137135 79% /mnt This looks like getting less than the whole source copied. :-p Hard links are definitely part of the source, glibc-locale if not more. Last I remember this shows up here (still unfixed over 5 years later): https://bugzilla.opensuse.org/show_bug.cgi?id=720150#c7 -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 03/31/2016 12:36 PM, Felix Miata wrote:
Very interesting. Looks like H was a missing key ingredient, [...]
Thanks for letting us know. BTW: as it's the initial copy - any reason you don't dd(1) the file system? Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Bernhard Voelker composed on 2016-03-31 12:43 (UTC+0200):
Felix Miata wrote:
Very interesting. Looks like H was a missing key ingredient, [...]
Thanks for letting us know. BTW: as it's the initial copy - any reason you don't dd(1) the file system?
1-I thought I knew rsync a lot better than dd. dd I rarely use. When I want a clone, it's usually a complete partition. For that I use my partitioner. 2-The whole point of the copy was to see if using EXT3 instead of EXT4 would make the problem go away (3+ minutes boot delays)[1]. Initial indication is it did indeed solve the problem, but with identical size target filesystem short of source by 10%, I have to wonder about other consequences of the rsync. [1] https://lists.opensuse.org/opensuse-kernel/2016-03/msg00024.html -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2016-03-31 11:57, Felix Miata wrote:
2-The whole point of the copy was to see if using EXT3 instead of EXT4 would make the problem go away (3+ minutes boot delays)[1]. Initial indication is it did indeed solve the problem, but with identical size target filesystem short of source by 10%, I have to wonder about other consequences of the rsync.
[1] https://lists.opensuse.org/opensuse-kernel/2016-03/msg00024.html
Have you done a quick compare of an ls -lR of each filesystem to see if they have the same contents? There's no particular reason why two different filesystem types should take the same amount of space to store the same information, even if their names are quite similar :) Cheers, Dave -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Dave Howorth composed on 2016-03-31 12:11 (UTC+0100):
Felix Miata wrote:
2-The whole point of the copy was to see if using EXT3 instead of EXT4 would make the problem go away (3+ minutes boot delays)[1]. Initial indication is it did indeed solve the problem, but with identical size target filesystem short of source by 10%, I have to wonder about other consequences of the rsync.
[1] https://lists.opensuse.org/opensuse-kernel/2016-03/msg00024.html
Have you done a quick compare of an ls -lR of each filesystem to see if they have the same contents?
Hadn't thought about it. Thx. :-)
There's no particular reason why two different filesystem types should take the same amount of space to store the same information, even if their names are quite similar :)
I tried same source, using 42.1 instead of 13.1, to two different targets thus: rsync -rlptgoDAHX --exclude='lost+found' /disks/stw/ /mnt First try, when done (including fstab backup to backup.07, then edit): Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda19 5655815 2934661 2430301 55% / /dev/sda23 5655815 4126642 1238319 77% /disks/stw EXT4 TW source /dev/sda29 5655815 4165031 1204026 78% /mnt EXT3 target # ls -lR /disks/stw | wc -l 199013 # ls -lR /mnt | wc -l 199014 Second try, before starting: Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda19 5655815 2934662 2430300 55% / /dev/sda23 5655815 4126642 1238319 77% /disks/stw EXT4 TW source /dev/sda30 5655815 3598 5361363 1% /mnt EXT4 target Second try, when done (including fstab backup to backup.07, then edit): Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda19 5655815 2934663 2430299 55% / /dev/sda23 5655815 4126642 1238319 77% /disks/stw EXT4 TW source /dev/sda30 5655815 4126449 1238512 77% /mnt EXT4 target # ls -lR /mnt | wc -l 199014 Both sda29 and sda30 are bootable as necessarily modified, so I guess it's not broken. :-) -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Bernhard Voelker -
Carlos E. R. -
Dave Howorth -
Felix Miata