[opensuse] 12.3 + ntfs + ext4 + USB3 + different copying speeds
I bought a Seagate 2TB USB3 external HDD today and formatted 75% of it with ext4 file system and 25% with ntfs - using latest version of Gparted. My two internal 1TB Seagates are SATA 3 and formatted with ext4 except for one partition (ntfs) which has XP installed, but which is rarely used (it's mainly there to bring me back to earth and to remind me how bloody aweful Windows is). I use mc (Midnight Commander) 99.99% of the time to manage my files. After I formatted the external HDD (as described above) I went to copy files from my main system to the new external HDD - which is to be used for backups. What I found - and which is what I am now asking about - is that when, using mc, that when copying files to the *ext4* formatted partition the files where being copied across at ~80MB/s. The files copied over were a mixture of ntfs and ext4 files (which I had copied from a collapsing 32-bit system to my current *Linux* system last year when I replaced the old computer). However, when I went to copy files to the *ntfs* formatted part of the new external HDD, the best transfer rate I got was ~28MB/s - which looks very much like what I used to get when using USB2! So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! What system am a I using? Look at my signature line; but I understand that some brain-dead mail clients drop the signature line so for these people with brain-dead mailers the sig. line is: Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 Any suggestions, please, as to why there is such a difference in the transfer rates between the 2 file systems, and how can this be overcome? BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin <blchupin@iinet.net.au> wrote:
I bought a Seagate 2TB USB3 external HDD today and formatted 75% of it with ext4 file system and 25% with ntfs - using latest version of Gparted.
My two internal 1TB Seagates are SATA 3 and formatted with ext4 except for one partition (ntfs) which has XP installed, but which is rarely used (it's mainly there to bring me back to earth and to remind me how bloody aweful Windows is).
I use mc (Midnight Commander) 99.99% of the time to manage my files.
After I formatted the external HDD (as described above) I went to copy files from my main system to the new external HDD - which is to be used
for backups.
What I found - and which is what I am now asking about - is that when, using mc, that when copying files to the *ext4* formatted partition the
files where being copied across at ~80MB/s. The files copied over were a mixture of ntfs and ext4 files (which I had copied from a collapsing 32-bit system to my current *Linux* system last year when I replaced the old computer).
However, when I went to copy files to the *ntfs* formatted part of the new external HDD, the best transfer rate I got was ~28MB/s - which looks very much like what I used to get when using USB2!
So, why the *big* difference in transfer rates between copying files to
a partition formatted in *ext4* and one formatted in *ntfs*?!
What system am a I using? Look at my signature line; but I understand that some brain-dead mail clients drop the signature line so for these people with brain-dead mailers the sig. line is:
Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1
Any suggestions, please, as to why there is such a difference in the transfer rates between the 2 file systems, and how can this be overcome?
BC Basil,
Ntfs-3g is a fuse based filesystem. That means the filesystem code actually is a userspace app, not a kernel module like most linux filesystems. I think that makes it inherently slower. Anyway, before you assume something is wrong with your system you should research what performance to expect from ntfs-3g. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 27/03/13 00:13, Greg Freemyer wrote:
Basil Chupin <blchupin@iinet.net.au> wrote:
I bought a Seagate 2TB USB3 external HDD today and formatted 75% of it with ext4 file system and 25% with ntfs - using latest version of Gparted.
My two internal 1TB Seagates are SATA 3 and formatted with ext4 except for one partition (ntfs) which has XP installed, but which is rarely used (it's mainly there to bring me back to earth and to remind me how bloody aweful Windows is).
I use mc (Midnight Commander) 99.99% of the time to manage my files.
After I formatted the external HDD (as described above) I went to copy files from my main system to the new external HDD - which is to be used
for backups.
What I found - and which is what I am now asking about - is that when, using mc, that when copying files to the *ext4* formatted partition the
files where being copied across at ~80MB/s. The files copied over were a mixture of ntfs and ext4 files (which I had copied from a collapsing 32-bit system to my current *Linux* system last year when I replaced the old computer).
However, when I went to copy files to the *ntfs* formatted part of the new external HDD, the best transfer rate I got was ~28MB/s - which looks very much like what I used to get when using USB2!
So, why the *big* difference in transfer rates between copying files to
a partition formatted in *ext4* and one formatted in *ntfs*?!
What system am a I using? Look at my signature line; but I understand that some brain-dead mail clients drop the signature line so for these people with brain-dead mailers the sig. line is:
Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1
Any suggestions, please, as to why there is such a difference in the transfer rates between the 2 file systems, and how can this be overcome?
BC Basil,
Ntfs-3g is a fuse based filesystem.
Thanks, Greg, but it's over it's over my head :-( . What's a fuse based filesystem?
That means the filesystem code actually is a userspace app, not a kernel module like most linux filesystems.
And again, way over my head: what is a "userspace app"?
I think that makes it inherently slower. Anyway, before you assume something is wrong with your system you should research what performance to expect from ntfs-3g.
Greg
OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions? BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin <blchupin@iinet.net.au> wrote:
On 27/03/13 00:13, Greg Freemyer wrote:
Basil Chupin <blchupin@iinet.net.au> wrote:
I bought a Seagate 2TB USB3 external HDD today and formatted 75% of
with ext4 file system and 25% with ntfs - using latest version of Gparted.
My two internal 1TB Seagates are SATA 3 and formatted with ext4 except for one partition (ntfs) which has XP installed, but which is rarely used (it's mainly there to bring me back to earth and to remind me how bloody aweful Windows is).
I use mc (Midnight Commander) 99.99% of the time to manage my files.
After I formatted the external HDD (as described above) I went to copy files from my main system to the new external HDD - which is to be used
for backups.
What I found - and which is what I am now asking about - is that when, using mc, that when copying files to the *ext4* formatted partition
files where being copied across at ~80MB/s. The files copied over
were
a mixture of ntfs and ext4 files (which I had copied from a collapsing 32-bit system to my current *Linux* system last year when I replaced the old computer).
However, when I went to copy files to the *ntfs* formatted part of
new external HDD, the best transfer rate I got was ~28MB/s - which looks very much like what I used to get when using USB2!
So, why the *big* difference in transfer rates between copying files to
a partition formatted in *ext4* and one formatted in *ntfs*?!
What system am a I using? Look at my signature line; but I understand that some brain-dead mail clients drop the signature line so for
it the the these
people with brain-dead mailers the sig. line is:
Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1
Any suggestions, please, as to why there is such a difference in the transfer rates between the 2 file systems, and how can this be overcome?
BC Basil,
Ntfs-3g is a fuse based filesystem.
Thanks, Greg, but it's over it's over my head :-( . What's a fuse based
filesystem?
That means the filesystem code actually is a userspace app, not a kernel module like most linux filesystems.
And again, way over my head: what is a "userspace app"?
Without getting complex, a linux based computer runs kernel code and userspace code. If you have data in the kernel and you need it in userspace, then typically you have to copy it from the kernel to userspace. That takes time. And if you need it back in the kernel, you have to copy it again (ie. More time). I don't know if fuse has any special accelerator techniques, but in general when people want top performance they write the code to be part of the kernel. The trouble is the kernel is very dynamic and code has to maintained much more actively. The ntfs-3g devs have chosen ease of development/support over performance.
I think that makes it inherently slower. Anyway, before you assume something is wrong with your system you should research what performance to expect from ntfs-3g.
Greg
OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions?
Assume your 28MB/sec performance is the fastest ntfs-3g can run on your computer. Usb-2 maxes out a little below that, so usb-2 was the bottleneck. Usb-3 is a lot faster, so now ntfs-3g is your bottleneck.
BC
If you find I'm wrong about ntfs-3g being the problem, please let us know. I too hit this bottleneck a lot. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions?
That would depend on where the bottle necks are. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions?
Thought you said it was SATA-3? Anyway, you might try mucking around in proc first you might try lowering /proc/sys/kernel/sched_latency_ns to 10,000,000 (enter it w/o the commas) .. What clock frequency are you running? Are you running tickless? Clock: HZ="$(zcat /proc/config.gz | grep "HZ_.*="|sed 's/CONFIG_HZ_// ; s/=y//')" echo $HZ What type of Pre-emption is your kernel running? zgrep PREEMPT /proc/config CONFIG_TREE_PREEMPT_RCU=y CONFIG_PREEMPT_RCU=y CONFIG_PREEMPT_NOTIFIERS=y # CONFIG_PREEMPT_NONE is not set # CONFIG_PREEMPT_VOLUNTARY is not set CONFIG_PREEMPT=y CONFIG_PREEMPT_COUNT=y # CONFIG_DEBUG_PREEMPT is not set # CONFIG_PREEMPT_TRACER is not set ---- I don't know if the delayacct tools will help or not. But you could try limiting a 'dd' transfer to 1 core and use 'time' in bash with the TIMEFORMAT set to: "%2Rsec %2Uusr %2Ssys (%P%% cpu)" i.e. in bash: export TIMEFORMAT="%2Rsec %2Uusr %2Ssys (%P%% cpu)" time dd if=/dev/zero of=testfile bs=1M count=16 oflag=direct for testing other way, try: time dd of=/dev/null if=testfile bs=1M count=16 iflag=direct You might find different bs's will work better To limit it to 1 core, use time schedtool -a 3 -e dd ..... (-a 3 runs on cpu 3) (-a 0x3 runs on cpu's 1+2 -- hex is a bit mask decimal is integer) You might play with some of the other SCHED policies if your kernel has them (see schedtool -h) delay accounting can be gotten with 'getdelays' from the delayacct-utils That might give you a few things to try... ;-) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 29/03/13 16:05, Linda Walsh wrote:
Basil Chupin wrote:
OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions?
Thought you said it was SATA-3?
I did. The internal HDDs are all SATA3. The external HDD is USB3.
Anyway, you might try mucking around in proc
first you might try lowering /proc/sys/kernel/sched_latency_ns to 10,000,000 (enter it w/o the commas) ..
That's now set to 24,000,000 so you suggest lowering it to 10m - OK, will try later.
What clock frequency are you running?
3.6MHz.
Are you running tickless?
My Master always uses this gunk on me so I have no ticks, thankfully. (Aside: "What's 'running tickless'?! Nevermind.....it's gotta be computer talk-type jargon...." :-) .) [pruned]
That might give you a few things to try... ;-)
That's it? That's all there is for me to try? :-D OK, I'll try this after I have worked out wot it all means :-D . But joking aside, thanks for your response - it's really appreciated. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
On 29/03/13 16:05, Linda Walsh wrote:
Basil Chupin wrote:
OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions?
Thought you said it was SATA-3?
I did. The internal HDDs are all SATA3. The external HDD is USB3.
Try measuring the "raw" disk READ times...(without any file system)... that should tell you how fast the disks can go w/o FS overhead. time dd if=/dev/sda of=/dev/null bs=16M count=256 --- that will do a raw read... Note if you read from a partition /dev/sda1, /dev/sda2... The higher numbered partitions will likely be slower (assuming (spinning hard disks) and not SolidStateDrives/SSD). Don't get your if (inputfile) and outputfile backwards on a read test... it can write garbage to your HD...(prolly know that, but I've made real bonehead mistakes before, so good to pay attention!)...
Anyway, you might try mucking around in proc
first you might try lowering /proc/sys/kernel/sched_latency_ns to 10,000,000 (enter it w/o the commas) ..
That's now set to 24,000,000 so you suggest lowering it to 10m - OK, will try later.
---- Yeah, that's what mine is--- I think that's normal ... you probably (am really guessing, but that likely means you have a 1000Hz clock. 24000,000 = 24ms = 24 ticks of a 1000Hz clock. Was just seeing/experimenting if a lower latency on switching might give you any better response in switching between kernel and user space (since you mentioned you were using the user-space NTFS driver). Everytime they switch there's more overhead... But, just a thought (or theory), that lowering the scheduling time-slice might help them switch back and forth more quickly -- (but I might be having you tweak the wrong value -- but don't worry -- it will reset when you reboot or just cat 24,000,000 into if you wanna put it back...
(Aside: "What's 'running tickless'?! Nevermind.....it's gotta be computer talk-type jargon...." :-) .)
Sorry, I never know how much the person I'm talking too knows, didn't mean to sound too techy....I can explain things to my mom and she's in her 80's!... so I have a bit of range....
[pruned]
That might give you a few things to try... ;-)
That's it? That's all there is for me to try? :-D
---- Well, if you really want more.... ;-) Just thought those might give you some solid numbers. I regularly use 'dd' for fastest-speed tests.
But joking aside, thanks for your response - it's really appreciated.
No prob -- understand the overload -- take it slow -- read up on what they do, it's the way to learn -- pokin around and exploring... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 22:41 -0700, Linda Walsh wrote:
Try measuring the "raw" disk READ times...(without any file system)... that should tell you how fast the disks can go w/o FS overhead.
time dd if=/dev/sda of=/dev/null bs=16M count=256 --- that will do a raw read...
I think you can do better with "hdparm -tT /dev/sda" :-)
Note if you read from a partition /dev/sda1, /dev/sda2... The higher numbered partitions will likely be slower (assuming (spinning hard disks) and not SolidStateDrives/SSD).
Not always true. Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3. I think I would have to rerun the test on other disks, to see if it repeats the same pattern. It will depend on how the tracks are actually translated. But it is a tedious test to run, unless you script it... - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFXaV0ACgkQtTMYHG2NR9XzeACeLwBNy5sKQYFGU/i2CMBxqxE4 TAkAn0gc8sv9nTsHRr5OZhiZYHACKqT9 =jOTY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-03-30 23:38 (GMT+0100) Carlos E. R. composed:
Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.
Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2013-03-30 at 19:26 -0400, Felix Miata wrote:
Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.
Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start.
No, it more probably is a result of the fixed rotational speed, the varying number of sectors depending where is the track, etc. We would need to have actual numbers to do the math. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFXe9MACgkQtTMYHG2NR9X4XACfWpSihRNg3MAeL0JPfQZTeM9J TrAAn3dnM/lBBEJ/akqNqqEPebi3+p8E =5aMx -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Felix Miata wrote:
On 2013-03-30 23:38 (GMT+0100) Carlos E. R. composed:
Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.
Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start. ==== Someone should go tell Tom's HW to update their graphs... Never seen one with a slower speed at the start...
That doesn't make any sense unless you have bad sectors there... or maybe they are putting spare sectors at the outside on some disks? I've never encountered those symptoms on any drives I've benched, but also I'm almost always running Enterprise drives for reliability. Same for your tests? Brands? I'm usually Hitachi, Fujitsu, Others are not ones I usually buy but have some...course now Hitachi is WD? ARG...wonder if they'll keep the facilities separate...never had good luck with WD's... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-03-30 21:29 (GMT-0700) Linda Walsh composed:
Felix Miata wrote:
On 2013-03-30 23:38 (GMT+0100) Carlos E. R. composed:
Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.
Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start. ==== Someone should go tell Tom's HW to update their graphs... Never seen one with a slower speed at the start...
#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the only such doc I saved. Note on its #2 the difference between front and middle is very small. All others I checked at http://fm.no-ip.com/PC/bench/Sysbench/ are fastest at front. Most of these were done shortly after receiving a new HD, prompting to test to see what speed to expect. ISTR seeing it happen more than once though. Otherwise I don't think I'd have remembered it ever happening.
That doesn't make any sense unless you have bad sectors there... or maybe they are putting spare sectors at the outside on some disks?
I've never encountered those symptoms on any drives I've benched, but also I'm almost always running Enterprise drives for reliability. Same for your tests? Brands? I'm usually Hitachi, Fujitsu, Others are not ones I usually buy but have some...course now Hitachi is WD? ARG...wonder if they'll keep the facilities separate...never had good luck with WD's...
Until the past year I had only ever bought WD twice, and one wasn't for any system of my own. Now the choices are just too limited not to consider WD. The last I bought was a 512 byte/sector HD155UI "refurb" 1.5T made in a Samsung factory several years ago but with Seagate on the label. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Felix Miata wrote:
Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.
Confirmed via observation here, closer maybe to 25%, but clearly slower
#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the only such doc I saved. Note on its #2 the difference between front and middle is very small. All others I checked at http://fm.no-ip.com/PC/bench/Sysbench/ are fastest at front. Most of these were done shortly after receiving a new HD, prompting to test to see what speed to expect. ISTR seeing it happen more than once though. Otherwise I don't think I'd have remembered it ever happening.
Just thought of something. Did you go from the beginning of the first partition or the beginning of the disk? Be default, or unless you've gone for DOS compatible partitioning, the first partition won't be track-aligned due to the boot sectors. Then usually, people (or software) opts for nearest track for end of 1st partition, so further partitions are aligned. It would make a slight difference, but the head would have to move over a track mid-read -- to read any track in the first partition, if such a situation were in place. I know when I partitioned with fdisk, I had to toggle a flag to make sure it was dos compat, otherwise, I got the use of 200 and some odd more sectors out of the first partition. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-03-31 13:14 (GMT-0700) Linda Walsh composed:
#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the
Just thought of something.
Did you go from the beginning of the first partition or the beginning of the disk?
I didn't do anything but start Sysbench. It just does what it does, which is to ignore partitions and filesystems. Also, that HD was made before manufacturers began configuring 4096 byte sectors on ordinary production units. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Felix Miata wrote:
On 2013-03-31 13:14 (GMT-0700) Linda Walsh composed:
#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the
Just thought of something.
Did you go from the beginning of the first partition or the beginning of the disk?
I didn't do anything but start Sysbench. It just does what it does, which is to ignore partitions and filesystems. Also, that HD was made before manufacturers began configuring 4096 byte sectors on ordinary production units.
Hmmm... I try sysbench and get: tools/sysbench/sysbench-0.4.12/sysbench> sysbench Missing required command argument. Usage: sysbench [general-options]... --test=<test-name> [test-options]... command General options: --num-threads=N number of threads to use [1] --max-requests=N limit for total number of requests [10000] --max-time=N limit for total execution time in seconds [0] --forced-shutdown=STRING amount of time to wait after --max-time before forcing shutdown [off] --thread-stack-size=SIZE size of stack per thread [32K] --init-rng=[on|off] initialize random number generator [off] --test=STRING test to run --debug=[on|off] print more debugging info [off] --validate=[on|off] perform validation checks where possible [off] --help=[on|off] print help and exit --version=[on|off] print version and exit Compiled-in tests: fileio - File I/O test cpu - CPU performance test memory - Memory functions speed test threads - Threads subsystem performance test mutex - Mutex performance test oltp - OLTP test Commands: prepare run cleanup help version See 'sysbench --test=<name> help' for a list of options for each test. ---- I don't see any option for disk testing... just fileio. And under it: --file-num=N number of files to create [128] --file-block-size=N block size to use in all IO operations [16384] --file-total-size=SIZE total size of files to create [2G] --file-test-mode=STRING test mode {seqwr, seqrewr, seqrd, rndrd, rndwr, rndrw} --file-io-mode=STRING file operations mode {sync,async,fastmmap,slowmmap} [sync] --file-async-backlog=N number of asynchronous operatons to queue per thread [128] --file-extra-flags=STRING additional flags to use on opening files {sync,dsync,direct} [] --file-fsync-freq=N do fsync() after this number of requests (0 - don't use fsync()) [100] --file-fsync-all=[on|off] do fsync() after each write operation [off] --file-fsync-end=[on|off] do fsync() at the end of test [on] --file-fsync-mode=STRING which method to use for synchronization {fsync, fdatasync} [fsync] --file-merged-requests=N merge at most this number of IO requests if possible (0 - don't merge) [0] --file-rw-ratio=N reads/writes ratio for combined test [1.5] Only file related tests... It doesn't test disks -- just files (and the other stuff listed...) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-04-01 15:08 (GMT-0700) Linda Walsh composed:
Felix Miata wrote:
On 2013-03-31 13:14 (GMT-0700) Linda Walsh composed:
#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the
Just thought of something.
Did you go from the beginning of the first partition or the beginning of the disk?
I didn't do anything but start Sysbench. It just does what it does, which is to ignore partitions and filesystems. Also, that HD was made before manufacturers began configuring 4096 byte sectors on ordinary production units.
Hmmm... I try sysbench and get: tools/sysbench/sysbench-0.4.12/sysbench> sysbench
Did you open my URL and notice the version at its top? It's not the same sysbench: http://www.os2warp.org/SysBench/ -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2013-03-31 at 13:14 -0700, Linda Walsh wrote:
I know when I partitioned with fdisk, I had to toggle a flag to make sure it was dos compat, otherwise, I got the use of 200 and some odd more sectors out of the first partition.
My test was done with default options. I first created an extended partition, then a number of logicals inside. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFYqeIACgkQtTMYHG2NR9UvWQCfRMcqsz50eEvng3QvY0ZT+iU7 h3MAnjRY4C+cp55H1ashU9CGKV9djqHX =REIj -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
My test was done with default options. I first created an extended partition, then a number of logicals inside.
---- partition tool? any file systems or did you just use raw sectors...? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2013-04-01 at 14:57 -0700, Linda Walsh wrote:
Carlos E. R. wrote:
My test was done with default options. I first created an extended partition, then a number of logicals inside.
---- partition tool?
fdisk.
any file systems or did you just use raw sectors...?
I don't remember, last time I did that was 2 years ago at least. IIRC I created a filesystem, but what I did not do was repeat the test with several filesystem types. Next time I buy a disk and I'm not on a hurry I'll do it :-) Hum, I found one of those tests. 9 Oct 2005. I used ext3, and the disk was 160GB. Wow, I found another test I did on 03 Dec 2009. I even made a graph! <http://susepaste.org/51338135> I found the data I tabulated, but not the script I used - I don't even know the name I used for it. I would have to search the entire disk for bash scripts, and then search all those inside for a recognizable string... that's a feature mc could have but it does not have. Maybe I'll create such a search script one day. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFaRgsACgkQtTMYHG2NR9VLlgCfcL7XtnrwNQWVzup4Rpn42fGF f2cAoIQIZhSFYOJjVt97tjaReZFID0iA =pqDT -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 02/04/13 13:44, Carlos E. R. wrote: [pruned]
I found the data I tabulated, but not the script I used - I don't even know the name I used for it. I would have to search the entire disk for bash scripts, and then search all those inside for a recognizable string... that's a feature mc could have but it does not have.
Sorry, but it does. Look again. Unless we are talking about 2 different things here - but I am talking about mc being able to search for text within files. Isn't this what you are saying mc cannot do?
Maybe I'll create such a search script one day.
BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
On 02/04/13 13:44, Carlos E. R. wrote:
[pruned]
I found the data I tabulated, but not the script I used - I don't even know the name I used for it. I would have to search the entire disk for bash scripts, and then search all those inside for a recognizable string... that's a feature mc could have but it does not have.
Sorry, but it does. Look again.
Unless we are talking about 2 different things here - but I am talking about mc being able to search for text within files. Isn't this what you are saying mc cannot do?
Dunno about mc, but won't this do: find / -iname <template for bash-scripts> -print0 | \ xargs -r -0 grep <a recognizable string> -- Per Jessen, Zürich (2.9°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 09:29 +0200, Per Jessen wrote:
Dunno about mc, but won't this do:
find / -iname <template for bash-scripts> -print0 | \ xargs -r -0 grep <a recognizable string>
What would the template be? I intended to use "file" output. cer@Telcontar:~> file bin/0_script_constructs bin/0_script_constructs: Bourne-Again shell script, UTF-8 Unicode text cer@Telcontar:~> Ie, locate all files of that type, then grep on those. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcGE8ACgkQtTMYHG2NR9W5TQCgmWNVnnkbOwAOJhoKYnCg5rlm g5oAn32d8cPLF3pdj/aacnpA06TTAp+m =+Rk4 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Wednesday, 2013-04-03 at 09:29 +0200, Per Jessen wrote:
Dunno about mc, but won't this do:
find / -iname <template for bash-scripts> -print0 | \ xargs -r -0 grep <a recognizable string>
What would the template be? I intended to use "file" output.
cer@Telcontar:~> file bin/0_script_constructs bin/0_script_constructs: Bourne-Again shell script, UTF-8 Unicode text cer@Telcontar:~>
Ie, locate all files of that type, then grep on those.
Hmm, how about this: find / -type f | xargs file | awk '/Bourne-Again/{print $1}' | \ tr -d ':' | xargs -r grep <a recognizable string> -- Per Jessen, Zürich (5.5°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 14:13 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Hmm, how about this:
find / -type f | xargs file | awk '/Bourne-Again/{print $1}' | \ tr -d ':' | xargs -r grep <a recognizable string>
Interesting. Let's try: cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' tr -d ':' | xargs -r grep "fdisk" awk: fatal: cannot open file `tr' for reading (No such file or directory) cer@Telcontar:~> which tr /usr/bin/tr cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' /usr/bin/tr -d ':' | xargs -r grep "fdisk" awk: cmd. line:1: fatal: cannot open file `-d' for reading (No such file or directory) cer@Telcontar:~> I'm not familiar with awk, so I don't know what is wrong above :-? - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcIvEACgkQtTMYHG2NR9VHYQCfYkQ8ijye0VKMwVnanNt0HEZE TTgAoIAKEeBy0T7zqxYoWvXAvyal4z9M =yex5 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. said the following on 04/03/2013 08:39 AM:
On Wednesday, 2013-04-03 at 14:13 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Hmm, how about this:
find / -type f | xargs file | awk '/Bourne-Again/{print $1}' | \ tr -d ':' | xargs -r grep <a recognizable string>
Interesting. Let's try:
cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' tr -d ':' | xargs -r grep "fdisk" awk: fatal: cannot open file `tr' for reading (No such file or directory) cer@Telcontar:~> which tr /usr/bin/tr cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' /usr/bin/tr -d ':' | xargs -r grep "fdisk" awk: cmd. line:1: fatal: cannot open file `-d' for reading (No such file or directory) cer@Telcontar:~>
I'm not familiar with awk, so I don't know what is wrong above :-?
There's a TYPO There's a missing "|" try find bin/* -type f | xargs file | \ awk '/Bourne-Again/{print $1}' |\ tr -d ':' | xargs -r grep "fdisk" If you look at the original - was it Per's ? - you'll see that the "|" is there -- If we believe absurdities, we shall commit atrocities. -- Voltaire -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 09:17 -0400, Anton Aylward wrote:
If you look at the original - was it Per's ? - you'll see that the "|" is there
You are right, I googed in the copy-paste. :-o It works: cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' | tr -d ':' | xargs -r grep "for i in" bin/CD-info:for i in 32768,7 32776,32 32808,32 32958,128 33086,128 33214,128 \ cer@Telcontar:~> :-))) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcMvkACgkQtTMYHG2NR9V8LACfT1zFHaK8gBRpJSmubwK+F3mt EUEAoIJi7UP8G8Isk6bbS0eqy4D6vPW9 =m60H -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 15:47 +0200, Carlos E. R. wrote:
:-)))
I have it running for real now. There is a little problem: cer@Telcontar:~> find /* -type f | xargs file | awk '/Bourne-Again/{print $1}'| tr -d ':' | tee listado_shells | xargs -r grep "mkfs.ext4 -L test_" find: `/boot/lost+found': Permission denied find: `/data/cripta/root': Permission denied xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option find: `/data/storage_b/cer/cosas/antiguo_cosas_nimrodel__mirar/old_theother/etc/opt/gnome/gconf/gconf.xml.mandantory': Permission denied That about the quotes in xargs :-? And I will have to run it as root to avoid the permission denied errors, too. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcNREACgkQtTMYHG2NR9XEsgCeJ7ek9F9r294YbVTQg0vE3ovi nhQAnjh4WaD9lb3uLSz3EBN2qAVaeOEt =kz0u -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 15:56 +0200, Carlos E. R. wrote:
That about the quotes in xargs :-?
And I will have to run it as root to avoid the permission denied errors, too.
It aborted after 5 minutes: Telcontar:~ # time nice find /* -type f | xargs file | awk '/Bourne-Again/{print $1}'| tr -d ':' | tee listado_shells | xargs -r grep "mkfs.ext4 -L test_" xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option find: `/home/cer/.gvfs': Permission denied find: `standard output': Broken pipe find: write error real 5m2.171s user 0m3.733s sys 0m21.042s Telcontar:~ # I'll put another tee to see where. Telcontar:~ # time nice find /* -type f | tee listado_total | xargs file | awk '/Bourne-Again/{print $1}'| tr -d ':' | tee listado_shells | xargs -r grep "mkfs.ext4 -L test_" xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option tee: standard output: Broken pipe find: `/home/cer/.gvfs': Permission denied tee: write error real 0m52.520s user 0m3.404s sys 0m10.551s Telcontar:~ # No luck. The last lines of 'listado_total' were: windows/OldA/not_mounted /windows/href.pif /windows/Worm_Gibe_C_1.zip /windows/D/not_mounted I don't know what it stuck at. The next entry in the sequence should have been "/windows/E/not_mounted". There are no files in the /windows directory, except those above and some more "not_mounted" 0 bytes flags. There is a file in the listing, early, like this: /data/storage_b/Backups_parciales_particiones_test/Elessar_sda9/home/cer/.local/share/Trash/info/"Custom Install.mpeg.rar".trashinfo That could be the one that produces the first error :-? No, there are several such entries: it would have complained more times. Hey, look at these entries: /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test! /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test. /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test: /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test\ /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/tes|t /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/[.test].test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/~test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/,}test3 /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/[.test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test::test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/"test" /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/12.{test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/te -st /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/te\*st /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/tes t /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/tes<>t /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test's /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test.. /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test:: Wow, difficult ones... :-o Maybe using the "-print0" option in find :-? - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcOtUACgkQtTMYHG2NR9UIpwCfS9y4RJ35KTdWNswnbcrZYeNV IJIAniRmNATXEUnZNlCax0fiK2KXcM+e =W2OP -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
There is a file in the listing, early, like this:
/data/storage_b/Backups_parciales_particiones_test/Elessar_sda9/home/cer/.local/share/Trash/info/"Custom Install.mpeg.rar".trashinfo
That could be the one that produces the first error :-? No, there are several such entries: it would have complained more times.
I don't think so, xargs will die right away.
Wow, difficult ones... :-o
Maybe using the "-print0" option in find :-?
Yes, that should work: find / -type f -print0 | xargs -0 file | \ awk '/Bourne-Again/{print $1}' | \ tr -d ':' | xargs -r grep <something> Only if some of those shell scripts have quotes in their names, this will break too. (second xargs) -- Per Jessen, Zürich (5.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 17:49 +0200, Per Jessen wrote:
Only if some of those shell scripts have quotes in their names, this will break too. (second xargs)
It worked :-) It found the file: /data/storage_b/Backups_parciales_particiones_test/Bombadil_sda1/root/bin/test_hd_make: #mkfs.ext4 -L test_$X /dev/sdb$X In a partial backup of a previous installation. Those are the scripts I used to measure hard disk speed across 20 partitions or so that I mentioned on another post. But it took too long: real 134m50.851s user 56m45.657s sys 3m7.112s If is only analyzing inside bash scripts, why did it took so long? - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcrjUACgkQtTMYHG2NR9X24wCdFjihH6ZZQM0KgimGteDV2vZe wtsAnRi0ZYC57iqf/3kagAwtbtQ7Oy/2 =mebw -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Content-ID: <alpine.LNX.2.00.1304040442080.21805@Telcontar.valinor> On Thursday, 2013-04-04 at 00:33 +0200, Carlos E. R. wrote:
But it took too long:
real 134m50.851s user 56m45.657s sys 3m7.112s
If is only analyzing inside bash scripts, why did it took so long?
I'll time it. time find / -type f -print0 | xargs -0 file | awk '/Bourne-Again/{print $1}' | tr -d ':' | xargs -r grep -e "mkfs.ext4 -L test_" +++···························· Telcontar:~ # time find / -type f -print0 > listadeficheros find: `/home/cer/.gvfs': Permission denied real 8m0.449s user 0m2.167s sys 0m29.063s Telcontar:~ # ····························++- That was reasonable. The listing has 184M. Next phase: +++···························· Telcontar:~ # time cat listadeficheros | xargs -0 file > lista.file_output real 129m13.901s user 56m40.918s sys 3m5.739s Telcontar:~ # ····························++- Wow, so it is the 'file' command that is slow! Seeing if a file is a script is easy, you only need to see the start of the first "line" (the shebang). But of course, we ask file to find what type it is, not to find only one particular type. Some files probably are harder to identify. +++···························· Telcontar:~ # time cat lista.file_output | awk '/Bourne-Again/{print $1}' > lista.awk_output real 0m2.866s user 0m1.457s sys 0m0.508s Telcontar:~ # ····························++- +++···························· Telcontar:~ # time cat lista.awk_output | tr -d ':' > lista.tr_output real 0m0.027s user 0m0.000s sys 0m0.002s Telcontar:~ # ····························++- +++···························· Telcontar:~ # time cat lista.tr_output | xargs -r grep -e "mkfs.ext4 -L test_" grep: /other/test_a2/home/cer/Documents/Recuerdos/Caricaturas: No such file or directory grep: /other/test_a2/home/cer/Documents/Recuerdos/Caricaturas: No such file or directory ... /data/storage_b/Backups_parciales_particiones_test/Bombadil_sda1/root/bin/test_hd_make: #mkfs.ext4 -L test_$X /dev/sdb$X /data/storage_b/Backups_parciales_particiones_test/Bombadil_sda1/root/bin/test_hd_make~: #mkfs.ext4 -L test_$X /dev/sdb$X real 0m18.306s user 0m0.014s sys 0m0.348s Telcontar:~ # ····························++- The final grep has errors because I have files and directories with spaces in the names, and the strings were truncted at the spaces. Well, in the sample above, a directory, which has one script inside. In the end, it did not matter because I found what I wanted. Sizes: +++···························· Telcontar:~ # ls -lh listadeficheros lista.file_output lista.awk_output lista.tr_output - -rw-r--r-- 1 root root 249K Apr 4 04:44 lista.awk_output - -rw-r--r-- 1 root root 346M Apr 4 04:41 lista.file_output - -rw-r--r-- 1 root root 245K Apr 4 04:46 lista.tr_output - -rw-r--r-- 1 root root 184M Apr 4 02:06 listadeficheros Telcontar:~ # ····························++- I still think it is faster than a brute force grep on the entire filesystem... I guess... - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFc7eQACgkQtTMYHG2NR9V89gCeIoxY9JNxuv2uexxzuVJW45wZ ayUAoJiSk0+8xoPuKHGwmz7O9ZLymZrw =dMWl -----END PGP SIGNATURE-----
Carlos E. R. wrote:
On Wednesday, 2013-04-03 at 17:49 +0200, Per Jessen wrote:
Only if some of those shell scripts have quotes in their names, this will break too. (second xargs)
It worked :-)
It found the file:
/data/storage_b/Backups_parciales_particiones_test/Bombadil_sda1/root/bin/test_hd_make: #mkfs.ext4 -L test_$X /dev/sdb$X
In a partial backup of a previous installation. Those are the scripts I used to measure hard disk speed across 20 partitions or so that I mentioned on another post.
But it took too long:
real 134m50.851s user 56m45.657s sys 3m7.112s
If is only analyzing inside bash scripts, why did it took so long?
Some possibilities - because 'file' has to process each and every file found. - because you have 25 gazillion files? :-) - because your disk is veeeerrrrryyyyyyyyy sssssssllllloooooowwwwww :-) If you want to know if it is 'xargs file' or 'xargs grep': find / -type f -print0 | xargs -0 file | \ awk '/Bourne-Again/{print $1}' >/dev/null How long does that take? -- Per Jessen, Zürich (2.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-04-04 at 07:46 +0200, Per Jessen wrote:
Carlos E. R. wrote:
If is only analyzing inside bash scripts, why did it took so long?
Some possibilities
I timed it on another post :-)
- because 'file' has to process each and every file found. - because you have 25 gazillion files? :-) - because your disk is veeeerrrrryyyyyyyyy sssssssllllloooooowwwwww :-)
If you want to know if it is 'xargs file' or 'xargs grep':
It is the file command. There are, of course, many files. I don't know how many, because: Telcontar:~ # wc -l listadeficheros 4 listadeficheros because they are null terminated strings ;-) The timing was: +++···························· Telcontar:~ # time cat listadeficheros | xargs -0 file > lista.file_output real 129m13.901s user 56m40.918s sys 3m5.739s Telcontar:~ # ····························++- comparing cpu time to real time, there is a large waiting time, to be expected. No, my disks are not slow. They are not enterprise class, but they are reasonably fast. <http://susepaste.org/51338135>. That's 125 MB/S max on the vertical axis :-) (the horizontal axis is the partition number) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFdNNcACgkQtTMYHG2NR9VqFACfaUAajpgj3kZ1QW7tlu+dL7LJ Xb0Anj6Tv+cbriS4Tl4lWkjeJqfl0bI2 =giYz -----END PGP SIGNATURE-----
On 04/04/2013 10:07 AM, Carlos E. R. wrote:
It is the file command. There are, of course, many files. I don't know how many, because:
Telcontar:~ # wc -l listadeficheros 4 listadeficheros
because they are null terminated strings ;-)
$ xargs -0 -n 1 echo < listadeficheros | wc -l ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-04-04 at 10:26 +0200, Bernhard Voelker wrote:
On 04/04/2013 10:07 AM, Carlos E. R. wrote:
It is the file command. There are, of course, many files. I don't know how many, because:
Telcontar:~ # wc -l listadeficheros 4 listadeficheros
because they are null terminated strings ;-)
$ xargs -0 -n 1 echo < listadeficheros | wc -l
;-)
Let's see... ... I have that running for 17 minutes and it has not finished. xargs is using 10% CPU, no disk activity. The listing is 184M. I abort it, try again without the last part of the pipe to see if there are errors. Telcontar:~ # time cat listadeficheros | xargs -0 -n 1 echo ... ... real 26m23.341s user 0m11.142s sys 2m15.679s Telcontar:~ # No error. Let's try again: Telcontar:~ # time cat listadeficheros | xargs -0 -n 1 echo | wc -l 2697454 real 25m10.859s user 0m11.499s sys 2m20.532s Telcontar:~ # Telcontar:~ # time cat listadeficheros > /dev/null real 0m0.041s user 0m0.000s sys 0m0.040s Telcontar:~ # It appears that xargs itself is slow... Anyway, I have 2697454 files :-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFdU40ACgkQtTMYHG2NR9V4IQCgmPSaRLZ8aDGeei5LcTmQPVIE Ql4AoJCIredXV98lmTwqM9O0ocTxyWqS =Ymw6 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2013-04-04 at 10:26 +0200, Bernhard Voelker wrote:
On 04/04/2013 10:07 AM, Carlos E. R. wrote:
It is the file command. There are, of course, many files. I don't know how many, because:
Telcontar:~ # wc -l listadeficheros 4 listadeficheros
because they are null terminated strings ;-)
$ xargs -0 -n 1 echo < listadeficheros | wc -l
;-)
Let's see...
I have that running for 17 minutes and it has not finished.
You have deliberately prolonged the agony by adding '-n 1' to xargs - that means xargs will fork 'echo' once per filename. Try this: xargs -0 printf "%s\n" <listadeficheros | wc -l or much better: wc -l --files0from=listadeficheros -- Per Jessen, Zürich (6.6°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-04-04 at 12:54 +0200, Per Jessen wrote:
Carlos E. R. wrote:
I have that running for 17 minutes and it has not finished.
You have deliberately prolonged the agony by adding '-n 1' to xargs - that means xargs will fork 'echo' once per filename.
It was not my code ;-)
Try this:
xargs -0 printf "%s\n" <listadeficheros | wc -l
What a difference! Telcontar:~ # time xargs -0 printf "%s\n" <listadeficheros | wc -l 2697454 real 0m2.794s user 0m2.279s sys 0m1.603s Telcontar:~ #
or much better:
wc -l --files0from=listadeficheros
Mmmm.... something wrong there. Telcontar:~ # wc -l --files0from=listadeficheros wc: unrecognized option '--files0from=listadeficheros' Try `wc --help' for more information. Telcontar:~ # wc -l --files0 from=listadeficheros wc: cannot open `from=listadeficheros' for reading: No such file or directory Telcontar:~ # wc -l --files0 --from=listadeficheros wc: cannot open `--from=listadeficheros' for reading: No such file or directory Telcontar:~ # Telcontar:~ # l listadeficheros - -rw-r--r-- 1 root root 192068567 Apr 4 02:06 listadeficheros [man wc...] Ah! It is: Telcontar:~ # time wc -l --files0-from=listadeficheros ... Had to abort it, it prints to the screen the entire list. There is no quiet option :-? Trying again.... Huh. It has got stuck somewhere in the middle. Locked. Telcontar:~ # time wc -l --files0-from=listadeficheros ... ... wc: /var/lib/ntp/proc/sysrq-trigger: Input/output error 0 /var/lib/ntp/proc/sysrq-trigger 1 /var/lib/ntp/proc/splash 48 /var/lib/ntp/proc/partitions 56 /var/lib/ntp/proc/diskstats 98 /var/lib/ntp/proc/crypto 1 /var/lib/ntp/proc/key-users 0 /var/lib/ntp/proc/kpageflags 32 /var/lib/ntp/proc/kpagecount <--- stuck for a minute in there, at least. ^C real 2m15.006s user 0m0.469s sys 0m1.976s Telcontar:~ # It is slower, because it prints to the screen, and it locks, producing no output. :-? - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFdX+MACgkQtTMYHG2NR9XF4wCdHRtm9kSAkiN9/oS0plq/q2y9 yJAAnjsDrAp8/vgdhSffdWnGvjeZIBC9 =/g5o -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/04/2013 01:11 PM, Carlos E. R. wrote:
Huh. It has got stuck somewhere in the middle. Locked.
Telcontar:~ # time wc -l --files0-from=listadeficheros
That command counts the lines numbers of each of you 2.6m files (instead of the number of files).
... ... wc: /var/lib/ntp/proc/sysrq-trigger: Input/output error 0 /var/lib/ntp/proc/sysrq-trigger 1 /var/lib/ntp/proc/splash 48 /var/lib/ntp/proc/partitions 56 /var/lib/ntp/proc/diskstats 98 /var/lib/ntp/proc/crypto 1 /var/lib/ntp/proc/key-users 0 /var/lib/ntp/proc/kpageflags 32 /var/lib/ntp/proc/kpagecount
<--- stuck for a minute in there, at least.
^C
real 2m15.006s user 0m0.469s sys 0m1.976s Telcontar:~ #
It is slower, because it prints to the screen, and it locks, producing no output. :-?
Bingo! That's where the file command is also stuck. The file list is wrong. You should limit it to where you really want to search for the shell scripts. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Thursday, 2013-04-04 at 12:54 +0200, Per Jessen wrote:
Carlos E. R. wrote:
I have that running for 17 minutes and it has not finished.
You have deliberately prolonged the agony by adding '-n 1' to xargs - that means xargs will fork 'echo' once per filename.
It was not my code ;-)
Try this:
xargs -0 printf "%s\n" <listadeficheros | wc -l
What a difference!
Telcontar:~ # time xargs -0 printf "%s\n" <listadeficheros | wc -l 2697454
real 0m2.794s user 0m2.279s sys 0m1.603s
Yep, that's more like it.
or much better:
wc -l --files0from=listadeficheros
Mmmm.... something wrong there. [snip] Ah! It is:
Telcontar:~ # time wc -l --files0-from=listadeficheros ...
Had to abort it, it prints to the screen the entire list. There is no quiet option :-?
My mistake, I misread what that option does. Ignore it. -- Per Jessen, Zürich (7.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/04/2013 12:54 PM, Per Jessen wrote:
You have deliberately prolonged the agony by adding '-n 1' to xargs - that means xargs will fork 'echo' once per filename.
Try this:
xargs -0 printf "%s\n" <listadeficheros | wc -l
or even better: $ tr '\0' '\n' < listadeficheros | wc -l Isn't it cool to have so many choices? ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Content-ID: <alpine.LNX.2.00.1304041355191.21805@Telcontar.valinor> On Thursday, 2013-04-04 at 13:29 +0200, Bernhard Voelker wrote:
On 04/04/2013 12:54 PM, Per Jessen wrote:
or even better:
$ tr '\0' '\n' < listadeficheros | wc -l
Telcontar:~ # time tr '\0' '\n' < listadeficheros | wc -l 2697454 real 0m0.638s user 0m0.167s sys 0m0.204s Yes, it is faster.
Isn't it cool to have so many choices? ;-)
:-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFdal8ACgkQtTMYHG2NR9VX4ACgk+2r3JyIumdbLXlLgQgTw7jW miMAni3l6BWHmCMBWUX5gt+0AoSalVju =hzSY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Thursday, 2013-04-04 at 13:29 +0200, Bernhard Voelker wrote:
or even better: $ tr '\0' '\n' < listadeficheros | wc -l Telcontar:~ # time tr '\0' '\n' < listadeficheros | wc -l 2697454 real 0m0.638s user 0m0.167s sys 0m0.204s
Yes, it is faster.
But it also defeats the point of using null-terminated file names instead of just printing them 1/line. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
It is the file command. There are, of course, many files. I don't know how many, because:
Telcontar:~ # wc -l listadeficheros 4 listadeficheros
because they are null terminated strings ;-)
You can use "--files0-from=listadeficheros"
The timing was:
+++···························· Telcontar:~ # time cat listadeficheros | xargs -0 file > lista.file_output
real 129m13.901s
No big surprise there. If you need to do this more than once, it would probably to filter out known non-bash file-extensions or only look for "^#!.*bash" in the first line of each file instead of using 'file'. It makes the script a little more complex, but I'm sure it would reduce execution time significantly. -- Per Jessen, Zürich (4.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-04-04 at 11:00 +0200, Per Jessen wrote:
No big surprise there.
If you need to do this more than once, it would probably to filter out known non-bash file-extensions or only look for "^#!.*bash" in the first line of each file instead of using 'file'. It makes the script a little more complex, but I'm sure it would reduce execution time significantly.
Oh, absolutely. Time permitting, I'll write a script or even a pascal program to do it or part of it. Looking for the shebang only should be trivial... - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFdRzoACgkQtTMYHG2NR9UwmACfdrHVSUd3DkDvbuXqF6xTDkyF pKQAn0d6ljiTQU7Rsl2bKGu5K44hdIUv =nGSN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
It aborted after 5 minutes:
--- Are you using the -mmap option? Do you tell it to skip devices and binary files? i.e. in your ENV, set : export GREP_OPTIONS="-D skip --binary-files=without-match" -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Wednesday, 2013-04-03 at 15:47 +0200, Carlos E. R. wrote:
:-)))
I have it running for real now. There is a little problem:
cer@Telcontar:~> find /* -type f | xargs file | awk '/Bourne-Again/{print $1}'| tr -d ':' | tee listado_shells | xargs -r grep "mkfs.ext4 -L test_" find: `/boot/lost+found': Permission denied find: `/data/cripta/root': Permission denied xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option find:
`/data/storage_b/cer/cosas/antiguo_cosas_nimrodel__mirar/old_theother/etc/opt/gnome/gconf/gconf.xml.mandantory':
Permission denied
That about the quotes in xargs :-?
I guess you are finding a filename or a filetype with quotes in it. Unless you have filenames with quotes in them, just use this tr instead: tr -d ':\'"' -- Per Jessen, Zürich (5.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 17:57 +1100, Basil Chupin wrote:
On 02/04/13 13:44, Carlos E. R. wrote:
[pruned]
I found the data I tabulated, but not the script I used - I don't even know the name I used for it. I would have to search the entire disk for bash scripts, and then search all those inside for a recognizable string... that's a feature mc could have but it does not have.
Sorry, but it does. Look again.
Unless we are talking about 2 different things here - but I am talking about mc being able to search for text within files. Isn't this what you are saying mc cannot do?
I want 'mc' to find a particular type of file (using the command 'file'), and only then do a text search inside. cer@Telcontar:~> file bin/0_script_constructs bin/0_script_constructs: Bourne-Again shell script, UTF-8 Unicode text cer@Telcontar:~> In this case I want to search inside bash scripts only. If I do a string search unrestricted on the entire disks it looks inside huge multimedia files, for example, taking hours to finish. By the way, if I try a CLI "grep" on my ~/bin directory, it locks: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND cer 23961 0.1 0.0 11820 3568 pts/2 S+ 13:46 0:00 | \_ grep --color=auto fdisk bin/0_script_constructs bin/0_script_constructs~ bin/200506_attrib.list bin/Antiguos bin/Bck200405_attrib.dat CPU usage is NIL, iotop doesn't see the grep as working. I just typed: cer@Telcontar:~> grep fdisk bin/* ... Got some hits, then it stuck. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcF50ACgkQtTMYHG2NR9XF+gCfV8BDX7y7oMM1uSmhNt0hTrbT vHkAni7azYxXa0PdYCGS3VdbcFLD/dPD =jaBV -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. said the following on 04/03/2013 07:50 AM:
By the way, if I try a CLI "grep" on my ~/bin directory, it locks:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND cer 23961 0.1 0.0 11820 3568 pts/2 S+ 13:46 0:00 | \_ grep --color=auto fdisk bin/0_script_constructs bin/0_script_constructs~ bin/200506_attrib.list bin/Antiguos bin/Bck200405_attrib.dat
CPU usage is NIL, iotop doesn't see the grep as working. I just typed:
cer@Telcontar:~> grep fdisk bin/* ...
Got some hits, then it stuck.
I've had that happen in other situations, most notably greping recursively through /etc :-( However if I use 'find' and only grep actual files, that is not some of the strange things that can be symlinked in there, it never hangs. I admit that I've never systematically searched to identity what the 'strange things' were. -- A distracted figure with a huge bushy beard blunders in just as you speak the word of ancient magic. The man wears loose clothing, and an expression of intense concentration. He is clutching his frizzy hair with one hand; his other hand grips an intricate grid - the object of his attention. His eyes brighten the word you've spoken reaches his ears. "Yes! Yes! That's it!" he exclaims as he draws out a pen and fills in a row of squares. "Now my hyperconstrained, double-acrostic, cryptic crossword is complete, and ready to puzzle others. That was all I needed - just a simple five-letter word, composed only of the letters 'X' 'Y' and 'Z,' that would fit here!" He grips your hand and shakes it fervently. "Thank you! Now that I've finished with that, I can get on to those other things I've been meaning to do, such as monkey-wrenching the demolition and saving recreational linguistics for future generations." He turns away and mutters, just before he departs, "I hope none of that will involve lying in front of a bulldozer..." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 08:08 -0400, Anton Aylward wrote:
Carlos E. R. said the following on 04/03/2013 07:50 AM:
By the way, if I try a CLI "grep" on my ~/bin directory, it locks:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND cer 23961 0.1 0.0 11820 3568 pts/2 S+ 13:46 0:00 | \_ grep --color=auto fdisk bin/0_script_constructs bin/0_script_constructs~ bin/200506_attrib.list bin/Antiguos bin/Bck200405_attrib.dat
CPU usage is NIL, iotop doesn't see the grep as working. I just typed:
cer@Telcontar:~> grep fdisk bin/* ...
Got some hits, then it stuck.
I've had that happen in other situations, most notably greping recursively through /etc :-(
However if I use 'find' and only grep actual files, that is not some of the strange things that can be symlinked in there, it never hangs.
I admit that I've never systematically searched to identity what the 'strange things' were.
I'm curious. Called it this way: cer@Telcontar:~> strace -o strace.out grep fdisk bin/* and I see it stuck at: close(3) = 0 stat("bin/fecha", {st_mode=S_IFREG|0755, st_size=39, ...}) = 0 open("bin/fecha", O_RDONLY) = 3 ioctl(3, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, 0x7fff54dc55f8) = -1 ENOTTY (Inappropriate ioctl for device) read(3, "#!/bin/bash\ndate --rfc-3339=seco"..., 5013504) = 39 read(3, "", 5013504) = 0 close(3) = 0 stat("bin/fifo", {st_mode=S_IFIFO|0666, st_size=0, ...}) = 0 open("bin/fifo", O_RDONLY I do have a spurious named pipe "fifo" there: cer@Telcontar:~> l bin/fifo prw-rw-rw- 1 cer users 0 Jan 31 2010 bin/fifo| cer@Telcontar:~> which I used long ago and forgot to remove. I delete it, restart grep, and now it completes :-) Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault? One of the reasons I use 'mc' for searches is that it is clever enough not to look inside those and does not get stuck. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcJd4ACgkQtTMYHG2NR9WVWwCbBMiHHbgI6tDJNZ4647UXJYO4 zeAAn1gVFMLbVmZAWlRAfLszLdPzqlpO =9HNg -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. said the following on 04/03/2013 08:51 AM:
Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?
No, its your fault. Grep can and _should_ read from whatever input you give it. How is a named pipe different from an unnamed pipe? cat /etc/passwd | grep "anton"
One of the reasons I use 'mc' for searches is that it is clever enough not to look inside those and does not get stuck.
Which, to my mind, says that 'mc' is broken. You problem is that the named pipe is not supplying a data stream and an end of file. Grep is going to keep reading from its input until it gets to the end of the file. You can do the same thing at the CLI with this $ grep anton Grep is now reading from STDIN. It will keep reading from STDIN until it sees and end of file -- type ctrl-D -- or until it is killed, perhaps with ctrl-C. There is nothing wrong with grep. It is behaving as it should. -- "No serious commentary will say that the user has no responsibility. We all have responsibilities to lock our doors in our homes and to buckle up when we get in cars." -- spokesman, Information Technology Association of America, Business Roundtable, AP, May 19, 2004 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 09:24 -0400, Anton Aylward wrote:
Carlos E. R. said the following on 04/03/2013 08:51 AM:
Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?
No, its your fault. Grep can and _should_ read from whatever input you give it.
Ok, but the man page does not say how to tell grep not to look inside not normal files :-) Of course, it can be done with a "find" in the line, but not with grep on its own. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcNp4ACgkQtTMYHG2NR9UlgwCggHhCzx4uHM8e4nRoURFl0BO3 anIAn2uCTmf1pxkNHQklP46Vxt3cLay9 =1keW -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?
No, its your fault. Grep can and _should_ read from whatever input you give it.
Ok, but the man page does not say how to tell grep not to look inside not normal files :-)
-D ACTION, --devices=ACTION If an input file is a device, FIFO or socket, use ACTION to process it. By default, ACTION is read, which means that devices are read just as if they were ordinary files. If ACTION is skip, devices are silently skipped. What's that then? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Content-ID: <alpine.LNX.2.00.1304031625101.21805@Telcontar.valinor> On Wednesday, 2013-04-03 at 15:17 +0100, Dave Howorth wrote:
Carlos E. R. wrote:
Ok, but the man page does not say how to tell grep not to look inside not normal files :-)
-D ACTION, --devices=ACTION If an input file is a device, FIFO or socket, use ACTION to process it. By default, ACTION is read, which means that devices are read just as if they were ordinary files. If ACTION is skip, devices are silently skipped.
What's that then?
Ha! Didn't see that one. Let's try - huh, no, I deleted that 3 year old named pipe. Ok, then: cer@Telcontar:~> grep -D skip fdisk /dev/* grep: /dev/core: Permission denied cer@Telcontar:~> I suppose that worked :-) That's one to put in my book. Never stop learning! :-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcPDQACgkQtTMYHG2NR9VtowCeIBrh0ACyf+3go+FjzUmfvPn6 h/8AoJJDQxeLnmY9EnbCYdYGQlcm2T7k =tjMD -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Wed, 03 Apr 2013 15:17:00 +0100 Dave Howorth <dhoworth@mrc-lmb.cam.ac.uk> пишет:
-D ACTION, --devices=ACTION
Oh. Never noticed it before. Thank you! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. said the following on 04/03/2013 10:03 AM:
No, its your fault. Grep can and _should_ read from whatever input you give it.
Ok, but the man page does not say how to tell grep not to look inside not normal files :-)
Ah, so you want a Swiss army knife? It reminds me of Rob Pike's "considered harmful" view of what Berkeley did to 'cat' I agree with the 'each thing should do one things and only one thing' approach to tool building. Not only does it make combining tools easier - the great power of UNIX is scripting, notable the shell -but each thing can be proven to be 'correct'. If you can make this case for grep then you can make it for cat and tr and many other programs. Perhaps you want to re-invent one of the mainframe OS's of the 50s and 60s? http://harmful.cat-v.org/cat-v/unix_prog_design.pdf -- May the itch of a thousand crabs affect the one who ruins your day and may their arms be too short to scratch. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 10:26 -0400, Anton Aylward wrote:
Carlos E. R. said the following on 04/03/2013 10:03 AM:
Ok, but the man page does not say how to tell grep not to look inside not normal files :-)
Ah, so you want a Swiss army knife?
:-) Ah, Dave posted the trick a minute ago :-) About grep and the Swiss army knife... I'm used to a special "grep" version made by Lucent Technologies (previous AT&T") that was named "cgrep". It is now available as free (well, actually "Lucent Public License"), sometime after the demise of Lucent and the disappearance of Bell Labs. This switch is what made it of particular use to me: 5ESS Project-Specific Options The following special-purpose options are provided for the convenience of those analyzing 5ESS International or Domestic ROP reports. -R Sets window delimiters and other options for 5ESS International or Domestic ROP reports. The -R option can also be used in conjunction with -nline or +nline to output preceding or subsequent ROP reports to the matched report, or both. In either case, nline-1 adjacent reports will also be printed. The -R option is almost equivalent to -DEMh +I2 -+w ´\+\+\+|^Y´ , but has one additional effect. Patterns of the form ´sm=n´ (the literal sm must be in lower case letters) will be expanded into regular expressions to match most if not all ROP reports pertaining to SM n. n must be an unsigned number without leading zeros, in which each digit may be either a simple digit or any one of a set of digits enclosed in square brackets. Within brackets, a hyphen can be used to indicate a range of digits. See the 5ESS EXAMPLES sec- tion for examples of the use of this feature. -T Same as -R, except also causes ROP event trails to be followed. If the string "EVENT=n" (n is a number of one or more digits, possibly preceded by a minus sign) or "EVENTO=n" (Spanish) appears in a matched ROP report, all subsequent reports with the same event number will be printed. Equivalent to the following: -R -t ´EVENTO?=-?[0-9]+´ . It is not the only addition. There are powerful options to capture context around a particular string found in the text, both before and after the match. I have used that feature in scripts of mine. The man page says that the author is "Bill Tanenbaum". The web page at the time I got the public version were these, according to the man page: http://www.bell-labs.com/project/wwexptools/cgrep/ Source Code, etc. http://www.bell-labs.com/cgi-user/wwexptools/gensnapshot?cgrep
It reminds me of Rob Pike's "considered harmful" view of what Berkeley did to 'cat'
I agree with the 'each thing should do one things and only one thing' approach to tool building. Not only does it make combining tools easier - the great power of UNIX is scripting, notable the shell -but each thing can be proven to be 'correct'.
True. Mmmm... systemd? >:-)
If you can make this case for grep then you can make it for cat and tr and many other programs.
X'-)
Perhaps you want to re-invent one of the mainframe OS's of the 50s and 60s?
Argh, 7 pages of dense PDF print... Later, perhaps. I have to do some accounting chores (which I hate). And I want to have my siesta about now, too ;-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcQXwACgkQtTMYHG2NR9Ww3gCgjrmkcHzF+wzAqjfGzJo6TgDa 8LQAoIsH330sphfaUjfjZ7vV/pi+y9fP =RZHA -----END PGP SIGNATURE-----
Carlos E. R. wrote:
Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?
You told it to look in there. Whenever I grep for something or other, I virtually always use the "find <specification> | xargs grep" combination. The xargs can also be parallelized, a big advantage if you've got multiple cores and many files. -- Per Jessen, Zürich (5.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 17:53 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?
You told it to look in there.
:-) Didn't know there was a named pipe in there :-)
Whenever I grep for something or other, I virtually always use the "find <specification> | xargs grep" combination. The xargs can also be parallelized, a big advantage if you've got multiple cores and many files.
Ah, I did not know that, either. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFceggACgkQtTMYHG2NR9WcEQCfWbRnfgLO4vC8hI0bxaLoVWLq xEIAnj97eZpTDVdSKsFVSs+LTMUQa2jq =8r3v -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
Didn't know there was a named pipe in there :-)
Whenever I grep for something or other, I virtually always use the "find <specification> | xargs grep" combination. The xargs can also be parallelized, a big advantage if you've got multiple cores and many files.
Ah, I did not know that, either.
But it is not automatic, you have to tell it how many jobs and when you do, it defaults to 1 command / process unless you tell it otherwise. Also, unless you have a slow-search expression, grep will usually be disk bound not cpu bound -- so spawning off extra copies to do simultaneous reads on different files isn't likely to give you as much increased benefit as you'd like... As an example, I have a script that runs an rpm query on every rpm in a directory to gather information.... As a result, I found that after about 9 Q's, I started losing overall speed -- that's 9 cpu's each running 1/9th of the list. Overall cpu usage for the job is about 300%, so 6 of those cpu's were waiting on disk -- not very efficient, but overall 9 Q's was best for this particular 'job'.....that's partly because the disk it was doing the queries on is a RAID50. If it was a single disk, probably no more than 3Q's before disk thrashing slows down overall progress... Anyway, if you set your GREP_OPTIONS="-D skip --binary-files=without-match" in your bashrc, grep will generally be better behaved... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Linda Walsh <suse@tlinx.org> wrote:
Felix Miata wrote:
Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.
Confirmed via observation here, closer maybe to 25%, but clearly slower
#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the only such doc I saved. Note on its #2 the difference between front and middle is very small. All others I checked at http://fm.no-ip.com/PC/bench/Sysbench/ are fastest at front. Most of these were done shortly after receiving a new HD, prompting to test to see what speed to expect. ISTR seeing it happen more than once though. Otherwise I don't think I'd have remembered it ever happening.
Just thought of something.
Did you go from the beginning of the first partition or the beginning of the disk?
Be default, or unless you've gone for DOS compatible partitioning, the first partition won't be track-aligned due to the boot sectors.
Then usually, people (or software) opts for nearest track for end of 1st partition, so further partitions are aligned.
It would make a slight difference, but the head would have to move over a track mid-read -- to read any track in the first partition, if such a situation were in place.
I know when I partitioned with fdisk, I had to toggle a flag to make sure it was dos compat, otherwise, I got the use of 200 and some odd more sectors out of the first partition.
Linda, It is time for you to enter the 21st century in respects to disk geometry. Back in the 80's and 90's CHS actually corresponded to physical reality. But LBA was introduced about 20 years ago because CHS was a pain. When disk sectors shrank to where more than 255 sectors per track became normal, the drive manufacturers just started lying when they reported the CHS geometry because 255 was the biggest reportable value and nobody cared enough at that point to fix the CHS standard. For at least the last decade the linear density of sectors is typically constant over the whole platter. Since you know circumference of a circle is PI * D, a track 3 inches from the center holds 3x the sectors as one 1 inch from the center. That means number of sectors per track isn't even a constant for a modern drive. That by the way is why start of the drive is faster than the end. The drives start at the outside edge, so for one revolution the head passes over a lot more sectors. The end of the drive is closer to the center, so it sees less sectors per revolution. And at least for now the rpms is kept constant, so the sectors per second speed is higher for the start of the drive. Anyway, if you think you have the ability to align partitions to tracks, you are simply mistaken. You can merely Fyi: both modern windows and linux align to 1MB boundaries with no concern how that aligns with tracks. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2013-03-31 at 23:35 -0400, Greg Freemyer wrote:
That by the way is why start of the drive is faster than the end. The drives start at the outside edge, so for one revolution the head passes over a lot more sectors.
True. However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?
Anyway, if you think you have the ability to align partitions to tracks, you are simply mistaken. You can merely
Yes, it is an illusion. Still, there are some partitioning sotware that does so. I mean, it says it does. I think it is sfdisk, it even complains if partitions do not end/start at track ends and makes you force the layout to accept it (if it is not sfdisk I can find out which, I have the text saved somewhere). - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFZA4gACgkQtTMYHG2NR9WZIQCghxsX2ifibB5htl9rZjCoFibM 6u0AmwRdL9Fl3ZEwsSsano3czlFhqTdK =8O/v -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
"Carlos E. R." <robin.listas@telefonica.net> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Sunday, 2013-03-31 at 23:35 -0400, Greg Freemyer wrote:
That by the way is why start of the drive is faster than the end. The drives start at the outside edge, so for one revolution the head passes over a lot more sectors.
True.
However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?
Anyway, if you think you have the ability to align partitions to tracks, you are simply mistaken. You can merely
Yes, it is an illusion.
Still, there are some partitioning sotware that does so. I mean, it says it does. I think it is sfdisk, it even complains if partitions do not end/start at track ends and makes you force the layout to accept it (if it is not sfdisk I can find out which, I have the text saved somewhere).
Windows made the jump to 1MB alignment with Vista. Opensuse a couple years ago. I'm sure there is software that still treats CHS geometry as meaningful, but that software is out of date. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2013-04-01 at 00:09 -0400, Greg Freemyer wrote:
"Carlos E. R." <> wrote:
Still, there are some partitioning sotware that does so. I mean, it says it does. I think it is sfdisk, it even complains if partitions do not end/start at track ends and makes you force the layout to accept it (if it is not sfdisk I can find out which, I have the text saved somewhere).
Windows made the jump to 1MB alignment with Vista. Opensuse a couple years ago.
I'm sure there is software that still treats CHS geometry as meaningful, but that software is out of date.
I know, of course they are out of date. I had to clone a partition table recently. I used sfdisk, like this - I have the text saved: minas-tirith:~ # sfdisk -d /dev/sdc > partition.sfdisk minas-tirith:~ # sfdisk --no-reread /dev/sda < partition.sfdisk ... sfdisk: I don't like these partitions - nothing changed. (If you really want this, use the --force option.) minas-tirith:~ # sfdisk --no-reread --force /dev/sda < partition.sfdisk .... Warning: partition 1 does not end at a cylinder boundary Successfully wrote the new partition table Unbelievable! I had tried other tools before, but that was the only one I found that cloned the exact same partition table (including the logicals), without at the same time changing things like attempting to align the partitions differently, which was not acceptable because then the cloning of the partitions themselves would fail. The clone worked fine. It is my laptop. Do you know of a modern tool that will clone a partition table using the exact same sectors as the original? Even if it thinks that the boundaries are wrong, because the old disk was done on CHS and the new one wants to use megabytes. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFaVksACgkQtTMYHG2NR9Wi+wCfWzl1byTbTQivb3J90Rz6909a yoEAnR0mu4HwmzZ3aUaaBiX3225DGqyr =3Sno -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-04-02 05:53 (GMT+0200) Carlos E. R. composed:
Do you know of a modern tool that will clone a partition table using the exact same sectors as the original? Even if it thinks that the boundaries are wrong, because the old disk was done on CHS and the new one wants to use megabytes.
The one I use permits every kind of alignment I've ever encountered, but it isn't FOSS. http://fm.no-ip.com/Tmp/Dfsee/dfslX-vizio.txt starts with output from its standard startup log. To it I've appended fdisk -l and hdparm -i output for comparison. It shows 1.5T and 2.0T devices that use 512 byte internal sectors. 1.5T uses legacy 255/63 geo while the 2T uses 64/32 "advanced format" 4k sector alignment. Both were partitioned 100% using DFSee, which comes with functionally equivalent binaries for DOS, OS/2, Win32, x86 Linux and Intel Mac. You might notice fdisk reports table out of order on sdb. That's on purpose. The STB it's normally used with can only recognize a partition defined by a first partition table entry. I partitioned and formatted and used it before I knew that fact, so since the NTFS needed to be "first", I used DFSee to swap the table entries. DFSee is one of my backup tools. I do a lot of partition and disk cloning with it, but it also creates compressed and uncompressed partition and disk images as desired. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
Do you know of a modern tool that will clone a partition table using the exact same sectors as the original? Even if it thinks that the boundaries are wrong, because the old disk was done on CHS and the new one wants to use megabytes.
dd will do that. -- Per Jessen, Zürich (4.3°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-04-02 at 13:09 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Do you know of a modern tool that will clone a partition table using the exact same sectors as the original? Even if it thinks that the boundaries are wrong, because the old disk was done on CHS and the new one wants to use megabytes.
dd will do that.
No, it doesn't. It clones the partition table itself, that is, the primary partition table only. The extended partition _table_ is not cloned unless you clone the entire disk. My backup image was not made of the entire disk, but of each single partition separately. Thus I have to recreate the partition table first, including the logicals. DD doesn't do that. Once the logical partitions are recreated, I can use dd to put their contents inside. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFa154ACgkQtTMYHG2NR9Wz6wCgitDZD/r8eLh9qvW1YHeBaVGc e8MAn0RX15q0UbTO19BkdFCeTibpsiby =mugJ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Tuesday, 2013-04-02 at 13:09 +0200, Per Jessen wrote:
Carlos E. R. wrote:
Do you know of a modern tool that will clone a partition table using the exact same sectors as the original? Even if it thinks that the boundaries are wrong, because the old disk was done on CHS and the new one wants to use megabytes.
dd will do that.
No, it doesn't.
It clones the partition table itself, that is, the primary partition table only. The extended partition _table_ is not cloned unless you clone the entire disk.
Sorry, I missed the bit about this being about extended partitions/tables. -- Per Jessen, Zürich (5.9°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-04-02 at 15:13 +0200, Per Jessen wrote:
Carlos E. R. wrote:
It clones the partition table itself, that is, the primary partition table only. The extended partition _table_ is not cloned unless you clone the entire disk.
Sorry, I missed the bit about this being about extended partitions/tables.
It took me by surprise. I had a disk going down fast, I was on a hurry to recover before total failure, I had done an image backup partition by partition (because one of them was damaged, and because of the distribution of the available space for the backup), I had dd-ed the partition table, I had a dd of the "extended" partition, too... I thought I had all I needed. And then I found out that I could not recover the stored images of the logical partitions because there were no partitions. And I could not recreate the partitions with fdisk because the original disk had been made by HP with old alignment, and fdisk wanted to do it on megabytes, skipping part of the initial region... If I allowed that, the images would simply not fit. sfdisk did the job, but I had to force it. Being an old and obsolete tool, I want to know what current tool will clone an entire partition table without questions. May have to post a new thread about that one day :-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFa5QwACgkQtTMYHG2NR9V2JACfUSN+qA2bH0qmCTpjiFjmoRg/ P2sAnAkYGFasXtitMeZGsvmFBGGp23MV =cjkA -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
And I could not recreate the partitions with fdisk because the original disk had been made by HP with old alignment, and fdisk wanted to do it on megabytes, skipping part of the initial region... If I allowed that, the images would simply not fit.
I'm not sure, I haven't tried it, but if you revert to DOS-compatible mode with command 'c', I think you might get to allocate partitions any which way you want. I'm sure I have seen a reference to that feature somewhere.
sfdisk did the job, but I had to force it. Being an old and obsolete tool, I want to know what current tool will clone an entire partition table without questions.
sfdisk isn't old nor obsolete, it's very useful. -- Per Jessen, Zürich (2.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-04-02 at 21:11 +0200, Per Jessen wrote:
Carlos E. R. wrote:
And I could not recreate the partitions with fdisk because the original disk had been made by HP with old alignment, and fdisk wanted to do it on megabytes, skipping part of the initial region... If I allowed that, the images would simply not fit.
I'm not sure, I haven't tried it, but if you revert to DOS-compatible mode with command 'c', I think you might get to allocate partitions any which way you want. I'm sure I have seen a reference to that feature somewhere.
Maybe I did not try that.
sfdisk did the job, but I had to force it. Being an old and obsolete tool, I want to know what current tool will clone an entire partition table without questions.
sfdisk isn't old nor obsolete, it's very useful.
It wants to align on tracks, complains bitterly if you don't, and does not support GPT. You can see what happened here: <https://forums.opensuse.org/english/get-technical-help-here/install-boot-login/482350-i-want-clone-disk.html#post2518815> At some point it erased the entire partition table of the sytem while it was runnning... - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFbXDwACgkQtTMYHG2NR9WnrACfZfXT38kBSoDCSyQ8YdWsBkF9 9bkAnRv+YbTG0R2CBIlFpcAlRz3WC6dD =ozqQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Tuesday, 2013-04-02 at 21:11 +0200, Per Jessen wrote:
Carlos E. R. wrote:
And I could not recreate the partitions with fdisk because the original disk had been made by HP with old alignment, and fdisk wanted to do it on megabytes, skipping part of the initial region... If I allowed that, the images would simply not fit.
I'm not sure, I haven't tried it, but if you revert to DOS-compatible mode with command 'c', I think you might get to allocate partitions any which way you want. I'm sure I have seen a reference to that feature somewhere.
Maybe I did not try that.
sfdisk did the job, but I had to force it. Being an old and obsolete tool, I want to know what current tool will clone an entire partition table without questions.
sfdisk isn't old nor obsolete, it's very useful.
It wants to align on tracks, complains bitterly if you don't, and does not support GPT.
Well, yeah. Still very useful when you want to automate things. -- Per Jessen, Zürich (2.9°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 09:27 +0200, Per Jessen wrote:
Carlos E. R. wrote:
It wants to align on tracks, complains bitterly if you don't, and does not support GPT.
Well, yeah. Still very useful when you want to automate things.
If there is no other tool that can clone an entire partition table, I'll use it. I hoped there would be a better tool. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcFRUACgkQtTMYHG2NR9UPggCfV+jd+5KEsvrXe3Md83xPogEA i6kAn3BLm4ChFR/Fnd9m/s3enOH1dYBs =yQp1 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2013 12:48 AM, Carlos E. R. wrote:
However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?
Does not matter really, as no sane human should use rotating disks anymore :-P -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-04-01 03:39 (GMT-0300) Cristian Rodríguez composed:
Carlos E. R. wrote:
However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?
Does not matter really, as no sane human should use rotating disks anymore :-P
When did sanity become a requirement for using a PC or Linux? Must be nice to have unlimited funds when you need another 2TB every 2-3 months, and trust that just because SS media has no moving parts that it will be materially or even any more reliable long-term than spinning platter storage. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2013-04-01 at 03:39 -0300, Cristian Rodríguez wrote:
On 04/01/2013 12:48 AM, Carlos E. R. wrote:
However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?
Does not matter really, as no sane human should use rotating disks anymore :-P
Can you lend me a few of those? I'm scarce on fundings, I have to say. >:-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFZb3oACgkQtTMYHG2NR9VdUwCgh0Q5EYWj61JgY/4GXZlJgdga 7AkAnA34LT5Qbcv4kXl8DX26/DR9yBlw =ATs+ -----END PGP SIGNATURE-----
Carlos E. R. said the following on 04/01/2013 07:28 AM:
Does not matter really, as no sane human should use rotating disks anymore :-P
Can you lend me a few of those? I'm scarce on fundings, I have to say. >:-)
+1 -- Parkinson's Law: Never Interview Emus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2013 07:43 AM, Anton Aylward pecked at the keyboard and wrote:
Carlos E. R. said the following on 04/01/2013 07:28 AM:
Does not matter really, as no sane human should use rotating disks anymore :-P
Can you lend me a few of those? I'm scarce on fundings, I have to say. >:-)
+1
I have been using disks without platters for over 10 years and they virtually cost nothing. Actually VirtualBox includes them for free as does VMware. :-) -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2013 11:20 AM, Ken Schneider - openSUSE pecked at the keyboard and wrote:
On 04/01/2013 07:43 AM, Anton Aylward pecked at the keyboard and wrote:
Carlos E. R. said the following on 04/01/2013 07:28 AM:
Does not matter really, as no sane human should use rotating disks anymore :-P
Can you lend me a few of those? I'm scarce on fundings, I have to say. >:-)
+1
I have been using disks without platters for over 10 years and they virtually cost nothing. Actually VirtualBox includes them for free as does VMware. :-)
Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-) -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Ken Schneider - openSUSE wrote:
Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-)
I guess those virtual bits cost a lot of money. ;-) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Ken Schneider - openSUSE said the following on 04/01/2013 11:28 AM:
Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-)
That's expensive! What a hell of a profit margin! http://www.everyjoe.com/2009/09/29/technology/hard-drive-cost-per-gigabyte-f... Even for a service .. http://www.hightechinthehub.com/2012/07/penny-per-gigabyte-per-month/ The $1/G is last years SSD price http://www.computerworld.com/s/article/9234744/SSD_prices_continue_to_plunge http://www.jcmit.com/diskprice.htm -- "Television is a medium because it is neither rare nor well done." -- Fred Friendly -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2013 11:44 AM, Anton Aylward pecked at the keyboard and wrote:
Ken Schneider - openSUSE said the following on 04/01/2013 11:28 AM:
Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-)
That's expensive! What a hell of a profit margin!
http://www.everyjoe.com/2009/09/29/technology/hard-drive-cost-per-gigabyte-f...
Even for a service .. http://www.hightechinthehub.com/2012/07/penny-per-gigabyte-per-month/
The $1/G is last years SSD price http://www.computerworld.com/s/article/9234744/SSD_prices_continue_to_plunge
Check for yourself: www.virtualbox.org/aprilfool -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Ken Schneider - openSUSE said the following on 04/01/2013 11:51 AM:
Check for yourself:
www.virtualbox.org/aprilfool
Getting a 404 is an april fool? -- There's a tendency today to absolve individuals of moral responsibility and treat them as victims of social circumstance. You buy that, you pay with your soul. -Tom Robbins, Still Life with Woodpecker -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 04/01/2013 12:12 PM, Anton Aylward pecked at the keyboard and wrote:
Ken Schneider - openSUSE said the following on 04/01/2013 11:51 AM:
Check for yourself:
www.virtualbox.org/aprilfool
Getting a 404 is an april fool?
No, taking this serious is being an April Fool. :-) -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 4/1/2013 11:44 AM, Anton Aylward wrote:
Ken Schneider - openSUSE said the following on 04/01/2013 11:28 AM:
Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-)
That's expensive! What a hell of a profit margin!
.... Well considering I can get a 1TB hard drive for ~$80 and 128G esata SSD for ~$145 (just now checked online), I think it's insane to send 15X the price for basic storage. My working storage is still just fine on disk, thank you; and my offline backup is, too. And I have much more than 128G in family and professional stuff I really want to keep -- not just trinkets like cute news and Hollywood photos and videos. And USB flash drives have made mobile computing very, very convenient -- I keep my 32G flash drive in my pocket for downloading interesting stuff I find when I'm online away from home or work. Yeah, there's a place for the SSD, but it's not in my systems yet. I'm subject neither to the fashion motives nor the high-reliability strictures that might justify the huge price difference. John Perry -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Cristian Rodríguez said the following on 04/01/2013 02:39 AM:
On 04/01/2013 12:48 AM, Carlos E. R. wrote:
However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?
Does not matter really, as no sane human should use rotating disks anymore :-P
HA! Another extremist statement! Any sane person know that rotating disks are an essential resource. Fridge Magnets! -- Virtually every major technological advance in the history of the human species-- back to the invention of stone tools and the domestication of fire-- has been ethically ambiguous. --Carl Sagan (The Demon-Haunted World) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Greg Freemyer wrote:
It is time for you to enter the 21st century in respects to disk geometry.
---- Greg, stop showing your ignorance. The LBA still corresponds to CHS *internally* -- just that the nums are internal to each HD.
For at least the last decade the linear density of sectors is typically constant over the whole platter. Since you know circumference of a circle is PI * D, a track 3 inches from the center holds 3x the sectors as one 1 inch from the center. That means number of sectors per track isn't even a constant for a modern drive.
---- Wrong. Modern HD's have been divided into zones for the past 15+years... Modern HD's have almost 1 zone/TB, but you can see a most modern examople at: http://media.bestofmicro.com/E/N/362111/original/throughput-diagram-wd4001FA... Latest 4TB HD... That isn't a smooth curve -- Each notch is a zone.. Across each zone the number of sectors is constant. Very odd on that graph... notice as some have noted the first 50-100GB are slightly lower than just past the 100GB mark -- for READS... writes seem to be very smooth but likely due to internal disk buffering...
Fyi: both modern windows and linux align to 1MB boundaries with no concern how that aligns with tracks.
---- Depends on what disk partition tool you use, but I don't know of any that use MB. The default display in fdisk:...sd Device Boot Start End Blocks Id System /dev/sdc1 63 25398764 12699351 83 Linux /dev/sdc2 25398765 42009974 8305605 83 Linux /dev/sdc3 * 42009975 44130554 1060290 83 Linux Shows POSIX standard 512 byte sectors. parted.. Number Start End Size Type File system Flags 1 32.3kB 13.0GB 13.0GB primary xfs type=83 2 13.0GB 21.5GB 8505MB primary xfs type=83 3 21.5GB 22.6GB 1086MB primary xfs boot, type=83 4 22.6GB 146GB 123GB extended type=05 5 22.6GB 31.2GB 8595MB logical linux-swap(v1) type=82 6 31.2GB 47.3GB 16.1GB logical xfs type=83 7 47.3GB 58.0GB 10.7GB logical xfs type=83 8 59.1GB 145GB 85.9GB logical type=83 Hmm....looks like kB is the lowest unit well...nope... (parted) help unit unit UNIT set the default unit to UNIT UNIT is one of: s, B, kB, MB, GB, TB, compact, cyl, chs, %, kiB, MiB, GiB, TiB default is 'compact'... As for my other two sd's, they both start at 17.4kB -- that's not a 1MB boundary either... it's actually 34 sectors -- I guess thats a gpt size -- and boot drive is sdc... starting at the 63 sector Looking at the # blocks/partition and dividing by 2*1024*1024 (sectors->MB): fdisk -l /dev/sdc|cut -c41-49|grep -P '^\s*[0-9]+$'|while read secs;do printf "%10d %s\n" "$secs" "$(pcalc "$secs/(2*1024*1024)"|grep -P '= \d')" done 12699351 = 6.05552244186401 8305605 = 3.96042108535767 1060290 = 0.505585670471191 120278655 = 57.3533320426941 8393931 = 4.00253820419312 15735636 = 7.50333595275879 10490413 = 5.00221872329712 83883366 = 39.9987058639526 ---- I would say those are track aligned -- That's a 21st century 15K SAS drive. MB alignment?... I don't think even windows is that dumb. Track alignment is still an issue -- ask anyone running RAID -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2013-04-01 at 14:52 -0700, Linda Walsh wrote:
I would say those are track aligned -- That's a 21st century 15K SAS drive.
That's an illusion. Not true. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFaOxAACgkQtTMYHG2NR9VpFACdEFyy1ycom0U7IMwGRjD29dDX 5fAAn1ME0PRu9WTA9zoJ6bSNQvIFMndv =SmhP -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On Monday, 2013-04-01 at 14:52 -0700, Linda Walsh wrote:
I would say those are track aligned -- That's a 21st century 15K SAS drive.
That's an illusion. Not true.
--- yeah right... You realize that if it wasn't, you couldn't have synchronized RAID disks. Try using desktop drives sometime in an LSI controller -- let me know how many 'pass' it's internal validation. If a disk has an extra few ms of latency compared to other members of the same array, LSI will kick the disk out as bad. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2013-03-30 at 21:29 -0700, Linda Walsh wrote:
I've never encountered those symptoms on any drives I've benched, but also I'm almost always running Enterprise drives for reliability. Same for your tests? Brands? I'm usually Hitachi,
I only use seagate. The normal kind. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFYaeAACgkQtTMYHG2NR9VsQwCcD86PwI+WsofC3R3XSaDYYJv8 G58AnixUEVRA9AWLh7aZmRLxIM/X5mEl =E7g9 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----Original Message----- From: Felix Miata <mrmazda@earthlink.net> To: opensuse@opensuse.org Subject: Re: [opensuse] 12.3 + ntfs + ext4 + USB3 + different copying speeds Date: Sat, 30 Mar 2013 19:26:40 -0400 On 2013-03-30 23:38 (GMT+0100) Carlos E. R. composed:
Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.
Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start. -----Original Message----- Where LBA 0 is located is probably not that much relevant. However, the distance between LBA-(n) and LBA-(n+1) could be. their relative position (interleaving), the rotational speed, cache-size, and the speed of the OS-driver should be well balanced. Without it, it "kind of works", and high-performance hardware might even work worse than cots-hardware. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote: ----- Oh -- and check out "xlatencytop'... see what that tells you... its a fun one... -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 29/03/13 16:08, Linda Walsh wrote:
Basil Chupin wrote: -----
Oh -- and check out "xlatencytop'... see what that tells you...
its a fun one...
But of course..... Sheesh, why didn't I think of "xlatencytop"?! :-) (Ignore -- I'm having one of *those* days today :-) .) Thanks for the suggestion re 'xlatencytop' - I'll try this one as well. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
El 26/03/13 10:36, Basil Chupin escribió:
Ntfs-3g is a fuse based filesystem.
Thanks, Greg, but it's over it's over my head :-( . What's a fuse based filesystem?
A FUSE based filesystem is a Filesystem in Userspace.. Contrary to most filesystems that are linux kernel modules (refered as "kernel space"), ntfs-3g runs in userspace. just like any other application such as "ls" or "kdm". In a __very__ simplified dummy way it goes something like this: in-Kernel FUSE module (generic interface) --> NTFS-3G (userspace)--> YOU. (obviously they real chain is more complicated ;) ) This has a cost and is probably the reason why you will see reduced performance. although if they loss is too dramatic it might be a just a bug. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Content-ID: <alpine.LNX.2.00.1303312327190.21805@Telcontar.valinor> On Sunday, 2013-03-31 at 17:47 -0300, Cristian Rodríguez wrote:
This has a cost and is probably the reason why you will see reduced performance. although if they loss is too dramatic it might be a just a bug.
3 times more slow... I call that "dramatic" ;-) Several people have explained it in detail, it is not a bug. It is how things are, that's all. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFYqpMACgkQtTMYHG2NR9VByQCeOzVFTvwIijh3Uay7l6fEVy6m 56wAn38IJtzzM+6wgqv5nXbhl/XxOZMO =quO1 -----END PGP SIGNATURE-----
Basil Chupin wrote:
So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?!
Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off. -- Per Jessen, Zürich (1.6°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 27/03/13 00:13, Per Jessen wrote:
Basil Chupin wrote:
So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off.
"Et tu, [Brute]?" :-) Wot is a "user-space filesystem"? I do know that from my use of Dolphin to copy to files and the use of mc I found that Dolphin is like walking thru molasses in winter when it comes to copying some files compared to how fast mc handles them -- but this "user-space" thingie has me spaced-out :-) . Why should whatever it this "user-space" thingie work the same as what one gets when using USB2 and not what one would expect when using USB3? BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
I doubt a little bit that the user space part really affects the performance, at least I cannot see any valid reason from a programmers point of view. It is more likely that the problem comes from the fact that the ntfs file system driver is reverse engineered (ntfs is proprietary and not a documented open standard) and simply runs suboptimal compared to windows where the microsoft engineers know the innards of their own file system a knowledge which is intentionally not made available the people who wrote ntfs-3g. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Mar 26, 2013 at 9:53 AM, Martin Helm <martin@null-a.de> wrote:
I doubt a little bit that the user space part really affects the performance, at least I cannot see any valid reason from a programmers point of view.
I don't know if you do any kernel level programming, but the kernel / userspace boundary is common bottleneck concern for kernel devs. I may be dated in my knowledge and there are tricks that can be played by the fuse devs that might overcome part of this. (I don't know how fuse works in detail, so take this with a big grain of salt): In particular a lot of DMA chips only support the moving data from a disk controller to/from the 1st GB of ram, thus the kernel ram buffers are restricted to being in the first GB of ram. Userspace in theory never needs to directly access hardware, so it is free to use all of ram. That means that userspace and the kernel can't easily share ram. So a disk write with MC to ntfs-3g would be: MC -> kernel (fuse) -> ntfs-3g -> kernel (fuse) -> dma -> disk controller. If the userspace -> kernel calls involve making a copy of the data every time, then that is a lot of overhead compared to: MC -> kernel (ext4) -> dma -> disk controller. Note that the dma aspect is typically handled by off-cpu logic, so the above only has a single cpu managed data copy going on. In the ntfs-3g case, it has 3 cpu managed data copies going on. Thus ntfs-3g has 3x the overhead of ext4. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
MC -> kernel (fuse) -> ntfs-3g -> kernel (fuse) -> dma -> disk controller. That is not 3 times overhead as the copy in copy out happens in RAM not on the disk (several orders of magnitude faster than the fastest
Am 26.03.2013 15:30, schrieb Greg Freemyer: transfer possible to any hd). But I see a point here in the additional CPU cycles, maybe the CPU cannot cope. A simple "top" during a large file transfer to the ntfs would probably tell something here. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Greg Freemyer wrote:
On Tue, Mar 26, 2013 at 9:53 AM, Martin Helm <martin@null-a.de> wrote:
I doubt a little bit that the user space part really affects the performance, at least I cannot see any valid reason from a programmers point of view.
I don't know if you do any kernel level programming, but the kernel / userspace boundary is common bottleneck concern for kernel devs.
There was a paper published in the Communications of the ACM: http://dl.acm.org/citation.cfm?id=1774130 http://www.csl.sri.com/users/gehani/papers/SAC-2010.FUSE.pdf I'm pretty certain it concludes that user-space file-systems (FUSE specifically) do perform measurably worse than kernel-space filesystems. /Per -- Per Jessen, Zürich (1.2°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, Mar 26, 2013 at 11:18 AM, Per Jessen <per@computer.org> wrote:
Greg Freemyer wrote:
On Tue, Mar 26, 2013 at 9:53 AM, Martin Helm <martin@null-a.de> wrote:
I doubt a little bit that the user space part really affects the performance, at least I cannot see any valid reason from a programmers point of view.
I don't know if you do any kernel level programming, but the kernel / userspace boundary is common bottleneck concern for kernel devs.
There was a paper published in the Communications of the ACM:
http://dl.acm.org/citation.cfm?id=1774130 http://www.csl.sri.com/users/gehani/papers/SAC-2010.FUSE.pdf
Per, I really appreciate the link to that PDF. Figure 3B of the pdf is pretty clear fuse "can" be a bottleneck. In their test setup, a native filesystem with 50-100MB files writes at about 850MB/sec. A fuse filesystem writes at about 350 MB/sec. But even 350 MB/sec is way above what Basil is seeing, so the bottleneck should not be in fuse directly. That leaves Martin's theory that it is ntfs-3g itself which is not optimally written. fyi: per the paper, they used a standard single disk as their storage media so any speeds above 115 MB/sec are because of disk caches. That's why the relatively small 50-100MB files achieve such apparently incredible write speeds. With bigger files or with cache clear on read, they are seeing much less difference between native and fuse. Thus basically for real world situations the disk is the bottleneck, not fuse. On the other hand, for a high performance disk setup that might achieve 500 MB/sec or above, fuse would become a major bottleneck. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-03-26 at 12:28 -0400, Greg Freemyer wrote:
Per, I really appreciate the link to that PDF.
Figure 3B of the pdf is pretty clear fuse "can" be a bottleneck. In their test setup, a native filesystem with 50-100MB files writes at about 850MB/sec.
A fuse filesystem writes at about 350 MB/sec.
But even 350 MB/sec is way above what Basil is seeing, so the bottleneck should not be in fuse directly. That leaves Martin's theory that it is ntfs-3g itself which is not optimally written.
Or both. As I said, on my older computer writing to NTFS put the single core cpu at 100%. It could not write faster. I can not see the same in my current computer, it is much more powerful. So the next thing is for Basil to watch 'top' while he copies files. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFR0CQACgkQtTMYHG2NR9Wr4ACeLT/rXPi1Lod+sG8d7ZT9m1KP i+AAniBmUnV0ESCTCEKiSmUs5kVZ2i7z =w/tv -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-03-26 at 14:53 +0100, Martin Helm wrote:
I doubt a little bit that the user space part really affects the performance, at least I cannot see any valid reason from a programmers point of view. It is more likely that the problem comes from the fact that the ntfs file system driver is reverse engineered (ntfs is proprietary and not a documented open standard) and simply runs suboptimal compared to windows where the microsoft engineers know the innards of their own file system a knowledge which is intentionally not made available the people who wrote ntfs-3g.
Yes, I agree. I noticed in my older computer that writing big files via "ntfs-3g" maxed the CPU. Basil, check your CPU load when you copy files to NTFS, compare it with writing to ext4. And yes, you did not notice the problem when using usb3 because usb2 is "so slow" itself that you did not notice the problem. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFRtg4ACgkQtTMYHG2NR9UL7wCePM3fhosi/5Apwmhqyj4AUd/H +BkAn2m1/T3y6s18sA/djmc3WjTuEAe8 =WjHG -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
On 27/03/13 00:13, Per Jessen wrote:
Basil Chupin wrote:
So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off.
"Et tu, [Brute]?" :-)
Wot is a "user-space filesystem"?
Why should whatever it this "user-space" thingie work the same as what one gets when using USB2 and not what one would expect when using USB3?
As Greg suggested, maybe USB-2 is the bottleneck, so going USB-3 won't give you anything extra. -- Per Jessen, Zürich (1.5°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
Basil Chupin wrote:
On 27/03/13 00:13, Per Jessen wrote:
Basil Chupin wrote:
So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off.
"Et tu, [Brute]?" :-)
Wot is a "user-space filesystem"?
Why should whatever it this "user-space" thingie work the same as what one gets when using USB2 and not what one would expect when using USB3?
As Greg suggested, maybe USB-2 is the bottleneck, so going USB-3 won't give you anything extra.
... maybe USB-2 is the _next_ bottleneck ... -- Per Jessen, Zürich (1.4°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 27/03/13 01:26, Per Jessen wrote:
Per Jessen wrote:
Basil Chupin wrote:
On 27/03/13 00:13, Per Jessen wrote:
Basil Chupin wrote:
So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off. "Et tu, [Brute]?" :-)
Wot is a "user-space filesystem"?
Why should whatever it this "user-space" thingie work the same as what one gets when using USB2 and not what one would expect when using USB3? As Greg suggested, maybe USB-2 is the bottleneck, so going USB-3 won't give you anything extra. ... maybe USB-2 is the _next_ bottleneck ...
Folks....... I thank each and all for the input so far to my question re the copying speed between to ext4 and ntfs formatted partitions. But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....". I am using USB3. The new external Seagate is USB3 and is connected to an USB3 port on the motherboard. The HDDs from which I am copying the files to the external USB3 Seagate are all SATA3 HDDS and recognised by the kernel as having UDMAs of 133. Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit). With this result nothing makes sense......:-( . Confused even more..... BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
Folks....... I thank each and all for the input so far to my question re the copying speed between to ext4 and ntfs formatted partitions.
But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....".
Greg mentioned it already - "Assume your 28MB/sec performance is the fastest ntfs-3g can run on your computer. Usb-2 maxes out a little below that, so usb-2 was the bottleneck. Usb-3 is a lot faster, so now ntfs-3g is your bottleneck."
Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit).
Any chance that the connection to your external drive is flaky? If it's throwing lots of errors and causing retries, that could explain your problem. -- Per Jessen, Zürich (-1.5°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 27/03/13 18:43, Per Jessen wrote:
Basil Chupin wrote:
Folks....... I thank each and all for the input so far to my question re the copying speed between to ext4 and ntfs formatted partitions.
But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....". Greg mentioned it already -
"Assume your 28MB/sec performance is the fastest ntfs-3g can run on your computer. Usb-2 maxes out a little below that, so usb-2 was the bottleneck. Usb-3 is a lot faster, so now ntfs-3g is your bottleneck."
Either you are not reading what I wrote or I am not being articulate enough to explain what just happened when I used the command line command 'cp' :-( . Go to the bottom where I respond to your last para.....
Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit). Any chance that the connection to your external drive is flaky? If it's throwing lots of errors and causing retries, that could explain your problem.
Yesterday I bought a brand new 2TB external Seagate HDD which uses USB3 - repeat, USB3 - and which is connected to one of the USB3 - repeat, USB3 - ports on the motherboard using a brand new USB3 cable. The HDD is also using a brand new power pack for its power source. My original post asked the question as to why I was getting different transfer rates using the above setup when copying to an ext4 formatted partition on that external Seagate compared to when doing so to a ntfs formatted partition. I provided the transfer rates as displayed in mc when using mc to do the copying. I haven't even attempted to use Dolphin to do any copying because past exeprience has shown me that it's quicker to transfer a large file using the old morse code dots and dashes than use Dolphin :-) . Earlier today, using the exactly the same setup and without any changes of any description, I copied a 7.1GB file to the ntfs formatted partition and this took around 4 minutes to do the copying versus last night's effort using mc which took around 15 minutes (actually, closer to 20 mins I think) to do a similar operation. What is missing in my explanation of the situation to keep the issue of USB2 being brought up? BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 27/03/13 19:53, Basil Chupin wrote: [pruned]
My original post asked the question as to why I was getting different transfer rates using the above setup when copying to an ext4 formatted partition on that external Seagate compared to when doing so to a ntfs formatted partition. I provided the transfer rates as displayed in mc when using mc to do the copying.
I haven't even attempted to use Dolphin to do any copying because past exeprience has shown me that it's quicker to transfer a large file using the old morse code dots and dashes than use Dolphin :-) .
Earlier today, using the exactly the same setup and without any changes of any description, I copied a 7.1GB file to the ntfs formatted partition and this took around 4 minutes to do the copying versus last night's effort using mc which took around 15 minutes (actually, closer to 20 mins I think) to do a similar operation.
I just did some 'scientifically' based tests which showed that one must never rely on memory when providing details about what (may have) occurred :-) . I used a stopwatch to check how long it took to copy a file to (a) a partition formatted in ntfs on an external USB3 HDD and (b) how long to copy to the same HDD but to a partition formatted in ext4. I copied a 11.437GB file to the ntfs partition using 3 methods and the results were: 1. copy using cp command................. 4 min 59.1 sec 2. copy using mc................................ 6 min 29.3 sec 3. copy using Dolphin......................... 5 min 13 sec Then I copied the same file to the ext4 formatted partition and the results were: 1. using cp command....................... 2 min 17.3 sec 2. using mc...................................... 2 min 18.1 sec 3. using Dolphin............................... 2 min 18.2 sec
What is missing in my explanation of the situation to keep the issue of USB2 being brought up?
BC
BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
I just did some 'scientifically' based tests which showed that one must never rely on memory when providing details about what (may have) occurred :-) .
I used a stopwatch to check how long it took to copy a file to (a) a partition formatted in ntfs on an external USB3 HDD and (b) how long to copy to the same HDD but to a partition formatted in ext4.
I copied a 11.437GB file to the ntfs partition using 3 methods and the results were:
1. copy using cp command................. 4 min 59.1 sec 2. copy using mc................................ 6 min 29.3 sec 3. copy using Dolphin......................... 5 min 13 sec
38Mb/sec, 29Mb/sec, 36Mb/sec.
Then I copied the same file to the ext4 formatted partition and the results were:
1. using cp command....................... 2 min 17.3 sec 2. using mc...................................... 2 min 18.1 sec 3. using Dolphin............................... 2 min 18.2 sec
All three about 83Mb/sec. So ext4 is more than twice as fast. I would ignore the differences between cp/mc/dolphin - they're interesting, but not in this context. I would use 'dd' to test IO instead - dd if=infile of=outfile bs=xxxx Try varying the blocksize to see if it makes a difference. You could also try using another filesystems - xfs or jfs instead of ext4. Then redo your test. If they show the same performance as ext4, ntfs is just slow, methinks. -- Per Jessen, Zürich (3.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-03-27 at 22:45 +1100, Basil Chupin wrote:
I used a stopwatch to check how long it took to copy a file to (a) a partition formatted in ntfs on an external USB3 HDD and (b) how long to copy to the same HDD but to a partition formatted in ext4.
I copied a 11.437GB file to the ntfs partition using 3 methods and the results were:
1. copy using cp command................. 4 min 59.1 sec
2. copy using mc................................ 6 min 29.3 sec
3. copy using Dolphin......................... 5 min 13 sec
Then I copied the same file to the ext4 formatted partition and the results were:
1. using cp command....................... 2 min 17.3 sec
2. using mc...................................... 2 min 18.1 sec
3. using Dolphin............................... 2 min 18.2 sec
The results are not surprising at all. It is about what I expected. Or we executed, most of all have told you about the same things :-) It is very simple: writing files on ntfs on Linux is slow because it uses more cpu. It does not matter to you why exactly - if it does matter, then try to understand all that talk about userspace vs kernelspace and single-double-triple buffering and reverse-engineering. Why different copy programs behave differently means that they do the copy differently. We would need a dev looking at the code to learn why some are more affected than others by this operation. It is curious, though. Repeat the above test having in another terminal "top" running and see what it shows...
What is missing in my explanation of the situation to keep the issue of USB2 being brought up?
Somewhere you mentioned that you did not have/see this issue when you used USB2 or some similar words. We said that using USB2, as it is slow itself, you do not hit the problem that you see in your table of ntfs-3g being slow. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFTlmkACgkQtTMYHG2NR9XrJwCfZ2z/qTjGCmjaXvE6mZOnkoxK E/MAnRFzok4x9UtAXWfdjc8pezwqo3LC =XiL+ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 28/03/13 12:01, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wednesday, 2013-03-27 at 22:45 +1100, Basil Chupin wrote:
I used a stopwatch to check how long it took to copy a file to (a) a partition formatted in ntfs on an external USB3 HDD and (b) how long to copy to the same HDD but to a partition formatted in ext4.
I copied a 11.437GB file to the ntfs partition using 3 methods and the results were:
1. copy using cp command................. 4 min 59.1 sec
2. copy using mc................................ 6 min 29.3 sec
3. copy using Dolphin......................... 5 min 13 sec
Then I copied the same file to the ext4 formatted partition and the results were:
1. using cp command....................... 2 min 17.3 sec
2. using mc...................................... 2 min 18.1 sec
3. using Dolphin............................... 2 min 18.2 sec
The results are not surprising at all. It is about what I expected. Or we executed, most of all have told you about the same things :-)
It is very simple: writing files on ntfs on Linux is slow because it uses more cpu. It does not matter to you why exactly - if it does matter, then try to understand all that talk about userspace vs kernelspace and single-double-triple buffering and reverse-engineering.
Why different copy programs behave differently means that they do the copy differently. We would need a dev looking at the code to learn why some are more affected than others by this operation. It is curious, though.
Repeat the above test having in another terminal "top" running and see what it shows...
To see how much each of the 8 cores are being worked? OK, I'll try that later.
What is missing in my explanation of the situation to keep the issue of USB2 being brought up?
Somewhere you mentioned that you did not have/see this issue when you used USB2 or some similar words. We said that using USB2, as it is slow itself, you do not hit the problem that you see in your table of ntfs-3g being slow.
Ah, so that is what those references to USB2 were all about :-) . ================= Actually, there may be more to all of this than meets the eye - to use an old cliche. I bought the Seagate because I bought a 2TB WD just before Christmas and I could NOT copy files to it without getting errors - with the screen going blood red in mc and the HDD being unmounted. These errors occurred when copying to either the ext4 or the ntfs partitions on that WD. The errors I would get were: "kernel I/O error (5)" most of the times but there were other very similar ones. And after battling these hassles for 3 months I got fed up, swore never to buy another WD HDD and went out and bought the Seagate a couple of days ago. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-03-28 at 15:49 +1100, Basil Chupin wrote:
On 28/03/13 12:01, Carlos E. R. wrote:
Repeat the above test having in another terminal "top" running and see what it shows...
To see how much each of the 8 cores are being worked? OK, I'll try that later.
My guess is that one core will be maxed :-)
What is missing in my explanation of the situation to keep the issue of USB2 being brought up?
Somewhere you mentioned that you did not have/see this issue when you used USB2 or some similar words. We said that using USB2, as it is slow itself, you do not hit the problem that you see in your table of ntfs-3g being slow.
Ah, so that is what those references to USB2 were all about :-) .
Right!
=================
Actually, there may be more to all of this than meets the eye - to use an old cliche.
I bought the Seagate because I bought a 2TB WD just before Christmas and I could NOT copy files to it without getting errors - with the screen going blood red in mc and the HDD being unmounted. These errors occurred when copying to either the ext4 or the ntfs partitions on that WD. The errors I would get were: "kernel I/O error (5)" most of the times but there were other very similar ones. And after battling these hassles for 3 months I got fed up, swore never to buy another WD HDD and went out and bought the Seagate a couple of days ago.
Ouch. I only buy Seagate myself. But they have bought out several manufacturers, I lost count of them. Means they are better than the competition, but once the competitors disappear, they will have no reason to surpass themselves... I'm off to bed. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFTzYMACgkQtTMYHG2NR9VXCgCcCWrqgciWjfr4PkglZH3heEL5 4EYAn2BZXEkYUd0VaV0xorzll3ShmOPz =QflZ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2013-03-28 at 15:49 +1100, Basil Chupin wrote:
On 28/03/13 12:01, Carlos E. R. wrote:
Repeat the above test having in another terminal "top" running and see what it shows...
To see how much each of the 8 cores are being worked? OK, I'll try that later.
My guess is that one core will be maxed :-)
Ditto. -- Per Jessen, Zürich (1.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen said the following on 03/27/2013 03:43 AM:
"Assume your 28MB/sec performance is the fastest ntfs-3g can run on your computer. Usb-2 maxes out a little below that, so usb-2 was the bottleneck. Usb-3 is a lot faster, so now ntfs-3g is your bottleneck."
Maybe showing it as a table, max speed vs technology, will help The point we're trying to get across to Basil is that even using the fastest file system, if you're going through usb2 then it doesn't matter. Hey, Basil, Try This: run timing on those large writes to a variety of file systems on such a drive over a usb-2 connection. Then the same on a usb-3 connection. Tabulate. -- Vegetables are not food; vegetables are what food eats. Fruit are vegetables that fool you by tasting good. Fish are fast-moving vegetables. Mushrooms are what grows on vegetables when food's done with them. -- Meat Eater's Credo -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-03-27 17:54 (GMT+1100) Basil Chupin composed:
Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit).
Were you perchance using MC's ftp connection to make its copy, but a Samba or NFS connection for the cp? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 27/03/13 18:58, Felix Miata wrote:
On 2013-03-27 17:54 (GMT+1100) Basil Chupin composed:
Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit).
Were you perchance using MC's ftp connection to make its copy, but a Samba or NFS connection for the cp?
I go to the icon which will start the terminal console. I type in mc and mc runs. I highlight the destination directory, and I highlight the source directory and file. I press F5 and mc starts to copy the file from the source to the destination, and as the copying progresses I get displayed the speed of the transfer and how much of the file has been copied so far and how much is yet to be copied and the ETA of the transfer. Re using the cli 'cp', I open a terminal, type in 'cp <file-to-be-copied> <destination-for-the-copy> and press ENTER. Now, you tell me if I was using ftp connection or a Samba or NFS connection? Not that long ago I modified my sig line but I decided to bring it back to read: Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU which should avoid having to explain what else my system is using. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-03-27 20:16 (GMT+1100) Basil Chupin composed:
On 27/03/13 18:58, Felix Miata wrote:
On 2013-03-27 17:54 (GMT+1100) Basil Chupin composed:
Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit).
Were you perchance using MC's ftp connection to make its copy, but a Samba or NFS connection for the cp?
I go to the icon which will start the terminal console.
I type in mc and mc runs.
I highlight the destination directory, and I highlight the source directory and file.
I press F5 and mc starts to copy the file from the source to the destination, and as the copying progresses I get displayed the speed of the transfer and how much of the file has been copied so far and how much is yet to be copied and the ETA of the transfer.
Re using the cli 'cp', I open a terminal, type in 'cp <file-to-be-copied> <destination-for-the-copy> and press ENTER.
Now, you tell me if I was using ftp connection or a Samba or NFS connection?
Insufficient data. Open either left or right MC menu, and you'll find listed among others "ftP link". Using this can result in either source dir or destination dir or both accessed via ftp. This is normally used only for non-local filesystems, so I did and do not expect that you had. The word "perchance" implies weak likelihood, yet possible. For example, I use this (ftpfs) for fetching installation kernel and initrd from ftp5.gwdg.de's opensuse/factory subdir prior to beginning an HTTP oS installation. In your case I expected and expect not, but had to ask just in case, as it is a possible reason for a copy speed differential compared to using cl cp. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, 27 Mar 2013, Basil Chupin wrote:
... But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....".
...
Ext4 is implemented completely within the kernel. Ntfs-3g is implemented outside of the kernel and runs as a user process - Filesystem in userspace (FUSE). Each time a process such as cp makes a kernel system call to do a file operation on a FUSE based filesystem, the call goes into the kernel and then the kernel passes the call back to a normal process via the FUSE API. The normal process then does more kernel system calls to the real IO behind the file system operation. This means that parameters and data in Ntfs-3g may need to be copied between kernel and userspace possible multiple times, before it makes it anywhere useful. Plus this requires a hierarchy of calls into and out of the kernel, all involving synchronisation between what's happening inside the kernel and inside the userspace. Contrast this with Ext4, where a system call to do a file system operation will enter the kernel only once and will run to completion entirely with the kernel - much faster. The Wikipedia entry on NTFS-3G states the bottleneck may be the CPU required by the FUSE implementation and that a fast enough CPU will yield faster results. Considering this is supposed to be about I/O this implies big overheads. FUSE based files system implementations are easier to write and more portable than writting kernel code. It only took me a few days to learn about FUSE and prototype a simple FUSED based filesystem (collectfs). It's just like writing any program that runs as a user process. Where as I've never had enough time to climb the learning curve for implementing filesystems that run inside the kernel. If you compare the performance of a kernel based implementation of an encrypted filesystem to a FUSE based implementation, I think you will see similar differences/limitations in performance. In a nutshell, a faster CPU might help narrow the difference. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 28/03/13 11:36, Michael Hamilton wrote:
On Wed, 27 Mar 2013, Basil Chupin wrote:
... But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....".
... Ext4 is implemented completely within the kernel. Ntfs-3g is implemented outside of the kernel and runs as a user process - Filesystem in userspace (FUSE).
Each time a process such as cp makes a kernel system call to do a file operation on a FUSE based filesystem, the call goes into the kernel and then the kernel passes the call back to a normal process via the FUSE API. The normal process then does more kernel system calls to the real IO behind the file system operation. This means that parameters and data in Ntfs-3g may need to be copied between kernel and userspace possible multiple times, before it makes it anywhere useful. Plus this requires a hierarchy of calls into and out of the kernel, all involving synchronisation between what's happening inside the kernel and inside the userspace. Contrast this with Ext4, where a system call to do a file system operation will enter the kernel only once and will run to completion entirely with the kernel - much faster.
The Wikipedia entry on NTFS-3G states the bottleneck may be the CPU required by the FUSE implementation and that a fast enough CPU will yield faster results. Considering this is supposed to be about I/O this implies big overheads.
FUSE based files system implementations are easier to write and more portable than writting kernel code. It only took me a few days to learn about FUSE and prototype a simple FUSED based filesystem (collectfs). It's just like writing any program that runs as a user process. Where as I've never had enough time to climb the learning curve for implementing filesystems that run inside the kernel.
If you compare the performance of a kernel based implementation of an encrypted filesystem to a FUSE based implementation, I think you will see similar differences/limitations in performance.
Thanks for this clear explanation. Duly taken onboard.
In a nutshell, a faster CPU might help narrow the difference.
You mean that this now may not be enough to run something like openSUSE 12.3?! :-) : Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-03-28 at 15:36 +1100, Basil Chupin wrote:
On 28/03/13 11:36, Michael Hamilton wrote:
You mean that this now may not be enough to run something like openSUSE 12.3?! :-) :
Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU
Why don't you simply open a terminal and run 'top' inside when you do a copy to the ntfs partition, then compare to what happens copying to the ext4 partition? And then tell us? :-) It is the third time I suggest you do this... I did it on my machine, so I know what to expect. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFTzAEACgkQtTMYHG2NR9V2KgCfVjbTNtZdfShQoEeM2DM8z56e stkAn1OVCvHZYwslxZT8EZVTNpXrM+vk =+bPZ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 28/03/13 15:50, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2013-03-28 at 15:36 +1100, Basil Chupin wrote:
On 28/03/13 11:36, Michael Hamilton wrote:
You mean that this now may not be enough to run something like openSUSE 12.3?! :-) :
Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU
Why don't you simply open a terminal and run 'top' inside when you do a copy to the ntfs partition, then compare to what happens copying to the ext4 partition? And then tell us?
:-)
It is the third time I suggest you do this... I did it on my machine, so I know what to expect.
OK, OK, OK already! I'll do it! :-) BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-03-28 at 16:05 +1100, Basil Chupin wrote:
On 28/03/13 15:50, Carlos E. R. wrote:
OK, OK, OK already! I'll do it! :-)
I'll wait impatiently :-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFUVqIACgkQtTMYHG2NR9VVYQCfbfiMF3qL8JafcymmTeEyVIdi U+gAn3lXmub3p0Bni0vyK+Gv+BQ+EQAd =tUjh -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Thu, Mar 28, 2013 at 10:41 AM, Carlos E. R. <robin.listas@telefonica.net> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2013-03-28 at 16:05 +1100, Basil Chupin wrote:
On 28/03/13 15:50, Carlos E. R. wrote:
OK, OK, OK already! I'll do it! :-)
I'll wait impatiently :-)
I just tried it on my laptop - USB3 - ntfs I got 25-30 MB/sec and mount.ntfs had about a 75% cpu load on one core. (total cpu load went up about only about 15% with a quad core., so 0.15 * 8 = about 120% of one core total cpu load increase.) I used dd if=/dev/zero of=test bs=1M count=20000 to create a 20 GB test file. The same on my laptop's internal drive was about 75 MB/sec with roughly a 20% total cpu load (quad core, so that is about 160% of one core's load.) So the interesting thing is introduction of mount.ntfs actually lowered the total cpu load of running dd. I assume because it introduced delays that reduced the data movement so much. To repeat: with dd -> ext4 75 MB/sec data transfer and 160% cpu load (uses multiple cores. No single core overloaded.) with dd -> ntfs 25-30 MB/sec data transfer and only 120% cpu load (one core running mount.ntfs at 75% load) I actually find that kind of strange, but it was repeatable. I think it is pretty clear that mount.ntfs is the bottleneck. The open (and unimportant) question if it is just poorly written or if the fact it is based on fuse makes it inefficient no matter what. fyi: this is a quad core laptop, but I was running a 4 worker processor intensive app at the same time, so I had a 50% untilization of my cpus as a base load. The 15% and 20% overall loads I mention brought the base load up to 65% and 70% respectively. Since they were just getting the hyper-threading cpu bounce, the loads may results may of been different without the base load running in the background. I don't actually know how that works. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, 29 Mar 2013, Greg Freemyer wrote:
On Thu, Mar 28, 2013 at 10:41 AM, Carlos E. R. <robin.listas@telefonica.net> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2013-03-28 at 16:05 +1100, Basil Chupin wrote:
On 28/03/13 15:50, Carlos E. R. wrote:
OK, OK, OK already! I'll do it! :-)
I'll wait impatiently :-)
I just tried it on my laptop - USB3 - ntfs
I got 25-30 MB/sec and mount.ntfs had about a 75% cpu load on one core. (total cpu load went up about only about 15% with a quad core., so 0.15 * 8 = about 120% of one core total cpu load increase.)
I used dd if=/dev/zero of=test bs=1M count=20000 to create a 20 GB test file.
The same on my laptop's internal drive was about 75 MB/sec with roughly a 20% total cpu load (quad core, so that is about 160% of one core's load.)
So the interesting thing is introduction of mount.ntfs actually lowered the total cpu load of running dd. I assume because it introduced delays that reduced the data movement so much.
To repeat:
with dd -> ext4 75 MB/sec data transfer and 160% cpu load (uses multiple cores. No single core overloaded.)
with dd -> ntfs 25-30 MB/sec data transfer and only 120% cpu load (one core running mount.ntfs at 75% load)
I actually find that kind of strange, but it was repeatable.
I think it is pretty clear that mount.ntfs is the bottleneck. The open (and unimportant) question if it is just poorly written or if the fact it is based on fuse makes it inefficient no matter what.
fyi: this is a quad core laptop, but I was running a 4 worker processor intensive app at the same time, so I had a 50% untilization of my cpus as a base load. The 15% and 20% overall loads I mention brought the base load up to 65% and 70% respectively. Since they were just getting the hyper-threading cpu bounce, the loads may results may of been different without the base load running in the background. I don't actually know how that works.
Greg
If you take a quick look at the list of operations in the the FUSE API you will see it just mirrors UNIX basic filesystem operations: http://fuse.sourceforge.net/doxygen/structfuse__operations.html Each one of these is normally a kernel operation, but FUSE is having each one call back to a userspace process and move data between userspace and kernel space. It's enormously inefficient to move data in/out of the kernel perhaps several times when a normal system call would just do that once, or perhaps not at all if memory mapped I/O takes place. Memory allocation and reclaim would be another big overhead. In kernel I/O is nicely designed for speed with all sorts of cleverness going on (see Linux Kernel Development by Robert Love - actually a reasonably easy/entertaining read for anyone with some Comp Sci awareness). If FUSE could come anywhere near kernel speeds there would be something wrong with the kernel design. FUSE is about ease of coding and making anything pretend to be a filesystem, it is not about speed. If someone goes the FUSE route for a normal filesystem they're choosing to be slower. Perhaps someone does care about fast NTFS for Linux, but not enough to do all the work required for a kernel implementation. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 09:01 +1300, Michael Hamilton wrote:
FUSE is about ease of coding and making anything pretend to be a filesystem, it is not about speed. If someone goes the FUSE route for a normal filesystem they're choosing to be slower. Perhaps someone does care about fast NTFS for Linux, but not enough to do all the work required for a kernel implementation.
I have the feeling that there are also legal reasons involved. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFUrIwACgkQtTMYHG2NR9WmuACfQgcmmBRFlxnf13s9bX/7ar2A dSgAn27jlyzbG7aD9N9INpJTEJ/M4rD/ =2AUC -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Michael Hamilton said the following on 03/28/2013 04:01 PM:
FUSE is about ease of coding and making anything pretend to be a filesystem, it is not about speed.
See also such exemplary "file systems" using FUSE as layered on top of SSH. And then we have https://github.com/goraxe/fuse-dns DNS uses fuse to create a directory structure using dns lookups and or zone files Not quite a file system OVER DNS as a channel, but never the less, don't expect it to be fast :-) See also: AS3FS IMAPFS NNTPFS and other 'useful' file systems. The point is that its much easier to use FUSE than write a kernel driver. -- A distracted figure with a huge bushy beard blunders in just as you speak the word of ancient magic. The man wears loose clothing, and an expression of intense concentration. He is clutching his frizzy hair with one hand; his other hand grips an intricate grid - the object of his attention. His eyes brighten the word you've spoken reaches his ears. "Yes! Yes! That's it!" he exclaims as he draws out a pen and fills in a row of squares. "Now my hyperconstrained, double-acrostic, cryptic crossword is complete, and ready to puzzle others. That was all I needed - just a simple five-letter word, composed only of the letters 'X' 'Y' and 'Z,' that would fit here!" He grips your hand and shakes it fervently. "Thank you! Now that I've finished with that, I can get on to those other things I've been meaning to do, such as monkey-wrenching the demolition and saving recreational linguistics for future generations." He turns away and mutters, just before he departs, "I hope none of that will involve lying in front of a bulldozer..." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Greg Freemyer wrote:
To repeat:
with dd -> ext4 75 MB/sec data transfer and 160% cpu load (uses multiple cores. No single core overloaded.)
with dd -> ntfs 25-30 MB/sec data transfer and only 120% cpu load (one core running mount.ntfs at 75% load)
I actually find that kind of strange, but it was repeatable.
I think it is pretty clear that mount.ntfs is the bottleneck.
I guess mount.ntfs is the FUSE binary?
The open (and unimportant) question if it is just poorly written or if the fact it is based on fuse makes it inefficient no matter what.
The FUSE cuases two context switches per operation, it is bound to have an impact. A factor 2.5-3 seems a lot, though. -- Per Jessen, Zürich (3.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil, This is a pretty technical response and doesn't really address your needs, so feel free to ignore it. (Per has to read it!) Per Jessen <per@computer.org> wrote:
Greg Freemyer wrote:
To repeat:
with dd -> ext4 75 MB/sec data transfer and 160% cpu load (uses multiple cores. No single core overloaded.)
with dd -> ntfs 25-30 MB/sec data transfer and only 120% cpu load (one core running mount.ntfs at 75% load)
I actually find that kind of strange, but it was repeatable.
I think it is pretty clear that mount.ntfs is the bottleneck.
I guess mount.ntfs is the FUSE binary?
I don't think fuse has a binary. It is in the kernel and provides a kernel api. There is a fuse library that binaries can use. But as you know when you issue a write(2) in a normal user application, that has to be mapped to specific disk sectors and the underlying sector write performed. And of course if the file grew because the write(2) extended the file, then various metadata has to be updated. With ext4 all of that logic is handled by the ext4 kernel driver/module. With fuse, the logic is pushed to userspace. I believe that mount.ntfs is the program that implements the logic required to read/write a ntfs filesystem. It uses the fuse api, but I would not call it the fuse binary. Fyi: I have at least one other tool that uses fuse. Ewfmount. It is a far simpler tool than ntfs, so if you want to see some source that uses the fuse api that might be a good choice to look at. It is in the ewftools subpackage of libewf and is in OBS. (Or I assume the srpm is on the dvd? I do all my source level work in conjunction with OBS, so I'm not sure where the srpms are.)
The open (and unimportant) question if it is just poorly written or if the fact it is based on fuse makes it inefficient no matter what.
The FUSE cuases two context switches per operation, it is bound to have an impact. A factor 2.5-3 seems a lot, though.
The logic to implement a filesystem is very complex even for something like ext2. Ntfs is more complex than ext4. Probably closer in complexity to btrfs. Ie. It is extent based, has a journal, and at least in windows supports snapshots/shadow copies of files within the filesystem. So in my test I used dd with 1 MB writes. That data has to be copied to the kernel when write(2) is called. I assume with mount.ntfs that the kernel in turn has to copy that data back out to userspace. Once there, mount.ntfs has to organize it the way it will exist on the disk. When this is done in the kernel, it is all handled by maintaining a page buffer and all the shuffleing is done by working with pointers. That really only works because the disk controllers have dma circuitry that accepts arrays of pointers that tell it where to pull the data from. They call that scatter/gather dma. Note that by design, disk controllers only read/write data in kernel buffers, they never directly access user space buffers. So mount.ntfs doesn't have a dma circuit it can call on, instead once it has decided how to manipulate that 1MB of data dd sent it, it has to call back into the kernel for it to copy that data back down into the kernel for it to interface with the disk controller and initiate the actual dma activity that moves the data out to the disk controller. While all that is happening the disk is spinning. Let's say 6000 rpm for ease of math. At that speed it takes 10 msecs for one rotation of the drive. That is a really long time in the world of cpus. So if while all the above logic is going on, an extra disk rotation slips in every once in a while, it causes big delays. In the 70's it was not realistic to write data to consecutive sectors as the disk spun because cpus were just to slow. The solution was interleaving multiple data streams. You may recall we used to format drives with a interleave factor of 2 or even 3. With a interleave of 3, when the cpu wanted to write consecutive data out to the drive it would write to sector 1, then 4, then 7, etc. The drive is spinning as it writes, so if you had 250 sectors per cylinder, then that 6000 rpm drive sees a new sector pass under the read head every 40 microseconds. With a interleave of 3, that meant the cpu had 120 microseconds to get the next sector of data out to the drive controller. For the last 20 years or so, cpus have been fast enough to do the job in the original 40 microseconds, so sectors are no longer interleaved. Also sectors have gotten physically smaller and smaller, so if there are now 2500 sectors per track, that is a sector every 4 microseconds, but rotation speeds haven't changed so it still takes 10 milliseconds per rotation. So with modern drives, if you miss that 4 microsecond deadline, a huge 10 millisecond penalty is handed to you by the disk (since it has to go all the way around again to let you try again). Back to mount.ntfs. When you introduce the delays associated with all that extra data movement caused by fuse, i'm sure there are times that the 4 microsecond deadline is missed. Every time that happens, a 10 millisecond delay of game penalty is thrown. As you can see its a complex dance, and the surprise to me is not that there is a 3x slow down, but that it is not way worse than that. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 08:42 -0400, Greg Freemyer wrote: ...
In the 70's it was not realistic to write data to consecutive sectors as the disk spun because cpus were just to slow. The solution was interleaving multiple data streams. You may recall we used to format drives with a interleave factor of 2 or even 3.
And more. I had one machine that I had to interleave at 8 or more. I actually formatted it several times at all values from 1 to 12, and then choose the faster one. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFVoT0ACgkQtTMYHG2NR9XPYACgjf6cwLuclESWz05oZfv/a72r OnAAn00Mm5Y/Lv92akDZf+F0etK1do1H =pmwM -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Greg Freemyer wrote:
Greg Freemyer wrote:
To repeat:
with dd -> ext4 75 MB/sec data transfer and 160% cpu load (uses multiple cores. No single core overloaded.)
with dd -> ntfs 25-30 MB/sec data transfer and only 120% cpu load (one core running mount.ntfs at 75% load)
I actually find that kind of strange, but it was repeatable.
I think it is pretty clear that mount.ntfs is the bottleneck.
I guess mount.ntfs is the FUSE binary?
I don't think fuse has a binary. It is in the kernel and provides a kernel api. There is a fuse library that binaries can use.
fuse does have a binary, that is the user space part. (I wrote a fuse myself a couple of years ago). The user-space binary is basically the code for the various filesystem operations: getattr, readdir, opendir, releasedir, open, release, read etc.
With fuse, the logic is pushed to userspace. I believe that mount.ntfs is the program that implements the logic required to read/write a ntfs filesystem. It uses the fuse api, but I would not call it the fuse binary.
Okay, I would call that the binary, but it doesn't matter. -- Per Jessen, Zürich (3.4°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen said the following on 03/29/2013 10:25 AM:
fuse does have a binary, that is the user space part.
ULTIMATELY everything gets executed by a binary. You could write a FUSE FS in perl, python, scala, ruby or even, $DEITY forbid!, javascript, but ULTIMATELY the interpreters for those languages are binaries. It strikes me that first working out the logic & algorithms of a new FS in a HLL makes a lot of sense. The old rubric: First get it RIGHT then get it fast. -- Be extremely subtle, even to the point of formlessness. Be extremely mysterious, even to the point of soundlessness. Thereby you can be the director of the opponent's fate. Sun-tzu, The Art of War. Emptiness and Fullness -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 29/03/13 23:42, Greg Freemyer wrote:
Basil,
This is a pretty technical response and doesn't really address your needs, so feel free to ignore it.
You're right - I'm ignoring it :-) . @ Per: you asked for it! :-D . [pruned] BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 29/03/13 01:41, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2013-03-28 at 16:05 +1100, Basil Chupin wrote:
On 28/03/13 15:50, Carlos E. R. wrote:
OK, OK, OK already! I'll do it! :-)
I'll wait impatiently :-)
Well, I used both "top" and the System Monitor>System Load simultaneously (each running in a different workspace) to see what was happening during the copy processes using mc to copy the 11.1GB file to ext4 and ntfs partitions. Sorry to disappoint, but I could not see any core "maxing out" during copying. The load was being distributed across 5 or possibly 6 cores (with wither 3 or 2 being almost idle). With the data jumping around from core to core it was just a bit hard to keep up :-) , but I did see that the max any single core reached in both cases (ext4, ntfs) was 74% - with, as I said, either 3 or 2 cores doing almost nothing). BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 13:20 +1100, Basil Chupin wrote:
Well, I used both "top" and the System Monitor>System Load simultaneously (each running in a different workspace) to see what was happening during the copy processes using mc to copy the 11.1GB file to ext4 and ntfs partitions.
Sorry to disappoint, but I could not see any core "maxing out" during copying. The load was being distributed across 5 or possibly 6 cores (with wither 3 or 2 being almost idle). With the data jumping around from core to core it was just a bit hard to keep up :-) , but I did see that the max any single core reached in both cases (ext4, ntfs) was 74% - with, as I said, either 3 or 2 cores doing almost nothing).
Curious. So the load was distributed on the cores... interesting. That's more complicated to analyze. There may be other issues in play. I'm guessing memory moves, and thus, board and memory bandwidth, but dunno how it can be diagnosed. Alternatively, the task is jumping from core to core, but nevertheless it is not running paralellized. On average the load would be low, but the cpu where the job is running, peaks. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFVCBIACgkQtTMYHG2NR9X4bQCff4peU1u03zz0VJDCpXPwuhse 2jQAniriwgFSR6J3f4F2QEZRAmJpcvbi =brEY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 29/03/13 14:18, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Friday, 2013-03-29 at 13:20 +1100, Basil Chupin wrote:
Well, I used both "top" and the System Monitor>System Load simultaneously (each running in a different workspace) to see what was happening during the copy processes using mc to copy the 11.1GB file to ext4 and ntfs partitions.
Sorry to disappoint, but I could not see any core "maxing out" during copying. The load was being distributed across 5 or possibly 6 cores (with wither 3 or 2 being almost idle). With the data jumping around from core to core it was just a bit hard to keep up :-) , but I did see that the max any single core reached in both cases (ext4, ntfs) was 74% - with, as I said, either 3 or 2 cores doing almost nothing).
Curious.
So the load was distributed on the cores... interesting. That's more complicated to analyze. There may be other issues in play. I'm guessing memory moves, and thus, board and memory bandwidth, but dunno how it can be diagnosed. Alternatively, the task is jumping from core to core, but nevertheless it is not running paralellized. On average the load would be low, but the cpu where the job is running, peaks.
I found this in the wikipedia which does not bode well for the argument of FUSE versus in-kernel thinige: Performance Benchmarks show that the driver's performance via FUSE is comparable to that of other filesystems' drivers in-kernel,[6] provided that the CPU is powerful enough. On embedded or old systems, the high processor usage can severely limit performance.[7] Current versions often show 100% CPU utilization on dealing with big files on fragmented NTFS file systems.[8] http://en.wikipedia.org/wiki/Ntfs-3g This leads to the question whether will all the fiddling about with changing where the USB devices are now mounted (in places other then the normal /media) and other stuffing around with changed paths this has a detrimental effect on the way ntfs-3g is now handled in openSUSE. (BTW,in case someone brings up the issue of the performance being hampered because the ntfs partition(s) have not been defragmented, well before I redid the tests a short time ago I defragmented all ntfs formatted partitions. The results were almost identical to those of yesterday.) BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 20:48 +1100, Basil Chupin wrote:
I found this in the wikipedia which does not bode well for the argument of FUSE versus in-kernel thinige:
Performance
Benchmarks show that the driver's performance via FUSE is comparable to that of other filesystems' drivers in-kernel,[6] provided that the CPU is powerful enough. On embedded or old systems, the high processor usage can severely limit performance.[7] Current versions often show 100% CPU utilization on dealing with big files on fragmented NTFS file systems.[8]
It actually agrees with what we said :-) You do not see speed degradation _if_ the CPU is powerful enough. In other words, you need a powerful CPU to use FUSE fast. And this is specially true with ntfs-3g. And I think that you not also you not only need a powerful CPU, but also fast memory and buses.
This leads to the question whether will all the fiddling about with changing where the USB devices are now mounted (in places other then the normal /media) and other stuffing around with changed paths this has a detrimental effect on the way ntfs-3g is now handled in openSUSE.
No, not at all. That would not have any effect at all. Mounting options, yes. Another posibility is USB3. I don't know if the design works in hardware or needs lots of CPU assistance. Some of the USB2 designs need more CPU than others. The wikipedia may talk about this.
(BTW,in case someone brings up the issue of the performance being hampered because the ntfs partition(s) have not been defragmented, well before I redid the tests a short time ago I defragmented all ntfs formatted partitions. The results were almost identical to those of yesterday.)
Your partitions were fresh, you said, so that should not be an issue. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFVdBgACgkQtTMYHG2NR9XOlACeOKPM+//YweidG7fsVioChmwr W5AAn1omjd5VEJuxC2SvYUcClqkzDeIV =xq86 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 29/03/13 21:59, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Friday, 2013-03-29 at 20:48 +1100, Basil Chupin wrote:
I found this in the wikipedia which does not bode well for the argument of FUSE versus in-kernel thinige:
Performance
Benchmarks show that the driver's performance via FUSE is comparable to that of other filesystems' drivers in-kernel,[6] provided that the CPU is powerful enough. On embedded or old systems, the high processor usage can severely limit performance.[7] Current versions often show 100% CPU utilization on dealing with big files on fragmented NTFS file systems.[8]
It actually agrees with what we said :-)
You do not see speed degradation _if_ the CPU is powerful enough. In other words, you need a powerful CPU to use FUSE fast. And this is specially true with ntfs-3g.
Sorry, but right at this moment my finances do not permit me buying a Cray :-) . Maybe next month, perhaps :-) .
And I think that you not also you not only need a powerful CPU, but also fast memory and buses.
Ummm, an 8-core 3.6GHz cpu with (16GB of) 1866MHz RAM and with a bus which can do 5.2GT/s still too slow, eh? Ah well, next time I'll do all this downwind so that perhaps I'll get some assistance from the wind to speed things up :-) .
This leads to the question whether will all the fiddling about with changing where the USB devices are now mounted (in places other then the normal /media) and other stuffing around with changed paths this has a detrimental effect on the way ntfs-3g is now handled in openSUSE.
No, not at all. That would not have any effect at all. Mounting options, yes.
Another posibility is USB3. I don't know if the design works in hardware or needs lots of CPU assistance. Some of the USB2 designs need more CPU than others. The wikipedia may talk about this.
(BTW,in case someone brings up the issue of the performance being hampered because the ntfs partition(s) have not been defragmented, well before I redid the tests a short time ago I defragmented all ntfs formatted partitions. The results were almost identical to those of yesterday.)
Your partitions were fresh, you said, so that should not be an issue.
But the partitions from which I was copying the files were not and which is why I defragmented them just to see if that made a difference. Leave nothing to chance, I always say :-) . BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
You do not see speed degradation _if_ the CPU is powerful enough. In other words, you need a powerful CPU to use FUSE fast. And this is specially true with ntfs-3g.
Sorry, but right at this moment my finances do not permit me buying a Cray :-) . Maybe next month, perhaps :-) .
And I think that you not also you not only need a powerful CPU, but also fast memory and buses.
Ummm, an 8-core 3.6GHz cpu with (16GB of) 1866MHz RAM and with a bus which can do 5.2GT/s still too slow, eh?
Probably. Never mind your eight cores, it's only one that matters when the workload is single-threaded. Basil, it's a pretty longwinded thread by now, I am having trouble working out the current status. Am I right in thinking this is still about explaining why you get different IO-rates on two different filesystems (ext4 and ntfs)? -- Per Jessen, Zürich (3.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 31/03/13 22:00, Per Jessen wrote:
Basil Chupin wrote:
You do not see speed degradation _if_ the CPU is powerful enough. In other words, you need a powerful CPU to use FUSE fast. And this is specially true with ntfs-3g. Sorry, but right at this moment my finances do not permit me buying a Cray :-) . Maybe next month, perhaps :-) .
And I think that you not also you not only need a powerful CPU, but also fast memory and buses. Ummm, an 8-core 3.6GHz cpu with (16GB of) 1866MHz RAM and with a bus which can do 5.2GT/s still too slow, eh? Probably. Never mind your eight cores, it's only one that matters when the workload is single-threaded.
Well, as I mentioned 5 or 6 cores were involved and not just one single one. I haven' seen anything to convince me that only 1 single core was involved or should only be involved.
Basil, it's a pretty longwinded thread by now, I am having trouble working out the current status.
The thread has taken on a life of its own. Or as they say on YouTube, the thread has gone "viral".
Am I right in thinking this is still about explaining why you get different IO-rates on two different filesystems (ext4 and ntfs)?
Yes, from my point of view the question still remains as to why there is such a difference between transfer rates. The entry in the wikipedia is indicating that such a large difference should not exist. I am not a technician, but to me to read a byte of data and then write it to another device and then to verify that what was written to the destination is the same as the original doesn't require a ming-boggling process of what some people are indicating happens with FUSE vs the kernel-thing. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
Well, as I mentioned 5 or 6 cores were involved and not just one single one. I haven' seen anything to convince me that only 1 single core was involved or should only be involved.
I think we can be quite certain that the ntfs fuse will occupy a max of one core unless the IO is multi-threaded. (and a single copy-operation isn't multi-threaded). That you are seeing fuse apparently run on more than one core simply means it is not being dispatched on the same core every time.
Basil, it's a pretty longwinded thread by now, I am having trouble working out the current status.
The thread has taken on a life of its own. Or as they say on YouTube, the thread has gone "viral".
Don't we just call it off-topic here? :-)
Am I right in thinking this is still about explaining why you get different IO-rates on two different filesystems (ext4 and ntfs)?
Yes, from my point of view the question still remains as to why there is such a difference between transfer rates. The entry in the wikipedia is indicating that such a large difference should not exist.
I am not a technician, but to me to read a byte of data and then write it to another device and then to verify that what was written to the destination is the same as the original doesn't require a ming-boggling process of what some people are indicating happens with FUSE vs the kernel-thing.
It may not require it, but that is how it works with FUSE. I don't think the difference should be as much as a factor 2.5-3, but I can offer no explanation other than poor ntfs-3g performance, partially due to the FUSE-implementation. To see if ntfs-3g actually maxes out a core, try locking "mount.ntfs" to one core (after you have started the copying) by using taskset, e.g.: taskset -p 0x1 $(pidof mount.ntfs) Then take a look at core#0 using 'top'. You can add cores to the mix using taskset -p 0x3 $(pidof mount.ntfs) (core#1+2) See 'man taskset'. -- Per Jessen, Zürich (5.3°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-04-04 at 20:35 +1100, Basil Chupin wrote:
On 31/03/13 22:00, Per Jessen wrote:
Basil Chupin wrote:
Probably. Never mind your eight cores, it's only one that matters when the workload is single-threaded.
Well, as I mentioned 5 or 6 cores were involved and not just one single one. I haven' seen anything to convince me that only 1 single core was involved or should only be involved.
Appeareances aside, it is only one core. I mean, the important process is not working in parallell on several cores, which would speed it up (hopefully), but instead it runs in one core, then jumps to another core, then jumps to another core. If this is happening fast, what you see is a small load on all cores.
Am I right in thinking this is still about explaining why you get different IO-rates on two different filesystems (ext4 and ntfs)?
Yes, from my point of view the question still remains as to why there is such a difference between transfer rates. The entry in the wikipedia is indicating that such a large difference should not exist.
I read is as the contrary :-)
I am not a technician, but to me to read a byte of data and then write it to another device and then to verify that what was written to the destination is the same as the original doesn't require a ming-boggling process of what some people are indicating happens with FUSE vs the kernel-thing.
With FUSE there are several more transfers, from memory to memory. Memory is fast, but it adds. Another issue that can happen is that the delay can be small, but enough so that the head in the rotating disk has already moved to the next sector by the time the kernel is ready to write it, and thus it has to wait for a full rotation before it writes. This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFe+SAACgkQtTMYHG2NR9Wj3wCdE5ksxaPmZkqJ95KPqgx9lUhB 88YAnj+ub1En41+iWR6aVS97MSogKl39 =4vQA -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes.
Disks don't do write-caching. That is left to hardware controllers with battery-backed cache. -- Per Jessen, Zürich (4.4°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Apr 5, 2013 at 12:27 PM, Per Jessen <per@computer.org> wrote:
Carlos E. R. wrote:
This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes.
Disks don't do write-caching. That is left to hardware controllers with battery-backed cache.
ISTR that sata drives have a very small write cache. For some reason I'm remembering up to 7 writes can be cached. For streamed data like BC is talking about, that is effectively nothing. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Greg Freemyer wrote:
On Fri, Apr 5, 2013 at 12:27 PM, Per Jessen <per@computer.org> wrote:
Carlos E. R. wrote:
This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes.
Disks don't do write-caching. That is left to hardware controllers with battery-backed cache.
ISTR that sata drives have a very small write cache. For some reason I'm remembering up to 7 writes can be cached. For streamed data like BC is talking about, that is effectively nothing.
Does it need to be explicitly enabled? Seems odd for the drive to report "data written to disk" unless it has? -- Per Jessen, Zürich (0.0°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Apr 5, 2013 at 3:39 PM, Per Jessen <per@computer.org> wrote:
Greg Freemyer wrote:
On Fri, Apr 5, 2013 at 12:27 PM, Per Jessen <per@computer.org> wrote:
Carlos E. R. wrote:
This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes.
Disks don't do write-caching. That is left to hardware controllers with battery-backed cache.
ISTR that sata drives have a very small write cache. For some reason I'm remembering up to 7 writes can be cached. For streamed data like BC is talking about, that is effectively nothing.
Does it need to be explicitly enabled? Seems odd for the drive to report "data written to disk" unless it has?
I just used hdparm -I /dev/sda to check my laptop drive. It has a 8MB cache and it is enabled. I'm running a default openSUSE install, so I'd say it defaults to on with openSUSE. OTOH, read http://yarchive.net/comp/linux/drive_caches.html. There Linus says that the cache is typically made of large segments on the order of an entire track. I think a full track is about 1 MB currently, so I've got roughly 8 tracks worth of write cache on my low end laptop drive. If I'm doing linear writes like in BC's workload, it should work great at accepting more data. If a single sectors data was late arriving, the cpu could just keep pushing data out while the platter is rotating back around. That is the cpu should not get hit with a 10 msec penalty just because it was a microsecond late pushing out a given sector. Instead the cpu can just go as fast as it can as long as there is a cache segment free. In the BC workload case where the drive is not running at full speed, there is probably always a few free segments in the drives cache. I "think" there are cache flush commands the kernel can push to the drive to ensure the cache is flushed prior to moving forward. I don't know if the linux kernel actually pushes those types of barrier calls out to sata drives or not. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
В Fri, 5 Apr 2013 16:00:49 -0400 Greg Freemyer <greg.freemyer@gmail.com> пишет:
I "think" there are cache flush commands the kernel can push to the drive to ensure the cache is flushed prior to moving forward. I don't know if the linux kernel actually pushes those types of barrier calls out to sata drives or not.
It does as long as drive claims to support FUA. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
BC, I happened to need to copy 1 TB of data between 2 NTFS USB-3 drives today. I only had a windows laptop handy and being who I am, I decided to boot a openSUSE boot cd/dvd instead of using windows. The boot cd/dvd I had handy was OS 12.2 based. I had built it myself (http://susestudio.com/a/eD1wrT/dfir-opensuse-gnome-desktop-32bit). I used the CLI and did cp -a /mnt/data /mnt1/data. I was running without X, just the alt-f2 style terminals. I got 90 MB/sec at the start of the process (per iostat -d 10). My data files were all 1.4GB each, so this was a large file copy test only. I haven't had the opportunity to test this with OS 12.3, but I'm very curious if OS 12.3 is slower than OS 12.2 for this or if you have something weird on your PC. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 11/04/13 13:26, Greg Freemyer wrote:
BC,
I happened to need to copy 1 TB of data between 2 NTFS USB-3 drives today. I only had a windows laptop handy and being who I am, I decided to boot a openSUSE boot cd/dvd instead of using windows.
The boot cd/dvd I had handy was OS 12.2 based. I had built it myself (http://susestudio.com/a/eD1wrT/dfir-opensuse-gnome-desktop-32bit).
I used the CLI and did cp -a /mnt/data /mnt1/data. I was running without X, just the alt-f2 style terminals.
I got 90 MB/sec at the start of the process (per iostat -d 10). My data files were all 1.4GB each, so this was a large file copy test only.
I haven't had the opportunity to test this with OS 12.3, but I'm very curious if OS 12.3 is slower than OS 12.2 for this or if you have something weird on your PC.
Well, after after you wrote I thought that I would do a test or two. Just to refresh the memories, I have an external USB3 HDD (a Seagate 2TB) which I split into 2 partitions: partition 1 formatted in ext4 and partition 2 formatted in ntfs. Copying files from the system's internal HDDs (2X SATA3 1TB Seagates) to the ntfs formatted partition on the external USB3 HDD in earlier attempts, using both the cp CLI command and copying using my preferred file manager mc (Midnight commander) produced: ~25MB/s using mc; and ~31MB/s using the CLI cp command Copying the same file to an USB3 memory stick formatted in ext4 produced ~73MB/s. After reading your post I booted the computer using SystemRescueCD v3.1.2, mounted the partitions on the internal HDDs and the ntfs and the ext4 partiotions on the external HDD and went thru the same exercise as above. The results were little different to the ones I quote above. On the other hand, copying the same large file 7,631,404,437 bytes big, to the partition on the external HDD formatted in ext4 produced a transfer speed of ~79MB/s (which was a bit faster than when, in the earlier trial I copied the file to an USB memory stick, I got a transfer speed of ~73MB/s). BTW, you mention at the beginning of your message that "I got 90 MB/sec at the start of the process " - but you don't mention what you finally ended up with :-) . For the first couple of seconds the thruput is mind-boggling while the cache in the HDD is being populated but after that...... :-) . And so....what has been resolved? Anything? Rather interesting that nobody with any "inside knowledge" about ntfs-3g is put together has commented or made suggestions as to why there are such differences in the transfer rates - especially in the light of what is written in the wikipedia (which I quoted). BC -- Using openSUSE 12.3 x86_64 KDE 4.10.2 & kernel 3.8.6-2 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-04-12 16:55 (GMT+1000) Basil Chupin composed:
Just to refresh the memories, I have an external USB3 HDD (a Seagate 2TB) which I split into 2 partitions: partition 1 formatted in ext4 and partition 2 formatted in ntfs.
Presumably, each is about half the entire HD? If so, and #1 is at the front of the HD and #2 brings up the rear, then you can expect #2 to be slower even if using the same filesystem type as #1. http://fm.no-ip.com/PC/bench/Sysbench/resul606.txt is typical of benchmarking I've done in the past, and clearly shows a big speed advantage near the front on two different 160GB HDs. I tried using the dbench utility to test my own Seagate 2TB HD on eSATA that is half EXT2 and half NTFS, but it produced even more bizarre results: front/EXT2 half of HD: http://fm.no-ip.com/PC/bench/dbench-ext2.txt Throughput 431.269 MB/sec last/NTFS half of HD: http://fm.no-ip.com/PC/bench/dbench-ntfs.txt Throughput 26.5151 MB/sec -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Felix Miata wrote:
On 2013-04-12 16:55 (GMT+1000) Basil Chupin composed:
Just to refresh the memories, I have an external USB3 HDD (a Seagate 2TB) which I split into 2 partitions: partition 1 formatted in ext4 and partition 2 formatted in ntfs.
Presumably, each is about half the entire HD? If so, and #1 is at the front of the HD and #2 brings up the rear, then you can expect #2 to be slower even if using the same filesystem type as #1.
Yes, I saw this too a while ago when I was testing some 3Tb drives. I don't think the differences were as significant as what Basil's been seeing though. -- Per Jessen, Zürich (10.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/04/13 18:54, Per Jessen wrote:
Felix Miata wrote:
On 2013-04-12 16:55 (GMT+1000) Basil Chupin composed:
Just to refresh the memories, I have an external USB3 HDD (a Seagate 2TB) which I split into 2 partitions: partition 1 formatted in ext4 and partition 2 formatted in ntfs. Presumably, each is about half the entire HD? If so, and #1 is at the front of the HD and #2 brings up the rear, then you can expect #2 to be slower even if using the same filesystem type as #1. Yes, I saw this too a while ago when I was testing some 3Tb drives. I don't think the differences were as significant as what Basil's been seeing though.
Look, I got all hot and bothered about this toing and froing about whether the front part of the HDD was formatted in whatever versus the backup being formatted in whatever (I have been reading the posts about the 1/3 part of an HDD being faster than the rest while someone else claimed that it is the 1/4 of the HDD which was the fastest while another claimed that it was all BS because the latest HDD don't do what the HDD did years ago, and so on and so forth, and dominus vobiscum etc.... So here is what I did. I took the 2TB external HDD and format the whole damn thing in (a) ntfs and then the whole damn thing in (b) ext4. I then used SystemRescueCD to copy the same file I mentioned in the earlier post (7,631,404,437 bytes) sitting on an *ntfs* formatted partition on an internal 1TB HDD to the external USB3 2TB HDD. With SystemRescueCD there was NO graphics involved. Copying the 7.6 GB file using mc (midnight commander) to the external HDD *all* of which was in ntfs, produced a result of ~27.4MB/s transfer rate. After I formatted the whole 2TB HDD in ext4, copying the same file using mc produced a transfer rate of ~79MB/s. In all these trials the same internal HDDs were used, the same USB3 cable was used, the same file(s) were copied using either the CLI command 'cp', or the app 'mc', and in the very latest trials (as in earlier today and tonight) using SystemRescue CD which does not use the openSUSE 12.2 or the 12.3 kernels and whatevers. The summary of all the results is that copying to an external USB3 device formatted in ntfs is 34% SLOWER than when copying the identical file to an ext4 formatted partition. Now, I am nowhere close to being an expert in any field, but trying to accept that copying a file to partition formatted in ntfs as against copying to an ext4 partition incurs an overhead of 34% is just a bit too much to accept gracefully. However, as I said, I am not an expert and will accept this (even though wikipedia doesn't support this difference in speeds). OK , taking all this into account, what can be wrong at *my* end, in my system, which could cause such a speed difference? BC -- Using openSUSE 12.3 x86_64 KDE 4.10.2 & kernel 3.8.6-2 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
OK , taking all this into account, what can be wrong at *my* end, in my
system, which could cause such a speed difference?
BC
Basil, I have to copy 500 GB of data from ntfs to ntfs via usb3 today. I will use opensuse 12.3 this time. Separately, ntfs has a auto-document indexing feature. I keep it disabled by default. Do you know if you have it enabled? I think a vanilla mkfs.ntfs has it enabled. There is a flag to disable it. I did not think opensuse even had support for that ntfs feature, but it is one of the few issues I can imagine it being. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 12/04/13 23:36, Greg Freemyer wrote:
OK , taking all this into account, what can be wrong at *my* end, in my
system, which could cause such a speed difference?
BC Basil,
I have to copy 500 GB of data from ntfs to ntfs via usb3 today.
I will use opensuse 12.3 this time.
Separately, ntfs has a auto-document indexing feature. I keep it disabled by default. Do you know if you have it enabled? I think a vanilla mkfs.ntfs has it enabled. There is a flag to disable it. I did not think opensuse even had support for that ntfs feature, but it is one of the few issues I can imagine it being.
Never heard of this "auto-document indexing" feature and, therefore, don't know if it is enabled or not. Where can one find this flag and how can one disable it if it does exist? BC -- Using openSUSE 12.3 x86_64 KDE 4.10.2 & kernel 3.8.6-2 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, Apr 12, 2013 at 9:36 AM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
OK , taking all this into account, what can be wrong at *my* end, in my
system, which could cause such a speed difference?
BC
Basil,
I have to copy 500 GB of data from ntfs to ntfs via usb3 today.
I will use opensuse 12.3 this time.
Separately, ntfs has a auto-document indexing feature. I keep it disabled by default. Do you know if you have it enabled? I think a vanilla mkfs.ntfs has it enabled. There is a flag to disable it. I did not think opensuse even had support for that ntfs feature, but it is one of the few issues I can imagine it being.
Greg
I had no speed issues with OS 12.3 either, so both OS 12.2 and 12.3 seem fine to me. ie. 90 MB/sec reading from NTFS USB-3 and writing to NTFS USB-3 via a simple cp -a command. This is with different disks, cables and laptop from what I did yesterday. (Yesterday was a 12 month old HP laptop. Today was a 18-month old Dell Laptop. Both have native / built-in USB-3 support for at least 2 ports.) Both yesterday and today, I was copying 1.4GB files. Yesterday I did about 1 TB, today turned out to only be 80 GB, but still big enough to be a good test. Basil, since you're willing to reformat the drive, try to format it with: mkfs.ntfs --no-indexing --fast /dev/sdb1 (Or whatever your device is). The drives I am reading / writing were formatted by the manufacturer, so I don't know off hand if they have the indexing flag set. And I only know how to check it via windows. In windows you just go to the properties window and there is a check box for indexing. Also, can you try a series of test like this with your 8 GB file: time dd if=<ext4-source-file> of=<ntfs-destfile> bs=4K conv=fdatasync time dd if=<ext4-source-file> of=<ntfs-destfile> bs=8K conv=fdatasync time dd if=<ext4-source-file> of=<ntfs-destfile> bs=1M conv=fdatasync Obviously replace the filenames with appropriate paths. You should not see much difference in throughput for those, but if something is wrong with the caching / buffering / readahead on your machine, maybe you will. To be honest I'm running out of ideas why your box is slow for this. Greg Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/04/13 02:49, Greg Freemyer wrote:
On Fri, Apr 12, 2013 at 9:36 AM, Greg Freemyer <greg.freemyer@gmail.com> wrote:
OK , taking all this into account, what can be wrong at *my* end, in my
system, which could cause such a speed difference?
BC Basil,
I have to copy 500 GB of data from ntfs to ntfs via usb3 today.
I will use opensuse 12.3 this time.
Separately, ntfs has a auto-document indexing feature. I keep it disabled by default. Do you know if you have it enabled? I think a vanilla mkfs.ntfs has it enabled. There is a flag to disable it. I did not think opensuse even had support for that ntfs feature, but it is one of the few issues I can imagine it being.
Greg
I had no speed issues with OS 12.3 either, so both OS 12.2 and 12.3 seem fine to me. ie. 90 MB/sec reading from NTFS USB-3 and writing to NTFS USB-3 via a simple cp -a command.
This is with different disks, cables and laptop from what I did yesterday. (Yesterday was a 12 month old HP laptop. Today was a 18-month old Dell Laptop. Both have native / built-in USB-3 support for at least 2 ports.)
Both yesterday and today, I was copying 1.4GB files. Yesterday I did about 1 TB, today turned out to only be 80 GB, but still big enough to be a good test.
Basil, since you're willing to reformat the drive, try to format it with:
mkfs.ntfs --no-indexing --fast /dev/sdb1 (Or whatever your device is).
OK, Time out! Stop the clock! Yesterday I did NOT do the above - reformat the drive(s) with the above cli but what I *did* do is to: a) hook up the *second* external HDD formatted in ntfs, and transferred the "test" file of 7,631,404,437 bytes to it from the internal SATA3 HDD partition which is formatted in ntfs - the transfer rate was the usual ~27MB/s; b) I then emulated what you did which was to cp that file from one external USB3 HDD to the other USB3 HDD using the SystemRescueCD and got a transfer rate of 109.0MB/s. Now, I simply do not understand any of this nor have an explanation for this - except to say that before I did the above test there were some updates made to the 12.3 - but this should not have anything to do with it because I used SystemRescueCD which is using a different version of the kernel and the 12.3 OS doesn't have anything to do with the copying processing - or those it? I really don't know. [pruned] BC -- Using openSUSE 12.3 x86_64 KDE 4.10.2 & kernel 3.8.7-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
Copying the 7.6 GB file using mc (midnight commander) to the external HDD *all* of which was in ntfs, produced a result of ~27.4MB/s transfer rate.
After I formatted the whole 2TB HDD in ext4, copying the same file using mc produced a transfer rate of ~79MB/s.
In all these trials the same internal HDDs were used, the same USB3 cable was used, the same file(s) were copied using either the CLI command 'cp', or the app 'mc', and in the very latest trials (as in earlier today and tonight) using SystemRescue CD which does not use the openSUSE 12.2 or the 12.3 kernels and whatevers.
The summary of all the results is that copying to an external USB3 device formatted in ntfs is 34% SLOWER than when copying the identical file to an ext4 formatted partition.
Now, I am nowhere close to being an expert in any field, but trying to accept that copying a file to partition formatted in ntfs as against copying to an ext4 partition incurs an overhead of 34% is just a bit too much to accept gracefully. However, as I said, I am not an expert and will accept this (even though wikipedia doesn't support this difference in speeds).
OK , taking all this into account, what can be wrong at *my* end, in my system, which could cause such a speed difference?
The bottle-neck is quite clearly in the writing to the NTFS filesystem, which means in the NTFS user-space side. I would try, as I think have suggested before, tying the mount.ntfs process to one core. It's hard to imagine switching core would cause this much of a reduction in performance, but when Greg sees much better numbers with a similar copy operation, we're running out of options. -- Per Jessen, Zürich (13.4°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, 12 Apr 2013 23:24:04 +1000 Basil Chupin wrote: <snipped>
Copying the 7.6 GB file using mc (midnight commander) to the external HDD *all* of which was in ntfs, produced a result of ~27.4MB/s transfer rate.
After I formatted the whole 2TB HDD in ext4, copying the same file using mc produced a transfer rate of ~79MB/s. <snipped> The summary of all the results is that copying to an external USB3 device formatted in ntfs is 34% SLOWER than when copying the identical file to an ext4 formatted partition.
Hi Basil, I admire your persistence, if not your math. :-) 27.4 MB/sec is 34.7% of 79.0 MB/sec. That's almost a 3:1 ratio, ext4:ntfs, so copying to the ntfs formatted device takes almost three times as long. *Now* I 'grok' the impetus of this investigation. It's 65.3% slower, not "34% SLOWER"! Great work, otherwise. Thanks for sharing. regards, Carl -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-04-12 10:54 (GMT+0200) Per Jessen composed:
Felix Miata wrote:
Presumably, each is about half the entire HD? If so, and #1 is at the front of the HD and #2 brings up the rear, then you can expect #2 to be slower even if using the same filesystem type as #1.
Yes, I saw this too a while ago when I was testing some 3Tb drives. I don't think the differences were as significant as what Basil's been seeing though.
Looks like you couldn't be bothered to open the link I provided to see what to most would be a significant difference, so I have to paste relevant excerpts from it: Disk I/O disk 0-1: Track 0 xfer rate fwd : 74.301 Megabytes/second Middle trk rate fwds. : 61.215 Megabytes/second Last track rate bwds. : 35.128 Megabytes/second Disk I/O disk 1-2: Track 0 xfer rate fwd : 60.338 Megabytes/second Middle trk rate fwds. : 49.043 Megabytes/second Last track rate bwds. : 34.297 Megabytes/second Those number are from drives 5+ years old. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Basil Chupin wrote:
This leads to the question whether will all the fiddling about with changing where the USB devices are now mounted (in places other then the normal /media) and other stuffing around with changed paths this has a detrimental effect on the way ntfs-3g is now handled in openSUSE.
Mounting point and filesystem paths have no bearing the performance. If you tie the ntfs-3g FUSE to one core (taskset -p), and repeat the copy operation (cp or dd), I expect you will see one core maxed out. The FUSE itself is not multi-threading, but it will handle multiple concurrent requests. One copy operation is almost certainly single-threaded, though. -- Per Jessen, Zürich (3.6°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, 29 Mar 2013, Basil Chupin wrote:
I found this in the wikipedia which does not bode well for the argument of FUSE versus in-kernel thinige:
Performance
Benchmarks show that the driver's performance via FUSE is comparable to that of other filesystems' drivers in-kernel,[6] provided that the CPU is powerful enough. On embedded or old systems, the high processor usage can severely limit performance.[7] Current versions often show 100% CPU utilization on dealing with big files on fragmented NTFS file systems.[8]
I read that too. Experience doesn't seem to bear this out. Have you checked the mount options? Especially related to things like sync and read/write sizes. Enter mount when the file system is mounted and see what options are enabled. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Friday, 2013-03-29 at 13:20 +1100, Basil Chupin wrote:
Well, I used both "top" and the System Monitor>System Load simultaneously (each running in a different workspace) to see what was happening during the copy processes using mc to copy the 11.1GB file to ext4 and ntfs partitions.
Sorry to disappoint, but I could not see any core "maxing out" during copying. The load was being distributed across 5 or possibly 6 cores (with wither 3 or 2 being almost idle). With the data jumping around from core to core it was just a bit hard to keep up :-) , but I did see that the max any single core reached in both cases (ext4, ntfs) was 74% - with, as I said, either 3 or 2 cores doing almost nothing).
Curious.
So the load was distributed on the cores... interesting. That's more complicated to analyze. There may be other issues in play. I'm guessing memory moves, and thus, board and memory bandwidth, but dunno how it can be diagnosed. Alternatively, the task is jumping from core to core, but nevertheless it is not running paralellized.
Every time the user-space filesystems needs to talk to the kernel, it needs two context-switches. (from unprivileged to privileged and back). If the FUSE does not have processor affinity, I guess it could be hopping about between processors. -- Per Jessen, Zürich (3.6°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-03-26 23:57 (GMT+1100) Basil Chupin composed:
I use mc (Midnight Commander) 99.99% of the time to manage my files.
What I found - and which is what I am now asking about - is that when, using mc, that when copying files to the *ext4* formatted partition the files where being copied across at ~80MB/s. The files copied over were a mixture of ntfs and ext4 files (which I had copied from a collapsing 32-bit system to my current *Linux* system last year when I replaced the old computer).
However, when I went to copy files to the *ntfs* formatted part of the new external HDD, the best transfer rate I got was ~28MB/s - which looks very much like what I used to get when using USB2!
Be glad you're not stuck needing to copy large media files from an AZBox running 2.6.15 kernel @ 3.2MB/s, like I am. Check out this hdparm -i AZBox output: PIO modes: pio0 pio3 pio4 DMA modes: *mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 * current active mode -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2013-03-26 23:57 (GMT+1100) Basil Chupin composed:
I use mc (Midnight Commander) 99.99% of the time to manage my files.
What I found - and which is what I am now asking about - is that when, using mc, that when copying files to the *ext4* formatted partition the files where being copied across at ~80MB/s. The files copied over were a mixture of ntfs and ext4 files (which I had copied from a collapsing 32-bit system to my current *Linux* system last year when I replaced the old computer).
However, when I went to copy files to the *ntfs* formatted part of the new external HDD, the best transfer rate I got was ~28MB/s - which looks very much like what I used to get when using USB2!
Connected to a SiI3512 eSATA port I have running @UDMA5 a 2TB Seagate divided 50/50 between NTFS (end partition) and EXT2 (start partition). Booted to 12.1's 3.1.10-1.16, on large file rsync from NTFS I got 54.22MB/s (top showing rsync using ~76% CPU) while from EXT2 only 50.16MB/s (top showing rsync using ~72% CPU). -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (18)
-
Andrey Borzenkov -
Anton Aylward -
Basil Chupin -
Bernhard Voelker -
Carl Hartung -
Carlos E. R. -
Cristian Rodríguez -
Dave Howorth -
Felix Miata -
Greg Freemyer -
Hans Witvliet -
James Knott -
John Perry -
Ken Schneider - openSUSE -
Linda Walsh -
Martin Helm -
Michael Hamilton -
Per Jessen