On 27/03/13 00:13, Greg Freemyer wrote:

Basil Chupin wrote:

...

I bought a Seagate 2TB USB3 external HDD today and formatted 75% of it with ext4 file system and 25% with ntfs - using latest version of Gparted.

My two internal 1TB Seagates are SATA 3 and formatted with ext4 except for one partition (ntfs) which has XP installed, but which is rarely used (it's mainly there to bring me back to earth and to remind me how bloody aweful Windows is).

I use mc (Midnight Commander) 99.99% of the time to manage my files.

After I formatted the external HDD (as described above) I went to copy files from my main system to the new external HDD - which is to be used

for backups.

What I found - and which is what I am now asking about - is that when, using mc, that when copying files to the *ext4* formatted partition the

files where being copied across at ~80MB/s. The files copied over were a mixture of ntfs and ext4 files (which I had copied from a collapsing 32-bit system to my current *Linux* system last year when I replaced the old computer).

However, when I went to copy files to the *ntfs* formatted part of the new external HDD, the best transfer rate I got was ~28MB/s - which looks very much like what I used to get when using USB2!

So, why the *big* difference in transfer rates between copying files to

a partition formatted in *ext4* and one formatted in *ntfs*?!

What system am a I using? Look at my signature line; but I understand that some brain-dead mail clients drop the signature line so for these people with brain-dead mailers the sig. line is:

Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1

Any suggestions, please, as to why there is such a difference in the transfer rates between the 2 file systems, and how can this be overcome?

BC Basil,

Ntfs-3g is a fuse based filesystem.

Thanks, Greg, but it's over it's over my head :-( . What's a fuse based filesystem?

That means the filesystem code actually is a userspace app, not a kernel module like most linux filesystems.

And again, way over my head: what is a "userspace app"?

I think that makes it inherently slower. Anyway, before you assume something is wrong with your system you should research what performance to expect from ntfs-3g.

Greg

OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions? BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Basil Chupin wrote:

OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions?

That would depend on where the bottle necks are. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 29/03/13 16:05, Linda Walsh wrote:

Basil Chupin wrote:

...

OK, I'll go checking what performance one can expect from ntfs-3g but if it worked in USB 2 conditions then why shouldn't it work in USB 3 conditions?

---

Thought you said it was SATA-3?

I did. The internal HDDs are all SATA3. The external HDD is USB3.

Anyway, you might try mucking around in proc

first you might try lowering /proc/sys/kernel/sched_latency_ns to 10,000,000 (enter it w/o the commas) ..

That's now set to 24,000,000 so you suggest lowering it to 10m - OK, will try later.

What clock frequency are you running?

3.6MHz.

Are you running tickless?

My Master always uses this gunk on me so I have no ticks, thankfully. (Aside: "What's 'running tickless'?! Nevermind.....it's gotta be computer talk-type jargon...." :-) .) [pruned]

That might give you a few things to try... ;-)

That's it? That's all there is for me to try? :-D OK, I'll try this after I have worked out wot it all means :-D . But joking aside, thanks for your response - it's really appreciated. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 22:41 -0700, Linda Walsh wrote:

Try measuring the "raw" disk READ times...(without any file system)... that should tell you how fast the disks can go w/o FS overhead.

time dd if=/dev/sda of=/dev/null bs=16M count=256 --- that will do a raw read...

I think you can do better with "hdparm -tT /dev/sda" :-)

Note if you read from a partition /dev/sda1, /dev/sda2... The higher numbered partitions will likely be slower (assuming (spinning hard disks) and not SolidStateDrives/SSD).

Not always true. Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3. I think I would have to rerun the test on other disks, to see if it repeats the same pattern. It will depend on how the tracks are actually translated. But it is a tedious test to run, unless you script it... - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFXaV0ACgkQtTMYHG2NR9XzeACeLwBNy5sKQYFGU/i2CMBxqxE4 TAkAn0gc8sv9nTsHRr5OZhiZYHACKqT9 =jOTY -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2013-03-30 at 19:26 -0400, Felix Miata wrote:

...

Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.

Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start.

No, it more probably is a result of the fixed rotational speed, the varying number of sectors depending where is the track, etc. We would need to have actual numbers to do the math. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFXe9MACgkQtTMYHG2NR9X4XACfWpSihRNg3MAeL0JPfQZTeM9J TrAAn3dnM/lBBEJ/akqNqqEPebi3+p8E =5aMx -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-03-30 21:29 (GMT-0700) Linda Walsh composed:

Felix Miata wrote:

...

On 2013-03-30 23:38 (GMT+0100) Carlos E. R. composed:

...

...

Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.

...

Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start. ==== Someone should go tell Tom's HW to update their graphs... Never seen one with a slower speed at the start...

#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the only such doc I saved. Note on its #2 the difference between front and middle is very small. All others I checked at http://fm.no-ip.com/PC/bench/Sysbench/ are fastest at front. Most of these were done shortly after receiving a new HD, prompting to test to see what speed to expect. ISTR seeing it happen more than once though. Otherwise I don't think I'd have remembered it ever happening.

That doesn't make any sense unless you have bad sectors there... or maybe they are putting spare sectors at the outside on some disks?

I've never encountered those symptoms on any drives I've benched, but also I'm almost always running Enterprise drives for reliability. Same for your tests? Brands? I'm usually Hitachi, Fujitsu, Others are not ones I usually buy but have some...course now Hitachi is WD? ARG...wonder if they'll keep the facilities separate...never had good luck with WD's...

Until the past year I had only ever bought WD twice, and one wasn't for any system of my own. Now the choices are just too limited not to consider WD. The last I bought was a 512 byte/sector HD155UI "refurb" 1.5T made in a Samsung factory several years ago but with Seagate on the label. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-03-31 13:14 (GMT-0700) Linda Walsh composed:

...

#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the

Just thought of something.

Did you go from the beginning of the first partition or the beginning of the disk?

I didn't do anything but start Sysbench. It just does what it does, which is to ignore partitions and filesystems. Also, that HD was made before manufacturers began configuring 4096 byte sectors on ordinary production units. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-04-01 15:08 (GMT-0700) Linda Walsh composed:

Felix Miata wrote:

...

On 2013-03-31 13:14 (GMT-0700) Linda Walsh composed:

...

...

...

#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the

...

...

Just thought of something.

...

...

Did you go from the beginning of the first partition or the beginning of the disk?

...

I didn't do anything but start Sysbench. It just does what it does, which is to ignore partitions and filesystems. Also, that HD was made before manufacturers began configuring 4096 byte sectors on ordinary production units.

---

Hmmm... I try sysbench and get: tools/sysbench/sysbench-0.4.12/sysbench> sysbench

Did you open my URL and notice the version at its top? It's not the same sysbench: http://www.os2warp.org/SysBench/ -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

My test was done with default options. I first created an extended partition, then a number of logicals inside.

---- partition tool? any file systems or did you just use raw sectors...? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 02/04/13 13:44, Carlos E. R. wrote: [pruned]

I found the data I tabulated, but not the script I used - I don't even know the name I used for it. I would have to search the entire disk for bash scripts, and then search all those inside for a recognizable string... that's a feature mc could have but it does not have.

Sorry, but it does. Look again. Unless we are talking about 2 different things here - but I am talking about mc being able to search for text within files. Isn't this what you are saying mc cannot do?

Maybe I'll create such a search script one day.

BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 09:29 +0200, Per Jessen wrote:

Dunno about mc, but won't this do:

find / -iname <template for bash-scripts> -print0 | \ xargs -r -0 grep <a recognizable string>

What would the template be? I intended to use "file" output. cer@Telcontar:~> file bin/0_script_constructs bin/0_script_constructs: Bourne-Again shell script, UTF-8 Unicode text cer@Telcontar:~> Ie, locate all files of that type, then grep on those. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcGE8ACgkQtTMYHG2NR9W5TQCgmWNVnnkbOwAOJhoKYnCg5rlm g5oAn32d8cPLF3pdj/aacnpA06TTAp+m =+Rk4 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 14:13 +0200, Per Jessen wrote:

Carlos E. R. wrote:

Hmm, how about this:

find / -type f | xargs file | awk '/Bourne-Again/{print $1}' | \ tr -d ':' | xargs -r grep <a recognizable string>

Interesting. Let's try: cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' tr -d ':' | xargs -r grep "fdisk" awk: fatal: cannot open file `tr' for reading (No such file or directory) cer@Telcontar:~> which tr /usr/bin/tr cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' /usr/bin/tr -d ':' | xargs -r grep "fdisk" awk: cmd. line:1: fatal: cannot open file `-d' for reading (No such file or directory) cer@Telcontar:~> I'm not familiar with awk, so I don't know what is wrong above :-? - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcIvEACgkQtTMYHG2NR9VHYQCfYkQ8ijye0VKMwVnanNt0HEZE TTgAoIAKEeBy0T7zqxYoWvXAvyal4z9M =yex5 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 09:17 -0400, Anton Aylward wrote:

If you look at the original - was it Per's ? - you'll see that the "|" is there

You are right, I googed in the copy-paste. :-o It works: cer@Telcontar:~> find bin/* -type f | xargs file | awk '/Bourne-Again/{print $1}' | tr -d ':' | xargs -r grep "for i in" bin/CD-info:for i in 32768,7 32776,32 32808,32 32958,128 33086,128 33214,128 \ cer@Telcontar:~> :-))) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcMvkACgkQtTMYHG2NR9V8LACfT1zFHaK8gBRpJSmubwK+F3mt EUEAoIJi7UP8G8Isk6bbS0eqy4D6vPW9 =m60H -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 15:56 +0200, Carlos E. R. wrote:

That about the quotes in xargs :-?

And I will have to run it as root to avoid the permission denied errors, too.

It aborted after 5 minutes: Telcontar:~ # time nice find /* -type f | xargs file | awk '/Bourne-Again/{print $1}'| tr -d ':' | tee listado_shells | xargs -r grep "mkfs.ext4 -L test_" xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option find: `/home/cer/.gvfs': Permission denied find: `standard output': Broken pipe find: write error real 5m2.171s user 0m3.733s sys 0m21.042s Telcontar:~ # I'll put another tee to see where. Telcontar:~ # time nice find /* -type f | tee listado_total | xargs file | awk '/Bourne-Again/{print $1}'| tr -d ':' | tee listado_shells | xargs -r grep "mkfs.ext4 -L test_" xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option tee: standard output: Broken pipe find: `/home/cer/.gvfs': Permission denied tee: write error real 0m52.520s user 0m3.404s sys 0m10.551s Telcontar:~ # No luck. The last lines of 'listado_total' were: windows/OldA/not_mounted /windows/href.pif /windows/Worm_Gibe_C_1.zip /windows/D/not_mounted I don't know what it stuck at. The next entry in the sequence should have been "/windows/E/not_mounted". There are no files in the /windows directory, except those above and some more "not_mounted" 0 bytes flags. There is a file in the listing, early, like this: /data/storage_b/Backups_parciales_particiones_test/Elessar_sda9/home/cer/.local/share/Trash/info/"Custom Install.mpeg.rar".trashinfo That could be the one that produces the first error :-? No, there are several such entries: it would have complained more times. Hey, look at these entries: /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test! /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test. /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test: /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test\ /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/tes|t /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/[.test].test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/~test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/,}test3 /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/[.test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test::test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/"test" /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/12.{test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/te -st /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/te\*st /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/tes t /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/tes<>t /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test's /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test.. /home1/cer/Compilaciones/FSlint-2.08/fslint/tstlnt/findnl/test:: Wow, difficult ones... :-o Maybe using the "-print0" option in find :-? - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcOtUACgkQtTMYHG2NR9UIpwCfS9y4RJ35KTdWNswnbcrZYeNV IJIAniRmNATXEUnZNlCax0fiK2KXcM+e =W2OP -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 17:49 +0200, Per Jessen wrote:

Only if some of those shell scripts have quotes in their names, this will break too. (second xargs)

It worked :-) It found the file: /data/storage_b/Backups_parciales_particiones_test/Bombadil_sda1/root/bin/test_hd_make: #mkfs.ext4 -L test_$X /dev/sdb$X In a partial backup of a previous installation. Those are the scripts I used to measure hard disk speed across 20 partitions or so that I mentioned on another post. But it took too long: real 134m50.851s user 56m45.657s sys 3m7.112s If is only analyzing inside bash scripts, why did it took so long? - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcrjUACgkQtTMYHG2NR9X24wCdFjihH6ZZQM0KgimGteDV2vZe wtsAnRi0ZYC57iqf/3kagAwtbtQ7Oy/2 =mebw -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On Wednesday, 2013-04-03 at 17:49 +0200, Per Jessen wrote:

...

Only if some of those shell scripts have quotes in their names, this will break too. (second xargs)

It worked :-)

It found the file:

/data/storage_b/Backups_parciales_particiones_test/Bombadil_sda1/root/bin/test_hd_make: #mkfs.ext4 -L test_$X /dev/sdb$X

In a partial backup of a previous installation. Those are the scripts I used to measure hard disk speed across 20 partitions or so that I mentioned on another post.

But it took too long:

real 134m50.851s user 56m45.657s sys 3m7.112s

If is only analyzing inside bash scripts, why did it took so long?

Some possibilities - because 'file' has to process each and every file found. - because you have 25 gazillion files? :-) - because your disk is veeeerrrrryyyyyyyyy sssssssllllloooooowwwwww :-) If you want to know if it is 'xargs file' or 'xargs grep': find / -type f -print0 | xargs -0 file | \ awk '/Bourne-Again/{print $1}' >/dev/null How long does that take? -- Per Jessen, Zürich (2.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 04/04/2013 10:07 AM, Carlos E. R. wrote:

It is the file command. There are, of course, many files. I don't know how many, because:

Telcontar:~ # wc -l listadeficheros 4 listadeficheros

because they are null terminated strings ;-)

$ xargs -0 -n 1 echo < listadeficheros | wc -l ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Thursday, 2013-04-04 at 10:26 +0200, Bernhard Voelker wrote:

...

On 04/04/2013 10:07 AM, Carlos E. R. wrote:

...

It is the file command. There are, of course, many files. I don't know how many, because:

Telcontar:~ # wc -l listadeficheros 4 listadeficheros

because they are null terminated strings ;-)

$ xargs -0 -n 1 echo < listadeficheros | wc -l

;-)

Let's see...

I have that running for 17 minutes and it has not finished.

You have deliberately prolonged the agony by adding '-n 1' to xargs - that means xargs will fork 'echo' once per filename. Try this: xargs -0 printf "%s\n" http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 04/04/2013 01:11 PM, Carlos E. R. wrote:

Huh. It has got stuck somewhere in the middle. Locked.

Telcontar:~ # time wc -l --files0-from=listadeficheros

That command counts the lines numbers of each of you 2.6m files (instead of the number of files).

... ... wc: /var/lib/ntp/proc/sysrq-trigger: Input/output error 0 /var/lib/ntp/proc/sysrq-trigger 1 /var/lib/ntp/proc/splash 48 /var/lib/ntp/proc/partitions 56 /var/lib/ntp/proc/diskstats 98 /var/lib/ntp/proc/crypto 1 /var/lib/ntp/proc/key-users 0 /var/lib/ntp/proc/kpageflags 32 /var/lib/ntp/proc/kpagecount

<--- stuck for a minute in there, at least.

^C

real 2m15.006s user 0m0.469s sys 0m1.976s Telcontar:~ #

It is slower, because it prints to the screen, and it locks, producing no output. :-?

Bingo! That's where the file command is also stuck. The file list is wrong. You should limit it to where you really want to search for the shell scripts. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 04/04/2013 12:54 PM, Per Jessen wrote:

You have deliberately prolonged the agony by adding '-n 1' to xargs - that means xargs will fork 'echo' once per filename.

Try this:

xargs -0 printf "%s\n"

or even better: $ tr '\0' '\n' < listadeficheros | wc -l Isn't it cool to have so many choices? ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On Thursday, 2013-04-04 at 13:29 +0200, Bernhard Voelker wrote:

...

or even better: $ tr '\0' '\n' < listadeficheros | wc -l Telcontar:~ # time tr '\0' '\n' < listadeficheros | wc -l 2697454 real 0m0.638s user 0m0.167s sys 0m0.204s

Yes, it is faster.

But it also defeats the point of using null-terminated file names instead of just printing them 1/line. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-04-04 at 11:00 +0200, Per Jessen wrote:

No big surprise there.

If you need to do this more than once, it would probably to filter out known non-bash file-extensions or only look for "^#!.*bash" in the first line of each file instead of using 'file'. It makes the script a little more complex, but I'm sure it would reduce execution time significantly.

Oh, absolutely. Time permitting, I'll write a script or even a pascal program to do it or part of it. Looking for the shebang only should be trivial... - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFdRzoACgkQtTMYHG2NR9UwmACfdrHVSUd3DkDvbuXqF6xTDkyF pKQAn0d6ljiTQU7Rsl2bKGu5K44hdIUv =nGSN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On Wednesday, 2013-04-03 at 15:47 +0200, Carlos E. R. wrote:

...

:-)))

I have it running for real now. There is a little problem:

cer@Telcontar:~> find /* -type f | xargs file | awk '/Bourne-Again/{print $1}'| tr -d ':' | tee listado_shells | xargs -r grep "mkfs.ext4 -L test_" find: `/boot/lost+found': Permission denied find: `/data/cripta/root': Permission denied xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option find:

`/data/storage_b/cer/cosas/antiguo_cosas_nimrodel__mirar/old_theother/etc/opt/gnome/gconf/gconf.xml.mandantory':

Permission denied

That about the quotes in xargs :-?

I guess you are finding a filename or a filetype with quotes in it. Unless you have filenames with quotes in them, just use this tr instead: tr -d ':\'"' -- Per Jessen, Zürich (5.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. said the following on 04/03/2013 07:50 AM:

By the way, if I try a CLI "grep" on my ~/bin directory, it locks:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND cer 23961 0.1 0.0 11820 3568 pts/2 S+ 13:46 0:00 | \_ grep --color=auto fdisk bin/0_script_constructs bin/0_script_constructs~ bin/200506_attrib.list bin/Antiguos bin/Bck200405_attrib.dat

CPU usage is NIL, iotop doesn't see the grep as working. I just typed:

cer@Telcontar:~> grep fdisk bin/* ...

Got some hits, then it stuck.

I've had that happen in other situations, most notably greping recursively through /etc :-( However if I use 'find' and only grep actual files, that is not some of the strange things that can be symlinked in there, it never hangs. I admit that I've never systematically searched to identity what the 'strange things' were. -- A distracted figure with a huge bushy beard blunders in just as you speak the word of ancient magic. The man wears loose clothing, and an expression of intense concentration. He is clutching his frizzy hair with one hand; his other hand grips an intricate grid - the object of his attention. His eyes brighten the word you've spoken reaches his ears. "Yes! Yes! That's it!" he exclaims as he draws out a pen and fills in a row of squares. "Now my hyperconstrained, double-acrostic, cryptic crossword is complete, and ready to puzzle others. That was all I needed - just a simple five-letter word, composed only of the letters 'X' 'Y' and 'Z,' that would fit here!" He grips your hand and shakes it fervently. "Thank you! Now that I've finished with that, I can get on to those other things I've been meaning to do, such as monkey-wrenching the demolition and saving recreational linguistics for future generations." He turns away and mutters, just before he departs, "I hope none of that will involve lying in front of a bulldozer..." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. said the following on 04/03/2013 08:51 AM:

Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?

No, its your fault. Grep can and _should_ read from whatever input you give it. How is a named pipe different from an unnamed pipe? cat /etc/passwd | grep "anton"

One of the reasons I use 'mc' for searches is that it is clever enough not to look inside those and does not get stuck.

Which, to my mind, says that 'mc' is broken. You problem is that the named pipe is not supplying a data stream and an end of file. Grep is going to keep reading from its input until it gets to the end of the file. You can do the same thing at the CLI with this $ grep anton Grep is now reading from STDIN. It will keep reading from STDIN until it sees and end of file -- type ctrl-D -- or until it is killed, perhaps with ctrl-C. There is nothing wrong with grep. It is behaving as it should. -- "No serious commentary will say that the user has no responsibility. We all have responsibilities to lock our doors in our homes and to buckle up when we get in cars." -- spokesman, Information Technology Association of America, Business Roundtable, AP, May 19, 2004 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

...

...

Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?

No, its your fault. Grep can and _should_ read from whatever input you give it.

Ok, but the man page does not say how to tell grep not to look inside not normal files :-)

-D ACTION, --devices=ACTION If an input file is a device, FIFO or socket, use ACTION to process it. By default, ACTION is read, which means that devices are read just as if they were ordinary files. If ACTION is skip, devices are silently skipped. What's that then? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

В Wed, 03 Apr 2013 15:17:00 +0100 Dave Howorth пишет:

-D ACTION, --devices=ACTION

Oh. Never noticed it before. Thank you! -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 10:26 -0400, Anton Aylward wrote:

Carlos E. R. said the following on 04/03/2013 10:03 AM:

...

Ok, but the man page does not say how to tell grep not to look inside not normal files :-)

Ah, so you want a Swiss army knife?

:-) Ah, Dave posted the trick a minute ago :-) About grep and the Swiss army knife... I'm used to a special "grep" version made by Lucent Technologies (previous AT&T") that was named "cgrep". It is now available as free (well, actually "Lucent Public License"), sometime after the demise of Lucent and the disappearance of Bell Labs. This switch is what made it of particular use to me: 5ESS Project-Specific Options The following special-purpose options are provided for the convenience of those analyzing 5ESS International or Domestic ROP reports. -R Sets window delimiters and other options for 5ESS International or Domestic ROP reports. The -R option can also be used in conjunction with -nline or +nline to output preceding or subsequent ROP reports to the matched report, or both. In either case, nline-1 adjacent reports will also be printed. The -R option is almost equivalent to -DEMh +I2 -+w ´\+\+\+|^Y´ , but has one additional effect. Patterns of the form ´sm=n´ (the literal sm must be in lower case letters) will be expanded into regular expressions to match most if not all ROP reports pertaining to SM n. n must be an unsigned number without leading zeros, in which each digit may be either a simple digit or any one of a set of digits enclosed in square brackets. Within brackets, a hyphen can be used to indicate a range of digits. See the 5ESS EXAMPLES sec- tion for examples of the use of this feature. -T Same as -R, except also causes ROP event trails to be followed. If the string "EVENT=n" (n is a number of one or more digits, possibly preceded by a minus sign) or "EVENTO=n" (Spanish) appears in a matched ROP report, all subsequent reports with the same event number will be printed. Equivalent to the following: -R -t ´EVENTO?=-?[0-9]+´ . It is not the only addition. There are powerful options to capture context around a particular string found in the text, both before and after the match. I have used that feature in scripts of mine. The man page says that the author is "Bill Tanenbaum". The web page at the time I got the public version were these, according to the man page: http://www.bell-labs.com/project/wwexptools/cgrep/ Source Code, etc. http://www.bell-labs.com/cgi-user/wwexptools/gensnapshot?cgrep

It reminds me of Rob Pike's "considered harmful" view of what Berkeley did to 'cat'

I agree with the 'each thing should do one things and only one thing' approach to tool building. Not only does it make combining tools easier - the great power of UNIX is scripting, notable the shell -but each thing can be proven to be 'correct'.

True. Mmmm... systemd? >:-)

If you can make this case for grep then you can make it for cat and tr and many other programs.

X'-)

Perhaps you want to re-invent one of the mainframe OS's of the 50s and 60s?

http://harmful.cat-v.org/cat-v/unix_prog_design.pdf

Argh, 7 pages of dense PDF print... Later, perhaps. I have to do some accounting chores (which I hate). And I want to have my siesta about now, too ;-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcQXwACgkQtTMYHG2NR9Ww3gCgjrmkcHzF+wzAqjfGzJo6TgDa 8LQAoIsH330sphfaUjfjZ7vV/pi+y9fP =RZHA -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 17:53 +0200, Per Jessen wrote:

Carlos E. R. wrote:

...

Now, is this a bug in grep, that should not look inside a named pipe, or is it my fault?

You told it to look in there.

:-) Didn't know there was a named pipe in there :-)

Whenever I grep for something or other, I virtually always use the "find <specification> | xargs grep" combination. The xargs can also be parallelized, a big advantage if you've got multiple cores and many files.

Ah, I did not know that, either. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFceggACgkQtTMYHG2NR9WcEQCfWbRnfgLO4vC8hI0bxaLoVWLq xEIAnj97eZpTDVdSKsFVSs+LTMUQa2jq =8r3v -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Linda Walsh wrote:

Felix Miata wrote:

...

...

...

...

Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.

...

...

Confirmed via observation here, closer maybe to 25%, but clearly slower

...

#1 on http://fm.no-ip.com/PC/bench/Sysbench/resul603.txt looks like the only such doc I saved. Note on its #2 the difference between front and middle is very small. All others I checked at http://fm.no-ip.com/PC/bench/Sysbench/ are fastest at front. Most of these were done shortly after receiving a new HD, prompting to test to see what speed to expect. ISTR seeing it happen more than once though. Otherwise I don't think I'd have remembered it ever happening.

---

Just thought of something.

Did you go from the beginning of the first partition or the beginning of the disk?

Be default, or unless you've gone for DOS compatible partitioning, the first partition won't be track-aligned due to the boot sectors.

Then usually, people (or software) opts for nearest track for end of 1st partition, so further partitions are aligned.

It would make a slight difference, but the head would have to move over a track mid-read -- to read any track in the first partition, if such a situation were in place.

I know when I partitioned with fdisk, I had to toggle a flag to make sure it was dos compat, otherwise, I got the use of 200 and some odd more sectors out of the first partition.

Linda, It is time for you to enter the 21st century in respects to disk geometry. Back in the 80's and 90's CHS actually corresponded to physical reality. But LBA was introduced about 20 years ago because CHS was a pain. When disk sectors shrank to where more than 255 sectors per track became normal, the drive manufacturers just started lying when they reported the CHS geometry because 255 was the biggest reportable value and nobody cared enough at that point to fix the CHS standard. For at least the last decade the linear density of sectors is typically constant over the whole platter. Since you know circumference of a circle is PI * D, a track 3 inches from the center holds 3x the sectors as one 1 inch from the center. That means number of sectors per track isn't even a constant for a modern drive. That by the way is why start of the drive is faster than the end. The drives start at the outside edge, so for one revolution the head passes over a lot more sectors. The end of the drive is closer to the center, so it sees less sectors per revolution. And at least for now the rpms is kept constant, so the sectors per second speed is higher for the start of the drive. Anyway, if you think you have the ability to align partitions to tracks, you are simply mistaken. You can merely Fyi: both modern windows and linux align to 1MB boundaries with no concern how that aligns with tracks. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

"Carlos E. R." wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Sunday, 2013-03-31 at 23:35 -0400, Greg Freemyer wrote:

...

That by the way is why start of the drive is faster than the end. The drives start at the outside edge, so for one revolution the head passes over a lot more sectors.

True.

However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?

...

Anyway, if you think you have the ability to align partitions to tracks, you are simply mistaken. You can merely

Yes, it is an illusion.

Still, there are some partitioning sotware that does so. I mean, it says it does. I think it is sfdisk, it even complains if partitions do not end/start at track ends and makes you force the layout to accept it (if it is not sfdisk I can find out which, I have the text saved somewhere).

Windows made the jump to 1MB alignment with Vista. Opensuse a couple years ago. I'm sure there is software that still treats CHS geometry as meaningful, but that software is out of date. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-04-02 05:53 (GMT+0200) Carlos E. R. composed:

Do you know of a modern tool that will clone a partition table using the exact same sectors as the original? Even if it thinks that the boundaries are wrong, because the old disk was done on CHS and the new one wants to use megabytes.

The one I use permits every kind of alignment I've ever encountered, but it isn't FOSS. http://fm.no-ip.com/Tmp/Dfsee/dfslX-vizio.txt starts with output from its standard startup log. To it I've appended fdisk -l and hdparm -i output for comparison. It shows 1.5T and 2.0T devices that use 512 byte internal sectors. 1.5T uses legacy 255/63 geo while the 2T uses 64/32 "advanced format" 4k sector alignment. Both were partitioned 100% using DFSee, which comes with functionally equivalent binaries for DOS, OS/2, Win32, x86 Linux and Intel Mac. You might notice fdisk reports table out of order on sdb. That's on purpose. The STB it's normally used with can only recognize a partition defined by a first partition table entry. I partitioned and formatted and used it before I knew that fact, so since the NTFS needed to be "first", I used DFSee to swap the table entries. DFSee is one of my backup tools. I do a lot of partition and disk cloning with it, but it also creates compressed and uncompressed partition and disk images as desired. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-04-02 at 13:09 +0200, Per Jessen wrote:

Carlos E. R. wrote:

...

Do you know of a modern tool that will clone a partition table using the exact same sectors as the original? Even if it thinks that the boundaries are wrong, because the old disk was done on CHS and the new one wants to use megabytes.

dd will do that.

No, it doesn't. It clones the partition table itself, that is, the primary partition table only. The extended partition _table_ is not cloned unless you clone the entire disk. My backup image was not made of the entire disk, but of each single partition separately. Thus I have to recreate the partition table first, including the logicals. DD doesn't do that. Once the logical partitions are recreated, I can use dd to put their contents inside. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFa154ACgkQtTMYHG2NR9Wz6wCgitDZD/r8eLh9qvW1YHeBaVGc e8MAn0RX15q0UbTO19BkdFCeTibpsiby =mugJ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-04-02 at 15:13 +0200, Per Jessen wrote:

Carlos E. R. wrote:

...

It clones the partition table itself, that is, the primary partition table only. The extended partition _table_ is not cloned unless you clone the entire disk.

Sorry, I missed the bit about this being about extended partitions/tables.

It took me by surprise. I had a disk going down fast, I was on a hurry to recover before total failure, I had done an image backup partition by partition (because one of them was damaged, and because of the distribution of the available space for the backup), I had dd-ed the partition table, I had a dd of the "extended" partition, too... I thought I had all I needed. And then I found out that I could not recover the stored images of the logical partitions because there were no partitions. And I could not recreate the partitions with fdisk because the original disk had been made by HP with old alignment, and fdisk wanted to do it on megabytes, skipping part of the initial region... If I allowed that, the images would simply not fit. sfdisk did the job, but I had to force it. Being an old and obsolete tool, I want to know what current tool will clone an entire partition table without questions. May have to post a new thread about that one day :-) - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFa5QwACgkQtTMYHG2NR9V2JACfUSN+qA2bH0qmCTpjiFjmoRg/ P2sAnAkYGFasXtitMeZGsvmFBGGp23MV =cjkA -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-04-02 at 21:11 +0200, Per Jessen wrote:

Carlos E. R. wrote:

...

And I could not recreate the partitions with fdisk because the original disk had been made by HP with old alignment, and fdisk wanted to do it on megabytes, skipping part of the initial region... If I allowed that, the images would simply not fit.

I'm not sure, I haven't tried it, but if you revert to DOS-compatible mode with command 'c', I think you might get to allocate partitions any which way you want. I'm sure I have seen a reference to that feature somewhere.

Maybe I did not try that.

...

sfdisk did the job, but I had to force it. Being an old and obsolete tool, I want to know what current tool will clone an entire partition table without questions.

sfdisk isn't old nor obsolete, it's very useful.

It wants to align on tracks, complains bitterly if you don't, and does not support GPT. You can see what happened here: https://forums.opensuse.org/english/get-technical-help-here/install-boot-log... At some point it erased the entire partition table of the sytem while it was runnning... - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFbXDwACgkQtTMYHG2NR9WnrACfZfXT38kBSoDCSyQ8YdWsBkF9 9bkAnRv+YbTG0R2CBIlFpcAlRz3WC6dD =ozqQ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2013-04-03 at 09:27 +0200, Per Jessen wrote:

Carlos E. R. wrote:

...

It wants to align on tracks, complains bitterly if you don't, and does not support GPT.

Well, yeah. Still very useful when you want to automate things.

If there is no other tool that can clone an entire partition table, I'll use it. I hoped there would be a better tool. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFcFRUACgkQtTMYHG2NR9UPggCfV+jd+5KEsvrXe3Md83xPogEA i6kAn3BLm4ChFR/Fnd9m/s3enOH1dYBs =yQp1 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-04-01 03:39 (GMT-0300) Cristian Rodríguez composed:

Carlos E. R. wrote:

...

However, some measurements say that disks are actually faster at about 1/4 or 1/3 of the way. I can make guesses, but I have not yet read an explanation of that :-?

Does not matter really, as no sane human should use rotating disks anymore :-P

When did sanity become a requirement for using a PC or Linux? Must be nice to have unlimited funds when you need another 2TB every 2-3 months, and trust that just because SS media has no moving parts that it will be materially or even any more reliable long-term than spinning platter storage. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. said the following on 04/01/2013 07:28 AM:

...

Does not matter really, as no sane human should use rotating disks anymore :-P

Can you lend me a few of those? I'm scarce on fundings, I have to say. >:-)

+1 -- Parkinson's Law: Never Interview Emus -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 04/01/2013 11:20 AM, Ken Schneider - openSUSE pecked at the keyboard and wrote:

On 04/01/2013 07:43 AM, Anton Aylward pecked at the keyboard and wrote:

...

Carlos E. R. said the following on 04/01/2013 07:28 AM:

...

...

Does not matter really, as no sane human should use rotating disks anymore :-P

Can you lend me a few of those? I'm scarce on fundings, I have to say. >:-)

+1

I have been using disks without platters for over 10 years and they virtually cost nothing. Actually VirtualBox includes them for free as does VMware. :-)

Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-) -- Ken Schneider SuSe since Version 5.2, June 1998 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Ken Schneider - openSUSE said the following on 04/01/2013 11:28 AM:

Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-)

That's expensive! What a hell of a profit margin! http://www.everyjoe.com/2009/09/29/technology/hard-drive-cost-per-gigabyte-f... Even for a service .. http://www.hightechinthehub.com/2012/07/penny-per-gigabyte-per-month/ The $1/G is last years SSD price http://www.computerworld.com/s/article/9234744/SSD_prices_continue_to_plunge http://www.jcmit.com/diskprice.htm -- "Television is a medium because it is neither rare nor well done." -- Fred Friendly -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Ken Schneider - openSUSE said the following on 04/01/2013 11:51 AM:

Check for yourself:

www.virtualbox.org/aprilfool

Getting a 404 is an april fool? -- There's a tendency today to absolve individuals of moral responsibility and treat them as victims of social circumstance. You buy that, you pay with your soul. -Tom Robbins, Still Life with Woodpecker -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 4/1/2013 11:44 AM, Anton Aylward wrote:

Ken Schneider - openSUSE said the following on 04/01/2013 11:28 AM:

...

Spoke to soon. Just checked virtualbox.org and found that Sun is going to start demanding $1US for every GB of virtual disk you create. Bummer. :-)

That's expensive! What a hell of a profit margin!

.... Well considering I can get a 1TB hard drive for ~$80 and 128G esata SSD for ~$145 (just now checked online), I think it's insane to send 15X the price for basic storage. My working storage is still just fine on disk, thank you; and my offline backup is, too. And I have much more than 128G in family and professional stuff I really want to keep -- not just trinkets like cute news and Hollywood photos and videos. And USB flash drives have made mobile computing very, very convenient -- I keep my 32G flash drive in my pocket for downloading interesting stuff I find when I'm online away from home or work. Yeah, there's a place for the SSD, but it's not in my systems yet. I'm subject neither to the fashion motives nor the high-reliability strictures that might justify the huge price difference. John Perry -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer wrote:

It is time for you to enter the 21st century in respects to disk geometry.

---- Greg, stop showing your ignorance. The LBA still corresponds to CHS *internally* -- just that the nums are internal to each HD.

For at least the last decade the linear density of sectors is typically constant over the whole platter. Since you know circumference of a circle is PI * D, a track 3 inches from the center holds 3x the sectors as one 1 inch from the center. That means number of sectors per track isn't even a constant for a modern drive.

---- Wrong. Modern HD's have been divided into zones for the past 15+years... Modern HD's have almost 1 zone/TB, but you can see a most modern examople at: http://media.bestofmicro.com/E/N/362111/original/throughput-diagram-wd4001FA... Latest 4TB HD... That isn't a smooth curve -- Each notch is a zone.. Across each zone the number of sectors is constant. Very odd on that graph... notice as some have noted the first 50-100GB are slightly lower than just past the 100GB mark -- for READS... writes seem to be very smooth but likely due to internal disk buffering...

Fyi: both modern windows and linux align to 1MB boundaries with no concern how that aligns with tracks.

---- Depends on what disk partition tool you use, but I don't know of any that use MB. The default display in fdisk:...sd Device Boot Start End Blocks Id System /dev/sdc1 63 25398764 12699351 83 Linux /dev/sdc2 25398765 42009974 8305605 83 Linux /dev/sdc3 * 42009975 44130554 1060290 83 Linux Shows POSIX standard 512 byte sectors. parted.. Number Start End Size Type File system Flags 1 32.3kB 13.0GB 13.0GB primary xfs type=83 2 13.0GB 21.5GB 8505MB primary xfs type=83 3 21.5GB 22.6GB 1086MB primary xfs boot, type=83 4 22.6GB 146GB 123GB extended type=05 5 22.6GB 31.2GB 8595MB logical linux-swap(v1) type=82 6 31.2GB 47.3GB 16.1GB logical xfs type=83 7 47.3GB 58.0GB 10.7GB logical xfs type=83 8 59.1GB 145GB 85.9GB logical type=83 Hmm....looks like kB is the lowest unit well...nope... (parted) help unit unit UNIT set the default unit to UNIT UNIT is one of: s, B, kB, MB, GB, TB, compact, cyl, chs, %, kiB, MiB, GiB, TiB default is 'compact'... As for my other two sd's, they both start at 17.4kB -- that's not a 1MB boundary either... it's actually 34 sectors -- I guess thats a gpt size -- and boot drive is sdc... starting at the 63 sector Looking at the # blocks/partition and dividing by 2*1024*1024 (sectors->MB): fdisk -l /dev/sdc|cut -c41-49|grep -P '^\s*[0-9]+$'|while read secs;do printf "%10d %s\n" "$secs" "$(pcalc "$secs/(2*1024*1024)"|grep -P '= \d')" done 12699351 = 6.05552244186401 8305605 = 3.96042108535767 1060290 = 0.505585670471191 120278655 = 57.3533320426941 8393931 = 4.00253820419312 15735636 = 7.50333595275879 10490413 = 5.00221872329712 83883366 = 39.9987058639526 ---- I would say those are track aligned -- That's a 21st century 15K SAS drive. MB alignment?... I don't think even windows is that dumb. Track alignment is still an issue -- ask anyone running RAID -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

On Monday, 2013-04-01 at 14:52 -0700, Linda Walsh wrote:

...

I would say those are track aligned -- That's a 21st century 15K SAS drive.

That's an illusion. Not true.

--- yeah right... You realize that if it wasn't, you couldn't have synchronized RAID disks. Try using desktop drives sometime in an LSI controller -- let me know how many 'pass' it's internal validation. If a disk has an extra few ms of latency compared to other members of the same array, LSI will kick the disk out as bad. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----Original Message----- From: Felix Miata To: opensuse@opensuse.org Subject: Re: [opensuse] 12.3 + ntfs + ext4 + USB3 + different copying speeds Date: Sat, 30 Mar 2013 19:26:40 -0400 On 2013-03-30 23:38 (GMT+0100) Carlos E. R. composed:

Once I partitioned a disk in 20 or 30 partitions, and tested speed on all of them. The faster region was about 1/3 of the disk. It was slower at the end of the disk than at the start; that I expected, but not that it would be faster at 1/3.

Confirmed via observation here, closer maybe to 25%, but clearly slower at the front than somewhat beyond the front, with the rear very clearly bringing up the rear in performance. It wouldn't surprise me if disk makers were putting LBA 0 somewhere other than the physical start. -----Original Message----- Where LBA 0 is located is probably not that much relevant. However, the distance between LBA-(n) and LBA-(n+1) could be. their relative position (interleaving), the rotational speed, cache-size, and the speed of the OS-driver should be well balanced. Without it, it "kind of works", and high-performance hardware might even work worse than cots-hardware. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 29/03/13 16:08, Linda Walsh wrote:

Basil Chupin wrote: -----

Oh -- and check out "xlatencytop'... see what that tells you...

its a fun one...

But of course..... Sheesh, why didn't I think of "xlatencytop"?! :-) (Ignore -- I'm having one of *those* days today :-) .) Thanks for the suggestion re 'xlatencytop' - I'll try this one as well. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Content-ID: On Sunday, 2013-03-31 at 17:47 -0300, Cristian Rodríguez wrote:

This has a cost and is probably the reason why you will see reduced performance. although if they loss is too dramatic it might be a just a bug.

3 times more slow... I call that "dramatic" ;-) Several people have explained it in detail, it is not a bug. It is how things are, that's all. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFYqpMACgkQtTMYHG2NR9VByQCeOzVFTvwIijh3Uay7l6fEVy6m 56wAn38IJtzzM+6wgqv5nXbhl/XxOZMO =quO1 -----END PGP SIGNATURE-----

On 27/03/13 00:13, Per Jessen wrote:

Basil Chupin wrote:

...

So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off.

"Et tu, [Brute]?" :-) Wot is a "user-space filesystem"? I do know that from my use of Dolphin to copy to files and the use of mc I found that Dolphin is like walking thru molasses in winter when it comes to copying some files compared to how fast mc handles them -- but this "user-space" thingie has me spaced-out :-) . Why should whatever it this "user-space" thingie work the same as what one gets when using USB2 and not what one would expect when using USB3? BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Tue, Mar 26, 2013 at 9:53 AM, Martin Helm wrote:

I doubt a little bit that the user space part really affects the performance, at least I cannot see any valid reason from a programmers point of view.

I don't know if you do any kernel level programming, but the kernel / userspace boundary is common bottleneck concern for kernel devs. I may be dated in my knowledge and there are tricks that can be played by the fuse devs that might overcome part of this. (I don't know how fuse works in detail, so take this with a big grain of salt): In particular a lot of DMA chips only support the moving data from a disk controller to/from the 1st GB of ram, thus the kernel ram buffers are restricted to being in the first GB of ram. Userspace in theory never needs to directly access hardware, so it is free to use all of ram. That means that userspace and the kernel can't easily share ram. So a disk write with MC to ntfs-3g would be: MC -> kernel (fuse) -> ntfs-3g -> kernel (fuse) -> dma -> disk controller. If the userspace -> kernel calls involve making a copy of the data every time, then that is a lot of overhead compared to: MC -> kernel (ext4) -> dma -> disk controller. Note that the dma aspect is typically handled by off-cpu logic, so the above only has a single cpu managed data copy going on. In the ntfs-3g case, it has 3 cpu managed data copies going on. Thus ntfs-3g has 3x the overhead of ext4. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer wrote:

On Tue, Mar 26, 2013 at 9:53 AM, Martin Helm wrote:

...

I doubt a little bit that the user space part really affects the performance, at least I cannot see any valid reason from a programmers point of view.

I don't know if you do any kernel level programming, but the kernel / userspace boundary is common bottleneck concern for kernel devs.

There was a paper published in the Communications of the ACM: http://dl.acm.org/citation.cfm?id=1774130 http://www.csl.sri.com/users/gehani/papers/SAC-2010.FUSE.pdf I'm pretty certain it concludes that user-space file-systems (FUSE specifically) do perform measurably worse than kernel-space filesystems. /Per -- Per Jessen, Zürich (1.2°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2013-03-26 at 12:28 -0400, Greg Freemyer wrote:

Per, I really appreciate the link to that PDF.

Figure 3B of the pdf is pretty clear fuse "can" be a bottleneck. In their test setup, a native filesystem with 50-100MB files writes at about 850MB/sec.

A fuse filesystem writes at about 350 MB/sec.

But even 350 MB/sec is way above what Basil is seeing, so the bottleneck should not be in fuse directly. That leaves Martin's theory that it is ntfs-3g itself which is not optimally written.

Or both. As I said, on my older computer writing to NTFS put the single core cpu at 100%. It could not write faster. I can not see the same in my current computer, it is much more powerful. So the next thing is for Basil to watch 'top' while he copies files. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFR0CQACgkQtTMYHG2NR9Wr4ACeLT/rXPi1Lod+sG8d7ZT9m1KP i+AAniBmUnV0ESCTCEKiSmUs5kVZ2i7z =w/tv -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Basil Chupin wrote:

On 27/03/13 00:13, Per Jessen wrote:

...

Basil Chupin wrote:

...

So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off.

"Et tu, [Brute]?" :-)

Wot is a "user-space filesystem"?

http://fuse.sourceforge.net/

Why should whatever it this "user-space" thingie work the same as what one gets when using USB2 and not what one would expect when using USB3?

As Greg suggested, maybe USB-2 is the bottleneck, so going USB-3 won't give you anything extra. -- Per Jessen, Zürich (1.5°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 27/03/13 01:26, Per Jessen wrote:

Per Jessen wrote:

...

Basil Chupin wrote:

...

On 27/03/13 00:13, Per Jessen wrote:

...

Basil Chupin wrote:

...

So, why the *big* difference in transfer rates between copying files to a partition formatted in *ext4* and one formatted in *ntfs*?! Isn't write-support in ntfs still somewhat iffy or not quite supported? Or is it because ntfs is a user-space filesystem perhaps? I don't use ntfs, I could be way off. "Et tu, [Brute]?" :-)

Wot is a "user-space filesystem"?

http://fuse.sourceforge.net/

...

Why should whatever it this "user-space" thingie work the same as what one gets when using USB2 and not what one would expect when using USB3? As Greg suggested, maybe USB-2 is the bottleneck, so going USB-3 won't give you anything extra. ... maybe USB-2 is the _next_ bottleneck ...

Folks....... I thank each and all for the input so far to my question re the copying speed between to ext4 and ntfs formatted partitions. But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....". I am using USB3. The new external Seagate is USB3 and is connected to an USB3 port on the motherboard. The HDDs from which I am copying the files to the external USB3 Seagate are all SATA3 HDDS and recognised by the kernel as having UDMAs of 133. Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit). With this result nothing makes sense......:-( . Confused even more..... BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 27/03/13 18:43, Per Jessen wrote:

Basil Chupin wrote:

...

Folks....... I thank each and all for the input so far to my question re the copying speed between to ext4 and ntfs formatted partitions.

But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....". Greg mentioned it already -

"Assume your 28MB/sec performance is the fastest ntfs-3g can run on your computer. Usb-2 maxes out a little below that, so usb-2 was the bottleneck. Usb-3 is a lot faster, so now ntfs-3g is your bottleneck."

Either you are not reading what I wrote or I am not being articulate enough to explain what just happened when I used the command line command 'cp' :-( . Go to the bottom where I respond to your last para.....

Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit). Any chance that the connection to your external drive is flaky? If it's throwing lots of errors and causing retries, that could explain your problem.

Yesterday I bought a brand new 2TB external Seagate HDD which uses USB3 - repeat, USB3 - and which is connected to one of the USB3 - repeat, USB3 - ports on the motherboard using a brand new USB3 cable. The HDD is also using a brand new power pack for its power source. My original post asked the question as to why I was getting different transfer rates using the above setup when copying to an ext4 formatted partition on that external Seagate compared to when doing so to a ntfs formatted partition. I provided the transfer rates as displayed in mc when using mc to do the copying. I haven't even attempted to use Dolphin to do any copying because past exeprience has shown me that it's quicker to transfer a large file using the old morse code dots and dashes than use Dolphin :-) . Earlier today, using the exactly the same setup and without any changes of any description, I copied a 7.1GB file to the ntfs formatted partition and this took around 4 minutes to do the copying versus last night's effort using mc which took around 15 minutes (actually, closer to 20 mins I think) to do a similar operation. What is missing in my explanation of the situation to keep the issue of USB2 being brought up? BC -- Using openSUSE 12.3 x86_64 with KDE 4.10.1 & kernel 3.8.4-1 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Basil Chupin wrote:

I just did some 'scientifically' based tests which showed that one must never rely on memory when providing details about what (may have) occurred :-) .

I used a stopwatch to check how long it took to copy a file to (a) a partition formatted in ntfs on an external USB3 HDD and (b) how long to copy to the same HDD but to a partition formatted in ext4.

I copied a 11.437GB file to the ntfs partition using 3 methods and the results were:

1. copy using cp command................. 4 min 59.1 sec 2. copy using mc................................ 6 min 29.3 sec 3. copy using Dolphin......................... 5 min 13 sec

38Mb/sec, 29Mb/sec, 36Mb/sec.

Then I copied the same file to the ext4 formatted partition and the results were:

1. using cp command....................... 2 min 17.3 sec 2. using mc...................................... 2 min 18.1 sec 3. using Dolphin............................... 2 min 18.2 sec

All three about 83Mb/sec. So ext4 is more than twice as fast. I would ignore the differences between cp/mc/dolphin - they're interesting, but not in this context. I would use 'dd' to test IO instead - dd if=infile of=outfile bs=xxxx Try varying the blocksize to see if it makes a difference. You could also try using another filesystems - xfs or jfs instead of ext4. Then redo your test. If they show the same performance as ext4, ntfs is just slow, methinks. -- Per Jessen, Zürich (3.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 28/03/13 12:01, Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Wednesday, 2013-03-27 at 22:45 +1100, Basil Chupin wrote:

...

I used a stopwatch to check how long it took to copy a file to (a) a partition formatted in ntfs on an external USB3 HDD and (b) how long to copy to the same HDD but to a partition formatted in ext4.

I copied a 11.437GB file to the ntfs partition using 3 methods and the results were:

1. copy using cp command................. 4 min 59.1 sec

2. copy using mc................................ 6 min 29.3 sec

3. copy using Dolphin......................... 5 min 13 sec

Then I copied the same file to the ext4 formatted partition and the results were:

1. using cp command....................... 2 min 17.3 sec

2. using mc...................................... 2 min 18.1 sec

3. using Dolphin............................... 2 min 18.2 sec

The results are not surprising at all. It is about what I expected. Or we executed, most of all have told you about the same things :-)

It is very simple: writing files on ntfs on Linux is slow because it uses more cpu. It does not matter to you why exactly - if it does matter, then try to understand all that talk about userspace vs kernelspace and single-double-triple buffering and reverse-engineering.

Why different copy programs behave differently means that they do the copy differently. We would need a dev looking at the code to learn why some are more affected than others by this operation. It is curious, though.

Repeat the above test having in another terminal "top" running and see what it shows...

To see how much each of the 8 cores are being worked? OK, I'll try that later.

...

What is missing in my explanation of the situation to keep the issue of USB2 being brought up?

Somewhere you mentioned that you did not have/see this issue when you used USB2 or some similar words. We said that using USB2, as it is slow itself, you do not hit the problem that you see in your table of ntfs-3g being slow.

Ah, so that is what those references to USB2 were all about :-) . ================= Actually, there may be more to all of this than meets the eye - to use an old cliche. I bought the Seagate because I bought a 2TB WD just before Christmas and I could NOT copy files to it without getting errors - with the screen going blood red in mc and the HDD being unmounted. These errors occurred when copying to either the ext4 or the ntfs partitions on that WD. The errors I would get were: "kernel I/O error (5)" most of the times but there were other very similar ones. And after battling these hassles for 3 months I got fed up, swore never to buy another WD HDD and went out and bought the Seagate a couple of days ago. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Thursday, 2013-03-28 at 15:49 +1100, Basil Chupin wrote:

...

On 28/03/13 12:01, Carlos E. R. wrote:

...

...

Repeat the above test having in another terminal "top" running and see what it shows...

To see how much each of the 8 cores are being worked? OK, I'll try that later.

My guess is that one core will be maxed :-)

Ditto. -- Per Jessen, Zürich (1.1°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-03-27 17:54 (GMT+1100) Basil Chupin composed:

Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit).

Were you perchance using MC's ftp connection to make its copy, but a Samba or NFS connection for the cp? -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-03-27 20:16 (GMT+1100) Basil Chupin composed:

On 27/03/13 18:58, Felix Miata wrote:

...

On 2013-03-27 17:54 (GMT+1100) Basil Chupin composed:

...

...

Now, just to throw the spanner into what otherwise would be a very dull situation :-) , I just copied a 7.1GB file to the external Seagate's ntfs formatted partition using the stock standard cp command and this took something like ~4 minutes as against what mc took to copy a smaller file last night of ~15 minutes (working from memory here I fully admit).

...

Were you perchance using MC's ftp connection to make its copy, but a Samba or NFS connection for the cp?

I go to the icon which will start the terminal console.

I type in mc and mc runs.

I highlight the destination directory, and I highlight the source directory and file.

I press F5 and mc starts to copy the file from the source to the destination, and as the copying progresses I get displayed the speed of the transfer and how much of the file has been copied so far and how much is yet to be copied and the ETA of the transfer.

Re using the cli 'cp', I open a terminal, type in 'cp <file-to-be-copied> <destination-for-the-copy> and press ENTER.

Now, you tell me if I was using ftp connection or a Samba or NFS connection?

Insufficient data. Open either left or right MC menu, and you'll find listed among others "ftP link". Using this can result in either source dir or destination dir or both accessed via ftp. This is normally used only for non-local filesystems, so I did and do not expect that you had. The word "perchance" implies weak likelihood, yet possible. For example, I use this (ftpfs) for fetching installation kernel and initrd from ftp5.gwdg.de's opensuse/factory subdir prior to beginning an HTTP oS installation. In your case I expected and expect not, but had to ask just in case, as it is a possible reason for a copy speed differential compared to using cl cp. -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 28/03/13 11:36, Michael Hamilton wrote:

On Wed, 27 Mar 2013, Basil Chupin wrote:

...

... But, honesty, none of it makes sense to me - all this talk about "user-space" and all that. It has only confused me more than I was ever confused when delving into grub and grub2. Especially this reference to "maybe USB-2 is the _next_bottleneck....".

... Ext4 is implemented completely within the kernel. Ntfs-3g is implemented outside of the kernel and runs as a user process - Filesystem in userspace (FUSE).

Each time a process such as cp makes a kernel system call to do a file operation on a FUSE based filesystem, the call goes into the kernel and then the kernel passes the call back to a normal process via the FUSE API. The normal process then does more kernel system calls to the real IO behind the file system operation. This means that parameters and data in Ntfs-3g may need to be copied between kernel and userspace possible multiple times, before it makes it anywhere useful. Plus this requires a hierarchy of calls into and out of the kernel, all involving synchronisation between what's happening inside the kernel and inside the userspace. Contrast this with Ext4, where a system call to do a file system operation will enter the kernel only once and will run to completion entirely with the kernel - much faster.

The Wikipedia entry on NTFS-3G states the bottleneck may be the CPU required by the FUSE implementation and that a fast enough CPU will yield faster results. Considering this is supposed to be about I/O this implies big overheads.

FUSE based files system implementations are easier to write and more portable than writting kernel code. It only took me a few days to learn about FUSE and prototype a simple FUSED based filesystem (collectfs). It's just like writing any program that runs as a user process. Where as I've never had enough time to climb the learning curve for implementing filesystems that run inside the kernel.

If you compare the performance of a kernel based implementation of an encrypted filesystem to a FUSE based implementation, I think you will see similar differences/limitations in performance.

Thanks for this clear explanation. Duly taken onboard.

In a nutshell, a faster CPU might help narrow the difference.

You mean that this now may not be enough to run something like openSUSE 12.3?! :-) : Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 28/03/13 15:50, Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Thursday, 2013-03-28 at 15:36 +1100, Basil Chupin wrote:

...

On 28/03/13 11:36, Michael Hamilton wrote:

...

You mean that this now may not be enough to run something like openSUSE 12.3?! :-) :

Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU

Why don't you simply open a terminal and run 'top' inside when you do a copy to the ntfs partition, then compare to what happens copying to the ext4 partition? And then tell us?

:-)

It is the third time I suggest you do this... I did it on my machine, so I know what to expect.

OK, OK, OK already! I'll do it! :-) BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Thu, Mar 28, 2013 at 10:41 AM, Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Thursday, 2013-03-28 at 16:05 +1100, Basil Chupin wrote:

...

On 28/03/13 15:50, Carlos E. R. wrote:

...

OK, OK, OK already! I'll do it! :-)

I'll wait impatiently :-)

I just tried it on my laptop - USB3 - ntfs I got 25-30 MB/sec and mount.ntfs had about a 75% cpu load on one core. (total cpu load went up about only about 15% with a quad core., so 0.15 * 8 = about 120% of one core total cpu load increase.) I used dd if=/dev/zero of=test bs=1M count=20000 to create a 20 GB test file. The same on my laptop's internal drive was about 75 MB/sec with roughly a 20% total cpu load (quad core, so that is about 160% of one core's load.) So the interesting thing is introduction of mount.ntfs actually lowered the total cpu load of running dd. I assume because it introduced delays that reduced the data movement so much. To repeat: with dd -> ext4 75 MB/sec data transfer and 160% cpu load (uses multiple cores. No single core overloaded.) with dd -> ntfs 25-30 MB/sec data transfer and only 120% cpu load (one core running mount.ntfs at 75% load) I actually find that kind of strange, but it was repeatable. I think it is pretty clear that mount.ntfs is the bottleneck. The open (and unimportant) question if it is just poorly written or if the fact it is based on fuse makes it inefficient no matter what. fyi: this is a quad core laptop, but I was running a 4 worker processor intensive app at the same time, so I had a 50% untilization of my cpus as a base load. The 15% and 20% overall loads I mention brought the base load up to 65% and 70% respectively. Since they were just getting the hyper-threading cpu bounce, the loads may results may of been different without the base load running in the background. I don't actually know how that works. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 09:01 +1300, Michael Hamilton wrote:

FUSE is about ease of coding and making anything pretend to be a filesystem, it is not about speed. If someone goes the FUSE route for a normal filesystem they're choosing to be slower. Perhaps someone does care about fast NTFS for Linux, but not enough to do all the work required for a kernel implementation.

I have the feeling that there are also legal reasons involved. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFUrIwACgkQtTMYHG2NR9WmuACfQgcmmBRFlxnf13s9bX/7ar2A dSgAn27jlyzbG7aD9N9INpJTEJ/M4rD/ =2AUC -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer wrote:

To repeat:

with dd -> ext4 75 MB/sec data transfer and 160% cpu load (uses multiple cores. No single core overloaded.)

with dd -> ntfs 25-30 MB/sec data transfer and only 120% cpu load (one core running mount.ntfs at 75% load)

I actually find that kind of strange, but it was repeatable.

I think it is pretty clear that mount.ntfs is the bottleneck.

I guess mount.ntfs is the FUSE binary?

The open (and unimportant) question if it is just poorly written or if the fact it is based on fuse makes it inefficient no matter what.

The FUSE cuases two context switches per operation, it is bound to have an impact. A factor 2.5-3 seems a lot, though. -- Per Jessen, Zürich (3.8°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Friday, 2013-03-29 at 08:42 -0400, Greg Freemyer wrote: ...

In the 70's it was not realistic to write data to consecutive sectors as the disk spun because cpus were just to slow. The solution was interleaving multiple data streams. You may recall we used to format drives with a interleave factor of 2 or even 3.

And more. I had one machine that I had to interleave at 8 or more. I actually formatted it several times at all values from 1 to 12, and then choose the faster one. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFVoT0ACgkQtTMYHG2NR9XPYACgjf6cwLuclESWz05oZfv/a72r OnAAn00Mm5Y/Lv92akDZf+F0etK1do1H =pmwM -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Per Jessen said the following on 03/29/2013 10:25 AM:

fuse does have a binary, that is the user space part.

ULTIMATELY everything gets executed by a binary. You could write a FUSE FS in perl, python, scala, ruby or even, $DEITY forbid!, javascript, but ULTIMATELY the interpreters for those languages are binaries. It strikes me that first working out the logic & algorithms of a new FS in a HLL makes a lot of sense. The old rubric: First get it RIGHT then get it fast. -- Be extremely subtle, even to the point of formlessness. Be extremely mysterious, even to the point of soundlessness. Thereby you can be the director of the opponent's fate. Sun-tzu, The Art of War. Emptiness and Fullness -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 29/03/13 01:41, Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Thursday, 2013-03-28 at 16:05 +1100, Basil Chupin wrote:

...

On 28/03/13 15:50, Carlos E. R. wrote:

...

OK, OK, OK already! I'll do it! :-)

I'll wait impatiently :-)

Well, I used both "top" and the System Monitor>System Load simultaneously (each running in a different workspace) to see what was happening during the copy processes using mc to copy the 11.1GB file to ext4 and ntfs partitions. Sorry to disappoint, but I could not see any core "maxing out" during copying. The load was being distributed across 5 or possibly 6 cores (with wither 3 or 2 being almost idle). With the data jumping around from core to core it was just a bit hard to keep up :-) , but I did see that the max any single core reached in both cases (ext4, ntfs) was 74% - with, as I said, either 3 or 2 cores doing almost nothing). BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 29/03/13 14:18, Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Friday, 2013-03-29 at 13:20 +1100, Basil Chupin wrote:

...

Well, I used both "top" and the System Monitor>System Load simultaneously (each running in a different workspace) to see what was happening during the copy processes using mc to copy the 11.1GB file to ext4 and ntfs partitions.

Sorry to disappoint, but I could not see any core "maxing out" during copying. The load was being distributed across 5 or possibly 6 cores (with wither 3 or 2 being almost idle). With the data jumping around from core to core it was just a bit hard to keep up :-) , but I did see that the max any single core reached in both cases (ext4, ntfs) was 74% - with, as I said, either 3 or 2 cores doing almost nothing).

Curious.

So the load was distributed on the cores... interesting. That's more complicated to analyze. There may be other issues in play. I'm guessing memory moves, and thus, board and memory bandwidth, but dunno how it can be diagnosed. Alternatively, the task is jumping from core to core, but nevertheless it is not running paralellized. On average the load would be low, but the cpu where the job is running, peaks.

I found this in the wikipedia which does not bode well for the argument of FUSE versus in-kernel thinige: Performance Benchmarks show that the driver's performance via FUSE is comparable to that of other filesystems' drivers in-kernel,[6] provided that the CPU is powerful enough. On embedded or old systems, the high processor usage can severely limit performance.[7] Current versions often show 100% CPU utilization on dealing with big files on fragmented NTFS file systems.[8] http://en.wikipedia.org/wiki/Ntfs-3g This leads to the question whether will all the fiddling about with changing where the USB devices are now mounted (in places other then the normal /media) and other stuffing around with changed paths this has a detrimental effect on the way ntfs-3g is now handled in openSUSE. (BTW,in case someone brings up the issue of the performance being hampered because the ntfs partition(s) have not been defragmented, well before I redid the tests a short time ago I defragmented all ntfs formatted partitions. The results were almost identical to those of yesterday.) BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.4-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 29/03/13 21:59, Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Friday, 2013-03-29 at 20:48 +1100, Basil Chupin wrote:

...

I found this in the wikipedia which does not bode well for the argument of FUSE versus in-kernel thinige:

Performance

Benchmarks show that the driver's performance via FUSE is comparable to that of other filesystems' drivers in-kernel,[6] provided that the CPU is powerful enough. On embedded or old systems, the high processor usage can severely limit performance.[7] Current versions often show 100% CPU utilization on dealing with big files on fragmented NTFS file systems.[8]

http://en.wikipedia.org/wiki/Ntfs-3g

It actually agrees with what we said :-)

You do not see speed degradation _if_ the CPU is powerful enough. In other words, you need a powerful CPU to use FUSE fast. And this is specially true with ntfs-3g.

Sorry, but right at this moment my finances do not permit me buying a Cray :-) . Maybe next month, perhaps :-) .

And I think that you not also you not only need a powerful CPU, but also fast memory and buses.

Ummm, an 8-core 3.6GHz cpu with (16GB of) 1866MHz RAM and with a bus which can do 5.2GT/s still too slow, eh? Ah well, next time I'll do all this downwind so that perhaps I'll get some assistance from the wind to speed things up :-) .

This leads to the question whether will all the fiddling about with changing where the USB devices are now mounted (in places other then the normal /media) and other stuffing around with changed paths this has a detrimental effect on the way ntfs-3g is now handled in openSUSE.

No, not at all. That would not have any effect at all. Mounting options, yes.

Another posibility is USB3. I don't know if the design works in hardware or needs lots of CPU assistance. Some of the USB2 designs need more CPU than others. The wikipedia may talk about this.

...

(BTW,in case someone brings up the issue of the performance being hampered because the ntfs partition(s) have not been defragmented, well before I redid the tests a short time ago I defragmented all ntfs formatted partitions. The results were almost identical to those of yesterday.)

Your partitions were fresh, you said, so that should not be an issue.

But the partitions from which I was copying the files were not and which is why I defragmented them just to see if that made a difference. Leave nothing to chance, I always say :-) . BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 31/03/13 22:00, Per Jessen wrote:

Basil Chupin wrote:

...

...

You do not see speed degradation _if_ the CPU is powerful enough. In other words, you need a powerful CPU to use FUSE fast. And this is specially true with ntfs-3g. Sorry, but right at this moment my finances do not permit me buying a Cray :-) . Maybe next month, perhaps :-) .

...

And I think that you not also you not only need a powerful CPU, but also fast memory and buses. Ummm, an 8-core 3.6GHz cpu with (16GB of) 1866MHz RAM and with a bus which can do 5.2GT/s still too slow, eh? Probably. Never mind your eight cores, it's only one that matters when the workload is single-threaded.

Well, as I mentioned 5 or 6 cores were involved and not just one single one. I haven' seen anything to convince me that only 1 single core was involved or should only be involved.

Basil, it's a pretty longwinded thread by now, I am having trouble working out the current status.

The thread has taken on a life of its own. Or as they say on YouTube, the thread has gone "viral".

Am I right in thinking this is still about explaining why you get different IO-rates on two different filesystems (ext4 and ntfs)?

Yes, from my point of view the question still remains as to why there is such a difference between transfer rates. The entry in the wikipedia is indicating that such a large difference should not exist. I am not a technician, but to me to read a byte of data and then write it to another device and then to verify that what was written to the destination is the same as the original doesn't require a ming-boggling process of what some people are indicating happens with FUSE vs the kernel-thing. BC -- Using openSUSE 12.3 x86_64 KDE 4.10.1 & kernel 3.8.5-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2013-04-04 at 20:35 +1100, Basil Chupin wrote:

On 31/03/13 22:00, Per Jessen wrote:

...

Basil Chupin wrote:

...

Probably. Never mind your eight cores, it's only one that matters when the workload is single-threaded.

Well, as I mentioned 5 or 6 cores were involved and not just one single one. I haven' seen anything to convince me that only 1 single core was involved or should only be involved.

Appeareances aside, it is only one core. I mean, the important process is not working in parallell on several cores, which would speed it up (hopefully), but instead it runs in one core, then jumps to another core, then jumps to another core. If this is happening fast, what you see is a small load on all cores.

...

Am I right in thinking this is still about explaining why you get different IO-rates on two different filesystems (ext4 and ntfs)?

Yes, from my point of view the question still remains as to why there is such a difference between transfer rates. The entry in the wikipedia is indicating that such a large difference should not exist.

I read is as the contrary :-)

I am not a technician, but to me to read a byte of data and then write it to another device and then to verify that what was written to the destination is the same as the original doesn't require a ming-boggling process of what some people are indicating happens with FUSE vs the kernel-thing.

With FUSE there are several more transfers, from memory to memory. Memory is fast, but it adds. Another issue that can happen is that the delay can be small, but enough so that the head in the rotating disk has already moved to the next sector by the time the kernel is ready to write it, and thus it has to wait for a full rotation before it writes. This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes. - -- Cheers, Carlos E. R. (from 12.1 x86_64 "Asparagus" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEARECAAYFAlFe+SAACgkQtTMYHG2NR9Wj3wCdE5ksxaPmZkqJ95KPqgx9lUhB 88YAnj+ub1En41+iWR6aVS97MSogKl39 =4vQA -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Apr 5, 2013 at 12:27 PM, Per Jessen wrote:

Carlos E. R. wrote:

...

This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes.

Disks don't do write-caching. That is left to hardware controllers with battery-backed cache.

ISTR that sata drives have a very small write cache. For some reason I'm remembering up to 7 writes can be cached. For streamed data like BC is talking about, that is effectively nothing. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, Apr 5, 2013 at 3:39 PM, Per Jessen wrote:

Greg Freemyer wrote:

...

On Fri, Apr 5, 2013 at 12:27 PM, Per Jessen wrote:

...

Carlos E. R. wrote:

...

This should be diminished by good disk hardware with a /write/ ram cache in the firmware. This cache is pretty small, and I'm not sure it caches writes.

Disks don't do write-caching. That is left to hardware controllers with battery-backed cache.

ISTR that sata drives have a very small write cache. For some reason I'm remembering up to 7 writes can be cached. For streamed data like BC is talking about, that is effectively nothing.

Does it need to be explicitly enabled? Seems odd for the drive to report "data written to disk" unless it has?

I just used hdparm -I /dev/sda to check my laptop drive. It has a 8MB cache and it is enabled. I'm running a default openSUSE install, so I'd say it defaults to on with openSUSE. OTOH, read http://yarchive.net/comp/linux/drive_caches.html. There Linus says that the cache is typically made of large segments on the order of an entire track. I think a full track is about 1 MB currently, so I've got roughly 8 tracks worth of write cache on my low end laptop drive. If I'm doing linear writes like in BC's workload, it should work great at accepting more data. If a single sectors data was late arriving, the cpu could just keep pushing data out while the platter is rotating back around. That is the cpu should not get hit with a 10 msec penalty just because it was a microsecond late pushing out a given sector. Instead the cpu can just go as fast as it can as long as there is a cache segment free. In the BC workload case where the drive is not running at full speed, there is probably always a few free segments in the drives cache. I "think" there are cache flush commands the kernel can push to the drive to ensure the cache is flushed prior to moving forward. I don't know if the linux kernel actually pushes those types of barrier calls out to sata drives or not. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

BC, I happened to need to copy 1 TB of data between 2 NTFS USB-3 drives today. I only had a windows laptop handy and being who I am, I decided to boot a openSUSE boot cd/dvd instead of using windows. The boot cd/dvd I had handy was OS 12.2 based. I had built it myself (http://susestudio.com/a/eD1wrT/dfir-opensuse-gnome-desktop-32bit). I used the CLI and did cp -a /mnt/data /mnt1/data. I was running without X, just the alt-f2 style terminals. I got 90 MB/sec at the start of the process (per iostat -d 10). My data files were all 1.4GB each, so this was a large file copy test only. I haven't had the opportunity to test this with OS 12.3, but I'm very curious if OS 12.3 is slower than OS 12.2 for this or if you have something weird on your PC. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 2013-04-12 16:55 (GMT+1000) Basil Chupin composed:

Just to refresh the memories, I have an external USB3 HDD (a Seagate 2TB) which I split into 2 partitions: partition 1 formatted in ext4 and partition 2 formatted in ntfs.

Presumably, each is about half the entire HD? If so, and #1 is at the front of the HD and #2 brings up the rear, then you can expect #2 to be slower even if using the same filesystem type as #1. http://fm.no-ip.com/PC/bench/Sysbench/resul606.txt is typical of benchmarking I've done in the past, and clearly shows a big speed advantage near the front on two different 160GB HDs. I tried using the dbench utility to test my own Seagate 2TB HD on eSATA that is half EXT2 and half NTFS, but it produced even more bizarre results: front/EXT2 half of HD: http://fm.no-ip.com/PC/bench/dbench-ext2.txt Throughput 431.269 MB/sec last/NTFS half of HD: http://fm.no-ip.com/PC/bench/dbench-ntfs.txt Throughput 26.5151 MB/sec -- "The wise are known for their understanding, and pleasant words are persuasive." Proverbs 16:21 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 12/04/13 18:54, Per Jessen wrote:

Felix Miata wrote:

...

On 2013-04-12 16:55 (GMT+1000) Basil Chupin composed:

...

Just to refresh the memories, I have an external USB3 HDD (a Seagate 2TB) which I split into 2 partitions: partition 1 formatted in ext4 and partition 2 formatted in ntfs. Presumably, each is about half the entire HD? If so, and #1 is at the front of the HD and #2 brings up the rear, then you can expect #2 to be slower even if using the same filesystem type as #1. Yes, I saw this too a while ago when I was testing some 3Tb drives. I don't think the differences were as significant as what Basil's been seeing though.

Look, I got all hot and bothered about this toing and froing about whether the front part of the HDD was formatted in whatever versus the backup being formatted in whatever (I have been reading the posts about the 1/3 part of an HDD being faster than the rest while someone else claimed that it is the 1/4 of the HDD which was the fastest while another claimed that it was all BS because the latest HDD don't do what the HDD did years ago, and so on and so forth, and dominus vobiscum etc.... So here is what I did. I took the 2TB external HDD and format the whole damn thing in (a) ntfs and then the whole damn thing in (b) ext4. I then used SystemRescueCD to copy the same file I mentioned in the earlier post (7,631,404,437 bytes) sitting on an *ntfs* formatted partition on an internal 1TB HDD to the external USB3 2TB HDD. With SystemRescueCD there was NO graphics involved. Copying the 7.6 GB file using mc (midnight commander) to the external HDD *all* of which was in ntfs, produced a result of ~27.4MB/s transfer rate. After I formatted the whole 2TB HDD in ext4, copying the same file using mc produced a transfer rate of ~79MB/s. In all these trials the same internal HDDs were used, the same USB3 cable was used, the same file(s) were copied using either the CLI command 'cp', or the app 'mc', and in the very latest trials (as in earlier today and tonight) using SystemRescue CD which does not use the openSUSE 12.2 or the 12.3 kernels and whatevers. The summary of all the results is that copying to an external USB3 device formatted in ntfs is 34% SLOWER than when copying the identical file to an ext4 formatted partition. Now, I am nowhere close to being an expert in any field, but trying to accept that copying a file to partition formatted in ntfs as against copying to an ext4 partition incurs an overhead of 34% is just a bit too much to accept gracefully. However, as I said, I am not an expert and will accept this (even though wikipedia doesn't support this difference in speeds). OK , taking all this into account, what can be wrong at *my* end, in my system, which could cause such a speed difference? BC -- Using openSUSE 12.3 x86_64 KDE 4.10.2 & kernel 3.8.6-2 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 12/04/13 23:36, Greg Freemyer wrote:

...

OK , taking all this into account, what can be wrong at *my* end, in my

system, which could cause such a speed difference?

BC Basil,

I have to copy 500 GB of data from ntfs to ntfs via usb3 today.

I will use opensuse 12.3 this time.

Separately, ntfs has a auto-document indexing feature. I keep it disabled by default. Do you know if you have it enabled? I think a vanilla mkfs.ntfs has it enabled. There is a flag to disable it. I did not think opensuse even had support for that ntfs feature, but it is one of the few issues I can imagine it being.

Never heard of this "auto-document indexing" feature and, therefore, don't know if it is enabled or not. Where can one find this flag and how can one disable it if it does exist? BC -- Using openSUSE 12.3 x86_64 KDE 4.10.2 & kernel 3.8.6-2 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On 13/04/13 02:49, Greg Freemyer wrote:

On Fri, Apr 12, 2013 at 9:36 AM, Greg Freemyer wrote:

...

...

OK , taking all this into account, what can be wrong at *my* end, in my

system, which could cause such a speed difference?

BC Basil,

I have to copy 500 GB of data from ntfs to ntfs via usb3 today.

I will use opensuse 12.3 this time.

Separately, ntfs has a auto-document indexing feature. I keep it disabled by default. Do you know if you have it enabled? I think a vanilla mkfs.ntfs has it enabled. There is a flag to disable it. I did not think opensuse even had support for that ntfs feature, but it is one of the few issues I can imagine it being.

Greg

I had no speed issues with OS 12.3 either, so both OS 12.2 and 12.3 seem fine to me. ie. 90 MB/sec reading from NTFS USB-3 and writing to NTFS USB-3 via a simple cp -a command.

This is with different disks, cables and laptop from what I did yesterday. (Yesterday was a 12 month old HP laptop. Today was a 18-month old Dell Laptop. Both have native / built-in USB-3 support for at least 2 ports.)

Both yesterday and today, I was copying 1.4GB files. Yesterday I did about 1 TB, today turned out to only be 80 GB, but still big enough to be a good test.

Basil, since you're willing to reformat the drive, try to format it with:

mkfs.ntfs --no-indexing --fast /dev/sdb1 (Or whatever your device is).

OK, Time out! Stop the clock! Yesterday I did NOT do the above - reformat the drive(s) with the above cli but what I *did* do is to: a) hook up the *second* external HDD formatted in ntfs, and transferred the "test" file of 7,631,404,437 bytes to it from the internal SATA3 HDD partition which is formatted in ntfs - the transfer rate was the usual ~27MB/s; b) I then emulated what you did which was to cp that file from one external USB3 HDD to the other USB3 HDD using the SystemRescueCD and got a transfer rate of 109.0MB/s. Now, I simply do not understand any of this nor have an explanation for this - except to say that before I did the above test there were some updates made to the 12.3 - but this should not have anything to do with it because I used SystemRescueCD which is using a different version of the kernel and the 12.3 OS doesn't have anything to do with the copying processing - or those it? I really don't know. [pruned] BC -- Using openSUSE 12.3 x86_64 KDE 4.10.2 & kernel 3.8.7-1 on a system with- AMD FX 8-core 3.6/4.2GHz processor 16GB PC14900/1866MHz Quad Channel Corsair "Vengeance" RAM Gigabyte AMD3+ m/board; Gigabyte nVidia GTX550Ti 1GB DDR5 GPU -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

On Fri, 12 Apr 2013 23:24:04 +1000 Basil Chupin wrote: <snipped>

Copying the 7.6 GB file using mc (midnight commander) to the external HDD *all* of which was in ntfs, produced a result of ~27.4MB/s transfer rate.

After I formatted the whole 2TB HDD in ext4, copying the same file using mc produced a transfer rate of ~79MB/s. <snipped> The summary of all the results is that copying to an external USB3 device formatted in ntfs is 34% SLOWER than when copying the identical file to an ext4 formatted partition.

Hi Basil, I admire your persistence, if not your math. :-) 27.4 MB/sec is 34.7% of 79.0 MB/sec. That's almost a 3:1 ratio, ext4:ntfs, so copying to the ntfs formatted device takes almost three times as long. *Now* I 'grok' the impetus of this investigation. It's 65.3% slower, not "34% SLOWER"! Great work, otherwise. Thanks for sharing. regards, Carl -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Basil Chupin wrote:

This leads to the question whether will all the fiddling about with changing where the USB devices are now mounted (in places other then the normal /media) and other stuffing around with changed paths this has a detrimental effect on the way ntfs-3g is now handled in openSUSE.

Mounting point and filesystem paths have no bearing the performance. If you tie the ntfs-3g FUSE to one core (taskset -p), and repeat the copy operation (cp or dd), I expect you will see one core maxed out. The FUSE itself is not multi-threading, but it will handle multiple concurrent requests. One copy operation is almost certainly single-threaded, though. -- Per Jessen, Zürich (3.6°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R. wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Friday, 2013-03-29 at 13:20 +1100, Basil Chupin wrote:

...

Well, I used both "top" and the System Monitor>System Load simultaneously (each running in a different workspace) to see what was happening during the copy processes using mc to copy the 11.1GB file to ext4 and ntfs partitions.

Sorry to disappoint, but I could not see any core "maxing out" during copying. The load was being distributed across 5 or possibly 6 cores (with wither 3 or 2 being almost idle). With the data jumping around from core to core it was just a bit hard to keep up :-) , but I did see that the max any single core reached in both cases (ext4, ntfs) was 74% - with, as I said, either 3 or 2 cores doing almost nothing).

Curious.

So the load was distributed on the cores... interesting. That's more complicated to analyze. There may be other issues in play. I'm guessing memory moves, and thus, board and memory bandwidth, but dunno how it can be diagnosed. Alternatively, the task is jumping from core to core, but nevertheless it is not running paralellized.

Every time the user-space filesystems needs to talk to the kernel, it needs two context-switches. (from unprivileged to privileged and back). If the FUSE does not have processor affinity, I guess it could be hopping about between processors. -- Per Jessen, Zürich (3.6°C) http://www.dns24.ch/ - free DNS hosting, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org