[opensuse-support] data moving tuning
Hi, all -- I am building a system for fast transfer of video files from MicroSD cards, and I am looking for tunables advice. It will also be my everyday desktop, since I can't afford two machines, but its primary mission is to move data fast. My plan is to use 15.1 and, without starting another religious war :-) and after the convenient recent file system discussion, leave my beloved ReiserFS to use XFS for the large data partition. The box has a single Ryzen CPU, an m.2 SSD, 16G of RAM that I will bump to 32G as soon as funding permits, 10 gigabit motherboard USB, and a second 10 gigabit USB daughter card. I'm limited to this hardware by budget, so there isn't much else that I can do there, but I wonder about parameters and settings in the OS or perhaps the file system config. Are there any knobs to turn to speed moving data across the bus? Is there a better version or even -- gasp -- a better flavor for this kind of work? Any suggestions in general? TIA & HAND :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
On Thu 19 Sep 2019 10:43:48 PM CDT, David T-G wrote:
Hi, all --
I am building a system for fast transfer of video files from MicroSD cards, and I am looking for tunables advice. It will also be my everyday desktop, since I can't afford two machines, but its primary mission is to move data fast.
<snip> Hi The only thing I would suggest is picking the best i/o scheduler to use for the SD card... -- Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890) Tumbleweed 20190918 | GNOME Shell 3.32.2 | 5.2.14-1-default Intel DQ77MK MB | Xeon E3-1245 V2 X8 @ 3.40 GHz | Intel/Nvidia up 5:17, 2 users, load average: 2.00, 2.26, 1.61 -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Malcolm, et al -- ...and then Malcolm said... % % On Thu 19 Sep 2019 10:43:48 PM CDT, David T-G wrote: % % >I am building a system for fast transfer of video files from MicroSD ... % <snip> % Hi % The only thing I would suggest is picking the best i/o scheduler to use % for the SD card... Thanks! Sounds good. Um, how do I select a scheduler? :-) Thanks again & HAND :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
On Sat 21 Sep 2019 12:05:42 AM CDT, David T-G wrote:
Malcolm, et al --
...and then Malcolm said... % % On Thu 19 Sep 2019 10:43:48 PM CDT, David T-G wrote: % % >I am building a system for fast transfer of video files from MicroSD ... % <snip> % Hi % The only thing I would suggest is picking the best i/o scheduler to use % for the SD card...
Thanks! Sounds good. Um, how do I select a scheduler? :-)
Thanks again & HAND
:-D Hi Here you go: https://doc.opensuse.org/documentation/leap/tuning/html/book.sle.tuning/cha....
I use mq-deadline for solid state, bfq for rotating rust. -- Cheers Malcolm °¿° SUSE Knowledge Partner (Linux Counter #276890) Tumbleweed 20190918 | GNOME Shell 3.32.2 | 5.2.14-1-default Intel DQ77MK MB | Xeon E3-1245 V2 X8 @ 3.40 GHz | Intel/Nvidia up 14:27, 2 users, load average: 0.36, 0.36, 0.62 -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Hello, On Sat, 21 Sep 2019, David T-G wrote:
Malcolm, et al -- % The only thing I would suggest is picking the best i/o scheduler to use % for the SD card...
Thanks! Sounds good. Um, how do I select a scheduler? :-)
RTFM: /usr/src/linux/Documentation/block/switching-sched.txt and /usr/src/linux/Documentation/kernel-parameters.txt (for elevator=) # cat /sys/block/sd?/queue/scheduler # echo cfq > /sys/block/sdX/queue/scheduler E.g. I have: # cat /sys/block/sd?/queue/scheduler [noop] deadline cfq noop deadline [cfq] [..] sda is an ssd, the rest is rotating rust. Oh, and I boot with: # cat /proc/cmdline [..] elevator=cfq [..] and then have ==== ssddevs=( /dev/disk/by-id/ata-... ) for dev in "${ssddevs[@]}"; do d=$(readlink "$dev" | sed 's@\.\./\.\./@@') echo noop > /sys/block/${d}/queue/scheduler done ==== in an early local boot script. HTH, -dnh -- Going on a year now, I ain't had nothing twixt my nethers weren't run on batteries -- Kaylee, Serenity -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
21.09.2019 15:59, David Haller пишет:
Hello,
On Sat, 21 Sep 2019, David T-G wrote:
Malcolm, et al -- % The only thing I would suggest is picking the best i/o scheduler to use % for the SD card...
Thanks! Sounds good. Um, how do I select a scheduler? :-)
RTFM: /usr/src/linux/Documentation/block/switching-sched.txt and /usr/src/linux/Documentation/kernel-parameters.txt (for elevator=)
# cat /sys/block/sd?/queue/scheduler # echo cfq > /sys/block/sdX/queue/scheduler
E.g. I have:
# cat /sys/block/sd?/queue/scheduler [noop] deadline cfq noop deadline [cfq]
15.1 switched to multiqueue framework, none of those schedulers is present here. -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Hello, On Sat, 21 Sep 2019, Andrei Borzenkov wrote:
21.09.2019 15:59, David Haller ?????:
Hello,
On Sat, 21 Sep 2019, David T-G wrote:
Malcolm, et al -- % The only thing I would suggest is picking the best i/o scheduler to use % for the SD card...
Thanks! Sounds good. Um, how do I select a scheduler? :-)
RTFM: /usr/src/linux/Documentation/block/switching-sched.txt and /usr/src/linux/Documentation/kernel-parameters.txt (for elevator=)
# cat /sys/block/sd?/queue/scheduler # echo cfq > /sys/block/sdX/queue/scheduler
E.g. I have:
# cat /sys/block/sd?/queue/scheduler [noop] deadline cfq noop deadline [cfq]
15.1 switched to multiqueue framework, none of those schedulers is present here.
That's why I used 'cat /sys/block/sd?/queue/scheduler' to have people have a look at the available schedulers (which depends on the precise kernel-config) and then use whatever they deem (or find out through experiments works) best. And I pointed to the actual documentation. -dnh -- /* After several hours of tedious analysis, the following hash * function won. Do not mess with it... -DaveM */ linux-2.2.16/fs/buffer.c -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
David, et al -- ...and then David Haller said... % % Hello, % % On Sat, 21 Sep 2019, David T-G wrote: % > % >Thanks! Sounds good. Um, how do I select a scheduler? :-) % % RTFM: /usr/src/linux/Documentation/block/switching-sched.txt % and /usr/src/linux/Documentation/kernel-parameters.txt (for elevator=) Thank you again :-) % % # cat /sys/block/sd?/queue/scheduler [snip] Looks like I have [none] mq-deadline kyber bfq and so mq-deadline for my nvme0n1 might be nice. But I'll go and read, too... HAND :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
On 20/09/2019 04.43, David T-G wrote:
Hi, all --
I am building a system for fast transfer of video files from MicroSD cards, and I am looking for tunables advice. It will also be my everyday desktop, since I can't afford two machines, but its primary mission is to move data fast.
My plan is to use 15.1 and, without starting another religious war :-) and after the convenient recent file system discussion, leave my beloved ReiserFS to use XFS for the large data partition. The box has a single Ryzen CPU, an m.2 SSD, 16G of RAM that I will bump to 32G as soon as funding permits, 10 gigabit motherboard USB, and a second 10 gigabit USB daughter card.
I'm limited to this hardware by budget, so there isn't much else that I can do there, but I wonder about parameters and settings in the OS or perhaps the file system config. Are there any knobs to turn to speed moving data across the bus? Is there a better version or even -- gasp -- a better flavor for this kind of work? Any suggestions in general?
TIA & HAND
:-D
Once I timed writing very large file (ISO image) from hard disk to an USB stick (which is the reverse of what you will be doing). What happened is that the stick are slow writing, but the entire machine was slowed up. The hypothesis we had is that the system cache memory was used up. I did a few tries, and this is what worked best (faster): dd if=ISO of=DEV bs=16M oflag=direct status=progress This allows to write several sticks at the same time without big impact. Not your case, but the idea is that you can try things :-) You are reading from card to hard disk. If it is USB3 to rotating rust, the limitation will be the write speed of the HD and you will hit it easily, perhaps doing 150 MB/S. If the destination is m.2, who knows. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
Carlos, et al -- ...and then Carlos E. R. said... % % Once I timed writing very large file (ISO image) from hard disk to an ... % slowed up. The hypothesis we had is that the system cache memory was % used up. I did a few tries, and this is what worked best (faster): That's interesting. On my laptop I tend to run into the same SERIOUS slowdown whenever pulling data, either via the built-in SD card slot or the USB readers. % % dd if=ISO of=DEV bs=16M oflag=direct status=progress Oho! That is *very* interesting. % % This allows to write several sticks at the same time without big impact. % % Not your case, but the idea is that you can try things :-) Indeed :-) Now I wonder if it might actually be more productive to image the cards to files on the SSD and then mount *those* to xfer at leisure. I'm definitely going to have to benchmark that! % % You are reading from card to hard disk. If it is USB3 to rotating rust, % the limitation will be the write speed of the HD and you will hit it % easily, perhaps doing 150 MB/S. If the destination is m.2, who knows. Yep. If I do go with spinning secondary, it will be straight striping (no fault protection) across multiple devices (the more heads the better for all practical purposes). Thanks again & HANN :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
On 24/09/2019 05.02, David T-G wrote:
Carlos, et al --
...and then Carlos E. R. said... % % Once I timed writing very large file (ISO image) from hard disk to an ... % slowed up. The hypothesis we had is that the system cache memory was % used up. I did a few tries, and this is what worked best (faster):
That's interesting. On my laptop I tend to run into the same SERIOUS slowdown whenever pulling data, either via the built-in SD card slot or the USB readers.
% % dd if=ISO of=DEV bs=16M oflag=direct status=progress
Oho! That is *very* interesting.
% % This allows to write several sticks at the same time without big impact. % % Not your case, but the idea is that you can try things :-)
Indeed :-) Now I wonder if it might actually be more productive to image the cards to files on the SSD and then mount *those* to xfer at leisure. I'm definitely going to have to benchmark that!
I believe that writing big enough files to SSD may hit a limitation on the SSD of continuous writing. In this case, I think that reading the files out of the card will be faster than imaging it first.
% % You are reading from card to hard disk. If it is USB3 to rotating rust, % the limitation will be the write speed of the HD and you will hit it % easily, perhaps doing 150 MB/S. If the destination is m.2, who knows.
Yep. If I do go with spinning secondary, it will be straight striping (no fault protection) across multiple devices (the more heads the better for all practical purposes).
Thanks again & HANN
:-D
Welcome :-) -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
On Thu, 19 Sep 2019 22:43:48 -0400 David T-G <davidtg-robot@justpickone.org> wrote:
Hi, all --
I am building a system for fast transfer of video files from MicroSD cards, and I am looking for tunables advice. It will also be my everyday desktop, since I can't afford two machines, but its primary mission is to move data fast.
My plan is to use 15.1 and, without starting another religious war :-) and after the convenient recent file system discussion, leave my beloved ReiserFS to use XFS for the large data partition. The box has a single Ryzen CPU, an m.2 SSD, 16G of RAM that I will bump to 32G as soon as funding permits, 10 gigabit motherboard USB, and a second 10 gigabit USB daughter card.
As well as the answers you get here, it might be worth asking on the XFS list as to how best to tune the filesystem. IIRC there are various tweaks possible with XFS, some of which may involve reformatting.
I'm limited to this hardware by budget, so there isn't much else that I can do there, but I wonder about parameters and settings in the OS or perhaps the file system config. Are there any knobs to turn to speed moving data across the bus? Is there a better version or even -- gasp -- a better flavor for this kind of work? Any suggestions in general?
TIA & HAND
:-D
-- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Dave, For *micro sd* card, the limits are * the card itself * the reader, usb3 being much better (many built-in card reader are not even usb2). Much better are the SD readers with real micro SD slot (I have one). Else, one have to use better recordings medium, like Atomos devices with ssd. But there recording format is prores or dnxhd 50 gb an hour 😫 (I also have one). The computer is not faulty. I have 16 gb ram and kdenlive do not use it well. If you only have to cut pieces, avidemux3 in copy mode is OK, but to really edit you need DaVinci Resolve (the free version is stunning) Jdd -- Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.
Jdd, et al -- ...and then Jdd said... % % Dave, % For *micro sd* card, the limits are % * the card itself True enough. You need something like these (UGH what obnoxious URLs) https://shop.progradedigital.com/products/prograde-digital-microsdx-uhs-ii-v... https://www.bhphotovideo.com/c/product/1463540-REG/lexar_lsdmi32gcbna1000a_3... to really get decent performance. The days of $8 cards are over :-/ % * the reader, usb3 being much better (many built-in card reader are not % even usb2). Much better are the SD readers with real micro SD slot (I % have one). I had to work pretty hard to track down not only card readers that would speak 3.1 Gen2 but a USB daughtercard to drive them. I think these will work nicely, though: https://www.frys.com/product/8951353 https://www.amazon.com/dp/B073WMMTB4 % Else, one have to use better recordings medium, like Atomos devices % with ssd. But there recording format is prores or dnxhd 50 gb an hour % ???? (I also have one). I wish, but the cameras only take uSD :-/ % The computer is not faulty. I have 16 gb ram and kdenlive do not use it % well. If you only have to cut pieces, avidemux3 in copy mode is OK, but % to really edit you need DaVinci Resolve (the free version is stunning) Fortunately, I don't do the editing; she does that on her iMac. I just have to shovel it :-) Thanks again & HANN :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Dave, et al -- ...and then Dave Howorth said... % % On Thu, 19 Sep 2019 22:43:48 -0400 % David T-G <davidtg-robot@justpickone.org> wrote: % ... % > My plan is to use 15.1 and, without starting another religious war :-) % > and after the convenient recent file system discussion, leave my % > beloved ReiserFS to use XFS for the large data partition. The box ... % % As well as the answers you get here, it might be worth asking on the % XFS list as to how best to tune the filesystem. IIRC there are various Thanks! That makes sense, and I'll track 'em down. % tweaks possible with XFS, some of which may involve reformatting. [snip] Fortunately, the system will be brand new, so reformatting won't be an issue :-) HANN :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Dave, et al -- ...and then Dave Howorth said... % % On Thu, 19 Sep 2019 22:43:48 -0400 % David T-G <davidtg-robot@justpickone.org> wrote: % ... % > and after the convenient recent file system discussion, leave my % > beloved ReiserFS to use XFS for the large data partition. The box ... % % As well as the answers you get here, it might be worth asking on the % XFS list as to how best to tune the filesystem. IIRC there are various % tweaks possible with XFS, some of which may involve reformatting. [snip] In the end, apparently it's best to just let mkfs.xfs do its thing since it's pretty smart and designed for the large-ish files I plan anyway. So that turned out to be really easy :-) and we'll see if I need future tuning once I have some real experience on this machine. Thanks again & HAND :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
Hi again, all -- Just a quick reply to say THANKS for the input so far. I look forward to all of this new reading, especially with pointers to TFM. Fortunately, this is a brand new system; I can reformat and reinstall as much as needed :-) Thanks again & stay tuned :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse-support+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-support+owner@opensuse.org
participants (7)
-
Andrei Borzenkov -
Carlos E. R. -
Dave Howorth -
David Haller -
David T-G -
Jdd -
Malcolm