On 01/01/2022 12.54, Carlos E. R. wrote:
On 31/12/2021 01.50, Bernhard Voelker wrote:
On 12/30/21 20:07, Carlos E. R. wrote:
...
with:
dd if=/dev/$1 status=progress bs=16M | \ tee mdpipe | zstd > $3.zst &"
No alternation! :-D
No constant reading speed, though. Maybe varies with compression rate of the current chunks.
(reading is from fast internal SSD; writing goes to external rotating rust over USB2)
Writing is intermittent. Means that the compression is doing its goal, compensating the slow i/o speed by having to write less bytes.
Very nice! :-D
I forgot to check CPU load.
2022-01-01 12:41:22+01:00 Starting dd: 2022-01-01 12:44:32+01:00 Ending compression phase: 2022-01-01 12:45:50+01:00 Ending verification phase and all:
time:
real 4m27.691s user 3m28.830s sys 0m48.674s
Size: 1.9GiB (2.1GB).
One more modification and test, after reading the manual a bit: dd if=/dev/$1 status=progress bs=16M | tee mdpipe | \ zstd --size-hint=$4 --adapt > $3.zst & --adapt[=min=#,max=#] zstd will dynamically adapt compression level to perceived I/O conditions. Com- pression level adaptation can be observed live by using command -v. Adaptation can be constrained between supplied min and max levels. The fea- ture works when combined with multi-threading and --long mode. It does not work with --single-thread. It sets window size to 8 MB by default (can be changed manually, see wlog). Due to the chaotic nature of dynamic adaptation, compressed result is not reproducible. note : at the time of this writing, --adapt can remain stuck at low speed when combined with multi- ple worker threads (>=2). --size-hint=# When handling input from a stream, zstd must guess how large the source size will be when optimizing compression parameters. If the stream size is rela- tively small, this guess may be a poor one, resulting in a higher compression ratio than expected. This feature allows for controlling the guess when needed. Exact guesses result in better compression ratios. Overestimates result in slightly degraded compression ratios, while underestimates may result in significant degradation. What it doesn't say is what units does the hint expect. I used "7G" and it did not complain. Now I have to consider how to calculate the size of a partition to feed that value automatically. 2022-01-01 13:34:42+01:00 Starting dd: 2022-01-01 13:37:02+01:00 Ending compression phase: 2022-01-01 13:39:37+01:00 Ending verification phase and all: real 4m55.131s user 1m25.702s sys 0m51.150s Size: 4.1GiB (4.3G) It actually takes more time to run, and the file size is doubled. Compression time is 2:20 (vs 3:10) Verification time is 2:35 (vs 1:18) Total time is 4:55 (vs 4:28) So, it is the verification that is much worse, the compression time is much better. But as the size is doubled, the read effort to do the verification suffers a lot, it is USB2. So, don't use --adapt. Instead, perhaps, try with different compression rations manually and compare the actual results - as I use a dedicated external disk for each computer that I backup, I can use tuned scripts for each. Or set min,max values for the hint switch - no, because the hint heuristics can not take into consideration the verification phase. (the default compression level is 3) -- Cheers / Saludos, Carlos E. R. (from 15.2 x86_64 at Telcontar)