-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2022-01-01 at 14:17 +0100, Carlos E. R. wrote:
On 01/01/2022 12.54, Carlos E. R. wrote:
On 31/12/2021 01.50, Bernhard Voelker wrote:
On 12/30/21 20:07, Carlos E. R. wrote:
...
So, it is the verification that is much worse, the compression time is much better. But as the size is doubled, the read effort to do the verification suffers a lot, it is USB2.
So, don't use --adapt. Instead, perhaps, try with different compression rations manually and compare the actual results - as I use a dedicated external disk for each computer that I backup, I can use tuned scripts for each.
Ok, doing some testing. The code is now: #!/bin/bash TIMESTAMPFILE=0_timestamp_dd_4 # Exit script on Control-C (signal 2) trap 'echo "Control-C pressed."; rm mdpipe ; exit 1;' 2 function hacer() { #parameters: device, estimated time, file name, device size, compression level DATE=`date --rfc-3339=seconds` SECS1=`date +%s` echo >> $TIMESTAMPFILE echo "Starting processing partition $1 ($2) to $3 (Level $5)" | tee -a $TIMESTAMPFILE echo " copying, compressing, and calculating md5..." | tee -a $TIMESTAMPFILE echo " $DATE Starting dd" | tee -a $TIMESTAMPFILE mkfifo mdpipe dd if=/dev/$1 status=progress bs=16M | tee mdpipe | zstd --size-hint=$4 -$5 > $3.zst & md5sum -b mdpipe | tee -a md5checksum_expanded wait rm mdpipe echo "$3" >> md5checksum_expanded SECS2=`date +%s` DATE=`date --rfc-3339=seconds` echo " $DATE Ending compression phase" | tee -a $TIMESTAMPFILE echo echo " Verifying..." zstd --test $3.zst DATE=`date --rfc-3339=seconds` SECS3=`date +%s` DIFF_COMP=`expr $SECS2 - $SECS1` DIFF_VERIF=`expr $SECS3 - $SECS2` DIFF_TOT=`expr $SECS3 - $SECS1` SIZE=`stat --printf="%s" $3.zst` SIZEM=`expr $SIZE / 1000000` echo " $DATE Ending verification phase and all ($DIFF_TOT = $DIFF_COMP + $DIFF_VERIF secs; $SIZEM MB)" | tee -a $TIMESTAMPFILE echo echo "·········" } echo "-----------" >> md5checksum_expanded DATE=`date --rfc-3339=seconds` SECS_TOT_1=`date +%s` echo | tee -a $TIMESTAMPFILE echo "==================" | tee -a $TIMESTAMPFILE echo "Start at $DATE" | tee -a $TIMESTAMPFILE echo "Doing test" time hacer sda9 "XX:YY" "sda9_Other_01" 7G 1 echo time hacer sda9 "XX:YY" "sda9_Other_02" 7G 2 echo ... time hacer sda9 "XX:YY" "sda9_Other_10" 7G 10 echo DATE3=`date --rfc-3339=seconds` SECS_TOT_2=`date +%s` DIFF_TOTAL=`expr $SECS_TOT_2 - $SECS_TOT_1` echo >> $TIMESTAMPFILE echo "End at $DATE3 (Total time= $DIFF_TOTAL S)" | tee -a $TIMESTAMPFILE There is no detection if the verification fails, don't know how to do that. Maybe an exit code from zstd? Could test for not zero. And the results are: (Level 1) (320 = 177 + 143 secs; 3027 MB) (Level 2) (318 = 176 + 142 secs; 3012 MB) <==== sweet point (Level 3) (335 = 195 + 140 secs; 3013 MB) (Level 4) (402 = 261 + 141 secs; 2851 MB) (Level 5) (431 = 303 + 128 secs; 2789 MB) (Level 6) (453 = 321 + 132 secs; 2785 MB) (Level 7) (458 = 325 + 133 secs; 2784 MB) (Level 8) (455 = 322 + 133 secs; 2784 MB) (Level 9) (600 = 469 + 131 secs; 2783 MB) (Level 10) (600 = 471 + 129 secs; 2783 MB) Of course, compressing a different partition with different files stored should alter the results a bit (sda9 is a small install of Leap 15.0 Beta, ext4 filesystem) Testing for verification failure: zstd --test $3.zst if [ ! $? ] then echo " Failed verification!" | tee -a $TIMESTAMPFILE fi Then I used "wxHexEditor sda9_Other_02.zst" and changed 2 bytes at random, and run the script skiping the writing phase. I get: Starting processing partition sda9 (XX:YY) to sda9_Other_02 (Level 2) copying, compressing, and calculating md5... 2022-01-01 22:47:26+01:00 Starting dd 2022-01-01 22:47:26+01:00 Ending compression phase Verifying... sda9_Other_02.zst : 2423 MB... sda9_Other_02.zst : Decoding error (36) : Destination buffer is too small 2022-01-01 22:47:58+01:00 Ending verification phase and all (done sda9 as sda9_Other_02 at level 2; 32 = 0 + 32 secs; 3012 MB) zstd prints an error to the screen, but the exitcode is still zero, no error detected. The importance is that my log file doesn't log the error; when compressing 9 partitions, if there is an error on the first one, it will flow out of the terminal and not be seen. So the function is now: function hacer() { #parameters: device, estimated time, file name, device size, compression level DATE=`date --rfc-3339=seconds` SECS1=`date +%s` echo >> $TIMESTAMPFILE echo "Starting processing partition $1 ($2) to $3 (Level $5)" | tee -a $TIMESTAMPFILE echo " copying, compressing, and calculating md5..." | tee -a $TIMESTAMPFILE echo " $DATE Starting dd" | tee -a $TIMESTAMPFILE mkfifo mdpipe dd if=/dev/$1 status=progress bs=16M | tee mdpipe | zstd --size-hint=$4 -$5 > $3.zst & md5sum -b mdpipe | tee -a md5checksum_expanded wait rm mdpipe echo "$3" >> md5checksum_expanded SECS2=`date +%s` DATE=`date --rfc-3339=seconds` echo " $DATE Ending compression phase" | tee -a $TIMESTAMPFILE echo echo " Verifying..." zstd --test $3.zst if [ ! $? ] then echo " Failed verification!" | tee -a $TIMESTAMPFILE fi DATE=`date --rfc-3339=seconds` SECS3=`date +%s` DIFF_COMP=`expr $SECS2 - $SECS1` DIFF_VERIF=`expr $SECS3 - $SECS2` DIFF_TOT=`expr $SECS3 - $SECS1` SIZE=`stat --printf="%s" $3.zst` SIZEM=`expr $SIZE / 1000000` echo " $DATE Ending verification phase and all (done $1 as $3 at level $5; $DIFF_TOT = $DIFF_COMP + $DIFF_VERIF secs; $SIZEM MB)" | tee -a $TIMESTAMPFILE echo echo "·········" } - -- Cheers, Carlos E. R. (from openSUSE 15.2 x86_64 at Telcontar) -----BEGIN PGP SIGNATURE----- iHoEARECADoWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCYdDO6hwccm9iaW4ubGlz dGFzQHRlbGVmb25pY2EubmV0AAoJELUzGBxtjUfViwYAn0UI+P/gi2/beuteXG0j t865eGexAJsH+04xyB7nIzfEP/IaZn+Dut2fNw== =STvO -----END PGP SIGNATURE-----