Hi, I had noticed that our rpm files and especially .drpm files can get a lot smaller by compressing them with xz or gzip. This is likely so, because all the metadata is uncompressed and only the payload (file content) is compressed, so that rpm -qp $RPMFILE can easily and quickly access all metadata. If you want to test yourself, you can use this: cat ~/bin/xzcurl #!/bin/sh url=$1 len=$(curl -s -L -I $url | awk 'BEGIN {RS="\r\n"} /^Content-Length: /{print $2}') comprlen=$(curl -s -L $url | xz | wc -c) echo "Len:$len Compr:$comprlen Saved:"$((len-$comprlen))\ " Ratio:$((100*$comprlen/$len))%" xzcurl http://download.opensuse.org/update/leap/15.0/oss/noarch/apache-pdfbox-javad... Len:216328 Compr:65144 Saved:151184 Ratio:30% xzcurl http://download.opensuse.org/update/leap/15.0/oss/noarch/apache-pdfbox-javad... Len:1379932 Compr:1224584 Saved:155348 Ratio:89% xzcurl http://download.opensuse.org/update/leap/15.0/oss/noarch/kernel-devel-4.12.1... Len:4766820 Compr:1229368 Saved:3537452 Ratio:26% xzcurl http://download.opensuse.org/update/leap/15.0/oss/noarch/kernel-devel-4.12.1... Len:14664108 Compr:11124512 Saved:3539596 Ratio:76% xzcurl http://download.opensuse.org/update/leap/15.0/oss/noarch/kernel-docs-html-4.... Len:3115424 Compr:957256 Saved:2158168 Ratio:31% xzcurl http://download.opensuse.org/update/leap/15.0/oss/noarch/kernel-docs-html-4.... Len:7771320 Compr:5609056 Saved:2162264 Ratio:72% xzcurl http://download.opensuse.org/update/leap/15.0/oss/x86_64/zutils-1.7-lp150.2.... Len:89184 Compr:79980 Saved:9204 Ratio:90% As you can see, the amount of saved bytes is nearly the same for the .drpm and .rpm files because they contain the same metadata. Now I was wondering: 1) how much effort would it be to patch librpm, libsolv, libzypp, createrepo, OBS and other tools to support .rpm.xz files. Maybe not all of them need a patch - e.g. if libzypp uncompresses files on the user side before passing them on to further processing, then libsolv and librpm dont need any change. 2) if it is worth the effort to save 10-70% of bandwidth and mirror storage space One simpler, yet efficient way to save bandwidth, would be to enable gzip compression during transfer via apache's mod_deflate (can be limited to the .drpm extension) E.g. kernel-docs-html-4.12.14-lp150.11.2_lp150.12.16.1.noarch.drpm still gives -64% savings with gzip => https://en.wikipedia.org/wiki/HTTP_compression With https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#precompressed or https://httpd.apache.org/docs/2.4/mod/mod_deflate.html#inflate this can even save disk-space by storing only .drpm.gz files on the server but delivering to clients whatever is requested. But then again, synchronizing such a mirror with rsync is harder, unless the rsync-source has .drpm.gz files. And that only works if all mirrors have mod_deflate configured properly... or libzypp is smartly patched I tested a bit with this on-the-fly compression: curl -v --compressed http://aw.zq1.de/rpm/test.rpm | wc -c curl -v -H "Accept-Encoding: gzip" http://aw.zq1.de/rpm/test.rpm | wc -c curl -v --compressed http://aw.zq1.de/rpm/kernel-docs-html.drpm | wc -c but both rpm -qpi http://aw.zq1.de/rpm/test.rpm zypper in -d http://aw.zq1.de/rpm/test.rpm request the uncompressed version atm https://build.opensuse.org/package/show/home:bmwiedemann:branches:zypp:Head/... has a tiny PoC compr.patch to allow it to save bandwidth towards servers that support it. But then, it would be nice to find a way to transfer compressed rpms that does not require mirror servers to be reconfigured. I guess, our distribution maintainers would not mind fitting 10-20% extra content on DVD / USB images. Would be possible with .rpm.xz or .rpm.gz files. Another place to save space + bandwidth is the repodata - those are currently compressed with gzip -9 --rsyncable but since xml file names change every time, rsyncable does not give a benefit there and we could just use xz instead. e.g. we get 35% savings there: wget -Oprimary.xml.gz http://download.opensuse.org/distribution/leap/15.0/repo/oss/repodata/71f232... gzip -cd primary.xml.gz | time xz -9 | wc -c 7071384/10883249 = 0.65 Any opinions on which way to go? Ciao Bernhard M. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org