New zypper media backend and parallel download support
Hello everyone, the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages: 1) zypper fetches packages during update/dup/install using parallel connections ( yes finally ). Currently the feature must be manually enabled by setting a environment variable like so: > env ZYPP_PCK_PRELOAD=1 zypper dup The number of concurrent connections is controlled by a setting in zypp.conf: download.max_concurrent_connections Currently the default is 5, but for me setting it higher performs even better. 2) A simpler more efficient media backend, it drops some of the overhead like fetching metalinks or splitting one file over multiple servers and should behave much better on connection reuse, in our internal test this always outperforms the current one. This is seperate from the parallel package downloads but should bring some benefits when fetching metadata from servers on zypper refresh. It's enabled with: > env ZYPP_CURL2=1 zypper ref Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower. Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0
It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now. Finally some numbers for those who enjoy it: Downloaded roughly 100Mb, around 250 Packages over my 100Mbit/s MultiCurl ( current default ): ------------------------------------------------------------------------------------------ Executed in 68.68 secs usr time 42.41 millis 0.38 millis 42.03 millis sys time 70.80 millis 1.02 millis 69.77 millis ZYPP_CURL2=1 enabled ( new experimental backend ): ------------------------------------------------------------------------------------------ Executed in 29.55 secs usr time 44.33 millis 0.00 millis 44.33 millis sys time 57.07 millis 1.42 millis 55.65 millis ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ------------------------------------------------------------------------------------------ Executed in 13.13 secs usr time 142.10 millis 0.38 millis 141.72 millis sys time 232.80 millis 1.02 millis 231.78 millis Download of 407 Packages ~1.02Gb into empty rootfs ( postgresql virtualbox go mariadb ruby ) MultiCurl ( current default ): ________________________________________________________ Executed in 281.15 secs usr time 83.34 millis 0.36 millis 82.99 millis sys time 186.04 millis 1.03 millis 185.01 millis ZYPP_CURL2=1 enabled ( new experimental backend ): ________________________________________________________ Executed in 208.51 secs usr time 76.77 millis 0.00 millis 76.77 millis sys time 168.91 millis 2.02 millis 166.89 millis ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ________________________________________________________ Executed in 119.60 secs usr time 1.92 secs 0.36 millis 1.92 secs sys time 3.58 secs 1.03 millis 3.58 secs Cheers, Benjamin
On 02. 03. 25, 23:57, Benjamin Zeller wrote:
> env ZYPP_PCK_PRELOAD=1 zypper dup ... > env ZYPP_CURL2=1 zypper ref
Hey man, that's super fast!
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/
Well, but none of non-oss, debug, update, or any other OBS repo I have contains metalink? So how do we use that feature there? thanks, -- js suse labs
03.03.2025 08:49, Jiri Slaby wrote:
On 02. 03. 25, 23:57, Benjamin Zeller wrote:
> env ZYPP_PCK_PRELOAD=1 zypper dup ... > env ZYPP_CURL2=1 zypper ref
Hey man, that's super fast!
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/
Well, but none of non-oss, debug, update, or any other OBS repo I have contains metalink?
My understanding is that it is generated on the fly. bor@bor-Latitude-E5450:~/src/dracut-ng$ curl https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 <?xml version="1.0" encoding="UTF-8"?> <metalink xmlns="urn:ietf:params:xml:ns:metalink"> <generator>MirrorCache</generator> <origin dynamic="true">https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml</origin> <published>2025-03-03T07:18:18Z</published> <publisher> <name>openSUSE</name> <url>download.opensuse.org</url> </publisher> <file name="repomd.xml"> ... So how do we use that feature there?
thanks,
On 3/3/25 08:20, Andrei Borzenkov wrote:
03.03.2025 08:49, Jiri Slaby wrote:
On 02. 03. 25, 23:57, Benjamin Zeller wrote:
> env ZYPP_PCK_PRELOAD=1 zypper dup ... > env ZYPP_CURL2=1 zypper ref
Hey man, that's super fast!
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/
Well, but none of non-oss, debug, update, or any other OBS repo I have contains metalink?
My understanding is that it is generated on the fly.
bor@bor-Latitude-E5450:~/src/dracut-ng$ curl https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 <?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink"> <generator>MirrorCache</generator> <origin dynamic="true">https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml</origin> <published>2025-03-03T07:18:18Z</published> <publisher> <name>openSUSE</name> <url>download.opensuse.org</url> </publisher> <file name="repomd.xml"> ...
So how do we use that feature there?
Zypp has some internal logic that it can use the URL for the repomd.xml metalink file as source for mirror informations for the whole repository, so it strips the repodata/repomd.xml.meta4 to get to the actual mirror URLs.. It then fills the base Url list for that repo with mirrors. Basically specifiying metalink= is the same as specifying multiple baseUrl entries in your repo file. So you give zypp a list of base Urls instead of just one it can select from or try when some of them fail. The code enabled with ZYPP_PCK_PRELOAD=1 takes advantage of that list and uses different mirrors for the parallel downloads. If you do not use metalink= but simply baseUrl=download.opensuse.org/repo then the server will automatically redirect you to a mirror for each request. Drawback with that is that the CURL connection cache might overflow and you need more http requests than going directly to the mirrors.
thanks,
On Mon, Mar 3, 2025 at 11:29 AM Benjamin Zeller <bzeller@suse.de> wrote:
Basically specifiying metalink= is the same as specifying multiple baseUrl entries in your repo file.
Does it mean I can manually set several preferred mirrors and use the new parallel download optoin? This would be some short term workaround when using tumbleweed-cli.
On Thu, Mar 6, 2025 at 5:41 AM Andrei Borzenkov <arvidjaar@gmail.com> wrote:
On Mon, Mar 3, 2025 at 11:29 AM Benjamin Zeller <bzeller@suse.de> wrote:
Basically specifiying metalink= is the same as specifying multiple baseUrl entries in your repo file.
Does it mean I can manually set several preferred mirrors and use the new parallel download optoin? This would be some short term workaround when using tumbleweed-cli.
That has always been possible, it's part of the yum repo file specification. You can choose to do one of the three choices: 1) use a list of baseurls by whitespace separated list (configparser style) 2) create a file of newline separated list and pass it in as mirrorlist= 3) use a metalink and pass it in as metalink= -- 真実はいつも一つ！/ Always, there's only one truth!
On 3/6/25 14:41, Andrei Borzenkov wrote:
On Mon, Mar 3, 2025 at 11:29 AM Benjamin Zeller <bzeller@suse.de> wrote:
Basically specifiying metalink= is the same as specifying multiple baseUrl entries in your repo file. Does it mean I can manually set several preferred mirrors and use the new parallel download optoin? This would be some short term workaround when using tumbleweed-cli. You can yes, but be aware that it will download gpg keys from the mirrors then.
One option would be to specifiy a local file that contains the mirrors you want . Format is just a simple textfile with one repo per line. Then replace baseUrl with metalink=/path/to/file. You also should be able to have multiple baseUrl= lines .. but I didn't verify that myself. Also don't forget that this feature is still in preview , we might change some behavior so don't rely too much on those workarounds :)
On 3/3/25 06:49, Jiri Slaby wrote:
On 02. 03. 25, 23:57, Benjamin Zeller wrote:
> env ZYPP_PCK_PRELOAD=1 zypper dup ... > env ZYPP_CURL2=1 zypper ref
Hey man, that's super fast!
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/
Well, but none of non-oss, debug, update, or any other OBS repo I have contains metalink? So how do we use that feature there? The metalink files are created on demand on download.opensuse.org, they are not part of the directory listing.
thanks,
-- Benjamin Zeller <bzeller@suse.de> Systems Programmer SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)
Benjamin Zeller kirjoitti 3.3.2025 klo 0.57:
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. ... > env ZYPP_PCK_PRELOAD=1 zypper dup ... > env ZYPP_CURL2=1 zypper ref ...
metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/ repomd.xml.meta4 ... ZYPP_PCK_PRELOAD=1 enabled (parallel downloads):
Executed in 119.60 secs
Awesome work, I've enabled them in my config, and installed libzypp and zypper separately first on my Tumbleweed so I was able to enjoy the features with zypper dup for the first time. So ZYPP_MEDIANETWORK=1 is gone now for good (https://github.com/openSUSE/libzypp/commit/f25304dee699a16b2fc018b59a00b3f6d...), replaced by ZYPP_CURL2? I do get a 'Backend: single_rpmtrans' line before "Continue" when running zypper (because using ZYPP_SINGLE_RPMTRANS=1), but I guess there is no such indication about PCK_PRELOAD os CURL2? Preloading is of course visible by the different download style ("Preloading" lines). Thank you for the hard work, I hope these get to be the default, maybe a phased rollout or something, at some suitable point in the future! Best Regards, Timo Jyrinki
On 3/3/25 09:48, Timo Jyrinki wrote:
Benjamin Zeller kirjoitti 3.3.2025 klo 0.57:
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. ... > env ZYPP_PCK_PRELOAD=1 zypper dup ... > env ZYPP_CURL2=1 zypper ref ...
metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/ repomd.xml.meta4 ... ZYPP_PCK_PRELOAD=1 enabled (parallel downloads):
Executed in 119.60 secs
Awesome work, I've enabled them in my config, and installed libzypp and zypper separately first on my Tumbleweed so I was able to enjoy the features with zypper dup for the first time.
So ZYPP_MEDIANETWORK=1 is gone now for good (https://github.com/openSUSE/libzypp/commit/f25304dee699a16b2fc018b59a00b3f6d...), replaced by ZYPP_CURL2?
Yes, ZYPP_MEDIANETWORK was a experimental backend where we tried to transparently build parallel downloads into the current serial media backend, but it never gave the performance boost we hoped for and the code was complicated. I removed it for good in favor of PCK_PRELOAD and CURL2. The PCK_PRELOAD code can parallelize better because it sits right in front of the transaction commit and knows the full list of packages it needs to download. CURL2 instead drops a lot of overhead the current default ( MultiCurl ) backend has and is faster just be being more lightweight, this helps with slow refreshes.
I do get a 'Backend: single_rpmtrans' line before "Continue" when running zypper (because using ZYPP_SINGLE_RPMTRANS=1), but I guess there is no such indication about PCK_PRELOAD os CURL2? Preloading is of course visible by the different download style ("Preloading" lines).
No this is about how zypp internally hands over the transaction to rpm/librpm. Legacy behavor was to execute rpm package by package, so every package started its own transaction. I added single_rpmtrans to make zypp behave exactly like upstream rpm would if you'd give it the full list of packages via commandline, it runs one transaction for all packages, so all the pre/post scripts and triggers work as intended, plus its much faster for big downloads. Singletrans is yet not default because especially the filesystem package relies on the legacy way of installing package by package, however we have some code there that detects if singletrans is enabled and the filesystem package is updated, in those cases it falls back to legacy mode.
Thank you for the hard work, I hope these get to be the default, maybe a phased rollout or something, at some suitable point in the future!
That is the plan, once we gave it some time for people to test and have all features ready we will first enable it for Tumbleweed and 16 as the default! Thanks for testing! Benjamin
On 3/3/25 09:48, Timo Jyrinki wrote:
I do get a 'Backend: single_rpmtrans' line before "Continue" when running zypper (because using ZYPP_SINGLE_RPMTRANS=1), but I guess there is no such indication about PCK_PRELOAD os CURL2? Preloading is of course visible by the different download style ("Preloading" lines).
I forgot to write: indicator for seeing the PCK_PRELOAD in action is if you get a progress bar saying "Preloading Packages / Preloading finished" and a bunch of "Preloading:" messages for each file being downloaded, like so
Preloading: bash-5.2.37-17.1.x86_64.rpm Preloading: compat-usrmerge-tools-84.87-5.21.x86_64.rpm Preloading: libncurses6-6.5.20250222-59.1.x86_64.rpm Preloading: terminfo-screen-6.5.20250222-59.1.x86_64.rpm Preloading: patterns-glibc-hwcaps-x86_64_v3-20230201-3.4.x86_64.rpm Preloading: patterns-glibc-hwcaps-x86_64_v3-20230201-3.4.x86_64.rpm [done] ------------------------ snip
Preloading: system-user-root-20190513-2.17.noarch.rpm [done] Preloading: filesystem-84.87-18.1.x86_64.rpm [done] Preloading: glibc-2.40-3.1.x86_64.rpm [done] Preload finished. [success (6.5 MiB/s) ] ............................................................................................................................................................................................................[done]
Am 03.03.25 um 10:19 schrieb Benjamin Zeller:
I forgot to write: indicator for seeing the PCK_PRELOAD in action is if you get a progress bar saying "Preloading Packages / Preloading finished" and a bunch of "Preloading:" messages for each file being downloaded, like so
Preloading: bash-5.2.37-17.1.x86_64.rpm Preloading: compat-usrmerge-tools-84.87-5.21.x86_64.rpm Preloading: libncurses6-6.5.20250222-59.1.x86_64.rpm Preloading: terminfo-screen-6.5.20250222-59.1.x86_64.rpm Preloading: patterns-glibc-hwcaps-x86_64_v3-20230201-3.4.x86_64.rpm Preloading: patterns-glibc-hwcaps-x86_64_v3-20230201-3.4.x86_64.rpm [done] ------------------------ snip
Preloading: system-user-root-20190513-2.17.noarch.rpm [done] Preloading: filesystem-84.87-18.1.x86_64.rpm [done] Preloading: glibc-2.40-3.1.x86_64.rpm [done] Preload finished. [success (6.5 MiB/ s) ] ............................................................................................................................................................................................................[done]
I suggest to throttle the "progress indicator" a little bit. My ssh connection to another machine preloading with only 100MBit connection was producing ~2MByte/s ssh traffic, just for the terminal output. Maybe updating it just a few times per second would be enough. Other than that it is pretty impressive, able to saturate my 600MBit downstream easily (and that even with me being lazy and not even bothering changing the repo files). -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On 04. 03. 25, 7:27, Stefan Seyfried via openSUSE Factory wrote:
I suggest to throttle the "progress indicator" a little bit. My ssh connection to another machine preloading with only 100MBit connection was producing ~2MByte/s ssh traffic, just for the terminal output.
Which results in 2 MBit download at most when using serial link :D.
Maybe updating it just a few times per second would be enough.
+1 -- js suse labs
Hey Benjamin, great work, love the speed increase. A few caveats though: sometimes the preloading fails although the file is there. an example: kerberos:/etc/zypp/repos.d # ZYPP_CURL2=1 ZYPP_PCK_PRELOAD=1 zypper dup Refreshing service 'NVIDIA'. Refreshing service 'openSUSE'. Loading repository data... Reading installed packages... Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command. Computing distribution upgrade... The following 15 packages are going to be upgraded: envsubst fastfetch fastfetch-bash-completion gettext-runtime gettext-tools gtk2-data gtk2-tools gtk2-tools-32bit libgtk-2_0-0 libgtk-2_0-0-32bit libtextstyle0 openSUSE-release openSUSE-release-ftp openSUSE-repos-Tumbleweed o penSUSE-repos-Tumbleweed-NVIDIA The following product is going to be upgraded: openSUSE Tumbleweed 20250303-0 -> 20250304-0 The following NEW package is going to be installed: gtk2-lang 15 packages to upgrade, 1 new. Package download size: 12.2 MiB Package install size change: | 40.2 MiB required by packages that will be installed 16.9 MiB | - 23.4 MiB released by packages that will be removed Backend: classic_rpmtrans Continue? [y/n/v/...? shows all options] (y): Preloading: envsubst-0.22.5-8.1.x86_64.rpm [done] Preloading: fastfetch-2.38.0-1.1.x86_64.rpm [done] Preloading: gtk2-data-2.24.33-14.1.noarch.rpm [done] Preloading: gtk2-tools-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libtextstyle0-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-ftp-20250304-3405.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-NVIDIA-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: fastfetch-bash-completion-2.38.0-1.1.noarch.rpm [done] Preloading: gtk2-tools-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-2.24.33-14.1.x86_64.rpm [done] Preloading: gettext-runtime-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-20250304-3405.1.x86_64.rpm [done] Preloading: gettext-tools-0.22.5-8.1.x86_64.rpm [done] Preloading: gtk2-lang-2.24.33-14.1.noarch.rpm [not found] Preload finished. [files missing] ............................................................................................................................................................................................................................................................................................[done] Installation has completed with error. it works if in just do a zypper dup: kerberos:/etc/zypp/repos.d # zypper dup Refreshing service 'NVIDIA'. Refreshing service 'openSUSE'. Loading repository data... Reading installed packages... Warning: You are about to do a distribution upgrade with all enabled repositories. Make sure these repositories are compatible before you continue. See 'man zypper' for more information about this command. Computing distribution upgrade... The following 15 packages are going to be upgraded: envsubst fastfetch fastfetch-bash-completion gettext-runtime gettext-tools gtk2-data gtk2-tools gtk2-tools-32bit libgtk-2_0-0 libgtk-2_0-0-32bit libtextstyle0 openSUSE-release openSUSE-release-ftp openSUSE-repos-Tumbleweed o penSUSE-repos-Tumbleweed-NVIDIA The following product is going to be upgraded: openSUSE Tumbleweed 20250303-0 -> 20250304-0 The following NEW package is going to be installed: gtk2-lang 15 packages to upgrade, 1 new. Package download size: 12.2 MiB Package install size change: | 40.2 MiB required by packages that will be installed 16.9 MiB | - 23.4 MiB released by packages that will be removed Backend: classic_rpmtrans Continue? [y/n/v/...? shows all options] (y): Retrieving: envsubst-0.22.5-8.1.x86_64 (repo-oss) (1/16), 42.6 KiB Retrieving: fastfetch-2.38.0-1.1.x86_64 (repo-oss) (2/16), 645.5 KiB Retrieving: gtk2-data-2.24.33-14.1.noarch (repo-oss) (3/16), 10.5 KiB Retrieving: gtk2-tools-32bit-2.24.33-14.1.x86_64 (repo-oss) (4/16), 13.3 KiB Retrieving: libgtk-2_0-0-32bit-2.24.33-14.1.x86_64 (repo-oss) (5/16), 2.1 MiB Retrieving: libtextstyle0-0.22.5-8.1.x86_64 (repo-oss) (6/16), 257.0 KiB Retrieving: openSUSE-release-ftp-20250304-3405.1.x86_64 (repo-oss) (7/16), 7.0 KiB Retrieving: openSUSE-repos-Tumbleweed-20250303.f74564e-1.1.x86_64 (repo-oss) (8/16), 18.2 KiB Retrieving: openSUSE-repos-Tumbleweed-NVIDIA-20250303.f74564e-1.1.x86_64 (repo-oss) (9/16), 16.9 KiB Retrieving: fastfetch-bash-completion-2.38.0-1.1.noarch (repo-oss) (10/16), 78.6 KiB Retrieving: gtk2-tools-2.24.33-14.1.x86_64 (repo-oss) (11/16), 14.6 KiB Retrieving: libgtk-2_0-0-2.24.33-14.1.x86_64 (repo-oss) (12/16), 2.1 MiB Retrieving: gettext-runtime-0.22.5-8.1.x86_64 (repo-oss) (13/16), 777.6 KiB Retrieving: openSUSE-release-20250304-3405.1.x86_64 (repo-oss) (14/16), 2.0 MiB Retrieving: gtk2-lang-2.24.33-14.1.noarch (repo-oss) (15/16), 2.3 MiB Retrieving: gtk2-lang-2.24.33-14.1.noarch.rpm ....................................................................................................................................................................................................................................................................[done (7.0 MiB/s)] Retrieving: gettext-tools-0.22.5-8.1.x86_64 (repo-oss) (16/16), 1.8 MiB Checking for file conflicts: .................................................................................................................................................................................................................................................................................................[done] ( 1/16) Installing: envsubst-0.22.5-8.1.x86_64 ...............................................................................................................................................................................................................................................................................[done] ( 2/16) Installing: fastfetch-2.38.0-1.1.x86_64 ..............................................................................................................................................................................................................................................................................[done] ( 3/16) Installing: gtk2-data-2.24.33-14.1.noarch ............................................................................................................................................................................................................................................................................[done] ( 4/16) Installing: gtk2-tools-32bit-2.24.33-14.1.x86_64 .....................................................................................................................................................................................................................................................................[done] ( 5/16) Installing: libgtk-2_0-0-32bit-2.24.33-14.1.x86_64 ...................................................................................................................................................................................................................................................................[done] ( 6/16) Installing: libtextstyle0-0.22.5-8.1.x86_64 ..........................................................................................................................................................................................................................................................................[done] ( 7/16) Installing: openSUSE-release-ftp-20250304-3405.1.x86_64 ..............................................................................................................................................................................................................................................................[done] ( 8/16) Installing: openSUSE-repos-Tumbleweed-20250303.f74564e-1.1.x86_64 ....................................................................................................................................................................................................................................................[done] ( 9/16) Installing: openSUSE-repos-Tumbleweed-NVIDIA-20250303.f74564e-1.1.x86_64 .............................................................................................................................................................................................................................................[done] (10/16) Installing: fastfetch-bash-completion-2.38.0-1.1.noarch ..............................................................................................................................................................................................................................................................[done] (11/16) Installing: gtk2-tools-2.24.33-14.1.x86_64 ...........................................................................................................................................................................................................................................................................[done] (12/16) Installing: libgtk-2_0-0-2.24.33-14.1.x86_64 .........................................................................................................................................................................................................................................................................[done] (13/16) Installing: gettext-runtime-0.22.5-8.1.x86_64 ........................................................................................................................................................................................................................................................................[done] (14/16) Installing: openSUSE-release-20250304-3405.1.x86_64 ..................................................................................................................................................................................................................................................................[done] (15/16) Installing: gtk2-lang-2.24.33-14.1.noarch ............................................................................................................................................................................................................................................................................[done] (16/16) Installing: gettext-tools-0.22.5-8.1.x86_64 ..........................................................................................................................................................................................................................................................................[done] now, preloading is fast even without the mmtealink. i tied to make it work with cdn domain but it doesn't like that [openSUSE:repo-oss] name=repo-oss enabled=1 autorefresh=1 baseurl=http://cdn.opensuse.org/tumbleweed/repo/oss service=openSUSE it only works with download.opensuse, is it differnet for cdn? anyway , its a great addition and im looking forward for further development. keep up the good work. | Paulo Dias | paulo.miguel.dias@gmail.com *Tempora mutantur, nos et mutamur in illis.* On Tue, Mar 4, 2025 at 8:14 AM Jiri Slaby <jslaby@suse.cz> wrote:
On 04. 03. 25, 7:27, Stefan Seyfried via openSUSE Factory wrote:
I suggest to throttle the "progress indicator" a little bit. My ssh connection to another machine preloading with only 100MBit connection was producing ~2MByte/s ssh traffic, just for the terminal output.
Which results in 2 MBit download at most when using serial link :D.
Maybe updating it just a few times per second would be enough.
+1
-- js suse labs
On 3/5/25 17:32, Paulo Dias wrote:
Hey Benjamin, great work, love the speed increase. Great to hear :)
A few caveats though:
sometimes the preloading fails although the file is there. an example:
Backend: classic_rpmtrans Continue? [y/n/v/...? shows all options] (y): Preloading: envsubst-0.22.5-8.1.x86_64.rpm [done] Preloading: fastfetch-2.38.0-1.1.x86_64.rpm [done] Preloading: gtk2-data-2.24.33-14.1.noarch.rpm [done] Preloading: gtk2-tools-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libtextstyle0-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-ftp-20250304-3405.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-NVIDIA-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: fastfetch-bash-completion-2.38.0-1.1.noarch.rpm [done] Preloading: gtk2-tools-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-2.24.33-14.1.x86_64.rpm [done] Preloading: gettext-runtime-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-20250304-3405.1.x86_64.rpm [done] Preloading: gettext-tools-0.22.5-8.1.x86_64.rpm [done] Preloading: gtk2-lang-2.24.33-14.1.noarch.rpm [not found] Preload finished. [files missing] ............................................................................................................................................................................................................................................................................................[done] Installation has completed with error.
If you still have the zypper log file containing that run feel free to open a bug report about it.
I do not, but i will create it if i see this again,, it happened twice now, so this isn't just one off. regards | Paulo Dias | paulo.miguel.dias@gmail.com *Tempora mutantur, nos et mutamur in illis.* On Wed, Mar 5, 2025 at 4:38 PM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/5/25 17:32, Paulo Dias wrote:
Hey Benjamin, great work, love the speed increase. Great to hear :)
A few caveats though:
sometimes the preloading fails although the file is there. an example:
Backend: classic_rpmtrans Continue? [y/n/v/...? shows all options] (y): Preloading: envsubst-0.22.5-8.1.x86_64.rpm [done] Preloading: fastfetch-2.38.0-1.1.x86_64.rpm [done] Preloading: gtk2-data-2.24.33-14.1.noarch.rpm [done] Preloading: gtk2-tools-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libtextstyle0-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-ftp-20250304-3405.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-NVIDIA-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: fastfetch-bash-completion-2.38.0-1.1.noarch.rpm [done] Preloading: gtk2-tools-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-2.24.33-14.1.x86_64.rpm [done] Preloading: gettext-runtime-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-20250304-3405.1.x86_64.rpm [done] Preloading: gettext-tools-0.22.5-8.1.x86_64.rpm [done] Preloading: gtk2-lang-2.24.33-14.1.noarch.rpm [not found] Preload finished. [files missing]
............................................................................................................................................................................................................................................................................................[done]
Installation has completed with error.
If you still have the zypper log file containing that run feel free to open a bug report about it.
Am Mittwoch, 5. März 2025, 17:38:12 Mitteleuropäische Normalzeit schrieb Benjamin Zeller:
On 3/5/25 17:32, Paulo Dias wrote:
Hey Benjamin, great work, love the speed increase. Great to hear :)
A few caveats though:
sometimes the preloading fails although the file is there. an example:
Backend: classic_rpmtrans Continue? [y/n/v/...? shows all options] (y): Preloading: envsubst-0.22.5-8.1.x86_64.rpm [done] Preloading: fastfetch-2.38.0-1.1.x86_64.rpm [done] Preloading: gtk2-data-2.24.33-14.1.noarch.rpm [done] Preloading: gtk2-tools-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-32bit-2.24.33-14.1.x86_64.rpm [done] Preloading: libtextstyle0-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-ftp-20250304-3405.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: openSUSE-repos-Tumbleweed-NVIDIA-20250303.f74564e-1.1.x86_64.rpm [done] Preloading: fastfetch-bash-completion-2.38.0-1.1.noarch.rpm [done] Preloading: gtk2-tools-2.24.33-14.1.x86_64.rpm [done] Preloading: libgtk-2_0-0-2.24.33-14.1.x86_64.rpm [done] Preloading: gettext-runtime-0.22.5-8.1.x86_64.rpm [done] Preloading: openSUSE-release-20250304-3405.1.x86_64.rpm [done] Preloading: gettext-tools-0.22.5-8.1.x86_64.rpm [done] Preloading: gtk2-lang-2.24.33-14.1.noarch.rpm [not found] Preload finished. [files missing] ............................................................................................................................................................................................................................................................................................[done] Installation has completed with error.
If you still have the zypper log file containing that run feel free to open a bug report about it.
Is there a bug report for this in between? Just stumbled over the same issue Cheers Axel
This is awesome news! Many thanks! On Sun, Mar 2, 2025 at 11:57 PM Benjamin Zeller <bzeller@suse.de> wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
1) zypper fetches packages during update/dup/install using parallel connections ( yes finally ). Currently the feature must be manually enabled by setting a environment variable like so:
> env ZYPP_PCK_PRELOAD=1 zypper dup
The number of concurrent connections is controlled by a setting in zypp.conf: download.max_concurrent_connections Currently the default is 5, but for me setting it higher performs even better.
2) A simpler more efficient media backend, it drops some of the overhead like fetching metalinks or splitting one file over multiple servers and should behave much better on connection reuse, in our internal test this always outperforms the current one. This is seperate from the parallel package downloads but should bring some benefits when fetching metadata from servers on zypper refresh. It's enabled with:
> env ZYPP_CURL2=1 zypper ref
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0
It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
Finally some numbers for those who enjoy it:
Downloaded roughly 100Mb, around 250 Packages over my 100Mbit/s
MultiCurl ( current default ): ------------------------------------------------------------------------------------------ Executed in 68.68 secs usr time 42.41 millis 0.38 millis 42.03 millis sys time 70.80 millis 1.02 millis 69.77 millis
ZYPP_CURL2=1 enabled ( new experimental backend ): ------------------------------------------------------------------------------------------ Executed in 29.55 secs usr time 44.33 millis 0.00 millis 44.33 millis sys time 57.07 millis 1.42 millis 55.65 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ------------------------------------------------------------------------------------------ Executed in 13.13 secs usr time 142.10 millis 0.38 millis 141.72 millis sys time 232.80 millis 1.02 millis 231.78 millis
Download of 407 Packages ~1.02Gb into empty rootfs ( postgresql virtualbox go mariadb ruby )
MultiCurl ( current default ): ________________________________________________________ Executed in 281.15 secs usr time 83.34 millis 0.36 millis 82.99 millis sys time 186.04 millis 1.03 millis 185.01 millis
ZYPP_CURL2=1 enabled ( new experimental backend ): ________________________________________________________ Executed in 208.51 secs usr time 76.77 millis 0.00 millis 76.77 millis sys time 168.91 millis 2.02 millis 166.89 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ________________________________________________________ Executed in 119.60 secs usr time 1.92 secs 0.36 millis 1.92 secs sys time 3.58 secs 1.03 millis 3.58 secs
Cheers,
Benjamin
-- Best regards Luboš Kocman openSUSE Leap Release Manager
On Sun, Mar 2, 2025 at 2:58 PM Benjamin Zeller <bzeller@suse.de> wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
1) zypper fetches packages during update/dup/install using parallel connections ( yes finally ). Currently the feature must be manually enabled by setting a environment variable like so:
> env ZYPP_PCK_PRELOAD=1 zypper dup
The number of concurrent connections is controlled by a setting in zypp.conf: download.max_concurrent_connections Currently the default is 5, but for me setting it higher performs even better.
2) A simpler more efficient media backend, it drops some of the overhead like fetching metalinks or splitting one file over multiple servers and should behave much better on connection reuse, in our internal test this always outperforms the current one. This is seperate from the parallel package downloads but should bring some benefits when fetching metadata from servers on zypper refresh. It's enabled with:
> env ZYPP_CURL2=1 zypper ref
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0
It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now.
Finally some numbers for those who enjoy it:
Downloaded roughly 100Mb, around 250 Packages over my 100Mbit/s
MultiCurl ( current default ): ------------------------------------------------------------------------------------------ Executed in 68.68 secs usr time 42.41 millis 0.38 millis 42.03 millis sys time 70.80 millis 1.02 millis 69.77 millis
ZYPP_CURL2=1 enabled ( new experimental backend ): ------------------------------------------------------------------------------------------ Executed in 29.55 secs usr time 44.33 millis 0.00 millis 44.33 millis sys time 57.07 millis 1.42 millis 55.65 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ------------------------------------------------------------------------------------------ Executed in 13.13 secs usr time 142.10 millis 0.38 millis 141.72 millis sys time 232.80 millis 1.02 millis 231.78 millis
Download of 407 Packages ~1.02Gb into empty rootfs ( postgresql virtualbox go mariadb ruby )
MultiCurl ( current default ): ________________________________________________________ Executed in 281.15 secs usr time 83.34 millis 0.36 millis 82.99 millis sys time 186.04 millis 1.03 millis 185.01 millis
ZYPP_CURL2=1 enabled ( new experimental backend ): ________________________________________________________ Executed in 208.51 secs usr time 76.77 millis 0.00 millis 76.77 millis sys time 168.91 millis 2.02 millis 166.89 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ________________________________________________________ Executed in 119.60 secs usr time 1.92 secs 0.36 millis 1.92 secs sys time 3.58 secs 1.03 millis 3.58 secs
Did you consider using librepo for this, out of curiosity? -- 真実はいつも一つ！/ Always, there's only one truth!
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now.
Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes.
Did you consider using librepo for this, out of curiosity? No, the code works with our existing http requester code and makes sure things like UrlResolver plugins work correctly. Also libzypp is big enough as it is, we can not introduce more dependencies.
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now.
Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes.
DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251 But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today. -- 真実はいつも一つ！/ Always, there's only one truth!
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now. Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away. However I'm working on fixing that problem: https://github.com/openSUSE/libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1... -- Benjamin Zeller <bzeller@suse.de> Systems Programmer SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)
On Mon, Mar 3, 2025 at 5:25 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now. Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away.
However I'm working on fixing that problem: https://github.com/openSUSE/libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1...
It sounds like a bad idea to do stuff like that. It's hard-coding unexpected divergent behavior. -- 真実はいつも一つ！/ Always, there's only one truth!
On 3/3/25 14:44, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:25 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
> [download.opensuse.org-oss] > name=Main Repository (OSS) > enabled=1 > autorefresh=1 > metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 > path=/ > type=rpm-md > keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now. Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away.
However I'm working on fixing that problem: https://github.com/openSUSE/libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1...
It sounds like a bad idea to do stuff like that. It's hard-coding unexpected divergent behavior.
I'm open for suggestions how to make it better... another idea would be if we could specify the path to the key and asc files in the repo configuration. Does dnf support something like that?
-- 真実はいつも一つ！/ Always, there's only one truth!
-- Benjamin Zeller <bzeller@suse.de> Systems Programmer SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)
On Mon, Mar 3, 2025 at 5:59 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:44, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:25 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
>> [download.opensuse.org-oss] >> name=Main Repository (OSS) >> enabled=1 >> autorefresh=1 >> metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 >> path=/ >> type=rpm-md >> keeppackages=0 > It replaces the baseUrl= key with metalink= key and URL. But be aware > that this will also download signature keys from the mirrors instead of > download.opensuse.org. So for production use better stick to baseUrl= > for now. > So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now. Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away.
However I'm working on fixing that problem: https://github.com/openSUSE/libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1...
It sounds like a bad idea to do stuff like that. It's hard-coding unexpected divergent behavior.
I'm open for suggestions how to make it better... another idea would be if we could specify the path to the key and asc files in the repo configuration.
Does dnf support something like that?
Yes. You specify them with gpgkey=. We don't have it for the asc file (that's always assumed to be the path resolved for the xml file), but gpg keys that you verify with are always retrieved in DNF from the gpgkey= list. You can see us doing this in the rpm-repos-openSUSE package. While we use local files in that one, http(s) is supported too. But I think it'd actually be safer if we stopped doing that for the distribution repositories and followed the practice that rpm-repos-openSUSE does. -- 真実はいつも一つ！/ Always, there's only one truth!
On 3/3/25 15:03, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:59 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:44, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:25 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
>>> [download.opensuse.org-oss] >>> name=Main Repository (OSS) >>> enabled=1 >>> autorefresh=1 >>> metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 >>> path=/ >>> type=rpm-md >>> keeppackages=0 >> It replaces the baseUrl= key with metalink= key and URL. But be aware >> that this will also download signature keys from the mirrors instead of >> download.opensuse.org. So for production use better stick to baseUrl= >> for now. >> > So this basically means repo files need to be configured the same way > for both DNF and Zypper to use metalinks now. Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away.
However I'm working on fixing that problem: https://github.com/openSUSE/libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1...
It sounds like a bad idea to do stuff like that. It's hard-coding unexpected divergent behavior. I'm open for suggestions how to make it better... another idea would be if we could specify the path to the key and asc files in the repo configuration.
Does dnf support something like that?
Yes. You specify them with gpgkey=.
We don't have it for the asc file (that's always assumed to be the path resolved for the xml file), but gpg keys that you verify with are always retrieved in DNF from the gpgkey= list. Ah right, zypp supports that line as well but its only used to search for a key if its not already known yet.
You can see us doing this in the rpm-repos-openSUSE package. While we use local files in that one, http(s) is supported too.
But I think it'd actually be safer if we stopped doing that for the distribution repositories and followed the practice that rpm-repos-openSUSE does.
Problem with stopping something we currently do as default always means breaking existing setups. If the users need to touch their repo files because we stopped auto fetching the keys we are already in trouble :/
On Mon, Mar 3, 2025 at 6:18 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 15:03, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:59 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:44, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:25 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote: >>>> [download.opensuse.org-oss] >>>> name=Main Repository (OSS) >>>> enabled=1 >>>> autorefresh=1 >>>> metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 >>>> path=/ >>>> type=rpm-md >>>> keeppackages=0 >>> It replaces the baseUrl= key with metalink= key and URL. But be aware >>> that this will also download signature keys from the mirrors instead of >>> download.opensuse.org. So for production use better stick to baseUrl= >>> for now. >>> >> So this basically means repo files need to be configured the same way >> for both DNF and Zypper to use metalinks now. > Not idea if dnf also uses the repomd.xml file to get to the metalink but > if it does you can > configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away.
However I'm working on fixing that problem: https://github.com/openSUSE/libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1...
It sounds like a bad idea to do stuff like that. It's hard-coding unexpected divergent behavior. I'm open for suggestions how to make it better... another idea would be if we could specify the path to the key and asc files in the repo configuration.
Does dnf support something like that?
Yes. You specify them with gpgkey=.
We don't have it for the asc file (that's always assumed to be the path resolved for the xml file), but gpg keys that you verify with are always retrieved in DNF from the gpgkey= list. Ah right, zypp supports that line as well but its only used to search for a key if its not already known yet.
DNF uses it as a filter. Repository content can *only* use those keys to validate, regardless of what's in the rpmdb. This ensures there isn't some kind of signature hijack.
You can see us doing this in the rpm-repos-openSUSE package. While we use local files in that one, http(s) is supported too.
But I think it'd actually be safer if we stopped doing that for the distribution repositories and followed the practice that rpm-repos-openSUSE does.
Problem with stopping something we currently do as default always means breaking existing setups. If the users need to touch their repo files because we stopped auto fetching the keys we are already in trouble :/
Yeah, but... that means it's possible to hijack repositories and users can't ever be really sure about the repository content. -- 真実はいつも一つ！/ Always, there's only one truth!
Am 03.03.25 um 14:25 schrieb Benjamin Zeller:
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/ repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now. Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away.
However I'm working on fixing that problem: https://github.com/openSUSE/ libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1f68d3d96bc7aa321e
hard coding host names? really? couldn't we use something like gpgkey=https://download.opensuse.org/tumbleweed/repo/oss/repodata/...? I see this in my RHEL-9 installs and it seems to work fine, so the metadata specification already has something that can do this? -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On 3/4/25 08:20, Stefan Seyfried via openSUSE Factory wrote:
Am 03.03.25 um 14:25 schrieb Benjamin Zeller:
On 3/3/25 14:20, Neal Gompa wrote:
On Mon, Mar 3, 2025 at 5:14 AM Benjamin Zeller <bzeller@suse.de> wrote:
> [download.opensuse.org-oss] > name=Main Repository (OSS) > enabled=1 > autorefresh=1 > metalink=https://download.opensuse.org/tumbleweed/repo/oss/ > repodata/repomd.xml.meta4 > path=/ > type=rpm-md > keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
So this basically means repo files need to be configured the same way for both DNF and Zypper to use metalinks now. Not idea if dnf also uses the repomd.xml file to get to the metalink but if it does you can configure them the same yes. DNF cannot do that yet, there is an RFE for it: https://github.com/rpm-software-management/librepo/issues/251
But if you use "metalink=", all resources are pulled from the mirrors. This is how Fedora and CentOS repositories work today.
Please note that when currently using metalink= zypp also will fetch all files from mirrors including .asc and .key files as I have mentioned in the first mail . This is one reason why we did not make it the default right away.
However I'm working on fixing that problem: https://github.com/openSUSE/ libzypp/pull/628/commits/dce6086c8fa91174f7ea5a1f68d3d96bc7aa321e
hard coding host names? really?
couldn't we use something like
gpgkey=https://download.opensuse.org/tumbleweed/repo/oss/repodata/...?
I see this in my RHEL-9 installs and it seems to work fine, so the metadata specification already has something that can do this?
Yes we could, it is one idea and we will discuss how we move on with that issue once everyone is back from vacation. Sadly a lot of the mirror and key handling is already hardcoded. Especially mirror handling.
On 3/2/25 5:57 PM, Benjamin Zeller wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
Do these new features work for those of us that are using the tumbleweed-cli package with the history repositories too ? -- Regards, Joe
On 3/5/25 17:36, Joe Salmeri wrote:
On 3/2/25 5:57 PM, Benjamin Zeller wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
Do these new features work for those of us that are using the tumbleweed-cli package with the history repositories too ?
If you have the libzypp-17.36.4 / zypper-1.14.87 combination installed then downloads from every repo should be parallelized using zypper. If you use any other frontend it would need to implement handling the preloading reports first. The code is only enabled if a handler is registered for it. Not idea what tumbleweed-cli does though.
On 3/5/25 11:43 AM, Benjamin Zeller wrote:
On 3/5/25 17:36, Joe Salmeri wrote:
On 3/2/25 5:57 PM, Benjamin Zeller wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
Do these new features work for those of us that are using the tumbleweed-cli package with the history repositories too ?
If you have the libzypp-17.36.4 / zypper-1.14.87 combination installed then downloads from every repo should be parallelized using zypper. If you use any other frontend it would need to implement handling the preloading reports first. The code is only enabled if a handler is registered for it.
Not idea what tumbleweed-cli does though.
Hi Benjamin, download.opensuse.org/history contains snapshots of the last 20 published builds of TW 20250304 20250303 20250302 20250301 20250228 20250227 20250226 20250225 20250224 20250223 20250222 20250221 20250220 20250219 20250218 20250217 20250216 20250211 20250209 20250207 The tumbleweed-cli package is just a simple script which lets you point your repositories to one of the history builds instead of the normal TW repos. This allows you to stay on that build and install other packages after the normal TW repos have moved to a newer build. When you're ready you just run the command to switch which history repos you want to point to and zypper dup to update to that TW build. So based on your reply it seems that once I have updated to the TW build which includes those new zypper versions then it should work fine and the same should be true for those that use the SlowRoll repos. Is there an ETA as to when you think this will become the default in TW? Thanks ! -- Regards, Joe
Op woensdag 5 maart 2025 17:51:48 Midden-Europese standaardtijd schreef Joe Salmeri:
The tumbleweed-cli package is just a simple script which lets you point your repositories to one of the history builds instead of the normal TW repos.
This allows you to stay on that build and install other packages after the normal TW repos have moved to a newer build. It is outright dangerous. I've seen quite some occasions where people completely borked their systems rolling backward and forward, locking packages and so on and ended up in some "Frankenstein Distro".
-- Gertjan Lettink a.k.a. Knurpht openSUSE Forums Team openSUSE Mods Team
On 3/5/25 11:58 AM, Knurpht-openSUSE wrote:
Op woensdag 5 maart 2025 17:51:48 Midden-Europese standaardtijd schreef Joe Salmeri:
The tumbleweed-cli package is just a simple script which lets you point your repositories to one of the history builds instead of the normal TW repos.
This allows you to stay on that build and install other packages after the normal TW repos have moved to a newer build. It is outright dangerous. I've seen quite some occasions where people completely borked their systems rolling backward and forward, locking packages and so on and ended up in some "Frankenstein Distro".
Hi Knurpht, I've been running it on multiple systems for several years now and never had a single problem. If someone installs TW today and then does not update for a week or two and then does a zypper dup then they move to the current build. If someone installs TW today, they would be on build 20250303. If they then install tumbleweed-cli they would also be on build 20250303, they just would be pointing to the history repos instead of the normal TW repos. A week later when newer builds have come out they would just pick which build they want to update to and tumbleweed switch 2025xxyy zypper dup The history repos are just a copy of what that TW build looked like when it was released and would be no different than if I had updated on the day it was released. Using tumbleweed-cli let's me pick what build to update too ( of the last 20 ) allowing me to pick a build which has less problems potentially than a more recent build. I've seen stories on this list of people that turned on a TW machine that was a few years old and run zypper dup and updated it to the current build without issues. It's one of the great features of TW. I can imagine that if someone rolled their system back and forward a bunch of times they could also bork their system too even if they weren't using tumbleweed-cli, it's not like the snapshots would be any different. Before I zypper dup, I delete all snapshots, create a "last" snapshot for the current build, and then do the zypper dup. If the new build had a serious problem, I'd just rollback to that "last" build and be exactly where I was before doing the zypper dup. I can definitely see your point if they locked certain important packages that they could get into trouble. I don't lock many packages and the ones I do are not critical ones that could cause that kind of problem. Also, I have better confidence in the build I am updating too because I check openqa for the stats and quality, therefore I generally don't end up in the situation where I need to rollback. tumbleweed-cli let's me be in more control and update more like slowroll before it existed. I would expect anyone using it the way I am to have a similar experience. -- Regards, Joe
On 3/5/25 17:51, Joe Salmeri wrote:
On 3/5/25 11:43 AM, Benjamin Zeller wrote:
On 3/5/25 17:36, Joe Salmeri wrote:
On 3/2/25 5:57 PM, Benjamin Zeller wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
Do these new features work for those of us that are using the tumbleweed-cli package with the history repositories too ?
If you have the libzypp-17.36.4 / zypper-1.14.87 combination installed then downloads from every repo should be parallelized using zypper. If you use any other frontend it would need to implement handling the preloading reports first. The code is only enabled if a handler is registered for it.
Not idea what tumbleweed-cli does though.
Hi Benjamin,
download.opensuse.org/history contains snapshots of the last 20 published builds of TW
20250304 20250303 20250302 20250301 20250228 20250227 20250226 20250225 20250224 20250223 20250222 20250221 20250220 20250219 20250218 20250217 20250216 20250211 20250209 20250207
The tumbleweed-cli package is just a simple script which lets you point your repositories to one of the history builds instead of the normal TW repos.
This allows you to stay on that build and install other packages after the normal TW repos have moved to a newer build. Ah interesting, thanks for the explanation :) . I knew those history repos are there but didn't know there was a tool to switch between them.
When you're ready you just run the command to switch which history repos you want to point to and zypper dup to update to that TW build.
So based on your reply it seems that once I have updated to the TW build which includes those new zypper versions then it should work fine and the same should be true for those that use the SlowRoll repos. Yes based on your explanation, as long as you do not downgrade zypper/libzypp too ofc :)
Is there an ETA as to when you think this will become the default in TW?
No, we will give it some time for people to play with it, then flip the switch for at least TW and 16, if that goes fine too we can enable it for the more stable releases as well.
05.03.2025 19:36, Joe Salmeri wrote:
On 3/2/25 5:57 PM, Benjamin Zeller wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
Do these new features work for those of us that are using the tumbleweed-cli package with the history repositories too ?
tumbleweed-cli manipulates baseurl=. Someone needs to enhance it to support metalink= as well.
Really awesome update 💯 Tested downloading texlive on 200 Mbps connection and 200 ms latency to cdn.o.o, 1798 packages, 649.6 MiB: Zypper with Preload: 5m13.371s Zypper Default: 31m11.073s 6x performance improvement when number of packages increased, on high latency connection. 🚀 The performance improvement was seen when testing with *only* ZYPP_PCK_PRELOAD=1. Adding ZYPP_CURL2=1 results in a performance decrease when both downloading packages (install -d) and refreshing packages (ref). Also, could this change be applied to zypper download in a future update? That would make things more usable from a scripting standpoint for downloading multiple packages. 📜 Thanks again for this much needed and appreciated update. 💫
Great improvement, zypper dup is definitely faster with the new parallel code, however image builds with kiwi seem to fail when ZYPP_PCK_PRELOAD=1
On 3/6/25 23:17, stephane.billiart@gmail.com wrote:
Great improvement, zypper dup is definitely faster with the new parallel code, however image builds with kiwi seem to fail when ZYPP_PCK_PRELOAD=1
The image builds fail? Can you share more information? IIRC kiwi downloads packages in "Download as needed" mode, parallelization can not happen there.
On Fri, Mar 7, 2025 at 6:26 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/6/25 23:17, stephane.billiart@gmail.com wrote:
Great improvement, zypper dup is definitely faster with the new parallel code, however image builds with kiwi seem to fail when ZYPP_PCK_PRELOAD=1
The image builds fail? Can you share more information? IIRC kiwi downloads packages in "Download as needed" mode, parallelization can not happen there.
We do whatever Zypper wants to do, we don't control the behavior like that in kiwi. -- 真実はいつも一つ！/ Always, there's only one truth!
On 3/7/25 15:27, Neal Gompa wrote:
On Fri, Mar 7, 2025 at 6:26 AM Benjamin Zeller <bzeller@suse.de> wrote:
On 3/6/25 23:17, stephane.billiart@gmail.com wrote:
Great improvement, zypper dup is definitely faster with the new parallel code, however image builds with kiwi seem to fail when ZYPP_PCK_PRELOAD=1 The image builds fail? Can you share more information? IIRC kiwi downloads packages in "Download as needed" mode, parallelization can not happen there.
We do whatever Zypper wants to do, we don't control the behavior like that in kiwi.
I can say more with logs and a bug report :)
On 3/7/25 8:26 AM, Benjamin Zeller wrote:
On 3/6/25 23:17, stephane.billiart@gmail.com wrote:
Great improvement, zypper dup is definitely faster with the new parallel code, however image builds with kiwi seem to fail when ZYPP_PCK_PRELOAD=1
The image builds fail? Can you share more information? IIRC kiwi downloads packages in "Download as needed" mode, parallelization can not happen there.
Hi, upgrading to Tumbleweed snapshot 20250308-0 and zypp.conf has commit.downloadMode = DownloadAsNeeded set here. Many a,r,i pauses so I decided to review this thread. I could swear that Retrieving speeds increased after ZYPP_PCK_PRELOAD=1 zypper dup . Above are you referring to only Kiwi? Thanks for your efforts🙂. I would like to ask more about the "hardcoded keys" mentioned above but will track down that post maybe and ask there. -Greatest Hopes
On 3/11/25 11:51, -pj via openSUSE Factory wrote:
On 3/7/25 8:26 AM, Benjamin Zeller wrote:
On 3/6/25 23:17, stephane.billiart@gmail.com wrote:
Great improvement, zypper dup is definitely faster with the new parallel code, however image builds with kiwi seem to fail when ZYPP_PCK_PRELOAD=1
The image builds fail? Can you share more information? IIRC kiwi downloads packages in "Download as needed" mode, parallelization can not happen there.
Hi, upgrading to Tumbleweed snapshot 20250308-0 and zypp.conf has commit.downloadMode = DownloadAsNeeded set here. Many a,r,i pauses so I decided to review this thread. I could swear that Retrieving speeds increased after ZYPP_PCK_PRELOAD=1 zypper dup . Above are you referring to only Kiwi?
Preload would not ask for a,r,i. Did you also set ZYPP_MEDIA_CURL2=1 ? That possibly would also increase speed. Prefetcher only makes sense if you have one synchro point where the code fetches all packages before starting the installation. If you use DownloadAsNeeded you do not have a point like that, because zypp will download and install package by package. So download Package A, install Package A, download Package B, install Package B etcetc. Where DownloadInAdvance first downloads everything and then installs the packages, which btw is the default upstream way of installing a related set of packages.
Thanks for your efforts🙂. I would like to ask more about the "hardcoded keys" mentioned above but will track down that post maybe and ask there.
You can also ask here though.
On 3/6/25 11:43, Pavin Joseph via openSUSE Factory wrote:
Really awesome update 💯
Tested downloading texlive on 200 Mbps connection and 200 ms latency to cdn.o.o, 1798 packages, 649.6 MiB:
Zypper with Preload: 5m13.371s Zypper Default: 31m11.073s 6x performance improvement when number of packages increased, on high latency connection. 🚀 Great to see those numbers :)
The performance improvement was seen when testing with *only* ZYPP_PCK_PRELOAD=1. Adding ZYPP_CURL2=1 results in a performance decrease when both downloading packages (install -d) and refreshing packages (ref).
ZYPP_CURL2=1 would only have impact on how metadata is fetched, not on the package preloader itself. For most people so far zypper ref also became a lot faster.
Also, could this change be applied to zypper download in a future update? That would make things more usable from a scripting standpoint for downloading multiple packages. 📜
Currently it works only when a transaction is commited, you can however download only with zypper in --download-only <list of packages>. I'll need to check how zypper implemented the download command.
Thanks again for this much needed and appreciated update. 💫
Happy that everyone likes it so far :)
03.03.2025 01:57, Benjamin Zeller wrote:
2) A simpler more efficient media backend, it drops some of the overhead like fetching metalinks or splitting one file over multiple servers and should behave much better on connection reuse, in our internal test this always outperforms the current one. This is seperate from the parallel package downloads but should bring some benefits when fetching metadata from servers on zypper refresh. It's enabled with:
> env ZYPP_CURL2=1 zypper ref
Apparently it is already forced by default in openSUSE repository service? https://github.com/openSUSE/openSUSE-repos/blob/main/opensuse-tumbleweed-rep...: <repo url="%{disturl}/%{distsub}/repo/oss?mediahandler=curl2"
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4
As I accidentally found, it works just fine with the traditional media backend.
On 3/15/25 15:44, Andrei Borzenkov wrote:
03.03.2025 01:57, Benjamin Zeller wrote:
2) A simpler more efficient media backend, it drops some of the overhead like fetching metalinks or splitting one file over multiple servers and should behave much better on connection reuse, in our internal test this always outperforms the current one. This is seperate from the parallel package downloads but should bring some benefits when fetching metadata from servers on zypper refresh. It's enabled with:
> env ZYPP_CURL2=1 zypper ref
Apparently it is already forced by default in openSUSE repository service?
https://github.com/openSUSE/openSUSE-repos/blob/main/opensuse-tumbleweed-rep...:
<repo url="%{disturl}/%{distsub}/repo/oss?mediahandler=curl2"
Yes, Lubos enabled it in the services. Zypp team does not maintain those.
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4
As I accidentally found, it works just fine with the traditional media backend.
Yes this feature was there a long time already, just not advertised. I guess mostly for compatibility with yum/dnf. There are some issues we currently work on with it. One being that if the list of mirrors change in any way a cache rebuild would be triggered. The bigger issue however would be that gpg keys would be fetched from the mirrors as well which is at least a theoretical attack point. We also work on that. If you can live with fetching keys from mirrors you can already use the feature.
Shout out to everyone involved — downloading packages is finally utilizing my connection's bandwidth, which is especially handy after the full rebuild, like the recent one. Amazing job, folks, can't wait for it to become the default implementation. On Sun, Mar 2, 2025 at 11:58 PM Benjamin Zeller <bzeller@suse.de> wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
1) zypper fetches packages during update/dup/install using parallel connections ( yes finally ). Currently the feature must be manually enabled by setting a environment variable like so:
> env ZYPP_PCK_PRELOAD=1 zypper dup
The number of concurrent connections is controlled by a setting in zypp.conf: download.max_concurrent_connections Currently the default is 5, but for me setting it higher performs even better.
2) A simpler more efficient media backend, it drops some of the overhead like fetching metalinks or splitting one file over multiple servers and should behave much better on connection reuse, in our internal test this always outperforms the current one. This is seperate from the parallel package downloads but should bring some benefits when fetching metadata from servers on zypper refresh. It's enabled with:
> env ZYPP_CURL2=1 zypper ref
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink= https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0
It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
Finally some numbers for those who enjoy it:
Downloaded roughly 100Mb, around 250 Packages over my 100Mbit/s
MultiCurl ( current default ):
------------------------------------------------------------------------------------------ Executed in 68.68 secs usr time 42.41 millis 0.38 millis 42.03 millis sys time 70.80 millis 1.02 millis 69.77 millis
ZYPP_CURL2=1 enabled ( new experimental backend ):
------------------------------------------------------------------------------------------ Executed in 29.55 secs usr time 44.33 millis 0.00 millis 44.33 millis sys time 57.07 millis 1.42 millis 55.65 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads):
------------------------------------------------------------------------------------------ Executed in 13.13 secs usr time 142.10 millis 0.38 millis 141.72 millis sys time 232.80 millis 1.02 millis 231.78 millis
Download of 407 Packages ~1.02Gb into empty rootfs ( postgresql virtualbox go mariadb ruby )
MultiCurl ( current default ): ________________________________________________________ Executed in 281.15 secs usr time 83.34 millis 0.36 millis 82.99 millis sys time 186.04 millis 1.03 millis 185.01 millis
ZYPP_CURL2=1 enabled ( new experimental backend ): ________________________________________________________ Executed in 208.51 secs usr time 76.77 millis 0.00 millis 76.77 millis sys time 168.91 millis 2.02 millis 166.89 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ________________________________________________________ Executed in 119.60 secs usr time 1.92 secs 0.36 millis 1.92 secs sys time 3.58 secs 1.03 millis 3.58 secs
Cheers,
Benjamin
On 3/17/25 08:03, Andrei Dziahel wrote:
Shout out to everyone involved — downloading packages is finally utilizing my connection's bandwidth, which is especially handy after the full rebuild, like the recent one. Amazing job, folks, can't wait for it to become the default implementation.
Great to hear, we are working on it :)
On Sun, Mar 2, 2025 at 11:58 PM Benjamin Zeller <bzeller@suse.de> wrote:
Hello everyone,
the new libzypp-17.36.4 / zypper-1.14.87 combination brings some features that hopefully will help to improve the slowness in fetching packages and metadata. They currently need to be enabled manually because they are still in early stages:
1) zypper fetches packages during update/dup/install using parallel connections ( yes finally ). Currently the feature must be manually enabled by setting a environment variable like so:
> env ZYPP_PCK_PRELOAD=1 zypper dup
The number of concurrent connections is controlled by a setting in zypp.conf: download.max_concurrent_connections Currently the default is 5, but for me setting it higher performs even better.
2) A simpler more efficient media backend, it drops some of the overhead like fetching metalinks or splitting one file over multiple servers and should behave much better on connection reuse, in our internal test this always outperforms the current one. This is seperate from the parallel package downloads but should bring some benefits when fetching metadata from servers on zypper refresh. It's enabled with:
> env ZYPP_CURL2=1 zypper ref
Since both features do not actively fetch mirrorlink informations from the server it is best to combine them with a existing feature, specifying the metalink/mirror information in each .repo file. Otherwise especially the parallel downloader will only fetch from the baseUrl which might be slower.
Example for my Tumbleweed repo file:
> [download.opensuse.org-oss] > name=Main Repository (OSS) > enabled=1 > autorefresh=1 > metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 > path=/ > type=rpm-md > keeppackages=0
It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org <http://download.opensuse.org>. So for production use better stick to baseUrl= for now.
Finally some numbers for those who enjoy it:
Downloaded roughly 100Mb, around 250 Packages over my 100Mbit/s
MultiCurl ( current default ): ------------------------------------------------------------------------------------------ Executed in 68.68 secs usr time 42.41 millis 0.38 millis 42.03 millis sys time 70.80 millis 1.02 millis 69.77 millis
ZYPP_CURL2=1 enabled ( new experimental backend ): ------------------------------------------------------------------------------------------ Executed in 29.55 secs usr time 44.33 millis 0.00 millis 44.33 millis sys time 57.07 millis 1.42 millis 55.65 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ------------------------------------------------------------------------------------------ Executed in 13.13 secs usr time 142.10 millis 0.38 millis 141.72 millis sys time 232.80 millis 1.02 millis 231.78 millis
Download of 407 Packages ~1.02Gb into empty rootfs ( postgresql virtualbox go mariadb ruby )
MultiCurl ( current default ): ________________________________________________________ Executed in 281.15 secs usr time 83.34 millis 0.36 millis 82.99 millis sys time 186.04 millis 1.03 millis 185.01 millis
ZYPP_CURL2=1 enabled ( new experimental backend ): ________________________________________________________ Executed in 208.51 secs usr time 76.77 millis 0.00 millis 76.77 millis sys time 168.91 millis 2.02 millis 166.89 millis
ZYPP_PCK_PRELOAD=1 enabled (parallel downloads): ________________________________________________________ Executed in 119.60 secs usr time 1.92 secs 0.36 millis 1.92 secs sys time 3.58 secs 1.03 millis 3.58 secs
Cheers,
Benjamin
-- Benjamin Zeller <bzeller@suse.de> Systems Programmer SUSE Software Solutions Germany GmbH Frankenstraße 146 90461 Nürnberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman (HRB 36809, AG Nürnberg)
Benjamin Zeller writes:
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0
It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now.
Bth cdn.opensuse.org and download.opensuse.org get redirected to mirrors these days it seems. I got redirected to mirror.hoasted.com as the first mirror and it didn't have a bunch of packages (for instance xz-lang-5.8.1-1.1 and probably not the only mirror with that problem, but I didn't check). With the preload active there seems to be no way to tell zypper to use a different mirror or fall back to downloading from the actual baseurl. --8<---------------cut here---------------start------------->8--- 3295:2025-04-12 08:54:00 <1> Gertrud(2093) [zypper++] Summary.cc(readPool):281 <install> U_Ts_(95149)xz-lang-5.8.1-1.1.noarch(Tumbleweed-OSS) 4214:2025-04-12 08:54:00 <1> Gertrud(2093) [zypper++] Summary.cc(readPool):281 <uninstall> I_TsUu(104871)xz-lang-5.6.4-1.2.noarch(@System) 12456:2025-04-12 08:56:46 <1> Gertrud(2093) [zypp-core] TmpPath.cc(Impl):53 /var/cache/zypp/packages/Tumbleweed-OSS/.preload/noarch/xz-lang-5.8.1-1.1.noarch.rpmmwV6oR 12459:2025-04-12 08:56:46 <1> Gertrud(2093) [zypp::media] commitpackagepreloader.cc(onRequestStarted):252 Request for https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch... started 12474:2025-04-12 08:56:46 <1> Gertrud(2093) [ZYPP_MEDIA_CURL++] request.cc(headerfunction):646 0x55e8dfbfc090 redirecting to https://mirror.hoasted.com/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1... 12506:2025-04-12 08:56:46 <1> Gertrud(2093) [zypp::media] commitpackagepreloader.cc(onRequestFinished):275 Request for https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch... finished. (File '/var/cache/zypp/packages/Tumbleweed-OSS/.preload/noarch/xz-lang-5.8.1-1.1.noarch.rpmmwV6oR' not found on medium 'https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch...' Curl error (22)) 12507:2025-04-12 08:56:46 <1> Gertrud(2093) [zypp::media] commitpackagepreloader.cc(onRequestFinished):328 Download from mirror failed for file https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch... trying to taint mirror and move on 12508:2025-04-12 08:56:46 <1> Gertrud(2093) [zypp-core] PathInfo.cc(unlink):711 unlink /var/cache/zypp/packages/Tumbleweed-OSS/.preload/noarch/xz-lang-5.8.1-1.1.noarch.rpmmwV6oR 15356:2025-04-12 08:57:11 <1> Gertrud(2093) [zypper++] Summary.cc(readPool):281 <install> U_Ts_(95149)xz-lang-5.8.1-1.1.noarch(Tumbleweed-OSS) 16275:2025-04-12 08:57:11 <1> Gertrud(2093) [zypper++] Summary.cc(readPool):281 <uninstall> I_TsUu(114056)xz-lang-5.6.4-1.2.noarch(@System) 17305:2025-04-12 08:57:17 <1> Gertrud(2093) [zypp-core] TmpPath.cc(Impl):53 /var/cache/zypp/packages/Tumbleweed-OSS/.preload/noarch/xz-lang-5.8.1-1.1.noarch.rpm0N9kg6 17308:2025-04-12 08:57:17 <1> Gertrud(2093) [zypp::media] commitpackagepreloader.cc(onRequestStarted):252 Request for https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch... started 17312:2025-04-12 08:57:17 <1> Gertrud(2093) [ZYPP_MEDIA_CURL++] request.cc(headerfunction):646 0x55e8df69a100 redirecting to https://mirror.hoasted.com/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1... 17444:2025-04-12 08:57:18 <1> Gertrud(2093) [zypp::media] commitpackagepreloader.cc(onRequestFinished):275 Request for https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch... finished. (File '/var/cache/zypp/packages/Tumbleweed-OSS/.preload/noarch/xz-lang-5.8.1-1.1.noarch.rpm0N9kg6' not found on medium 'https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch...' Curl error (22)) 17445:2025-04-12 08:57:18 <1> Gertrud(2093) [zypp::media] commitpackagepreloader.cc(onRequestFinished):328 Download from mirror failed for file https://cdn.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.noarch... trying to taint mirror and move on 17446:2025-04-12 08:57:18 <1> Gertrud(2093) [zypp-core] PathInfo.cc(unlink):711 unlink /var/cache/zypp/packages/Tumbleweed-OSS/.preload/noarch/xz-lang-5.8.1-1.1.noarch.rpm0N9kg6 --8<---------------cut here---------------end--------------->8--- It seems to have tried to get a different mirror, but ended up suing the same one again anyway the second time. Same, but different with the CURL2 backend: --8<---------------cut here---------------start------------->8--- 5326:2025-04-12 09:28:20 <1> Gertrud(8043) [ZYPP_MEDIA_CURL++] request.cc(headerfunction):646 0x559e5bddaad0 redirecting to https://mirror.hoasted.com/opensuse/tumbleweed/repo/oss/noarch/python311-mis... 5356:2025-04-12 09:29:02 <1> Gertrud(8043) [ZYPP_MEDIA_CURL] MediaCurl2.cc(MediaCurl2):84 MediaCurl2::MediaCurl2(https://mirror.hoasted.com/opensuse/tumbleweed/repo, ) 5357:2025-04-12 09:29:02 <1> Gertrud(8043) [zypp::media] MediaHandlerFactory.cc(createHandler):171 Opened: https://mirror.hoasted.com/opensuse/tumbleweed/repo not attached; localRoot "" 5358:2025-04-12 09:29:02 <1> Gertrud(8043) [zypp::media++] MediaManager.cc(open):327 Opened new media access using id 9 to https://mirror.hoasted.com/opensuse/tumbleweed/repo 5363:2025-04-12 09:29:02 <1> Gertrud(8043) [zypp::media] MediaHandler.cc(attach):654 Attached: https://mirror.hoasted.com/opensuse/tumbleweed/repo attached; localRoot "/var/adm/mount/AP_0x4Im2Wg" 5370:2025-04-12 09:29:02 <1> Gertrud(8043) [ZYPP_MEDIA_CURL++] MediaCurl2.cc(doGetFileCopy):352 URL: https://mirror.hoasted.com/opensuse/tumbleweed/repo/noarch/python311-mistune... 5373:2025-04-12 09:29:02 <5> Gertrud(8043) [zypp-core] Exception.cc(log):219 MediaCurl2.cc(executeRequest):629 THROW (EXCPTR): File '/opensuse/tumbleweed/repo/noarch/python311-mistune-3.1.3-1.1.noarch.rpm' not found on medium 'https://mirror.hoasted.com/opensuse/tumbleweed/repo' 5375:2025-04-12 09:29:02 <5> Gertrud(8043) [zypp-core] Exception.cc(log):219 MediaCurl2.cc(getFileCopy):251 RETHROW: File '/opensuse/tumbleweed/repo/noarch/python311-mistune-3.1.3-1.1.noarch.rpm' not found on medium 'https://mirror.hoasted.com/opensuse/tumbleweed/repo' 5376:2025-04-12 09:29:02 <5> Gertrud(8043) [zypp-core] Exception.cc(log):219 MediaSetAccess.cc(provide):285 CAUGHT: File '/opensuse/tumbleweed/repo/noarch/python311-mistune-3.1.3-1.1.noarch.rpm' not found on medium 'https://mirror.hoasted.com/opensuse/tumbleweed/repo' 5380:2025-04-12 09:29:02 <1> Gertrud(8043) [zypp::media++] MediaHandler.cc(release):714 Request to release attached media https<https://mirror.hoasted.com/opensuse/tumbleweed/repo>, use count=1 5381:2025-04-12 09:29:02 <1> Gertrud(8043) [zypp::media++] MediaHandler.cc(release):721 Releasing media https<https://mirror.hoasted.com/opensuse/tumbleweed/repo> 5382:2025-04-12 09:29:02 <1> Gertrud(8043) [zypp::media] MediaHandler.cc(disconnect):693 Disconnected: https://mirror.hoasted.com/opensuse/tumbleweed/repo attached; localRoot "/var/adm/mount/AP_0x4Im2Wg" 5386:2025-04-12 09:29:02 <1> Gertrud(8043) [zypp::media] MediaHandler.cc(release):775 Released: https://mirror.hoasted.com/opensuse/tumbleweed/repo not attached; localRoot "" 5388:2025-04-12 09:29:02 <1> Gertrud(8043) [zypper++] media.cc(requestMedia):204 medium problem, url: https://mirror.hoasted.com/opensuse/tumbleweed/repo, error 1, label 'Tumbleweed-OSS', #1 5394:2025-04-12 09:29:53 <1> Gertrud(8043) [zypp::media] MediaManager.cc(close):116 Close: https(https://mirror.hoasted.com/opensuse/tumbleweed/repo not attached; localRoot "") (OK) 5396:2025-04-12 09:29:53 <1> Gertrud(8043) [zypp::media] MediaManager.cc(close):116 Close: https(https://mirror.hoasted.com/opensuse/tumbleweed/repo not attached; localRoot "") (OK) --8<---------------cut here---------------end--------------->8--- I ended up switching off the preload and tell it to use a different URL when it asked for how to handle the error (note: that option does not seem to be avilable with all backends?). On another machine I had the same problem and here using the MultiCurl backend skipped the two mirrors that did not have the file (apparentlyy because it was requesting the a single block with the same range from all mirrors, which seems wasteful): --8<---------------cut here---------------start------------->8--- 2025-04-12 22:54:30 <1> Gerda(18610) [zypper++] Summary.cc(readPool):281 <install> U_Ts_r(102515)xz-lang-5.8.1-1.1.noarch(download.opensuse.org-oss_2) 2025-04-12 22:54:30 <1> Gerda(18610) [zypper++] Summary.cc(readPool):281 <uninstall> I_TsU(108695)xz-lang-5.6.4-1.2.noarch(@System) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp] PackageProvider.cc(providePackage):424 provide Package (102515)xz-lang-5.8.1-1.1.noarch(download.opensuse.org-oss_2) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] RepoProvideFile.cc(provideFile):253 [1]./noarch/xz-lang-5.8.1-1.1.noarch.rpm{455.5 KiB|sha512-4f436b4cbbc28e4365efa14d649e8fa5eb3c6432616e4b12e6a935fe530526dfec8a51260ec7faef53fd060f78c9dbbfa40442c15f06de91cf771d3559f06488|} 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::fetcher++] MediaSetAccess.cc(provide):275 Going to try to provide file ./noarch/xz-lang-5.8.1-1.1.noarch.rpm from media number 1 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1510 dest: /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1511 temp: /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm.new.zypp.MK8JI4 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1064 ./noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1074 URL: http://download.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.no... 2025-04-12 22:54:33 <1> Gerda(18610) [zypp-curl] metalinkparser.cc(parse):445 Parsed 10 mirrors from /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm.new.zypp.MK8JI4 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://mirror.hoasted.com/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.uni-bayreuth.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-lang..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.rz.uni-wuerzburg.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-..., ) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://opensuse.schlundtech.de/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1...., ) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.tu-chemnitz.de/pub/linux/opensuse/tumbleweed/repo/oss/noarch/xz-l..., ) 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #0: state: 6 received: 0 url: http://mirror.hoasted.com/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #1: state: 3 received: 86024 url: http://ftp.uni-bayreuth.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-lang... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #2: state: 4 received: 466439 url: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #3: state: 3 received: 402773 url: http://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #4: state: 3 received: 408960 url: http://ftp.rz.uni-wuerzburg.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #5: state: 6 received: 0 url: http://opensuse.schlundtech.de/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #6: state: 3 received: 13396 url: http://ftp.tu-chemnitz.de/pub/linux/opensuse/tumbleweed/repo/oss/noarch/xz-l... 2025-04-12 22:54:34 <1> Gerda(18610) [zypp-core] PathInfo.cc(rename):753 rename /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm.new.zypp.MK8JI4 -> /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1713 done: /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm{- 0644 0/0 size 466439} 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media++] MediaHandler.cc(provideFile):977 provideFile([1]./noarch/xz-lang-5.8.1-1.1.noarch.rpm{455.5 KiB|sha512-4f436b4cbbc28e4365efa14d649e8fa5eb3c6432616e4b12e6a935fe530526dfec8a51260ec7faef53fd060f78c9dbbfa40442c15f06de91cf771d3559f06488|}) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::fetcher] Fetcher.cc(validate):393 Checking job [/var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm] (2 checkers ) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp-core] PathInfo.cc(hardlinkCopy):921 hardlinkCopy /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm -> /var/cache/zypp/packages/download.opensuse.org-oss_2/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::fetcher++] MediaSetAccess.cc(releaseFile):91 Going to release file ./noarch/xz-lang-5.8.1-1.1.noarch.rpm from media number 1 2025-04-12 22:54:34 <1> Gerda(18610) [zypp-core] PathInfo.cc(unlink):711 unlink /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media] RepoProvideFile.cc(provideFile):321 provideFile at /var/cache/zypp/packages/download.opensuse.org-oss_2/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp] PackageProvider.cc(providePackage):525 provided Package (102515)xz-lang-5.8.1-1.1.noarch(download.opensuse.org-oss_2) at /var/cache/zypp/packages/download.opensuse.org-oss_2/noarch/xz-lang-5.8.1-1.1.noarch.rpm --8<---------------cut here---------------end--------------->8--- Generally, sharding a request across servers doesn't make sense unless both the overhead of the request is low and/or it is known in advance that the servers will throttle the rate to significantly less than the capacity of the receiving system. Sharding a request to the same server should not improve download speed much for the type of mirrors that zypper targets also. Empirically, the best heuristic is to select a server that delivers at line rate after a fast TCP ramp up for large files and re-use that connection for as long as it holds that rate. If you know the size of the request beforehand and HTTP/2 can be used, "burning the candle from two ends" is an easy way of keeping the connection fully utilized: start with the largest file and smallest file and keep both queues busy with requests from their respective end. Adding parallel request queues for either or both ends might be useful if the server has a long latency for accepting a request or has a rate limit for individual requests that is significantly below the line rate at the receiving end. As long as the connection gets reused, in my experience you get quickly diminishing returns between 4 or 8 parallel requests, although keeping a 2GBit line busy might require something on the order of 32. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
Benjamin Zeller writes:
Example for my Tumbleweed repo file:
[download.opensuse.org-oss] name=Main Repository (OSS) enabled=1 autorefresh=1 metalink=https://download.opensuse.org/tumbleweed/repo/oss/repodata/repomd.xml.meta4 path=/ type=rpm-md keeppackages=0 It replaces the baseUrl= key with metalink= key and URL. But be aware that this will also download signature keys from the mirrors instead of download.opensuse.org. So for production use better stick to baseUrl= for now. Bth cdn.opensuse.org and download.opensuse.org get redirected to mirrors these days it seems. I got redirected to mirror.hoasted.com as the first mirror and it didn't have a bunch of packages (for instance xz-lang-5.8.1-1.1 and probably not the only mirror with that problem, but I didn't check). With the preload active there seems to be no way to tell zypper to use a different mirror or fall back to downloading from the actual baseurl. I know, that's why I wrote in my initial mail that the preloader is best to be combined with the metalink setting in the repo file. Alternatively you could also put in a list of Urls into the baseUrl key. With MultiCurl we downloaded a list of mirrors per file, which made the process really slow and connection reuse almost impossible. Now that we do not have that file anymore and we want less magic the users will need to configure a Url that points to a repo mirrors list or provide the list themselves.
I'm currently experimenting with adding some code to the preloader to fetch mirrors if it does not have them that we can use during transition time when users do not have the metalink configuration. Not all mirrors might have all the files at the point in time where refresh or download is started.
It seems to have tried to get a different mirror, but ended up suing the same one again anyway the second time. Same, but different with the CURL2 backend:
Basically the same problem as above, thats why the new backend and downloader is not yet the default. We have code in review to solve that issue. Using metalink instead of baseUrl will fix that issue with the new code.
I ended up switching off the preload and tell it to use a different URL when it asked for how to handle the error (note: that option does not seem to be avilable with all backends?). On another machine I had the same problem and here using the MultiCurl backend skipped the two mirrors that did not have the file (apparentlyy because it was requesting the a single block with the same range from all mirrors, which seems wasteful):
--8<---------------cut here---------------start------------->8--- 2025-04-12 22:54:30 <1> Gerda(18610) [zypper++] Summary.cc(readPool):281 <install> U_Ts_r(102515)xz-lang-5.8.1-1.1.noarch(download.opensuse.org-oss_2) 2025-04-12 22:54:30 <1> Gerda(18610) [zypper++] Summary.cc(readPool):281 <uninstall> I_TsU(108695)xz-lang-5.6.4-1.2.noarch(@System) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp] PackageProvider.cc(providePackage):424 provide Package (102515)xz-lang-5.8.1-1.1.noarch(download.opensuse.org-oss_2) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] RepoProvideFile.cc(provideFile):253 [1]./noarch/xz-lang-5.8.1-1.1.noarch.rpm{455.5 KiB|sha512-4f436b4cbbc28e4365efa14d649e8fa5eb3c6432616e4b12e6a935fe530526dfec8a51260ec7faef53fd060f78c9dbbfa40442c15f06de91cf771d3559f06488|} 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::fetcher++] MediaSetAccess.cc(provide):275 Going to try to provide file ./noarch/xz-lang-5.8.1-1.1.noarch.rpm from media number 1 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1510 dest: /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1511 temp: /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm.new.zypp.MK8JI4 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1064 ./noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media++] MediaCurl.cc(doGetFileCopyFile):1074 URL: http://download.opensuse.org/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.no... 2025-04-12 22:54:33 <1> Gerda(18610) [zypp-curl] metalinkparser.cc(parse):445 Parsed 10 mirrors from /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm.new.zypp.MK8JI4 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://mirror.hoasted.com/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.uni-bayreuth.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-lang..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1..., ) 2025-04-12 22:54:33 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.rz.uni-wuerzburg.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-..., ) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://opensuse.schlundtech.de/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1...., ) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media] MediaCurl.cc(MediaCurl):217 MediaCurl::MediaCurl(http://ftp.tu-chemnitz.de/pub/linux/opensuse/tumbleweed/repo/oss/noarch/xz-l..., ) 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #0: state: 6 received: 0 url: http://mirror.hoasted.com/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #1: state: 3 received: 86024 url: http://ftp.uni-bayreuth.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-lang... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #2: state: 4 received: 466439 url: http://ftp.uni-erlangen.de/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #3: state: 3 received: 402773 url: http://ftp.gwdg.de/pub/opensuse/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #4: state: 3 received: 408960 url: http://ftp.rz.uni-wuerzburg.de/linux/opensuse/tumbleweed/repo/oss/noarch/xz-... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #5: state: 6 received: 0 url: http://opensuse.schlundtech.de/tumbleweed/repo/oss/noarch/xz-lang-5.8.1-1.1.... 2025-04-12 22:54:34 <2> Gerda(18610) [zypp::media] MediaMultiCurl.cc(run):1358 #6: state: 3 received: 13396 url: http://ftp.tu-chemnitz.de/pub/linux/opensuse/tumbleweed/repo/oss/noarch/xz-l... 2025-04-12 22:54:34 <1> Gerda(18610) [zypp-core] PathInfo.cc(rename):753 rename /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm.new.zypp.MK8JI4 -> /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media++] MediaMultiCurl.cc(doGetFileCopy):1713 done: /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm{- 0644 0/0 size 466439} 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media++] MediaHandler.cc(provideFile):977 provideFile([1]./noarch/xz-lang-5.8.1-1.1.noarch.rpm{455.5 KiB|sha512-4f436b4cbbc28e4365efa14d649e8fa5eb3c6432616e4b12e6a935fe530526dfec8a51260ec7faef53fd060f78c9dbbfa40442c15f06de91cf771d3559f06488|}) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::fetcher] Fetcher.cc(validate):393 Checking job [/var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm] (2 checkers ) 2025-04-12 22:54:34 <1> Gerda(18610) [zypp-core] PathInfo.cc(hardlinkCopy):921 hardlinkCopy /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm -> /var/cache/zypp/packages/download.opensuse.org-oss_2/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::fetcher++] MediaSetAccess.cc(releaseFile):91 Going to release file ./noarch/xz-lang-5.8.1-1.1.noarch.rpm from media number 1 2025-04-12 22:54:34 <1> Gerda(18610) [zypp-core] PathInfo.cc(unlink):711 unlink /var/tmp/AP_0xAtcA8z/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp::media] RepoProvideFile.cc(provideFile):321 provideFile at /var/cache/zypp/packages/download.opensuse.org-oss_2/noarch/xz-lang-5.8.1-1.1.noarch.rpm 2025-04-12 22:54:34 <1> Gerda(18610) [zypp] PackageProvider.cc(providePackage):525 provided Package (102515)xz-lang-5.8.1-1.1.noarch(download.opensuse.org-oss_2) at /var/cache/zypp/packages/download.opensuse.org-oss_2/noarch/xz-lang-5.8.1-1.1.noarch.rpm --8<---------------cut here---------------end--------------->8---
Generally, sharding a request across servers doesn't make sense unless both the overhead of the request is low and/or it is known in advance that the servers will throttle the rate to significantly less than the capacity of the receiving system. Sharding a request to the same server should not improve download speed much for the type of mirrors that zypper targets also. Empirically, the best heuristic is to select a server that delivers at line rate after a fast TCP ramp up for large files and re-use that connection for as long as it holds that rate. If you know the size of the request beforehand and HTTP/2 can be used, "burning the candle from two ends" is an easy way of keeping the connection fully utilized: start with the largest file and smallest file and keep both queues busy with requests from their respective end. Adding parallel request queues for either or both ends might be useful if the server has a long latency for accepting a request or has a rate limit for individual requests that is significantly below the line rate at the receiving end. As long as the connection gets reused, in my experience you get quickly diminishing returns between 4 or 8 parallel requests, although keeping a 2GBit line busy might require something on the order of 32.
The MultiCurl backend will no longer be used moving forward. The pending PR removes it completely. The point of multicurl never was to be really fast but to be more resilient. It's from a time where the internet connections were not as reliable as they are now. The PR is here if you want to follow https://github.com/openSUSE/libzypp/pull/628 Regards, Benjamin
participants (16)
-
-pj -
Andrei Borzenkov -
Andrei Dziahel -
ASSI -
Axel Braun -
Benjamin Zeller -
Jiri Slaby -
Joe Salmeri -
Knurpht-openSUSE -
Lubos Kocman -
Neal Gompa -
Paulo Dias -
Pavin Joseph -
Stefan Seyfried -
stephane.billiart＠gmail.com -
Timo Jyrinki