Thanks for this tech preview. I did three tests up to now: "regular" dup = zypper dup "super" dup = env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup test 1 notebook: lock libreoffice, regular dup to get snapshot 20210830, remove lock, super dup to get libreoffice update. test 2 desktop: super dup to get multimedia packages update from packman. test 3 notebook: super dup to get snapshot 20210831. desktop: regular dup to get snapshot 20210831. All tests were successful. On test 3, the download with super dup was four times slower than regular dup. The snapshot was fresh so the "additional mirror rating" must have made a poor choice due to the ongoing mirror syncing. Regards, Bruno.

On Tue, 2021-08-31 at 15:53 +0200, Benjamin Zeller wrote:

In the last months the zypp team worked on the basics for speeding up and modernizing the zypper/libzypp codebase. Some of those projects are now available to Tumbleweed and can be enabled and tested.

Please keep in mind that those changes currently are tech previews, so zypper might crash or even break your system.

New RPM transaction backend for zypper: Traditionally zypper executes rpm separately for each operation in a transaction. This requires some hacks and workarounds and is not fully compatible to running all operations in one single transaction.

It is also a lot slower for big transactions ( like dup with many packages ). Therefore we have implemented a new backend that runs all operations in a single big transaction via librpm. It can be enabled by setting: ZYPP_SINGLE_RPMTRANS=1 in zyppers environment.

Issues/Changes with ZYPP_SINGLE_RPMTRANS=1 zypper ... : - Changes how file conflicts work, since they are now executed by librpm we are not able to generate a callback to ask the user if the error should be ignored.    In the presence of file conflicts, the transaction will report an error and will not execute unless --replacefiles is also used.

- Currently there is also a issue that could potentially break your system in very specific preconditions. We put in a safety net that will protect you on Tumbleweed,    however it still remains a issue for all systems that are PREUsrMerge and use the single transaction mode to apply the update.    For example if you are on Leap and want to  upgrade to Tumbleweed.

   This is the bug in question:    https://bugzilla.suse.com/show_bug.cgi?id=1189788

- Breaks Yast / PackageKit softwaremanager if enabled system wide: Due to changes in the reporting , especially new reports, this would break the Yast software manager and Packagekit,    so if you use one of the two technologies enable this feature explicitly per command: `env ZYPP_SINGLE_RPMTRANS=1 zypper dup`.

New zypper HTTP backend: Another project we have been working on is parallelizing downloads, for this a new async downloader backend was implemented. While it currently won't have massive impact on performance due to the frontend code not requesting files asynchronously, it will do some additional mirror rating and as soon as we update the frontend code will bring more benefits.

This can be enabled via setting the env var: ZYPP_MEDIANETWORK=1

If you find any bugs or issues please let us know.

Preferably via:    https://bugzilla.opensuse.org/    Product:   openSUSE Tumbleweed    Component: libzypp (or zypper if available)

   And please attach the /var/log/zypper.log to the bugreport.

it seem to sometimes fail while retriving packages from my own home:sp1rit OBS repo. ``` zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp- core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {}]: Assertion `this->impl()->exec_count_ == 0' failed. [1] 26832 IOT instruction sudo env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup ``` https://bugzilla.opensuse.org/show_bug.cgi?id=1190199 Regards, Florian "sp1rit" -- $\int_\text{now}^{+\infty}\text{Keep trying}$ Matrix: @sp1rit:tchncs.de D248BF2F4C6A82A1E0569D897D8C1CD573166D09 BBDE032EAAFBFC627FB7E635B1F4055D8460CE34

I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only` And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup` Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!

* Patrick Shanahan [09-08-21 18:55]:

* Martin Jungblut [09-08-21 03:11]:

...

I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`

And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`

Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!

I tried these setting today and received download errors until reverting. I do use download-only. I will try to capture the errors and provide them.

Retrieving: libeditorconfig0-0.12.5-1.2.x86_64.rpm --------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped) http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856... Retrieving: openSUSE-release-ftp-20210906-1145.1.x86_64.rpm -----------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856... Retrieving: libc++abi1-12.0.1-1.4.x86_64.rpm --------------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856... -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode

* Patrick Shanahan [09-08-21 20:02]:

* Patrick Shanahan [09-08-21 19:02]:

...

* Patrick Shanahan [09-08-21 18:55]:

...

* Martin Jungblut [09-08-21 03:11]:

...

I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`

And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`

Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!

I tried these setting today and received download errors until reverting. I do use download-only. I will try to capture the errors and provide them.

Retrieving: libeditorconfig0-0.12.5-1.2.x86_64.rpm --------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped) http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...

Retrieving: openSUSE-release-ftp-20210906-1145.1.x86_64.rpm -----------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...

Retrieving: libc++abi1-12.0.1-1.4.x86_64.rpm --------------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...

sorry, read permission missing, corrected. coredumps should be available.

from another machine, Retrieving: readcd-3.02~a10-180.1.x86_64.rpm -----------------------------------------------------------------------------------------[| (388 B/s)] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal::~Signal() [with R = void; T = {}]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped) coredump not saved -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode

I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`

And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`

Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work! Thanks for the report, there was a issue with caching in the new code. This should be fixed with the upcoming libzypp release, for reference:   https://bugzilla.suse.com/show_bug.cgi?id=1190356

-- Benjamin Zeller Systems Programmer SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuremberg, Germany Tel: +49-911-74053-0; Fax: +49-911-7417755; https://www.suse.com/ (HRB 36809, AG Nürnberg)

Benjamin Zeller writes:

New RPM transaction backend for zypper: […] It can be enabled by setting: ZYPP_SINGLE_RPMTRANS=1 in zyppers environment.

I've been updating to Tumbleweed machines with this backend for some time and it works really well.

New zypper HTTP backend: Another project we have been working on is parallelizing downloads, for this a new async downloader backend was implemented.

This on the other hand has been problematic. The backend often choses mirrors that are extremely slow (on the other side of the globe or something that seems to be feeding its upload through a DSL line) and just won't stop using these. The most common symptom is that the download of a large file start off at full wire speed until it's somewhere in the 80% region and then drops off to modem speeds until finally all bits have arrived. If one of the servers of a multi-chunk download is a lot slower than the others, that download should be retried elsewhere.

While it currently won't have massive impact on performance due to the frontend code not requesting files asynchronously, it will do some additional mirror rating and as soon as we update the frontend code will bring more benefits.

At least a download-only dup should be able to fire off the downloads of multiple files in parallel. This is quite well supported in libcurl (you don't need multiple threads for that). If you complete the downloads in issue order, the output could be kept the way it appears today with the synchronous backend (but you wouldn't see the progress on the downloads that are started after the current front-of-queue one). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra