Call for testers: zypp tech preview
In the last months the zypp team worked on the basics for speeding up and modernizing the zypper/libzypp codebase. Some of those projects are now available to Tumbleweed and can be enabled and tested. Please keep in mind that those changes currently are tech previews, so zypper might crash or even break your system. New RPM transaction backend for zypper: Traditionally zypper executes rpm separately for each operation in a transaction. This requires some hacks and workarounds and is not fully compatible to running all operations in one single transaction. It is also a lot slower for big transactions ( like dup with many packages ). Therefore we have implemented a new backend that runs all operations in a single big transaction via librpm. It can be enabled by setting: ZYPP_SINGLE_RPMTRANS=1 in zyppers environment. Issues/Changes with ZYPP_SINGLE_RPMTRANS=1 zypper ... : - Changes how file conflicts work, since they are now executed by librpm we are not able to generate a callback to ask the user if the error should be ignored. In the presence of file conflicts, the transaction will report an error and will not execute unless --replacefiles is also used. - Currently there is also a issue that could potentially break your system in very specific preconditions. We put in a safety net that will protect you on Tumbleweed, however it still remains a issue for all systems that are PREUsrMerge and use the single transaction mode to apply the update. For example if you are on Leap and want to upgrade to Tumbleweed. This is the bug in question: https://bugzilla.suse.com/show_bug.cgi?id=1189788 - Breaks Yast / PackageKit softwaremanager if enabled system wide: Due to changes in the reporting , especially new reports, this would break the Yast software manager and Packagekit, so if you use one of the two technologies enable this feature explicitly per command: `env ZYPP_SINGLE_RPMTRANS=1 zypper dup`. New zypper HTTP backend: Another project we have been working on is parallelizing downloads, for this a new async downloader backend was implemented. While it currently won't have massive impact on performance due to the frontend code not requesting files asynchronously, it will do some additional mirror rating and as soon as we update the frontend code will bring more benefits. This can be enabled via setting the env var: ZYPP_MEDIANETWORK=1 If you find any bugs or issues please let us know. Preferably via: https://bugzilla.opensuse.org/ Product: openSUSE Tumbleweed Component: libzypp (or zypper if available) And please attach the /var/log/zypper.log to the bugreport. -- Benjamin Zeller<bzeller@suse.de> Systems Programmer SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuremberg, Germany Tel: +49-911-74053-0; Fax: +49-911-7417755;https://www.suse.com/ (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer
Thanks for this tech preview. I did three tests up to now: "regular" dup = zypper dup "super" dup = env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup test 1 notebook: lock libreoffice, regular dup to get snapshot 20210830, remove lock, super dup to get libreoffice update. test 2 desktop: super dup to get multimedia packages update from packman. test 3 notebook: super dup to get snapshot 20210831. desktop: regular dup to get snapshot 20210831. All tests were successful. On test 3, the download with super dup was four times slower than regular dup. The snapshot was fresh so the "additional mirror rating" must have made a poor choice due to the ongoing mirror syncing. Regards, Bruno.
Hi Bruno, thanks for the feedback! As for test 3 , its hard to reliable profile speed on a living server that sometimes has high load and sometimes low. But the mirror rating is just a very simple and early version, might be interesting to see your zypper log for the slow case to check if any timeouts were firing. Basically we only profile connection time and resort mirrors we got from download.opensuse.org according to that. Also the new media backend is writting with async requests in mind and since the frontend does not schedule those yet it won't have a lot of benefit, we are working on it :). Benjamin Am 02.09.21 um 04:20 schrieb Bruno Damasceno Freire:
Thanks for this tech preview.
I did three tests up to now:
"regular" dup = zypper dup "super" dup = env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup
test 1 notebook: lock libreoffice, regular dup to get snapshot 20210830, remove lock, super dup to get libreoffice update.
test 2 desktop: super dup to get multimedia packages update from packman.
test 3 notebook: super dup to get snapshot 20210831. desktop: regular dup to get snapshot 20210831.
All tests were successful. On test 3, the download with super dup was four times slower than regular dup. The snapshot was fresh so the "additional mirror rating" must have made a poor choice due to the ongoing mirror syncing.
Regards, Bruno.
-- Benjamin Zeller <bzeller@suse.de> Systems Programmer SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuremberg, Germany Tel: +49-911-74053-0; Fax: +49-911-7417755; https://www.suse.com/ (HRB 36809, AG Nürnberg) Managing Director: Felix Imendörffer
On Tue, 2021-08-31 at 15:53 +0200, Benjamin Zeller wrote:
In the last months the zypp team worked on the basics for speeding up and modernizing the zypper/libzypp codebase. Some of those projects are now available to Tumbleweed and can be enabled and tested.
Please keep in mind that those changes currently are tech previews, so zypper might crash or even break your system.
New RPM transaction backend for zypper: Traditionally zypper executes rpm separately for each operation in a transaction. This requires some hacks and workarounds and is not fully compatible to running all operations in one single transaction.
It is also a lot slower for big transactions ( like dup with many packages ). Therefore we have implemented a new backend that runs all operations in a single big transaction via librpm. It can be enabled by setting: ZYPP_SINGLE_RPMTRANS=1 in zyppers environment.
Issues/Changes with ZYPP_SINGLE_RPMTRANS=1 zypper ... : - Changes how file conflicts work, since they are now executed by librpm we are not able to generate a callback to ask the user if the error should be ignored. In the presence of file conflicts, the transaction will report an error and will not execute unless --replacefiles is also used.
- Currently there is also a issue that could potentially break your system in very specific preconditions. We put in a safety net that will protect you on Tumbleweed, however it still remains a issue for all systems that are PREUsrMerge and use the single transaction mode to apply the update. For example if you are on Leap and want to upgrade to Tumbleweed.
This is the bug in question: https://bugzilla.suse.com/show_bug.cgi?id=1189788
- Breaks Yast / PackageKit softwaremanager if enabled system wide: Due to changes in the reporting , especially new reports, this would break the Yast software manager and Packagekit, so if you use one of the two technologies enable this feature explicitly per command: `env ZYPP_SINGLE_RPMTRANS=1 zypper dup`.
New zypper HTTP backend: Another project we have been working on is parallelizing downloads, for this a new async downloader backend was implemented. While it currently won't have massive impact on performance due to the frontend code not requesting files asynchronously, it will do some additional mirror rating and as soon as we update the frontend code will bring more benefits.
This can be enabled via setting the env var: ZYPP_MEDIANETWORK=1
If you find any bugs or issues please let us know.
Preferably via: https://bugzilla.opensuse.org/ Product: openSUSE Tumbleweed Component: libzypp (or zypper if available)
And please attach the /var/log/zypper.log to the bugreport.
it seem to sometimes fail while retriving packages from my own home:sp1rit OBS repo. ``` zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp- core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {}]: Assertion `this->impl()->exec_count_ == 0' failed. [1] 26832 IOT instruction sudo env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup ``` https://bugzilla.opensuse.org/show_bug.cgi?id=1190199 Regards, Florian "sp1rit" -- $\int_\text{now}^{+\infty}\text{Keep trying}$ Matrix: @sp1rit:tchncs.de <sp1rit@disroot.org> D248BF2F4C6A82A1E0569D897D8C1CD573166D09 <sp1rit@national.shitposting.agency> BBDE032EAAFBFC627FB7E635B1F4055D8460CE34
I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only` And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup` Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!
I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only` And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup` Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!
* Martin Jungblut <martinjungblut@gmail.com> [09-08-21 03:11]:
I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`
And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`
Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!
I tried these setting today and received download errors until reverting. I do use download-only. I will try to capture the errors and provide them. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode
* Patrick Shanahan <paka@opensuse.org> [09-08-21 18:55]:
* Martin Jungblut <martinjungblut@gmail.com> [09-08-21 03:11]:
I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`
And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`
Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!
I tried these setting today and received download errors until reverting. I do use download-only. I will try to capture the errors and provide them.
Retrieving: libeditorconfig0-0.12.5-1.2.x86_64.rpm --------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped) http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856... Retrieving: openSUSE-release-ftp-20210906-1145.1.x86_64.rpm -----------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856... Retrieving: libc++abi1-12.0.1-1.4.x86_64.rpm --------------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856... -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode
* Patrick Shanahan <paka@opensuse.org> [09-08-21 19:02]:
* Patrick Shanahan <paka@opensuse.org> [09-08-21 18:55]:
* Martin Jungblut <martinjungblut@gmail.com> [09-08-21 03:11]:
I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`
And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`
Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!
I tried these setting today and received download errors until reverting. I do use download-only. I will try to capture the errors and provide them.
Retrieving: libeditorconfig0-0.12.5-1.2.x86_64.rpm --------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped) http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...
Retrieving: openSUSE-release-ftp-20210906-1145.1.x86_64.rpm -----------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...
Retrieving: libc++abi1-12.0.1-1.4.x86_64.rpm --------------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...
sorry, read permission missing, corrected. coredumps should be available. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode
* Patrick Shanahan <paka@opensuse.org> [09-08-21 20:02]:
* Patrick Shanahan <paka@opensuse.org> [09-08-21 19:02]:
* Patrick Shanahan <paka@opensuse.org> [09-08-21 18:55]:
* Martin Jungblut <martinjungblut@gmail.com> [09-08-21 03:11]:
I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`
And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`
Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work!
I tried these setting today and received download errors until reverting. I do use download-only. I will try to capture the errors and provide them.
Retrieving: libeditorconfig0-0.12.5-1.2.x86_64.rpm --------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped) http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...
Retrieving: openSUSE-release-ftp-20210906-1145.1.x86_64.rpm -----------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...
Retrieving: libc++abi1-12.0.1-1.4.x86_64.rpm --------------------------------------------------------------------------------------------[starting] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {} ]: Assertion `this->impl()->exec_count_ == 0' failed. http://wahoo.no-ip.org/~paka/core.Zypp-main.0.b9956c991631416ab63133a35a0856...
sorry, read permission missing, corrected. coredumps should be available.
from another machine, Retrieving: readcd-3.02~a10-180.1.x86_64.rpm -----------------------------------------------------------------------------------------[| (388 B/s)] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {}]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped) coredump not saved -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode
Hi Patrick, Am 09.09.21 um 16:05 schrieb Patrick Shanahan
from another machine,
Retrieving: readcd-3.02~a10-180.1.x86_64.rpm -----------------------------------------------------------------------------------------[| (388 B/s)] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {}]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped)
coredump not saved sorry for the late answer. We had some issues with an assertion in the code, which should be fixed now. Do you still run into those issues?
I just ran: `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup --download-only`
And trying to run a regular dup (without --download-only) did not display the packages as being already cached. I had to re-download them all again when running `env ZYPP_SINGLE_RPMTRANS=1 ZYPP_MEDIANETWORK=1 zypper dup`
Other than that, the upgrade went flawlessly, and these features are really useful. Thanks a lot for the incredible work! Thanks for the report, there was a issue with caching in the new code. This should be fixed with the upcoming libzypp release, for reference: https://bugzilla.suse.com/show_bug.cgi?id=1190356
-- Benjamin Zeller <bzeller@suse.de> Systems Programmer SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuremberg, Germany Tel: +49-911-74053-0; Fax: +49-911-7417755; https://www.suse.com/ (HRB 36809, AG Nürnberg)
* Benjamin Zeller <bzeller@suse.de> [10-14-21 05:58]:
Hi Patrick,
Am 09.09.21 um 16:05 schrieb Patrick Shanahan
from another machine,
Retrieving: readcd-3.02~a10-180.1.x86_64.rpm -----------------------------------------------------------------------------------------[| (388 B/s)] zypper: /home/abuild/rpmbuild/BUILD/libzypp-17.28.2/zypp-core/zyppng/base/signals.h:193: zyppng::Signal<R(T ...)>::~Signal() [with R = void; T = {}]: Assertion `this->impl()->exec_count_ == 0' failed. Aborted (core dumped)
coredump not saved sorry for the late answer. We had some issues with an assertion in the code, which should be fixed now. Do you still run into those issues?
tks, just tested on three tw boxes successfully. good work. added export ZYPP_SINGLE_RPMTRANS=1 export ZYPP_MEDIANETWORK=1 to .bashrc -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode
Benjamin Zeller writes:
New RPM transaction backend for zypper: […] It can be enabled by setting: ZYPP_SINGLE_RPMTRANS=1 in zyppers environment.
I've been updating to Tumbleweed machines with this backend for some time and it works really well.
New zypper HTTP backend: Another project we have been working on is parallelizing downloads, for this a new async downloader backend was implemented.
This on the other hand has been problematic. The backend often choses mirrors that are extremely slow (on the other side of the globe or something that seems to be feeding its upload through a DSL line) and just won't stop using these. The most common symptom is that the download of a large file start off at full wire speed until it's somewhere in the 80% region and then drops off to modem speeds until finally all bits have arrived. If one of the servers of a multi-chunk download is a lot slower than the others, that download should be retried elsewhere.
While it currently won't have massive impact on performance due to the frontend code not requesting files asynchronously, it will do some additional mirror rating and as soon as we update the frontend code will bring more benefits.
At least a download-only dup should be able to fire off the downloads of multiple files in parallel. This is quite well supported in libcurl (you don't need multiple threads for that). If you complete the downloads in issue order, the output could be kept the way it appears today with the synchronous backend (but you wouldn't see the progress on the downloads that are started after the current front-of-queue one). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra
* Achim Gratz <Stromeko@nexgo.de> [01-29-22 15:23]:
Benjamin Zeller writes:
New RPM transaction backend for zypper: […] It can be enabled by setting: ZYPP_SINGLE_RPMTRANS=1 in zyppers environment.
I've been updating to Tumbleweed machines with this backend for some time and it works really well.
I have been using that setting since Benjamin Zeller announced it 31 Aug last and have not noticed a problem. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc What sort of day was it? A day like all days, filled with those events that alter and illuminate our times...
participants (6)
-
Achim Gratz -
Benjamin Zeller -
Bruno Damasceno Freire -
Martin Jungblut -
Patrick Shanahan -
sp1rit