Enable delta RPMs for TW (was Re: [opensuse-factory] New Tumbleweed snapshot 20160117 released!)
Hi list, I'd like to raise a topic from almost a year ago, http://lists.opensuse.org/opensuse-factory/2015-02/msg00032.html
In the meantime `zypper lu` says almost (no, I didn't go through each of ~2900 lines of it) every package got just build number increased. Not package version, not upstream version, only build number. And I've got that gut feeling that build number increment is only change for 99% of affected packages. I wonder how big would delta RPMs with such changes be, something like 6K, correct? Imagine the bandwith (and more importantly, the time) they'd save, if someone would enable building and publishing them for Tumbleweed. So, could delta RPMs be built and published in Tumbleweed? Or at least can we have a steady resolution on that topic? Thank you in advance. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 21.01.2016 11:56, Andrei Dziahel wrote:
Whenever there is a full rebuild, the wish for delta rpms is brought up - and every time I explain that it needs someone to drive it. And then silence ;( Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Just to make sure I've got you right: "someone" means "someone who got admin access to OBS", correct? 2016-01-21 15:04 GMT+03:00 Stephan Kulow <coolo@suse.de>:
-- Regards, Andrei Dziahel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 21 January 2016 at 18:51, Andrei Dziahel <develop7@develop7.info> wrote:
Just to make sure I've got you right: "someone" means "someone who got admin access to OBS", correct?
I believe Coolo was trying to point out that if people want delta RPMs, someone has to drive it. And this isn't just a simple technical 'I need access rights' perspective, but also a practical one like.. 'delta RPMs against what'? If you declared one snapshot as the 'master' forever, and only ever did Deltas from that, you'd soon end up with bigger problems than the occational rebuild If you declared one snapshot per month as the master, and did deltas from that, you'd be having a similar problem to this one, just monthly rather than 'whenever we need to' Someone needs to decide what is wanted, how to achieve that, and then do it Access rights isn't an issue - You have 'admin access' to your own home: project in the build service, and that should be sufficient for any proof of concept to demonstrate that a chosen approach is viable. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Ok, I feel we've been there before :) 2016-01-21 21:04 GMT+03:00 Richard Brown <RBrownCCB@opensuse.org>:
In order to avoid redownloading thousands of packages due to mass rebuild, I'd answer "against previous snapshot, period". If you're not following TW snapshots, you probably don't care anyway.
What would serve best way — a wiki page, an RFC right here, and openFATE idea or something else?
unfortunately no — drpms can be built in OBS, but they are not published regardless: https://gist.github.com/develop7/9d854840a404b67ae661 (drpms are in place (http://download.opensuse.org/repositories/home:/develop7:/branches:/GNOME:/A...) but there's no mention of them in [repodata](http://download.opensuse.org/repositories/home:/develop7:/branches:/GNOME:/A...), which was mentioned in http://lists.opensuse.org/opensuse-factory/2015-02/msg00183.html) -- Regards, Andrei Dziahel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jan 22, 2016 at 10:56:56AM +0300, Andrei Dziahel wrote:
That's actually a bug in the publisher: you just added the delta option in the project config, but no package was rebuilt. So the publisher didn't re-publish the repo. If you look at your openSUSE_Factory instead of openSUSE_Tumbleweed, you'll find a "prestodelta" entry. AFAIK this "prestodelta" file will be ignored by the zypp stack, though, as it's called "deltainfo" for SUSE. So please change the config back to "Repotype: rpm-md:deltainfo" for SUSE. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jan 22, 2016 at 11:16:50AM +0100, Michael Schroeder wrote:
Oh, and could you please remote that "Support: build-mkdrpms deltarpm" line? The deltas created by the build process are completly useless, you need the ones created by the OBS publisher. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Except when we talk about publishing openSUSE:Factory - this project is not published in the normal way by the OBS publisher, but done through the media builder using kiwi... I assume in this case we will need the deltarpm generation done during build - not in the publisher run, and kiwi can then collect the files and add them to the repository and the metadata, no? Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jan 22, 2016 at 12:34:48PM +0100, Dominique Leuenberger / DimStar wrote:
Yes, Tumbleweed is a kiwi product, so the deltas need to be done in a different way. I was talking about the home:develop7:branches:GNOME:Apps project. For Tumbleweed, deltas from the build process are ok, I guess. I can change the obs worker to also download the deltas, but somebody needs to patch kiwi and create_package_descr so that the deltas are in the created repos. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Done. And guess what: it actually worked — deltas were actually downloaded and applied by zypper. Thanks for suggestion. 2016-01-22 13:16 GMT+03:00 Michael Schroeder <mls@suse.de>:
-- Regards, Andrei Dziahel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 25.01.2016 13:27, Andrei Dziahel wrote:
Done. And guess what: it actually worked — deltas were actually downloaded and applied by zypper. Thanks for suggestion.
Now comes the catch: TW is not a repo-md repo in OBS, but a product built by kiwi. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 2016-01-22 at 11:16 +0100, Michael Schroeder wrote:
This happens to work in Andrei's home project as the drpm are generated against the packages in the PARENT project - but NOT against old packages inside the same project atself as far as I see. So, even though there are now 'drpm' packages created there, they do not appear to be exactly what the expectation would be imho (this works fine for the :Update repo. Besides all the other topics already discussed that openSUSE Tumbleweed anyway is a kiwi product, and no rpm-md repo: the main question remains: against what should the .drpm be created? (my vote would be 'last published snapshot, so openSUSE:Factory/snapshot, but making sure that kiwi has access to this info won't be much easier) Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Jan 25, 2016 at 04:30:30PM +0100, Dominique Leuenberger / DimStar wrote:
That depends. They are built against the last published state, but the if a repo is published again (because some other package is built), the last published version will be identical and thus no delta is created. This works for the update repos, as they contain all the old versions as well. The code should be changed so that it keeps the old delta if no other version is present. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 21.01.2016 18:51, Andrei Dziahel wrote:
Just to make sure I've got you right: "someone" means "someone who got admin access to OBS", correct?
No. Someone has to find out what's missing in OBS (e.g. by deploying OBS himself and playing with the delta feature on FTP trees) and then drive the solution. The last part is then bothering OBS admins, yes. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 21.01.2016 um 11:56 schrieb Andrei Dziahel:
If the build number would be the only change, build-compare would have prevented the publisheing of the packages, I assume.
Go ahead, send a pull request! :-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jan 22, Stefan Seyfried wrote:
If the build number would be the only change, build-compare would have prevented the publisheing of the packages, I assume.
It will catch only a very small subset of all packages. Everything which is using either c++ or cmake will most likely differ because the objects are linked in unpredictable order. Olaf -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Olaf Hering <olaf@aepfle.de> writes:
There is nothing inherent in C++ that causes unpredictable link order. Andreas. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 21.01.2016 11:56, Andrei Dziahel wrote:
Whenever there is a full rebuild, the wish for delta rpms is brought up - and every time I explain that it needs someone to drive it. And then silence ;( Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Just to make sure I've got you right: "someone" means "someone who got admin access to OBS", correct? 2016-01-21 15:04 GMT+03:00 Stephan Kulow <coolo@suse.de>:
-- Regards, Andrei Dziahel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 21 January 2016 at 18:51, Andrei Dziahel <develop7@develop7.info> wrote:
Just to make sure I've got you right: "someone" means "someone who got admin access to OBS", correct?
I believe Coolo was trying to point out that if people want delta RPMs, someone has to drive it. And this isn't just a simple technical 'I need access rights' perspective, but also a practical one like.. 'delta RPMs against what'? If you declared one snapshot as the 'master' forever, and only ever did Deltas from that, you'd soon end up with bigger problems than the occational rebuild If you declared one snapshot per month as the master, and did deltas from that, you'd be having a similar problem to this one, just monthly rather than 'whenever we need to' Someone needs to decide what is wanted, how to achieve that, and then do it Access rights isn't an issue - You have 'admin access' to your own home: project in the build service, and that should be sufficient for any proof of concept to demonstrate that a chosen approach is viable. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Ok, I feel we've been there before :) 2016-01-21 21:04 GMT+03:00 Richard Brown <RBrownCCB@opensuse.org>:
In order to avoid redownloading thousands of packages due to mass rebuild, I'd answer "against previous snapshot, period". If you're not following TW snapshots, you probably don't care anyway.
What would serve best way — a wiki page, an RFC right here, and openFATE idea or something else?
unfortunately no — drpms can be built in OBS, but they are not published regardless: https://gist.github.com/develop7/9d854840a404b67ae661 (drpms are in place (http://download.opensuse.org/repositories/home:/develop7:/branches:/GNOME:/A...) but there's no mention of them in [repodata](http://download.opensuse.org/repositories/home:/develop7:/branches:/GNOME:/A...), which was mentioned in http://lists.opensuse.org/opensuse-factory/2015-02/msg00183.html) -- Regards, Andrei Dziahel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jan 22, 2016 at 10:56:56AM +0300, Andrei Dziahel wrote:
That's actually a bug in the publisher: you just added the delta option in the project config, but no package was rebuilt. So the publisher didn't re-publish the repo. If you look at your openSUSE_Factory instead of openSUSE_Tumbleweed, you'll find a "prestodelta" entry. AFAIK this "prestodelta" file will be ignored by the zypp stack, though, as it's called "deltainfo" for SUSE. So please change the config back to "Repotype: rpm-md:deltainfo" for SUSE. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jan 22, 2016 at 11:16:50AM +0100, Michael Schroeder wrote:
Oh, and could you please remote that "Support: build-mkdrpms deltarpm" line? The deltas created by the build process are completly useless, you need the ones created by the OBS publisher. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Except when we talk about publishing openSUSE:Factory - this project is not published in the normal way by the OBS publisher, but done through the media builder using kiwi... I assume in this case we will need the deltarpm generation done during build - not in the publisher run, and kiwi can then collect the files and add them to the repository and the metadata, no? Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, Jan 22, 2016 at 12:34:48PM +0100, Dominique Leuenberger / DimStar wrote:
Yes, Tumbleweed is a kiwi product, so the deltas need to be done in a different way. I was talking about the home:develop7:branches:GNOME:Apps project. For Tumbleweed, deltas from the build process are ok, I guess. I can change the obs worker to also download the deltas, but somebody needs to patch kiwi and create_package_descr so that the deltas are in the created repos. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Done. And guess what: it actually worked — deltas were actually downloaded and applied by zypper. Thanks for suggestion. 2016-01-22 13:16 GMT+03:00 Michael Schroeder <mls@suse.de>:
-- Regards, Andrei Dziahel -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 25.01.2016 13:27, Andrei Dziahel wrote:
Done. And guess what: it actually worked — deltas were actually downloaded and applied by zypper. Thanks for suggestion.
Now comes the catch: TW is not a repo-md repo in OBS, but a product built by kiwi. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Fri, 2016-01-22 at 11:16 +0100, Michael Schroeder wrote:
This happens to work in Andrei's home project as the drpm are generated against the packages in the PARENT project - but NOT against old packages inside the same project atself as far as I see. So, even though there are now 'drpm' packages created there, they do not appear to be exactly what the expectation would be imho (this works fine for the :Update repo. Besides all the other topics already discussed that openSUSE Tumbleweed anyway is a kiwi product, and no rpm-md repo: the main question remains: against what should the .drpm be created? (my vote would be 'last published snapshot, so openSUSE:Factory/snapshot, but making sure that kiwi has access to this info won't be much easier) Cheers, Dominique -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Mon, Jan 25, 2016 at 04:30:30PM +0100, Dominique Leuenberger / DimStar wrote:
That depends. They are built against the last published state, but the if a repo is published again (because some other package is built), the last published version will be identical and thus no delta is created. This works for the update repos, as they contain all the old versions as well. The code should be changed so that it keeps the old delta if no other version is present. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (8)
-
Andreas Schwab -
Andrei Dziahel -
Dominique Leuenberger / DimStar -
Michael Schroeder -
Olaf Hering -
Richard Brown -
Stefan Seyfried -
Stephan Kulow