[opensuse-buildservice] OBS scheduler features
I have two patches for review: 1. Fix a problem with different names - OBS package name and RPM name described in the spec-file. For example: OBS package name 'linaro-gcc', but name of rpm is 'gcc'. 2. Change a scheduler mechanism. The build of non-cyclic packets wait the builds of cyclic packets. This has some advantages: reduce time of project build, reducing number of packages rebuilds. I tested the patches on the project with 129 packages (41 cyclic) - the build time was reduced by ~33%. I would like to know your opinion about the patches. -- Best regards, Oleg Ogurtsov. Samsung R&D Institute Russia, e-mail: o.ogurtsov@samsung.com
Hi! First of all thank you for looking into the gory parts of the open build service! ;) On Fri, Jul 21, 2017 at 09:01:44AM +0300, Oleg Ogurtsov wrote:
I have two patches for review: 1. Fix a problem with different names - OBS package name and RPM name described in the spec-file. For example: OBS package name 'linaro-gcc', but name of rpm is 'gcc'.
This should work. So you'll need to be much more specific on where the problem is. Your patch does a couple of things: - it changes the way the $notready hash is used, it also uses the OBS package name. I don't understand why this is needed. The current policy is to put rpm source names into the hash, because we can get the source names from the repository metadata. - it changes the way package sorting works Yes, I think there is indeed a bug in the current code, it seems to mix rpm source names with OBS package names. You might also have a problem in your code, because multiple OBS packages can build the same source (this is common when more than one version of a package is built). So your %name2pkgid hash must map to an array, not scalar. Setting up a dep2pkgid hash might be easier, though.
2. Change a scheduler mechanism. The build of non-cyclic packets wait the builds of cyclic packets. This has some advantages: reduce time of project build, reducing number of packages rebuilds. I tested the patches on the project with 129 packages (41 cyclic) - the build time was reduced by ~33%.
Hmm, I remember mailing with Pavel Kopyl from Samsung about improving the handling of cycles. Is this related? I think the patch does way to much to implement this. My suggestion from the mail thread was just to tweak the code so that non-cycle packages come after the cycle ones, and then change the code so that "$notready" is set of all cycle packages if one of them is building/blocked. I'll check how much change that would be next week, we can then discuss things further. Again, thanks for daring to improve the messy parts of the backend code. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello. Sorry for the long silence. I continued Pavel's work. At first he faced a problem of names and solved it. I made optimization on the basis of his patch. 1. Currently, web-interface does not show cycles for obs packets with different names OBS and rpm. For example, 3 obs-packets: obs packages names are obs-package1, obs-package2, obs-package3, and rpm names are package1, package2, package3 (described in the spec - Name: package1). The cycle - package1 have build requires package2, package2 have build requires package3, package3 have build requires package1. Need to separate entities obs-package and rpm-name. Thus, we need obs-package -> rpm-name -> rpm-packages(many rpm files, from one spec file). The 'notready' hash must contain obs package names. And we always can get all data: rpm-name->obs-package and back. This patch fix a problem. I tested also with linked obs-package. And did not see any problems.
You might also have a problem in your code, OBS packages can build the same source (this is common when more than one version of a package is built). So your %name2pkgid hash must map to an array, not scalar. Setting up a dep2pkgid hash might be easier, though. Can you describe in more detail?
2. The code in Checker.pm changes the sequence of processing. The code in Package.pm adds a checking for non-cyclic packages - block the build of package, if all cyclic packages are not ready. This is more effective. Best regards, Oleg Ogurtsov. Samsung R&D Institute Russia, e-mail: o.ogurtsov@samsung.com -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello. I want to report strange results with different versions of the library BSSolv. I developed on the OBS version 2.7 with library BSSolv 0.9(git tag). And tested on the OBS 2.8 from iso image, which, as I understood, has also BSSolv version 0.9. I wrote about results earlier. Then I had to test another project (very similar on the previous: 133 packages, 41 cyclic). I installed a last version BSSolv from sources (0.13). And I saw a worsening of work with my patches - increase in a build time of the project. I have a log parser for dispatcher and saw an increase rebuilds for cyclic packets. I suppose this is related with changes in the BSSolv library. Also it is related to the first patch, because it fixes the names and two packages become cyclically dependent (such as linaro-gcc). Can someone tell me something about this? p.s. Perhaps, I haven't noticed something. Best regards, Oleg Ogurtsov. Samsung R&D Institute Russia, e-mail: o.ogurtsov@samsung.com -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hello. This is ping for "02/08/2017 19:05" e-mail. On 10/08/2017 18:16, Oleg Ogurtsov wrote:
I installed a last version BSSolv from sources (0.13). And I saw a worsening of work with my patches - increase in a build time of the project.
I meant perl-BSSolv.I tested with different versions of the perl-BSSlov library and concluded - versions above 0.30.0 (or 0.10 git-tag) break the scheduler algorithm with phases 1,2,3 for cyclic packages (%cycpass). Packets don't wait the phase 3. And after the build, scheduler set up their immediately on the rebuild. Please tell me if it is not so. On 28/07/2017 18:29, Michael Schroeder wrote:
Hmm, I remember mailing with Pavel Kopyl from Samsung about improving the handling of cycles. Is this related?
On 02/08/2017 19:05, Oleg Ogurtsov wrote:
I continued Pavel's work. At first he faced a problem of names and solved it. I made optimization on the basis of his patch.
1. I think, Pavel's patch fix a problem with names. Although it significantly increases the build time, as in my case with linaro-gcc(relatively long build time). The patch itself may worsen the performance, but it fixes problem in internal package names. So, in my opinion, it is significant for OBS. On 28/07/2017 18:29, Michael Schroeder wrote:
I think the patch does way to much to implement this. My suggestion from the mail thread was just to tweak the code so that non-cycle packages come after the cycle ones, and then change the code so that "$notready" is set of all cycle packages if one of them is building/blocked.
I'll check how much change that would be next week, we can then discuss things further.On 02/08/2017 19:05, Oleg Ogurtsov wrote: 2. The code in Checker.pm changes the sequence of processing. The code in Package.pm adds a checking for non-cyclic packages - block the build of package, if all cyclic packages are not ready.
2. The main idea - build non-cyclic packages and their dependencies only once. Changing the processing sequence is not enough. But changes in Package.pm also do not guarantee the expected result. A project configuration may contain lines that exclude packages from list of expanded dependencies, because of Preinstall section. It can do is added as an option to a project configuration - enabling/disabling. -- Best regards, Oleg Ogurtsov. Samsung R&D Institute Russia, e-mail: o.ogurtsov@samsung.com -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Ping On 08/25/2017 03:51 PM, Oleg Ogurtsov wrote:
Hello. This is ping for "02/08/2017 19:05" e-mail.
On 10/08/2017 18:16, Oleg Ogurtsov wrote:
I installed a last version BSSolv from sources (0.13). And I saw a worsening of work with my patches - increase in a build time of the project.
I meant perl-BSSolv.I tested with different versions of the perl-BSSlov library and concluded - versions above 0.30.0 (or 0.10 git-tag) break the scheduler algorithm with phases 1,2,3 for cyclic packages (%cycpass). Packets don't wait the phase 3. And after the build, scheduler set up their immediately on the rebuild. Please tell me if it is not so.
On 28/07/2017 18:29, Michael Schroeder wrote:
Hmm, I remember mailing with Pavel Kopyl from Samsung about improving the handling of cycles. Is this related?
On 02/08/2017 19:05, Oleg Ogurtsov wrote:
I continued Pavel's work. At first he faced a problem of names and solved it. I made optimization on the basis of his patch.
1. I think, Pavel's patch fix a problem with names. Although it significantly increases the build time, as in my case with linaro-gcc(relatively long build time). The patch itself may worsen the performance, but it fixes problem in internal package names. So, in my opinion, it is significant for OBS.
On 28/07/2017 18:29, Michael Schroeder wrote:
I think the patch does way to much to implement this. My suggestion from the mail thread was just to tweak the code so that non-cycle packages come after the cycle ones, and then change the code so that "$notready" is set of all cycle packages if one of them is building/blocked.
I'll check how much change that would be next week, we can then discuss things further.On 02/08/2017 19:05, Oleg Ogurtsov wrote: 2. The code in Checker.pm changes the sequence of processing. The code in Package.pm adds a checking for non-cyclic packages - block the build of package, if all cyclic packages are not ready.
2. The main idea - build non-cyclic packages and their dependencies only once. Changing the processing sequence is not enough. But changes in Package.pm also do not guarantee the expected result. A project configuration may contain lines that exclude packages from list of expanded dependencies, because of Preinstall section. It can do is added as an option to a project configuration - enabling/disabling.
Best regards, Oleg Ogurtsov. Samsung R&D Institute Russia, e-mail: o.ogurtsov@samsung.com -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mittwoch, 13. September 2017, 09:29:54 CEST wrote Oleg Ogurtsov:
Ping
Sorry, but this is not on our top prio list atm (because not from a contracted customer;) and it is also at a cruical place (esp. performance wise). So we don't want to rush it here. But we admit that it should be changed. Sorry, some more patience is needed :/ adrian
On 08/25/2017 03:51 PM, Oleg Ogurtsov wrote:
Hello. This is ping for "02/08/2017 19:05" e-mail.
On 10/08/2017 18:16, Oleg Ogurtsov wrote:
I installed a last version BSSolv from sources (0.13). And I saw a worsening of work with my patches - increase in a build time of the project.
I meant perl-BSSolv.I tested with different versions of the perl-BSSlov library and concluded - versions above 0.30.0 (or 0.10 git-tag) break the scheduler algorithm with phases 1,2,3 for cyclic packages (%cycpass). Packets don't wait the phase 3. And after the build, scheduler set up their immediately on the rebuild. Please tell me if it is not so.
On 28/07/2017 18:29, Michael Schroeder wrote:
Hmm, I remember mailing with Pavel Kopyl from Samsung about improving the handling of cycles. Is this related?
On 02/08/2017 19:05, Oleg Ogurtsov wrote:
I continued Pavel's work. At first he faced a problem of names and solved it. I made optimization on the basis of his patch.
1. I think, Pavel's patch fix a problem with names. Although it significantly increases the build time, as in my case with linaro-gcc(relatively long build time). The patch itself may worsen the performance, but it fixes problem in internal package names. So, in my opinion, it is significant for OBS.
On 28/07/2017 18:29, Michael Schroeder wrote:
I think the patch does way to much to implement this. My suggestion from the mail thread was just to tweak the code so that non-cycle packages come after the cycle ones, and then change the code so that "$notready" is set of all cycle packages if one of them is building/blocked.
I'll check how much change that would be next week, we can then discuss things further.On 02/08/2017 19:05, Oleg Ogurtsov wrote: 2. The code in Checker.pm changes the sequence of processing. The code in Package.pm adds a checking for non-cyclic packages - block the build of package, if all cyclic packages are not ready.
2. The main idea - build non-cyclic packages and their dependencies only once. Changing the processing sequence is not enough. But changes in Package.pm also do not guarantee the expected result. A project configuration may contain lines that exclude packages from list of expanded dependencies, because of Preinstall section. It can do is added as an option to a project configuration - enabling/disabling.
Best regards, Oleg Ogurtsov.
Samsung R&D Institute Russia, e-mail: o.ogurtsov@samsung.com
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Adrian Schröter -
Michael Schroeder -
Oleg Ogurtsov