[opensuse-buildservice] The details for scheduling option "rebuild" in meta configuration
Hello everyone, I have been confused for long time in terms of the rebuild (or I can say 'scheduling') mechanism of OBS. The most confused point is why a full rebuild is needed for software release on OBS. It is said in the document which describes the scheduling strategies that if we don't use the default value of "rebuild" and "block" option for scheduling, there will be some build or dependency problem occurred. I can not imagine what kind of the scenario will cause the problem. I will be appreciate if someone can describe the reason or can provide some material for that. Thanks Jidiablo -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Donnerstag, 26. Februar 2015, 14:59:21 wrote Ji Xiang:
Hello everyone, I have been confused for long time in terms of the rebuild (or I can say 'scheduling') mechanism of OBS. The most confused point is why a full rebuild is needed for software release on OBS.
Well if it is "needed" can always be discussed. But it is needed _IF_ you want to be sure that you are able to rebuild it after releasing it (eg. the newer gcc may just crash or header files suddenly provide conflicting definitions). Also some changes could apply which are not expressed in the dependencies of packages.
It is said in the document which describes the scheduling strategies that if we don't use the default value of "rebuild" and "block" option for scheduling, there will be some build or dependency problem occurred.
block option is mostly only an optimization of build power => do not build something if you know you will need to rebuild it soon (at least in default (transitive) rebuild strategy).
I can not imagine what kind of the scenario will cause the problem.
I will be appreciate if someone can describe the reason or can provide some material for that.
It is a little bit documented here: http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.build_s...
Thanks Jidiablo
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Thursday, February 26, 2015 08:26:09 AM Adrian Schröter wrote:
On Donnerstag, 26. Februar 2015, 14:59:21 wrote Ji Xiang:
Hello everyone, I have been confused for long time in terms of the rebuild (or I can say 'scheduling') mechanism of OBS. The most confused point is why a full rebuild is needed for software release on OBS.
Well if it is "needed" can always be discussed. But it is needed _IF_ you want to be sure that you are able to rebuild it after releasing it (eg. the newer gcc may just crash or header files suddenly provide conflicting definitions). Also some changes could apply which are not expressed in the dependencies of packages.
Many thanks for your kind of answer, do you mind give me a example what kind of the change which are not included in the dependency? The major reason I asked this question is we are using OBS, however we found that it takes quite long time to do a full build(some packages are built many times due to the dependency), and if there are some changes happened at low layer(like glibc , qt etc), all the packages will be built, we have to wait for long time. It can not met the requirement of fast release process. So I am trying to understand how the scheduling mechanism of OBS working, and how to optimize the build time to make OBS well support the fast release process. This is why I tried the 'rebuild' and 'block' options, but it is said in the document set both to the non-default value will cause some build and dependency problem. If both option are not 'security', then I can not use it to optimize the build speed.
It is said in the document which describes the scheduling strategies that if we don't use the default value of "rebuild" and "block" option for scheduling, there will be some build or dependency problem occurred.
block option is mostly only an optimization of build power => do not build something if you know you will need to rebuild it soon (at least in default (transitive) rebuild strategy).
I can not imagine what kind of the scenario will cause the problem.
I will be appreciate if someone can describe the reason or can provide some material for that.
It is a little bit documented here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.build_s cheduling_and_dispatching.htm
Thanks Jidiablo
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi, Some comments inline, but I will say the automatic rebuilds triggered in OBS are definitely a feature and not a defect :) On Thu, 26 Feb 2015 16:26:38 +0800 Ji Xiang <jixiang@syberos.com> wrote:
On Thursday, February 26, 2015 08:26:09 AM Adrian Schröter wrote:
On Donnerstag, 26. Februar 2015, 14:59:21 wrote Ji Xiang:
Hello everyone, I have been confused for long time in terms of the rebuild (or I can say 'scheduling') mechanism of OBS. The most confused point is why a full rebuild is needed for software release on OBS.
Well if it is "needed" can always be discussed. But it is needed _IF_ you want to be sure that you are able to rebuild it after releasing it (eg. the newer gcc may just crash or header files suddenly provide conflicting definitions). Also some changes could apply which are not expressed in the dependencies of packages.
Many thanks for your kind of answer, do you mind give me a example what kind of the change which are not included in the dependency? The major reason I asked this question is we are using OBS, however we found that it takes quite long time to do a full build(some packages are built many times due to the dependency), and if there are some changes happened at low layer(like glibc , qt etc), all the packages will be built, we have to wait for long time. It can not met the requirement of fast release process.
If glibc or Qt are rebuilt for some reason, you *defintely* want packages which depend on them rebuilt. This the magic of OBS which guarantees all packages publish in a coherent state. If you do not you may have symbol mismatches or location errors among other nastiness. Changes might include a patch which changes something with loading a library or other functional issue which is not simply expressed a by a version update.
So I am trying to understand how the scheduling mechanism of OBS working, and how to optimize the build time to make OBS well support the fast release process.
You very well need more build power in your workers, faster machines, more memory, fast disks or all of the above. You did not state how much time is "slow" nor how you are running the OBS systems. I strongly advise everyone running a private OBS be running on bare metal. If you gave a description of your hardware layout and what machines are running which components, we can gladly give some advise to optimize the hardware. I've implemented a few private OBS instances and you would be suprised how much stress OBS puts on hardware.
This is why I tried the 'rebuild' and 'block' options, but it is said in the document set both to the non-default value will cause some build and dependency problem. If both option are not 'security', then I can not use it to optimize the build speed.
It is said in the document which describes the scheduling strategies that if we don't use the default value of "rebuild" and "block" option for scheduling, there will be some build or dependency problem occurred.
block option is mostly only an optimization of build power => do not build something if you know you will need to rebuild it soon (at least in default (transitive) rebuild strategy).
I can not imagine what kind of the scenario will cause the problem.
I will be appreciate if someone can describe the reason or can provide some material for that.
It is a little bit documented here:
http://openbuildservice.org/help/manuals/obs-reference-guide/cha.obs.build_s cheduling_and_dispatching.htm
Thanks Jidiablo
Hope that helps, Peter -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Donnerstag, 26. Februar 2015, 15:05:27 wrote plinnell:
Hi,
Some comments inline, but I will say the automatic rebuilds triggered in OBS are definitely a feature and not a defect :)
sure ;)
On Thu, 26 Feb 2015 16:26:38 +0800 Ji Xiang <jixiang@syberos.com> wrote:
On Thursday, February 26, 2015 08:26:09 AM Adrian Schröter wrote:
On Donnerstag, 26. Februar 2015, 14:59:21 wrote Ji Xiang:
Hello everyone, I have been confused for long time in terms of the rebuild (or I can say 'scheduling') mechanism of OBS. The most confused point is why a full rebuild is needed for software release on OBS.
Well if it is "needed" can always be discussed. But it is needed _IF_ you want to be sure that you are able to rebuild it after releasing it (eg. the newer gcc may just crash or header files suddenly provide conflicting definitions). Also some changes could apply which are not expressed in the dependencies of packages.
Many thanks for your kind of answer, do you mind give me a example what kind of the change which are not included in the dependency? The major reason I asked this question is we are using OBS, however we found that it takes quite long time to do a full build(some packages are built many times due to the dependency), and if there are some changes happened at low layer(like glibc , qt etc), all the packages will be built, we have to wait for long time. It can not met the requirement of fast release process.
If glibc or Qt are rebuilt for some reason, you *defintely* want packages which depend on them rebuilt. This the magic of OBS which guarantees all packages publish in a coherent state. If you do not you may have symbol mismatches or location errors among other nastiness.
It depends heavily on your expectations here. You can argue of course that glibc and Qt must not become backward incompatible. Even worse you may hide this incompability which is not acceptable for a stable product. Therefore we do NOT rebuild everything for stable products (aka maintenance updates). But we definitive want a rebuild for all upstream work. If there is a breakage we want to know as early as possible. Not month later and for sure not after the product got released. Listen to talks from other distros who have the problem that a significant number of packages which were part of a release do not actually build anymore. And that is really the worst case scenaria when you need to do way more changes in a released product then needed for a simple fix. It increases the risk of side effects a lot and requires way more QA testing then. -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Friday, February 27, 2015 07:17:24 AM Adrian Schröter wrote:
On Donnerstag, 26. Februar 2015, 15:05:27 wrote plinnell:
Hi,
Some comments inline, but I will say the automatic rebuilds triggered in OBS are definitely a feature and not a defect :) Peter, Many thanks for your help. Your conclusion for rebuild is really clean, it assumed me the dependency rebuild is needed:)
For the performance related issue, I think we can turn to another perspective to evaluate it. OBS is handy and easy to use for open source distribution, however for a commercial product, it may still be slower. Commercial product requires low-end package to be submitted frequently, that will cause OBS to fully build all the packages every day. And some package will not just built for once due to the dependency(for example: A required B and C , then it will be rebuild twice after B and C are built successfully). Compare to android build system coupled with distributed compiling, it just take half hour to finish the full build and image creation. Is that possible that OBS can reach such fast delivery speed? Our product is a mobile system, that implies the automation test will be more difficult, we have to import more effort for manual test. So to deliver our test team a new image time and again in one day have became a major fact for improving the productivity for whole team. Do you know if there are some successful case for that? =================================================== Below is the hardware layout for my obs server: OBS version: 2.3 8 local obs-worker are assigned this server, no remote worker are using. Memory: 16GB Hard Disk: 7TB CPU: Intel E5-2650 ====================================================
sure ;)
On Thu, 26 Feb 2015 16:26:38 +0800
Ji Xiang <jixiang@syberos.com> wrote:
On Thursday, February 26, 2015 08:26:09 AM Adrian Schröter wrote:
On Donnerstag, 26. Februar 2015, 14:59:21 wrote Ji Xiang:
Hello everyone, I have been confused for long time in terms of the rebuild (or I can say 'scheduling') mechanism of OBS. The most confused point is why a full rebuild is needed for software release on OBS.
Well if it is "needed" can always be discussed. But it is needed _IF_ you want to be sure that you are able to rebuild it after releasing it (eg. the newer gcc may just crash or header files suddenly provide conflicting definitions). Also some changes could apply which are not expressed in the dependencies of packages.
Many thanks for your kind of answer, do you mind give me a example what kind of the change which are not included in the dependency? The major reason I asked this question is we are using OBS, however we found that it takes quite long time to do a full build(some packages are built many times due to the dependency), and if there are some changes happened at low layer(like glibc , qt etc), all the packages will be built, we have to wait for long time. It can not met the requirement of fast release process.
If glibc or Qt are rebuilt for some reason, you *defintely* want packages which depend on them rebuilt. This the magic of OBS which guarantees all packages publish in a coherent state. If you do not you may have symbol mismatches or location errors among other nastiness.
It depends heavily on your expectations here. You can argue of course that glibc and Qt must not become backward incompatible. Even worse you may hide this incompability which is not acceptable for a stable product.
Therefore we do NOT rebuild everything for stable products (aka maintenance updates).
But we definitive want a rebuild for all upstream work. If there is a breakage we want to know as early as possible. Not month later and for sure not after the product got released.
Listen to talks from other distros who have the problem that a significant number of packages which were part of a release do not actually build anymore. And that is really the worst case scenaria when you need to do way more changes in a released product then needed for a simple fix. It increases the risk of side effects a lot and requires way more QA testing then. Adrian, many thanks for your suggestion. Due to it is a commercial product. I think our exception is make our daily release as rapid as possible, so I am thinking over to disable the dependency rebuild for daily release. Small risk from it is acceptable. But I am not sure if this kind of 'risk' is small enough to be accept.
I use the single OBS project hierarchy which contain 7 project to hold about 700 packages, more than 300 package are set to as project base, those will not to be rebuild anymore. According to you suggestion, seems it is not a good way for commercial release:) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Freitag, 27. Februar 2015, 15:57:11 wrote Ji Xiang:
On Friday, February 27, 2015 07:17:24 AM Adrian Schröter wrote:
On Donnerstag, 26. Februar 2015, 15:05:27 wrote plinnell:
Hi,
Some comments inline, but I will say the automatic rebuilds triggered in OBS are definitely a feature and not a defect :) Peter, Many thanks for your help. Your conclusion for rebuild is really clean, it assumed me the dependency rebuild is needed:)
For the performance related issue, I think we can turn to another perspective to evaluate it. OBS is handy and easy to use for open source distribution, however for a commercial product, it may still be slower.
OBS or the transitive build mode? I mean, you are aware that you can switch just to build the packages with changed sources, right? (if not, see my link to the book chapter :) ...
Adrian, many thanks for your suggestion. Due to it is a commercial product. I think our exception is make our daily release as rapid as possible, so I am thinking over to disable the dependency rebuild for daily release. Small risk from it is acceptable. But I am not sure if this kind of 'risk' is small enough to be accept.
I use the single OBS project hierarchy which contain 7 project to hold about 700 packages, more than 300 package are set to as project base, those will not to be rebuild anymore. According to you suggestion, seems it is not a good way for commercial release:)
It depends of course a lot what you want to do with it after release. If no maintenance updates are planed, there is no problem. If you need to do anyway a full testing of everything neither. But the reallity for us is that changes must be kept as small as possible to avoid sideeffects. But this is maybe just true for SUSE. Another option is to switch between the modes, for example keep rebuild="local" for the daily snapshots, but have some milestones in between with clean transitive builds. In any case, I would check for build dependency cycles. If you have such dependency cycles it increases the build time a lot. Some packagee splitting might help a alot here to avoid them. Or some Ignore: statements in prjconf. -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Friday, February 27, 2015 09:14:47 AM Adrian Schröter wrote:
On Freitag, 27. Februar 2015, 15:57:11 wrote Ji Xiang:
On Friday, February 27, 2015 07:17:24 AM Adrian Schröter wrote:
On Donnerstag, 26. Februar 2015, 15:05:27 wrote plinnell:
Hi,
Some comments inline, but I will say the automatic rebuilds triggered in OBS are definitely a feature and not a defect :)
Peter, Many thanks for your help. Your conclusion for rebuild is really clean, it assumed me the dependency rebuild is needed:)
For the performance related issue, I think we can turn to another perspective to evaluate it. OBS is handy and easy to use for open source distribution, however for a commercial product, it may still be slower.
OBS or the transitive build mode? I mean, you are aware that you can switch just to build the packages with changed sources, right? (if not, see my link to the book chapter :)
I understand what you asked, I think what I mean is transitive mode:) We are using transitive to do daily release, the low speed has affect our productivity a lot.
...
Adrian, many thanks for your suggestion. Due to it is a commercial product. I think our exception is make our daily release as rapid as possible, so I am thinking over to disable the dependency rebuild for daily release. Small risk from it is acceptable. But I am not sure if this kind of 'risk' is small enough to be accept.
I use the single OBS project hierarchy which contain 7 project to hold about 700 packages, more than 300 package are set to as project base, those will not to be rebuild anymore. According to you suggestion, seems it is not a good way for commercial release:)
It depends of course a lot what you want to do with it after release. If no maintenance updates are planed, there is no problem. If you need to do anyway a full testing of everything neither. But the reallity for us is that changes must be kept as small as possible to avoid sideeffects. But this is maybe just true for SUSE.
Another option is to switch between the modes, for example keep rebuild="local" for the daily snapshots, but have some milestones in between with clean transitive builds.
In any case, I would check for build dependency cycles. If you have such dependency cycles it increases the build time a lot. Some packagee splitting might help a alot here to avoid them. Or some Ignore: statements in prjconf.
Thanks for your tips :) I will try to enable the option: rebuild="local" and to see if the side effect is acceptable:) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Adrian Schröter -
Ji Xiang -
plinnell