[opensuse-arm] Change to the rebuild behavior of openSUSE:Factory:ARM
Hi, We had a couple of issues related to getting a new build of Factory into openqa recently. This was partially caused by ARM not able to catch up with the check in frequency of x86. To some extend this was bad luck (the critial long running build job hitting the slowest / misconfigured worker taking forever to finish) and to some extend we just can't keep up I guess. Over the last few days Alex and I have reconfigured the workers to be less overcommitted and redirecting more resource intensive jobs to the better performing variants. hopefully this helps with the overall build speed and gets us to catch up. In addition, there is a rebuild counter sync endless-loop issue that isn't fully understood yet, but apparently either aarch64 or armv7l were playing ping pong rebuilds with each other. we've stabilized that by disabling armv7l build temporarily (it is reenabled now). Last but not least as an experiment we've changed the source rebuild behavior. previously, Factory:ARM was directly using the sources from Factory, so whenever there was a source change there it immediately propagated to ARM. that contributed to rarely be able to finish a ftp tree build (because that one only starts when nothing else is building, and it takes a couple of hours to complete). without a ftp tree build, new DVDs are not entering openqa. So as an experiment we've switched to frozen links. this means we're linking sources from a particular level (0201 right now) of factory, and won't get automatic updates. That seems to work fine. unfortunately the "updating link" operation crashes the build service with an error 500, so we can't update anymore, and need an admin to fix that for us. The command to update the source state to build against is this: osc api -X POST '/source/openSUSE:Factory:ARM?cmd=freezelink' It needs to be run by someone who has maintainer rights in that project. currently it crashes though. Unfortunately I'll be out for a few days (vacation), so I hope alex or Andreas can figure out together with the OBS team on how to get that issue fixed. The plan is to add that into the totest manager, so that we're only taking source changes once when we just have handed off a build to openqa. this way we might not try to do every snapshot that x86 publishes, but if we were able to finish one, we take the next one from aarch64 (potentially skipping one or two interim ones on x86). this part isn't implemented yet though, I'll look at that end of next week. TIA, Dirk -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Hi, Thank you for the notification. Few questions below. Le 03/02/2018 à 09:42, Dirk Müller a écrit :
Hi,
We had a couple of issues related to getting a new build of Factory into openqa recently. This was partially caused by ARM not able to catch up with the check in frequency of x86. To some extend this was bad luck (the critial long running build job hitting the slowest / misconfigured worker taking forever to finish) and to some extend we just can't keep up I guess.
Over the last few days Alex and I have reconfigured the workers to be less overcommitted and redirecting more resource intensive jobs to the better performing variants. hopefully this helps with the overall build speed and gets us to catch up.
In addition, there is a rebuild counter sync endless-loop issue that isn't fully understood yet, but apparently either aarch64 or armv7l were playing ping pong rebuilds with each other. we've stabilized that by disabling armv7l build temporarily (it is reenabled now).
All packages are unresolvable in Factory:ARM armv7 and armv6 is still disabled. Is it intended?
Last but not least as an experiment we've changed the source rebuild behavior. previously, Factory:ARM was directly using the sources from Factory, so whenever there was a source change there it immediately propagated to ARM. that contributed to rarely be able to finish a ftp tree build (because that one only starts when nothing else is building, and it takes a couple of hours to complete). without a ftp tree build, new DVDs are not entering openqa.
So as an experiment we've switched to frozen links. this means we're linking sources from a particular level (0201 right now) of factory, and won't get automatic updates. That seems to work fine. unfortunately the "updating link" operation crashes the build service with an error 500, so we can't update anymore, and need an admin to fix that for us.
Not sure what is the 0201. Is it a kind of version number? How could we check the diff between Factory and Factory:ARM?
The command to update the source state to build against is this:
osc api -X POST '/source/openSUSE:Factory:ARM?cmd=freezelink'
It needs to be run by someone who has maintainer rights in that project. currently it crashes though.
Unfortunately I'll be out for a few days (vacation), so I hope alex or Andreas can figure out together with the OBS team on how to get that issue fixed.
The plan is to add that into the totest manager, so that we're only taking source changes once when we just have handed off a build to openqa. this way we might not try to do every snapshot that x86 publishes, but if we were able to finish one, we take the next one from aarch64 (potentially skipping one or two interim ones on x86). this part isn't implemented yet though, I'll look at that end of next week.
Sounds good! :) Guillaume -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Op zaterdag 3 februari 2018 14:04:25 CET schreef Guillaume Gardet:
Le 03/02/2018 à 09:42, Dirk Müller a écrit :
So as an experiment we've switched to frozen links. this means we're linking sources from a particular level (0201 right now) of factory, and won't get automatic updates. That seems to work fine. unfortunately the "updating link" operation crashes the build service with an error 500, so we can't update anymore, and need an admin to fix that for us.
Not sure what is the 0201. Is it a kind of version number? How could we check the diff between Factory and Factory:ARM?
looks like Februari 1st, which is the latest Tumbleweed version. Still the repository for aarch64 is not up-to-date, it still contains versions of packages not later than January 5th. -- fr.gr. member openSUSE Freek de Kruijf -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
On Samstag, 3. Februar 2018, 09:42:13 CET wrote Dirk Müller: ...
The command to update the source state to build against is this:
osc api -X POST '/source/openSUSE:Factory:ARM?cmd=freezelink'
It needs to be run by someone who has maintainer rights in that project. currently it crashes though.
it seems we have a regression which makes this call dog-slow ... (it parses the existing freeze link file again and again). It seems still to work though, you experience most likely just a timeout of the api. The backend should finish correct later on. So your link should be fine, but we have to speed this up again of course. Btw, just updating the link in :ARM because I did not found an exception mail... -- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
03.02.2018 11:42, Dirk Müller пишет:
Hi,
We had a couple of issues related to getting a new build of Factory into openqa recently. This was partially caused by ARM not able to catch up with the check in frequency of x86. To some extend this was bad luck (the critial long running build job hitting the slowest / misconfigured worker taking forever to finish) and to some extend we just can't keep up I guess.
Over the last few days Alex and I have reconfigured the workers to be less overcommitted and redirecting more resource intensive jobs to the better performing variants. hopefully this helps with the overall build speed and gets us to catch up.
In addition, there is a rebuild counter sync endless-loop issue that isn't fully understood yet, but apparently either aarch64 or armv7l were playing ping pong rebuilds with each other. we've stabilized that by disabling armv7l build temporarily (it is reenabled now).
Last but not least as an experiment we've changed the source rebuild behavior. previously, Factory:ARM was directly using the sources from Factory, so whenever there was a source change there it immediately propagated to ARM. that contributed to rarely be able to finish a ftp tree build (because that one only starts when nothing else is building, and it takes a couple of hours to complete). without a ftp tree build, new DVDs are not entering openqa.
So as an experiment we've switched to frozen links. this means we're linking sources from a particular level (0201 right now) of factory, and won't get automatic updates. That seems to work fine. unfortunately the "updating link" operation crashes the build service with an error 500, so we can't update anymore, and need an admin to fix that for us.
The command to update the source state to build against is this:
osc api -X POST '/source/openSUSE:Factory:ARM?cmd=freezelink'
It needs to be run by someone who has maintainer rights in that project. currently it crashes though.
How often will it be triggered?
Unfortunately I'll be out for a few days (vacation), so I hope alex or Andreas can figure out together with the OBS team on how to get that issue fixed.
The plan is to add that into the totest manager, so that we're only taking source changes once when we just have handed off a build to openqa. this way we might not try to do every snapshot that x86 publishes, but if we were able to finish one, we take the next one from aarch64 (potentially skipping one or two interim ones on x86). this part isn't implemented yet though, I'll look at that end of next week.
TIA, Dirk
-- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Hi! Are there such severe problems with armv6hl that it's still remaining on openSUSE-release-20180124-1.2.armv6hl (including conflicts with the ruby version) instead of having the same state as armv7hl openSUSE-release-20180214-1.1.armv7hl (without ruby conflicts) ? Best regards, Ralph Am 09.02.2018 um 21:58 schrieb Matwey V. Kornilov:
03.02.2018 11:42, Dirk Müller пишет:
Hi,
We had a couple of issues related to getting a new build of Factory into openqa recently. This was partially caused by ARM not able to catch up with the check in frequency of x86. To some extend this was bad luck (the critial long running build job hitting the slowest / misconfigured worker taking forever to finish) and to some extend we just can't keep up I guess.
Over the last few days Alex and I have reconfigured the workers to be less overcommitted and redirecting more resource intensive jobs to the better performing variants. hopefully this helps with the overall build speed and gets us to catch up.
In addition, there is a rebuild counter sync endless-loop issue that isn't fully understood yet, but apparently either aarch64 or armv7l were playing ping pong rebuilds with each other. we've stabilized that by disabling armv7l build temporarily (it is reenabled now).
Last but not least as an experiment we've changed the source rebuild behavior. previously, Factory:ARM was directly using the sources from Factory, so whenever there was a source change there it immediately propagated to ARM. that contributed to rarely be able to finish a ftp tree build (because that one only starts when nothing else is building, and it takes a couple of hours to complete). without a ftp tree build, new DVDs are not entering openqa.
So as an experiment we've switched to frozen links. this means we're linking sources from a particular level (0201 right now) of factory, and won't get automatic updates. That seems to work fine. unfortunately the "updating link" operation crashes the build service with an error 500, so we can't update anymore, and need an admin to fix that for us.
The command to update the source state to build against is this:
osc api -X POST '/source/openSUSE:Factory:ARM?cmd=freezelink'
It needs to be run by someone who has maintainer rights in that project. currently it crashes though. How often will it be triggered?
Unfortunately I'll be out for a few days (vacation), so I hope alex or Andreas can figure out together with the OBS team on how to get that issue fixed.
The plan is to add that into the totest manager, so that we're only taking source changes once when we just have handed off a build to openqa. this way we might not try to do every snapshot that x86 publishes, but if we were able to finish one, we take the next one from aarch64 (potentially skipping one or two interim ones on x86). this part isn't implemented yet though, I'll look at that end of next week.
TIA, Dirk
-- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Hi! Are there such severe problems with armv6hl that it's still remaining on openSUSE-release-20180124-1.2.armv6hl (including conflicts with the ruby version) instead of having the same state as armv7hl openSUSE-release-20180214-1.1.armv7hl (without ruby conflicts) ? Best regards, Ralph Am 09.02.2018 um 21:58 schrieb Matwey V. Kornilov:
03.02.2018 11:42, Dirk Müller пишет:
Hi,
We had a couple of issues related to getting a new build of Factory into openqa recently. This was partially caused by ARM not able to catch up with the check in frequency of x86. To some extend this was bad luck (the critial long running build job hitting the slowest / misconfigured worker taking forever to finish) and to some extend we just can't keep up I guess.
Over the last few days Alex and I have reconfigured the workers to be less overcommitted and redirecting more resource intensive jobs to the better performing variants. hopefully this helps with the overall build speed and gets us to catch up.
In addition, there is a rebuild counter sync endless-loop issue that isn't fully understood yet, but apparently either aarch64 or armv7l were playing ping pong rebuilds with each other. we've stabilized that by disabling armv7l build temporarily (it is reenabled now).
Last but not least as an experiment we've changed the source rebuild behavior. previously, Factory:ARM was directly using the sources from Factory, so whenever there was a source change there it immediately propagated to ARM. that contributed to rarely be able to finish a ftp tree build (because that one only starts when nothing else is building, and it takes a couple of hours to complete). without a ftp tree build, new DVDs are not entering openqa.
So as an experiment we've switched to frozen links. this means we're linking sources from a particular level (0201 right now) of factory, and won't get automatic updates. That seems to work fine. unfortunately the "updating link" operation crashes the build service with an error 500, so we can't update anymore, and need an admin to fix that for us.
The command to update the source state to build against is this:
osc api -X POST '/source/openSUSE:Factory:ARM?cmd=freezelink'
It needs to be run by someone who has maintainer rights in that project. currently it crashes though. How often will it be triggered?
Unfortunately I'll be out for a few days (vacation), so I hope alex or Andreas can figure out together with the OBS team on how to get that issue fixed.
The plan is to add that into the totest manager, so that we're only taking source changes once when we just have handed off a build to openqa. this way we might not try to do every snapshot that x86 publishes, but if we were able to finish one, we take the next one from aarch64 (potentially skipping one or two interim ones on x86). this part isn't implemented yet though, I'll look at that end of next week.
TIA, Dirk
-- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
Hi Ralph, We're currently debugging build consistency issues with armv6, so the builds are disabled for now to not corrupt our tree. Once we solved those, we'll happily enable it again and have working builds that should hopefully build as far as their armv7 counterparts. Thanks, Alex On 15.02.18 09:48, BWC Illmensee GmbH - Ralph Gauer wrote:
Hi!
Are there such severe problems with armv6hl that it's still remaining on openSUSE-release-20180124-1.2.armv6hl (including conflicts with the ruby version) instead of having the same state as armv7hl openSUSE-release-20180214-1.1.armv7hl (without ruby conflicts) ?
Best regards, Ralph
Am 09.02.2018 um 21:58 schrieb Matwey V. Kornilov:
03.02.2018 11:42, Dirk Müller пишет:
Hi,
We had a couple of issues related to getting a new build of Factory into openqa recently. This was partially caused by ARM not able to catch up with the check in frequency of x86. To some extend this was bad luck (the critial long running build job hitting the slowest / misconfigured worker taking forever to finish) and to some extend we just can't keep up I guess.
Over the last few days Alex and I have reconfigured the workers to be less overcommitted and redirecting more resource intensive jobs to the better performing variants. hopefully this helps with the overall build speed and gets us to catch up.
In addition, there is a rebuild counter sync endless-loop issue that isn't fully understood yet, but apparently either aarch64 or armv7l were playing ping pong rebuilds with each other. we've stabilized that by disabling armv7l build temporarily (it is reenabled now).
Last but not least as an experiment we've changed the source rebuild behavior. previously, Factory:ARM was directly using the sources from Factory, so whenever there was a source change there it immediately propagated to ARM. that contributed to rarely be able to finish a ftp tree build (because that one only starts when nothing else is building, and it takes a couple of hours to complete). without a ftp tree build, new DVDs are not entering openqa.
So as an experiment we've switched to frozen links. this means we're linking sources from a particular level (0201 right now) of factory, and won't get automatic updates. That seems to work fine. unfortunately the "updating link" operation crashes the build service with an error 500, so we can't update anymore, and need an admin to fix that for us.
The command to update the source state to build against is this:
osc api -X POST '/source/openSUSE:Factory:ARM?cmd=freezelink'
It needs to be run by someone who has maintainer rights in that project. currently it crashes though. How often will it be triggered?
Unfortunately I'll be out for a few days (vacation), so I hope alex or Andreas can figure out together with the OBS team on how to get that issue fixed.
The plan is to add that into the totest manager, so that we're only taking source changes once when we just have handed off a build to openqa. this way we might not try to do every snapshot that x86 publishes, but if we were able to finish one, we take the next one from aarch64 (potentially skipping one or two interim ones on x86). this part isn't implemented yet though, I'll look at that end of next week.
TIA, Dirk
-- To unsubscribe, e-mail: opensuse-arm+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-arm+owner@opensuse.org
participants (8)
-
Adrian Schröter -
Alexander Graf -
BWC Illmensee GmbH - Ralph Gauer -
Dirk Müller -
Freek de Kruijf -
Guillaume Gardet -
Matwey V. Kornilov -
ralph＠gauernet.de