[opensuse-buildservice] project hasn't starting building for 20 hours and counting...
https://build.opensuse.org/project/monitor?project=home:sipfoundry:4.4 It's gotten progressively slower over last month. Questions 1.) Was there was a change to the queueing algorithm? 2.) Anyone else see slower than normal build scheduling? 3.) Is queue algorithm published somewhere? 4.) I think home projects don't show up in global monitor? If so, is there a way home project can see where they are in the total queue? 5.) Is there anyway someone can bump my build up, there is a nasty bug i'd like to get out to folks. Yes, i know, it's a free service, but just thought I'd ask. Thanks -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thursday 02 December 2010, 23:03:08 Douglas Hubler wrote:
https://build.opensuse.org/project/monitor?project=home:sipfoundry:4. 4
It's gotten progressively slower over last month.
Yes, it is. Horribly declining in the last week.
Questions 1.) Was there was a change to the queueing algorithm? 2.) Anyone else see slower than normal build scheduling?
Me.
3.) Is queue algorithm published somewhere?
It _should_ be all in the sources.
4.) I think home projects don't show up in global monitor? If so, is there a way home project can see where they are in the total queue?
They do, and no, there's no order prediction (since the priorities seem to change all the time, hence it's simply not possible, I guess).
5.) Is there anyway someone can bump my build up, there is a nasty bug i'd like to get out to folks. Yes, i know, it's a free service, but just thought I'd ask.
From my observations, it seems that new checkins are priorized some way, but this this on the other hand leads to projects, where there is no progress for days! (in my projects, there are about 100 packages in scheduled state, and no progress for two days). osc pr home\:frispete\:PyQt | python -c 'import sys;print sum([l.count(" s ") for l in sys.stdin]);' OTOH, monster projects like windows:mingw:win{32,64} seem to be preferred for some reason. At least, they built several hundred packages during the last two days, while _none_ of mine. Douglas, see, it still can get worser. We'll have to be patient and should be happy, if it finally builds something for us, too.. Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag 03 Dezember 2010 schrieb Hans-Peter Jansen:
OTOH, monster projects like windows:mingw:win{32,64} seem to be preferred for some reason. At least, they built several hundred packages during the last two days, while _none_ of mine.
Douglas, see, it still can get worser.
We'll have to be patient and should be happy, if it finally builds something for us, too..
This is a problem we discussed just yesterday - we see it too and want to solve it, but we're unsure how. One thing to do surely is to clean up stuff that noone touched for years. Adrian sent around a list of projects to get rid of repositories, but so far it wasn't yet done - but this will only help a bit. There are two some key facts about the dispatching: - repositories that see a lot of downloads are preferred - projects that create a lot of load get punished - packages that were touched in the last 24 hours are preferred The number of downloads that are the base of the priorities, you can find here: http://www.suse.de/~coolo/repo.list You have to scroll a lot to find home:sipfoundry - which leads to the fact that it doesn't get _any_ priority. It's just one of many home projects and as such it gets build power only when everything else is finished (not exactly that black & white, but towards that). That windows:mingw:win32 gets _so_ much build power shouldn't happen either (its openSUSE_Factory repo #1279 in the repo list, so it's ok if it's build more often than random home projects not downloaded, but it should get a "fair" share, which I guess is less than what it gets right now). One additional thing we discussed was prefering projects that have recent changes assuming that it's more likely that people look at it. home:sipfoundry:4.4 is a good example: The packaged changed have all built, but one scheduled package blocks the others and the final pushing - frustrating the one wanting to test the change. In this case, the scheduled one should get a little higher priority because it's in a project together with recently changed packages. At least higher than all these kernels and libqt4 packages that people linked into their home projects a year ago ;( BTW: the dispatching didn't change and the priorities have been this way for a long time, but we see a lot more packages building now as we're more relaxed in what packages we checkin into openSUSE:Factory, so a lot of repositories building against openSUSE:Factory create a lot of build jobs - and if these repositories are even downloaded, they get preferrence. This might be something we have to decrease. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Le vendredi 03 décembre 2010 à 11:06 +0100, Stephan Kulow a écrit :
Am Freitag 03 Dezember 2010 schrieb Hans-Peter Jansen:
OTOH, monster projects like windows:mingw:win{32,64} seem to be preferred for some reason. At least, they built several hundred packages during the last two days, while _none_ of mine.
Douglas, see, it still can get worser.
We'll have to be patient and should be happy, if it finally builds something for us, too..
This is a problem we discussed just yesterday - we see it too and want to solve it, but we're unsure how.
One thing to do surely is to clean up stuff that noone touched for years. Adrian sent around a list of projects to get rid of repositories, but so far it wasn't yet done - but this will only help a bit.
There are two some key facts about the dispatching: - repositories that see a lot of downloads are preferred - projects that create a lot of load get punished - packages that were touched in the last 24 hours are preferred
The number of downloads that are the base of the priorities, you can find here: http://www.suse.de/~coolo/repo.list
You have to scroll a lot to find home:sipfoundry - which leads to the fact that it doesn't get _any_ priority. It's just one of many home projects and as such it gets build power only when everything else is finished (not exactly that black & white, but towards that).
That windows:mingw:win32 gets _so_ much build power shouldn't happen either (its openSUSE_Factory repo #1279 in the repo list, so it's ok if it's build more often than random home projects not downloaded, but it should get a "fair" share, which I guess is less than what it gets right now).
One additional thing we discussed was prefering projects that have recent changes assuming that it's more likely that people look at it.
Well, it is a problem for me : I've been working on cleaning Sugar project since Tuesday, creating a branch ( home:fcrozat:branches:X11:Sugar ) for all package in X11:Sugar and disabling publishing until I had something working. And now, I'm stucked with packages in scheduled for two days. I've tried to enable publishing but of course, it won't happen until the full project is built :( -- Frederic Crozat <fcrozat@novell.com> Novell -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag 03 Dezember 2010 schrieb Frederic Crozat:
Le vendredi 03 décembre 2010 à 11:06 +0100, Stephan Kulow a écrit :
Am Freitag 03 Dezember 2010 schrieb Hans-Peter Jansen:
OTOH, monster projects like windows:mingw:win{32,64} seem to be preferred for some reason. At least, they built several hundred packages during the last two days, while _none_ of mine.
Douglas, see, it still can get worser.
We'll have to be patient and should be happy, if it finally builds something for us, too..
This is a problem we discussed just yesterday - we see it too and want to solve it, but we're unsure how.
One thing to do surely is to clean up stuff that noone touched for years. Adrian sent around a list of projects to get rid of repositories, but so far it wasn't yet done - but this will only help a bit.
There are two some key facts about the dispatching: - repositories that see a lot of downloads are preferred - projects that create a lot of load get punished - packages that were touched in the last 24 hours are preferred
The number of downloads that are the base of the priorities, you can find here: http://www.suse.de/~coolo/repo.list
You have to scroll a lot to find home:sipfoundry - which leads to the fact that it doesn't get _any_ priority. It's just one of many home projects and as such it gets build power only when everything else is finished (not exactly that black & white, but towards that).
That windows:mingw:win32 gets _so_ much build power shouldn't happen either (its openSUSE_Factory repo #1279 in the repo list, so it's ok if it's build more often than random home projects not downloaded, but it should get a "fair" share, which I guess is less than what it gets right now).
One additional thing we discussed was prefering projects that have recent changes assuming that it's more likely that people look at it.
Well, it is a problem for me : I've been working on cleaning Sugar project since Tuesday, creating a branch ( home:fcrozat:branches:X11:Sugar ) for all package in X11:Sugar and disabling publishing until I had something working. And now, I'm stucked with packages in scheduled for two days. I've tried to enable publishing but of course, it won't happen until the full project is built :(
The main problem here was that home: projects had an additional penalty, so you have to fight against all the other other !home projects for the load. And with that many packages you easily create enough load to get punished - it's hard to differ people "working on cleaning Sugar" and people linking all of factory to get their personal distribution. While the later surely wait for their packages too, we have to protect us from getting overrun by such experiments. The home penalty is gone and I see your project is almost through. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Dear Stephan, first of all thank you very much for taking your precious time to respond to this issue. On Friday 03 December 2010, 11:06:27 Stephan Kulow wrote:
Am Freitag 03 Dezember 2010 schrieb Hans-Peter Jansen:
OTOH, monster projects like windows:mingw:win{32,64} seem to be preferred for some reason. At least, they built several hundred packages during the last two days, while _none_ of mine.
Douglas, see, it still can get worser.
We'll have to be patient and should be happy, if it finally builds something for us, too..
This is a problem we discussed just yesterday - we see it too and want to solve it, but we're unsure how.
One thing to do surely is to clean up stuff that noone touched for years. Adrian sent around a list of projects to get rid of repositories, but so far it wasn't yet done - but this will only help a bit.
There are two some key facts about the dispatching: - repositories that see a lot of downloads are preferred - projects that create a lot of load get punished - packages that were touched in the last 24 hours are preferred
The number of downloads that are the base of the priorities, you can find here: http://www.suse.de/~coolo/repo.list
Hmm, that heuristic sounds reasonable at first. I think, it could be improved by adding a fairness/charity ratio: a fixed amount of build resources, let's say 5%, that that weights by project finished ratio (packages missing / packages done) preferring high values, combined with one that prefers the longest waiting one. The fixed amount of resources is only taken, if there are any aspirants in this group. By turning the knob, the overall fairness can be controlled on the fly.
You have to scroll a lot to find home:sipfoundry - which leads to the fact that it doesn't get _any_ priority. It's just one of many home projects and as such it gets build power only when everything else is finished (not exactly that black & white, but towards that).
That windows:mingw:win32 gets _so_ much build power shouldn't happen either (its openSUSE_Factory repo #1279 in the repo list, so it's ok if it's build more often than random home projects not downloaded, but it should get a "fair" share, which I guess is less than what it gets right now).
One additional thing we discussed was prefering projects that have recent changes assuming that it's more likely that people look at it.
This is a bit dangerous, since it could lead to forced changes that might trigger _more_ rebuilds and to an forcibly increased overall load. It might probably be combineable with the other ideas towards fairness.
home:sipfoundry:4.4 is a good example: The packaged changed have all built, but one scheduled package blocks the others and the final pushing - frustrating the one wanting to test the change. In this case, the scheduled one should get a little higher priority because it's in a project together with recently changed packages. At least higher than all these kernels and libqt4 packages that people linked into their home projects a year ago ;(
BTW: the dispatching didn't change and the priorities have been this way for a long time, but we see a lot more packages building now as we're more relaxed in what packages we checkin into openSUSE:Factory, so a lot of repositories building against openSUSE:Factory create a lot of build jobs - and if these repositories are even downloaded, they get preferrence. This might be something we have to decrease.
At least, the fairness to the poor projects should be raised a bit. Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag 03 Dezember 2010 schrieb Hans-Peter Jansen:
Hmm, that heuristic sounds reasonable at first.
I think, it could be improved by adding a fairness/charity ratio: a fixed amount of build resources, let's say 5%, that that weights by project finished ratio (packages missing / packages done) preferring high values, combined with one that prefers the longest waiting one. The fixed amount of resources is only taken, if there are any aspirants in this group. By turning the knob, the overall fairness can be controlled on the fly.
While this sounds fair, it will make your situation worse. Because it will lead to all projects finishing at the same time - if I understood your proposal correct, because you seem to say that we should prefer the project that has 90 scheduled over those that has 1 scheduled. But it doesn't matter how many are scheduled - in the end you want a published repo - when _all_ are done. So what I consider more fair is that the build power is spent on stuff that people are interested at. This can be either people downloading or developers testing. Right now I notice way too many rebuilds of stuff I expect noone to care for. What the dispatcher should aim for is decreasing the average time of waiting for a package result - if noone waits for it, there is no reason to touch it. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, 2010-12-03 at 13:46 +0100, Stephan Kulow wrote:
What the dispatcher should aim for is decreasing the average time of waiting for a package result - if noone waits for it, there is no reason to touch it.
I hope you mean "there is no need to be in a rush to rebuild it". I have a few packages that I know have minimal usage. Still, they do get used. It would be nice if they eventually get updated. There is no urgency that it be done quickly. But it would be nice if they eventually get rebuilt. -- Roger Oberholtzer OPQ Systems / Ramböll RST Ramböll Sverige AB Krukmakargatan 21 P.O. Box 17009 SE-104 62 Stockholm, Sweden Office: Int +46 10-615 60 20 Mobile: Int +46 70-815 1696 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Am Freitag 03 Dezember 2010 schrieb Roger Oberholtzer:
On Fri, 2010-12-03 at 13:46 +0100, Stephan Kulow wrote:
What the dispatcher should aim for is decreasing the average time of waiting for a package result - if noone waits for it, there is no reason to touch it.
I hope you mean "there is no need to be in a rush to rebuild it". I have a few packages that I know have minimal usage. Still, they do get used. It would be nice if they eventually get updated. There is no urgency that it be done quickly. But it would be nice if they eventually get rebuilt.
Sure, we're only talking about priorities. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, Dec 3, 2010 at 8:55 AM, Stephan Kulow <coolo@suse.de> wrote:
Am Freitag 03 Dezember 2010 schrieb Roger Oberholtzer:
On Fri, 2010-12-03 at 13:46 +0100, Stephan Kulow wrote:
What the dispatcher should aim for is decreasing the average time of waiting for a package result - if noone waits for it, there is no reason to touch it.
I hope you mean "there is no need to be in a rush to rebuild it". I have a few packages that I know have minimal usage. Still, they do get used. It would be nice if they eventually get updated. There is no urgency that it be done quickly. But it would be nice if they eventually get rebuilt.
Sure, we're only talking about priorities.
Greetings, Stephan
On the gui, there is a rebuild now button. Could it or some other explicit mechanism be used for the user to tell the scheduler to bump the priority one notch. Sort of like nice -1. That way packages that don't have a real user asking for them to build get an inherently lower priority. Greg -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Friday 03 December 2010, 14:55:23 Stephan Kulow wrote:
Am Freitag 03 Dezember 2010 schrieb Roger Oberholtzer:
On Fri, 2010-12-03 at 13:46 +0100, Stephan Kulow wrote:
What the dispatcher should aim for is decreasing the average time of waiting for a package result - if noone waits for it, there is no reason to touch it.
I hope you mean "there is no need to be in a rush to rebuild it". I have a few packages that I know have minimal usage. Still, they do get used. It would be nice if they eventually get updated. There is no urgency that it be done quickly. But it would be nice if they eventually get rebuilt.
Sure, we're only talking about priorities.
Stephan, the latest changes make a _tremendous_ difference from a BS usability POV (as a packager of not so popular packages..). Thanks, greatly appreciated. Pete -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, 3 Dec 2010 11:06:27 +0100 Stephan Kulow <coolo@suse.de> wrote:
The number of downloads that are the base of the priorities, you can find here: http://www.suse.de/~coolo/repo.list
Hi Stephan So does that mean from my Python repository there have been 14,210 downloads? 14210 home:malcolmlewis:Python::openSUSE_11.3 @OP, I tend to build/use locally against the various repositories to validate, then push to the OBS to try and reduce my load on the servers. So maybe some suggestions to other users (somehow) so that they try building locally first? -- Cheers Malcolm °¿° (Linux Counter #276890) SUSE Linux Enterprise Desktop 11 (x86_64) Kernel 2.6.32.24-0.2-default up 10 days 16:08, 3 users, load average: 0.03, 0.09, 0.04 GPU GeForce 8600 GTS Silent - Driver Version: 260.19.21 -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hello, There was a discussion a while ago on speeding up some of the slow sections of obs See these references during the discussion http://lists.opensuse.org/opensuse-buildservice/2010-09/msg00074.html http://lists.opensuse.org/opensuse-buildservice/2010-09/msg00062.html Glenn -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (8)
-
doiggl＠velocitynet.com.au -
Douglas Hubler -
Frederic Crozat -
Greg Freemyer -
Hans-Peter Jansen -
Malcolm -
Roger Oberholtzer -
Stephan Kulow