All jobs in https://build.opensuse.org/project/monitor?project=home%3Abeekhof were in the "finished' state for nearly an hour. On Nov 28, 2008, at 1:21 PM, Adrian Schröter wrote:

On Freitag 28 November 2008 13:01:34 Andrew Beekhof wrote:

...

Looks like the job that publishes completed builds isn't functioning. Can someone kick it please?

What exactly is the problem ?

The publisher service is running fine and I am not aware of a bugreport describing your issue (because it just very vague descriped above).

bye adrian

--

Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de

-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Nov 28, 2008, at 1:43 PM, Adrian Schröter wrote:

On Freitag 28 November 2008 13:27:43 Andrew Beekhof wrote:

...

All jobs in https://build.opensuse.org/project/monitor?project=home%3Abeekhof were in the "finished' state for nearly an hour.

The x86_64 are okay, because the scheduler was reloading to take the new code we need. (this takes about two hours with the size of our service). We do this for this reason per architecture, so only one is hanging for a while.

The i586 scheduler has processed it meanwhile, I saw that it had more than 2000 medium events some hours ago (means that it has to recalculate 2000 project repositories due to changed packages). If you can provide a patch which makes this faster, I am happy to apply. (small sidenote, a compiler which compiles the entire distribution instantly would be also appricated :).

You're missing the point. This is not about scheduler reloads taking too long. They're necessary, we get that... they take a while, we get that too... just tell us when they (and other maintenance events) are happening. Preferably in advance so we can plan around an expected outage, but even telling us (via email or a status page update) after the event helps a great deal because at least we know that what we're seeing is expected and we should go do something else for the next few hours. The current information vacuum is causing a lot of frustration. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Nov 28, 2008, at 3:19 PM, Adrian Schröter wrote:

On Freitag 28 November 2008 14:32:55 Andrew Beekhof wrote:

...

On Nov 28, 2008, at 1:43 PM, Adrian Schröter wrote:

...

On Freitag 28 November 2008 13:27:43 Andrew Beekhof wrote:

...

All jobs in https://build.opensuse.org/project/monitor?project=home%3Abeekhof were in the "finished' state for nearly an hour.

The x86_64 are okay, because the scheduler was reloading to take the new code we need. (this takes about two hours with the size of our service). We do this for this reason per architecture, so only one is hanging for a while.

The i586 scheduler has processed it meanwhile, I saw that it had more than 2000 medium events some hours ago (means that it has to recalculate 2000 project repositories due to changed packages). If you can provide a patch which makes this faster, I am happy to apply. (small sidenote, a compiler which compiles the entire distribution instantly would be also appricated :).

You're missing the point. This is not about scheduler reloads taking too long.

They're necessary, we get that... they take a while, we get that too... just tell us when they (and other maintenance events) are happening.

You can see this always on the monitor page.

No. You can't. The last i586 scheduler restart was 4 days ago and the dispatcher/ publisher was also over a day. So nothing on that page would indicate why i586 packages were not moving from the finished to the success state around lunch time today.

...

Preferably in advance so we can plan around an expected outage, but even telling us (via email or a status page update) after the event helps a great deal because at least we know that what we're seeing is expected and we should go do something else for the next few hours.

The current information vacuum is causing a lot of frustration.

You may should consider to use "osc build" more often.

As a matter of fact, I use osc build quite a lot. But when it comes to installing clusters or doing releases, it is no more helpful than no build service at all.

We have often enough multiple thousands packages to be build, waiting for them will quite often not work (and this is independend of any kind of maintenance).

The status page is the first thing I look at when something seems to be taking a while (when I checked today the queue lengths were single digits) and if I see a high load the immediate reaction is: "Oh, its just really busy. I'll do something else then". Which again illustrates my point - if we know what's going on, we'll be more understanding. Is it really so much trouble to add a one-liner to the status messages? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Le vendredi 28 novembre 2008, à 15:19 +0100, Adrian Schröter a écrit :

You may should consider to use "osc build" more often. We have often enough multiple thousands packages to be build, waiting for them will quite often not work (and this is independend of any kind of maintenance).

FWIW, I usually build stuff locally, but then, when I need to submit a package to openSUSE:Factory, I prefer to wait that it also builds fine on the build service (on all architectures) to make sure everything is okay. (I have an osc plugin that monitors the build so I don't have to wait myself -- I let osc wait for me ;-)) Anyway, the wait time is generally okay -- it's only really long when a rebuild just started for factory (at least for me). Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Le lundi 01 décembre 2008, à 09:18 +0100, Adrian Schröter a écrit :

Hm, this sounds like you poll the build service.

Yep. One request every minute.

Do you know that you can configure notifications at http://hermes.opensuse.org, so you do not need to wait/poll ? You just get a mail for example, if it fails (or succeeds, depending on your settings).

I know this -- but it's not really possible to make hermes notifications work with an osc plugin that can run on any computer. At least, I don't know how to achieve that. Vincent -- Les gens heureux ne sont pas pressés. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Freitag 28 November 2008 14:51:50 Lars Marowsky-Bree wrote:

On 2008-11-28T13:43:20, Adrian Schröter wrote:

...

The i586 scheduler has processed it meanwhile, I saw that it had more than 2000 medium events some hours ago (means that it has to recalculate 2000 project repositories due to changed packages). If you can provide a patch which makes this faster, I am happy to apply. (small sidenote, a compiler which compiles the entire distribution instantly would be also appricated :).

It is below 100 meanwhile, that means that should currently faster catch up.

There's also a weird issue when jobs have just finished (possibly failed).

The logs shown then - for a couple of minutes - will refer to the N-1 build. (They essentially revert from the "live log" to the stored log, I guess, and the copy is not immediate, while the status got updated already.)

This can be very confusing while the sync is catching up, and noone can tell me that copying a logfile takes a long time ;-)

This is a bug, please create a report. (The file does not need even to be copied on the server, it just get renamed).

Maybe the scheduler could be partitioned, so that it doesn't have to process everything in one go? Maybe the sync could be partitioned too, so that smaller increments are synced but the status is always correct?

The data export is partitioned in a seperate part (called publisher). The status can not be always complete, except we block accepting the build result until it has been recalculated. Just to clarify, I am happy about any constructive suggesstion how to speed up the process. But we have by intention a batch job system and not a life interactive system in this regard to be handle the load at all. Why does it take so long ? When ever a build job finishes, the build result can provide and require anything. So it needs recalculate all related packages. This are all packages inside of the project repo and dependending on the build repo (while there is a prio on the packages inside of the repo). It can not start any other build in this project before this has been done. Recalculating of one openSUSE:Factory repo does take about 90 seconds. (small project repos are much faster and usually in the 0-2 seconds area). In short, the scheduler has always a complete view of all packages in all projects including all dependencies at any time. It updates the view on every change (source or binarie). These dependencies take currently about 750MB in memory. -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Freitag 28 November 2008 17:06:29 John Kelly wrote:

On Fri, 28 Nov 2008 15:17:12 +0100, Adrian Schröter

wrote:

...

Just to clarify, I am happy about any constructive suggesstion how to speed up the process. But we have by intention a batch job system and not a life interactive system in this regard to be handle the load at all.

Your most fundamental problem is having 3000+ opensuse distro packages all in one big repo. SUSE needs a minimal build system which can host and bootstrap itself.

Actually, it is way more. Since we have at least one source link to each package, also this one is affected and needs to recalculated. In addition to this also repos depending on other repos. So, the scheduler is not really interessted in which project a package lives. It always calculates the dependency graph for the entire universe (all package in all projects in all repos for all architectures). Now you can say that this is a silly setup. But it is also the power of the service that you can always see if it does still work with the current version. This gives a lot functionality which helps us to test and develop stuff and to improve our quality.

Base:build is a start, but without meta packages like rpmlint-mini and friends, it can't bootstrap itself within the OBS. And those packages add many dependencies on .*-tk and xorg-.*

It does not really matter if the packages are in one project or multiple. The scheduler knows all of them and needs to calculate the dependencies also across the projects. (due to source links or depending repositories). To make it short and let me leave for weekend: Yes, we have ideas how to make it faster. But no, it is not obvious simple. If you are interested in more details we could someday write a longer paper or you could look inside of the scheduler code and could make suggestions how to improve it. It is open source ! ;) -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

John Kelly wrote:

On Fri, 28 Nov 2008 17:18:20 +0100, Adrian Schröter wrote:

...

On Freitag 28 November 2008 17:06:29 John Kelly wrote:

...

On Fri, 28 Nov 2008 15:17:12 +0100, Adrian Schröter

wrote:

...

Just to clarify, I am happy about any constructive suggesstion how to speed up the process. But we have by intention a batch job system and not a life interactive system in this regard to be handle the load at all.

Your most fundamental problem is having 3000+ opensuse distro packages all in one big repo. SUSE needs a minimal build system which can host and bootstrap itself.

Actually, it is way more. Since we have at least one source link to each package, also this one is affected and needs to recalculated.

In addition to this also repos depending on other repos.

So, the scheduler is not really interessted in which project a package lives. It always calculates the dependency graph for the entire universe (all package in all projects in all repos for all architectures).

Now you can say that this is a silly setup. But it is also the power of the service that you can always see if it does still work with the current version. This gives a lot functionality which helps us to test and develop stuff and to improve our quality.

...

Base:build is a start, but without meta packages like rpmlint-mini and friends, it can't bootstrap itself within the OBS. And those packages add many dependencies on .*-tk and xorg-.*

It does not really matter if the packages are in one project or multiple. The scheduler knows all of them and needs to calculate the dependencies also across the projects. (due to source links or depending repositories).

Schedule the universe if you want to. But SUSE still needs a minimal self hosting repo that other people can use. Many of us don't want to see the whole universe!

If "self hosting" and "minimal" is the issue here, than you always have the option to install a local buildservice on your machines. OBS is even opensource, so feel free to improve the "slow algorithms" and the "buggy code", that results in a "borked" service. We are heavily looking for developers and testers in this area. Otherwise, if it is a service that is commonly used, it has to serve many needs. So individual needs may not be 100% satisfied. Martin -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

John Kelly wrote:

On Fri, 28 Nov 2008 17:42:04 +0100, Martin Mohring wrote:

...

If "self hosting" and "minimal" is the issue here, than you always have the option to install a local buildservice on your machines. OBS is even opensource, so feel free to improve the "slow algorithms" and the "buggy code", that results in a "borked" service. We are heavily looking for developers and testers in this area.

To work for free, while you earn a salary? I will be happy to hear a job offer which includes a competitive salary.

But that makes it neccesary that you tell us what is your qualification, and what you would like to be :-)

...

Otherwise, if it is a service that is commonly used, it has to serve many needs. So individual needs may not be 100% satisfied.

You miss the point. A minimal self hosting repo would be useful to a wide audience, not just a few.

It seems to me, OBS developers are defensive of their own turf, and do not see the big picture.

Please explain me, what turf do the OBS developers defend.

The scheduling "power" Adrian mentioned is academic; the reality is: OBS is TOO SLOW.

This is a relative thing. TOO SLOW towards what measure.

Better code is not the only solution. You need a better practical approach. Split the big opensuse project into smaller sub projects. That may not improve scheduling speed, but think beyond the code.

Also here: according to what technical requirements and what measure?

If people won't use opensuse/OBS, because the repo is 3000+ big, and unwieldy, there's no point of working hard on code. Try not to hide from yourselves, the fundamental needs which are obvious to everyone else.

Thats all relative: I also want a car that needs only 1l / 100km, can drive lightspeed and costs nothing. But nobody yet fulfilled my practical need. Nevertheless, cars that do not fulfil this seem to be of practical relevance. Martin -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org