[opensuse-packaging] OBS cleanup -> removal of outdated repositories
Hi, Please excuse the cross post. You may have noticed that we have higher and higher build load on build.o.o. While the new dispatcher algorithms are still good to prefer the "right" jobs, it is still a sign that we may should cleanup. My current plan is to remove all repositories in home projects, which have not been touched this year. Not touched means, no source submission and no meta data has been altered at all. Additionally also all non-home projects, which have not been touch in last two years. This will affect 5668 out of 15668 projects. Why remove repos and not just disable the build ? This will free disk space on our servers and also on all mirrors. As result we can be mirrored more easily. Will any source get lost ? No. How to enable it again ? Just add the wanted repos again. Which projects will get affected exactly ? Find a full list here: http://www.suse.de/~adrian/OBS-remove-list-candidates There is a project which has not been touched, but the repos are still anyway important ! Just drop me a mail .... Why not drop the entire project now that we have an undelete function ? I thought about that, but currently the webui just says that the project does not exist. It does not offer to undelete it, so that might be too agressive for now. Why not drop people a mail and ask them to remove it ? Way to many accounts have no valid email adress and past experience showed that people are often not react when they lost interesst in their project. May plan is to do the removal end of this week, except more discussion about this is needed. Just tell me your opinion, also when you support this ;) thanks adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Tue, Nov 23, 2010 at 11:45 AM, Adrian Schröter
<snip>
May plan is to do the removal end of this week, except more discussion about this is needed.
Just tell me your opinion, also when you support this ;)
thanks adrian
Adrian, I'm surprised you have so many untouched projects. I assume anyone that has added a recent distro repo will not see their packages deleted? Or does that happen at the project level, and thus even maintainers that have been adding new distros to their repo list will see packages deleted? I consider that a bad thing. Also, just a comment that this is a major holiday week in the US and lots of people are offline already. And starting Wed. COB, many will not touch a computer again before Monday AM. I don't know if that impacts the timing of your process or not. Greg -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Dienstag, 23. November 2010, 17:56:39 schrieb Greg Freemyer:
On Tue, Nov 23, 2010 at 11:45 AM, Adrian Schröter
wrote: <snip>
May plan is to do the removal end of this week, except more discussion about this is needed.
Just tell me your opinion, also when you support this ;)
thanks adrian
Adrian,
I'm surprised you have so many untouched projects. I assume anyone that has added a recent distro repo will not see their packages deleted? Or does that happen at the project level, and thus even maintainers that have been adding new distros to their repo list will see packages deleted?
No, who ever touch any package source or touch project meta data (by adding a repo) should not be affected. The list may contain quite some projects, where never a real package have been built. Just people who clicked wildly in the webui ;) bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
El 23/11/10 13:45, Adrian Schröter escribió:
Hi,
Please excuse the cross post. You may have noticed that we have higher and higher build load on build.o.o. While the new dispatcher algorithms are still good to prefer the "right" jobs, it is still a sign that we may should cleanup.
My current plan is to remove all repositories in home projects, which have not been touched this year. Not touched means, no source submission and no meta data has been altered at all. Additionally also all non-home projects, which have not been touch in last two years.
This will affect 5668 out of 15668 projects.
I just sent a osc rdelete loop to remove mines. ;) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 11/23/2010 06:45 PM, Adrian Schröter wrote:
Hi,
Please excuse the cross post. You may have noticed that we have higher and higher build load on build.o.o. While the new dispatcher algorithms are still good to prefer the "right" jobs, it is still a sign that we may should cleanup.
My current plan is to remove all repositories in home projects, which have not been touched this year. Not touched means, no source submission and no meta data has been altered at all. Additionally also all non-home projects, which have not been touch in last two years.
This will affect 5668 out of 15668 projects.
Why remove repos and not just disable the build ? This will free disk space on our servers and also on all mirrors. As result we can be mirrored more easily.
Will any source get lost ? No.
How to enable it again ? Just add the wanted repos again.
Which projects will get affected exactly ? Find a full list here: http://www.suse.de/~adrian/OBS-remove-list-candidates
There is a project which has not been touched, but the repos are still anyway important ! Just drop me a mail ....
Why not drop the entire project now that we have an undelete function ? I thought about that, but currently the webui just says that the project does not exist. It does not offer to undelete it, so that might be too agressive for now.
Why not drop people a mail and ask them to remove it ? Way to many accounts have no valid email adress and past experience showed that people are often not react when they lost interesst in their project.
May plan is to do the removal end of this week, except more discussion about this is needed.
Just tell me your opinion, also when you support this ;)
thanks adrian
If it helps to stop a few of my wip home project packages from staying on scheduled, sometimes for more than a day, I'm +1 Dave P -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On 11/23/10 7:59 PM, Dave Plater wrote:
On 11/23/2010 06:45 PM, Adrian Schröter wrote:
Hi,
Please excuse the cross post. You may have noticed that we have higher and higher build load on build.o.o. While the new dispatcher algorithms are still good to prefer the "right" jobs, it is still a sign that we may should cleanup.
My current plan is to remove all repositories in home projects, which have not been touched this year. Not touched means, no source submission and no meta data has been altered at all. Additionally also all non-home projects, which have not been touch in last two years.
This will affect 5668 out of 15668 projects.
Why remove repos and not just disable the build ? This will free disk space on our servers and also on all mirrors. As result we can be mirrored more easily.
Will any source get lost ? No.
How to enable it again ? Just add the wanted repos again.
Which projects will get affected exactly ? Find a full list here: http://www.suse.de/~adrian/OBS-remove-list-candidates
There is a project which has not been touched, but the repos are still anyway important ! Just drop me a mail ....
Why not drop the entire project now that we have an undelete function ? I thought about that, but currently the webui just says that the project does not exist. It does not offer to undelete it, so that might be too agressive for now.
Why not drop people a mail and ask them to remove it ? Way to many accounts have no valid email adress and past experience showed that people are often not react when they lost interesst in their project.
May plan is to do the removal end of this week, except more discussion about this is needed.
Just tell me your opinion, also when you support this ;)
thanks adrian
If it helps to stop a few of my wip home project packages from staying on scheduled, sometimes for more than a day, I'm +1 Dave P
Hi Adrian, My only question is why it took so long :) I had a look at that list and it is to my eyes a perfectly legitimate move to delete these repos. On the other hand, it shows you are victim of your OBS's popularity... Cheers, Peter -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Tue, 23 Nov 2010 17:45:30 +0100
Adrian Schröter
Which projects will get affected exactly ? Find a full list here: http://www.suse.de/~adrian/OBS-remove-list-candidates
I just committed an unimportant fix (smp_mflags) to home:seife:cross, to trigger a rebuild. With current speed of the buildservice, this will take about two weeks to build (because it's a big project, and I consciously avoid changing it if not strictly necessary, but you basically asked for it so you get ist ;)
May plan is to do the removal end of this week, except more discussion about this is needed.
I expect you to check the candidates again for activity before doing this... :) -- Stefan Seyfried "Any ideas, John?" "Well, surrounding them's out." -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
* Adrian Schröter
May plan is to do the removal end of this week, except more discussion about this is needed.
Does the ability exist to determine usage (download) of a repo? Changes/reguilds may have not been necessary.
Just tell me your opinion, also when you support this ;)
absolutely. -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
El 23/11/10 13:45, Adrian Schröter escribió:
Hi,
Please excuse the cross post. You may have noticed that we have higher and higher build load on build.o.o. While the new dispatcher algorithms are still good to prefer the "right" jobs, it is still a sign that we may should cleanup.
To cleanup your repos by your own try: curl http://www.suse.de/\~adrian/OBS-remove-list-candidates | awk -F " " '{print $1}' | sed -e s@.pkg@@g | grep yourusername | while read repo; do osc rdelete --force "$repo"; done -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Dienstag, 23. November 2010, 21:28:59 schrieb Stefan Seyfried:
On Tue, 23 Nov 2010 17:45:30 +0100 Adrian Schröter
wrote: Which projects will get affected exactly ? Find a full list here: http://www.suse.de/~adrian/OBS-remove-list-candidates
I just committed an unimportant fix (smp_mflags) to home:seife:cross, to trigger a rebuild. With current speed of the buildservice, this will take about two weeks to build (because it's a big project, and I consciously avoid changing it if not strictly necessary, but you basically asked for it so you get ist ;)
The way I collected the data was just by checking the sources, independend from build results. So it does not matter if it builds or not until then. I removed home:seife:cross from my delete list now.
May plan is to do the removal end of this week, except more discussion about this is needed.
I expect you to check the candidates again for activity before doing this... :)
k, but it is in any cases better to drop me a mail in such cases. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hello, Am Dienstag, 23. November 2010 schrieb Adrian Schröter:
You may have noticed that we have higher and higher build load on build.o.o.
Well, at least on the x86_64 build hosts. The ppc64 hosts are bored ;-) Which brings me to a question and suggestion: How to handle noarch packages? As a rough number, on my 11.3 laptop I have 1940 x86_64 and 318 noarch packages - that's about 15% noarch packages. (I'll take this number instead of checking all packages in the distribution in the rest of this mail.) 1) build targets The noarch packges I checked in Factory are build for i586 _and_ x86_64, and later only one of them is really used. The other one just wastes build service power. Wouldn't it be enough to build for one target, say i586? That would save half of 15% = 7.5% of all builds, and is probably easy to implement (scan specfile for noarch, then disable x86_64 builds for them). 2) where to build noarch packages Noarch packages could be build on any idle host, including ppc64, with the same result. This would move some load to the currently idle ppc64 build hosts and remove some burden from the overloaded x86_64 hosts. 15% would mean that about 3 x86_64 build hosts would be freed for other build jobs as long as there are enough idle ppc64 build hosts. I don't know how difficult this would be on the technical side (scheduler etc.), but hey, that's the advantage if you come up with ideas without knowing all the technical details *g* Regards, Christian Boltz -- Du weißt aber schon, was T-ONLINE heißt - oder? T raurig, - O bwohl N ichts L äuft, I st N iemand E rreichbar [Peter Geerds in suse-linux] -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Wednesday 24 November 2010 02:00:40 pm Christian Boltz wrote:
Hello,
Am Dienstag, 23. November 2010 schrieb Adrian Schröter:
You may have noticed that we have higher and higher build load on build.o.o.
Well, at least on the x86_64 build hosts. The ppc64 hosts are bored ;-)
Which brings me to a question and suggestion:
How to handle noarch packages?
As a rough number, on my 11.3 laptop I have 1940 x86_64 and 318 noarch packages - that's about 15% noarch packages. (I'll take this number instead of checking all packages in the distribution in the rest of this mail.)
1) build targets
The noarch packges I checked in Factory are build for i586 _and_ x86_64, and later only one of them is really used. The other one just wastes build service power.
Wouldn't it be enough to build for one target, say i586?
That would save half of 15% = 7.5% of all builds, and is probably easy to implement (scan specfile for noarch, then disable x86_64 builds for them).
2) where to build noarch packages
Noarch packages could be build on any idle host, including ppc64, with the same result.
This would move some load to the currently idle ppc64 build hosts and remove some burden from the overloaded x86_64 hosts.
15% would mean that about 3 x86_64 build hosts would be freed for other build jobs as long as there are enough idle ppc64 build hosts.
I don't know how difficult this would be on the technical side (scheduler etc.), but hey, that's the advantage if you come up with ideas without knowing all the technical details *g*
Sounds like a really great idea! Which I wholeheartedly second, also knowing nothing about the technical details ;) -- Jean Delvare Suse L3 -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Mittwoch, 24. November 2010, 14:00:40 schrieb Christian Boltz:
Hello,
Am Dienstag, 23. November 2010 schrieb Adrian Schröter:
You may have noticed that we have higher and higher build load on build.o.o.
Well, at least on the x86_64 build hosts. The ppc64 hosts are bored ;-)
Which brings me to a question and suggestion:
How to handle noarch packages?
As a rough number, on my 11.3 laptop I have 1940 x86_64 and 318 noarch packages - that's about 15% noarch packages. (I'll take this number instead of checking all packages in the distribution in the rest of this mail.)
1) build targets
The noarch packges I checked in Factory are build for i586 _and_ x86_64, and later only one of them is really used. The other one just wastes build service power.
Wouldn't it be enough to build for one target, say i586?
yes ... but if you need it for building x86_64 packages you need to maintain export filters manually. And you can build dependency loops via cross architecture, which are harder to detect and would increase the build time a lot. So far we always believed it is too much trouble. But you can try it of course. bye adrian -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hello, on Mittwoch, 24. November 2010, Adrian Schröter wrote:
Wouldn't it be enough to build for one target, say i586?
yes ... but if you need it for building x86_64 packages you need to maintain export filters manually. And you can build dependency loops via cross architecture, which are harder to detect and would increase the build time a lot.
Hmm, what about a "first build wins" concept? If a noarch package needs a rebuild, schedule it for every arch where a rebuild is needed (i586, x86_64 and maybe additionally ppc64 if you want to move some load there). So far, nothing new here except the additional build job for ppc64. Let's assume the build on ppc64 is finished first. As soon as the package from ppc64 is finished, - copy the RPM to i586 and x86_64 - remove the build of that package from the i586 and x86_64 scheduler queue and mark it as successfully built. (You could call it a very fast build ;-)) - (in case i586 build has already started, do not copy over the package built on ppc64 to i586.) Yes, I know that sounds hacky, but if I get your description right, it should cause less problems than using a separate noarch scheduler. If a BuildRequire'd package changes on one arch after copying in the package from the ppc64 build, re-add it to the scheduler for rebuild - but this will probably happen already.
So far we always believed it is too much trouble. But you can try it of course.
Adrian, there's a reason why I wrote "that's the advantage if you come up with ideas without knowing all the technical details" in my first mail ;-) As long as we talk about the concept or some not-too-creative perl sniplets, I can most probably follow you. However, I'm afraid my perl knownledge is not good enough for something like the scheduler, and (bigger problem) I don't have enough time to read and understand all the code. Regards, Christian Boltz --
Axel, algerisch gegen neuen Linkschreibung Die neue recht Schreibung hat aber einen nicht unter schaetzbaren vor Teil gegen ueber der alten recht Schreibung: so werden zum bei Spiel viele lange Woerter nicht mehr zu Samen geschrieben. [Axel Woelke und Vlad Berditchevskiy in datk] -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (9)
-
Adrian Schröter
-
Christian Boltz
-
Cristian Rodríguez
-
Dave Plater
-
Greg Freemyer
-
Jean Delvare
-
Patrick Shanahan
-
Peter Linnell
-
Stefan Seyfried