good article and interesting idea. I think I also commented original
mail about rings. I am not sure if it is right direction how it should
go, because there is still too much human interaction. I think that in
optimal case all work should do machines and humans only solve
exceptions or special cases.
Of course it depends if we have enough build power to keep all work to
My idea is much simplier:
1) for every package submission to factory create own COW copy of
factory where try to build package and all of its dependencies ( so
quick for leaf package and slow for core package )
2f) if this fail reject submission, then create from copy staging
project and fix all problems related to failure
2o) if everything is fine, then create ISO and test it in openQA and
follow to 3
3) all automatic tests passes, all requiring manual check is done, then
repeat automatic test before real submit to minimalize race conditions
this way factory should always pass openQA, so have at least basic
For problem with related commits I propose simple solution - express
dependency in requires and have bot, that check if factory copy can
satisfy dependencies and if not, search all submit request if there is
new enough version and merge submits together. This way you have proper
dependencies and need no manual human intervention.
I think it is important to handle so many package in factory to have
way how to automatize as much as possible. ( I think that some of my
ideas requires writing a code or modify BS code, but for future it is
more important then introduce more human interaction ).
Example is e.g. in Yast which have almost 1M lines of source codes and
without automatization we cannot keep it up to date and find problems
quick enough. So we work quite hard to making things working without
our interaction and only if problem is found we make action.
On Thu, 28 Nov 2013 14:49:32 +0100
Stephan Kulow <coolo(a)suse.de> wrote:
It's Thanksgiving today and we had a great harvest - "Bottle" rocks.
But today is also a good time to plant new seeds for even better
openSUSE releases. Factory has a healthy growth, but we have to make
sure it's growing in the right direction and so the openSUSE team
at SUSE had an on-and-off discussion basically since 12.3 on how
to improve things.
But first let me give you some background, that you might not be aware
of: 10.3 had 3334 packages, 11.1 3746, 3605 for 11.2, 3807 for 11.3,
4784 in 12.1, 5710 in 12.2, 6246 in 12.3, 6678 in 13.1, 6800 right now
in Factory. If you need a picture, look at http://s.kulow.org/packages
Integrating these to make a good distribution is real work. And one of
my favourite songs (in that context) goes:
No one said it would be easy
But no one said it'd be this hard
No one said it would be easy
No one thought we'd come this far
In that song Sheryl Crow sings "It's just a question of eliminating
obstacles", so what did we in the openSUSE Team do to help? We focused
on getting a grip on testing by improving openqa
), but we soon found out that it was
not good enough to test Factory ISOs. Factory is broken often enough
not to produce ISOs at all, ISOs can't be installed and once these
problems are sorted out, we found in openqa very basic things to be
broken, but it was too late to protect factory users to run into them.
One thing I tried was to setup "rings" to help easing the very painful
staging projects (with 6800 packages, every staging project as we use
them is a monster). That experiment has shown rings to be worthy way
to check, but they won't work as I thought with the OBS as it is. We
need to think bigger. So we tried to come up with an idea on how to
improve the factory development process that includes a more clever
way to utilize staging projects and openQA.
As this development process is a bit hard to explain in email, Alberto
and Ancor prepared an interactive diagram:
We basically want to put the pressure on the submitting packager not
the user. Using factory should be safe, for this we want to revive a
thing that has been lost on the way: Bernhard's factory-tested
And we want to open another submission path: from staging projects.
Consider a situation where recent updates to GNOME and automake
'clash', causing problems when they're installed together. Right now
we throw both into Factory, breaking both for our Factory users until
we solve the issues. Instead, we think working on these issues in a
separate 'staging project' could be the solution. None of us knows how
*exactly* it will look alike because we need to get a conversation
with the OBS team going. But the basic idea is:
- GNOME:Factory stays the devel project of things that relate to
- devel:tools stays the devel project of things that relate to
- on automake updates, we open up a new devel project that stages
GNOME packages for the new automake. In there, GNOME devs and
automake experts work together to fix them and updates in there
are submitted either back to GNOME:Factory and are integrated right
into Factory. Or if that's not possible, the automake update is
"grouped" with various GNOME package updates and these updates end
in Factory together.
There are several problems with the current "everything through devel
project" approach we need to solve. Our ideas are just ideas, but I
had several discussions in various places and nobody offered a better
idea. So we really would like to start with it and I would like to
hear your concerns so they can be part of the final solution.
We have more ideas, but we can only achieve that if we get help, so
let me finish with another favourite of mine
What would you think if I sang out of tune?
Would you stand up and walk out on me?
Lend me your ears and I'll sing you a song
And I'll try not to sing out of key
Oh, I get by with a little help from my friends
To unsubscribe, e-mail: opensuse-factory+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-factory+owner(a)opensuse.org