On 11/28/2013 08:49 AM, Stephan Kulow wrote:
It's Thanksgiving today and we had a great harvest - "Bottle" rocks.
But today is also a good time to plant new seeds for even better
openSUSE releases. Factory has a healthy growth, but we have to make
sure it's growing in the right direction and so the openSUSE team
at SUSE had an on-and-off discussion basically since 12.3 on how
to improve things.
But first let me give you some background, that you might not be aware
of: 10.3 had 3334 packages, 11.1 3746, 3605 for 11.2, 3807 for 11.3,
4784 in 12.1, 5710 in 12.2, 6246 in 12.3, 6678 in 13.1, 6800 right now
in Factory. If you need a picture, look at http://s.kulow.org/packages
Integrating these to make a good distribution is real work. And one of
my favourite songs (in that context) goes:
No one said it would be easy
But no one said it'd be this hard
No one said it would be easy
No one thought we'd come this far
In that song Sheryl Crow sings "It's just a question of eliminating
obstacles", so what did we in the openSUSE Team do to help? We focused
on getting a grip on testing by improving openqa
), but we soon found out that it was not
good enough to test Factory ISOs. Factory is broken often enough not to
produce ISOs at all, ISOs can't be installed and once these problems are
sorted out, we found in openqa very basic things to be broken, but it
was too late to protect factory users to run into them.
One thing I tried was to setup "rings" to help easing the very painful
staging projects (with 6800 packages, every staging project as we use
them is a monster). That experiment has shown rings to be worthy way
to check, but they won't work as I thought with the OBS as it is. We
need to think bigger. So we tried to come up with an idea on how to
improve the factory development process that includes a more clever way
to utilize staging projects and openQA.
As this development process is a bit hard to explain in email, Alberto
and Ancor prepared an interactive diagram:
We basically want to put the pressure on the submitting packager not the
user. Using factory should be safe, for this we want to revive a thing
that has been lost on the way: Bernhard's factory-tested project.
I have two primary concerns, one with the "put pressure on the
submitter" and two with the "staging approach".
1.) Putting pressure on the submitter
In principal this is probably a good idea, however in practice this is
very difficult to follow through. When submission A breaks package B and
the submitter A is no responsible for fixing the problem in B than the
- must be notified that B broke (not even easy in staging architecture)
- the submitter must have some knowledge of B, the code inside of B
or acquire that knowledge
Putting pressure on the submitter is a good concept to avoid "dump and
run scenarios", i.e. put you code in and everyone has to fix the fall
out. However, stated as such the submitter can very easily feel
overwhelmed and left alone, thus the submission may never take place.
What I think we need is a process/environment that holds the submitter
sufficiently responsible to avoid "dump and run" while at the same time
providing enough support such that the submitter does not feel left
alone and overwhelmed.
In a staging model I have no idea how to get there.
2.) The staging approach
I can only speak from experience and thus this might sound a little
lame, sorry. I have seen two implementations of the staging model in
action at companies that produce large software suites. In both cases I
consider the approach as a failure.
The problem in both cases is that the number of staging
trees/branches/projects has an ever increasing slope, thus consuming
ever more manpower to manage the ever increasing number of staging
projects. While the original problem of "how do we deal with unknown
adverse interactions between updates" remains unresolved. The "solution"
to this problem taken in one case was to have intermediate staging trees
where "known risky updates" were tested together. Yes, staging trees
upon staging trees. But this only solves the problem superficially as
the target tree will move ahead and thus the staging tree by definition
is always out of date. Unless the target tree is frozen until a
particular staging tree is merged. Anyway it is a maze that requires
potentially a lot of people.
The other problem with the staging model is that the "potentially risky
interactions" knowledge is an implicit set of interactions that the
staging tree managers happen to know. This is not expressed anywhere and
thus makes it difficult for other people to learn. We have this problem
today and from my point of view this will not be resolved with more
The staging model will not catch adverse interactions reliably. The
reason is that by definition the staging tree is always out of date,
unless the target tree is frozen and after one staging tree is accepted
all other staging trees get rebuilt. This is not conducive to parallel
What do I have to offer other than concerns?
There is the component model that I had proposed a while back. The
component model may merge well with the idea of rings, that's something
that could be explored.
Anyway, I do not necessarily need to revive that discussion, we can of
course if people are interested. My main point was to express my
concerns about the from my point of view to primary changes in the model
- more pressure on submitters
- more staging trees
Robert Schweikert MAY THE SOURCE BE WITH YOU
SUSE-IBM Software Integration Center LINUX
Public Cloud Architect
To unsubscribe, e-mail: opensuse-factory+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-factory+owner(a)opensuse.org