Mailinglist Archive: opensuse-features (41 mails)

< Previous Next >
[openFATE 315250] Make it easier to move around kiwi files in OBS
Feature changed by: Adrian Schröter (adrianSuSE)
Feature #315250, revision 27
Title: Make it easier to move around kiwi files in OBS

- Buildservice: Implementation
+ Buildservice: Done
Milestone: 2.5
Priority
Requester: Mandatory
Projectmanager: Mandatory

Requested by: Adrian Schröter (adriansuse)
Partner organization: openSUSE.org

Description:
We have often problems when moving around kiwi files in OBS, since they
often need a changed repository list then.
OBS will support a directive to use the repositories from the project
repository definition instead. This can be triggered via defining an
"obsrepositories:/" repository in the kiwi file. OBS (and osc) will
patch then the kiwi file when a build starts to use the expanded
repository list from the project definition.
It will work therefore with all kiwi version, but we should add some
proper error message to kiwi, if someone tries to run kiwi manually on
such a file.

Discussion:
#1: Adrian Schröter (adriansuse) (2013-06-28 10:18:06)
It is half implemented and testable at build-test.opensuse.org.
Missing is:
* product image support
* osc support
* correct kiwi error message


#2: Robert Schweikert (rjschwei) (2013-06-28 08:48:07)
As the feature is partially implemented I guess it is best to move
forward with this. However, I think we should re-think this and the
existing setup.
At present we have:
opensuse and obs
acronyms comprehended by both KIWI and the build service. With this we
get another acronym
obsrepositories
This acronym however is only comprehended by the build service, if I
understand correctly. The build service will change obsrepositories
into obs:// with the appropriate path before handing the config file to
kiwi.
I have the following concerns about this:
1.) We mangle the user data 2.) It is really difficult to explain the
difference between obs:/ and obsrepositories:/ 3.) Building images in
the build service is getting ever more complicated and drifting father
away from a kiwi build on the command line

#3: Adrian Schröter (adriansuse) (2013-06-28 14:57:14) (reply to #2)
we can still drop the entire feature (it is just in the master branch
for now), if there is a better solution. I just do not see one :)
Yes, we mangle the user data, but only when the user decided that we
should do so by using the obsrepositories:/ directive. Please note that
the file in the source repo will be untouched, the mangling does only
happen on the build host.
The general problem here is that the repository list is in the kiwi
file usually while OBS users expect it in the project meta. That is
what I want to solve with the directive.

#4: Robert Schweikert (rjschwei) (2013-06-28 10:21:31) (reply to #3)
If the actual goal is to have the image build fit better into the
concepts prevalent in OBS then it would be easier to tell kiwi to
ignore the configured repositories and add new repositories that are
taken from the project config on the command line.
Thus when OBS invokes kiwi the command line switch --ignore-repos would
be use as well as --add-repo, --add-repoalias, --add-repopriority, and
--add-repotype.
This has the following advantages:
1.) It is very easy to explain and document "During an image build in
OBS all repositories configured in the kiwi configuration file will be
ignored. Repositories for the image build are added on the kiwi command
line and are taken from the project configuration."
2.) It gets OBS out of the business of having to comprehend the
repository configuration of the kiwi configuration file
3.) A user can check in a configuration file that will build inside and
outside of OBS
The only thing that's left to solve in this case is how one build
images for multiple distributions in the same project. But this can
probably be deducted from some attribute settings.

#5: Marcus Schaefer (sax2) (2013-07-01 09:42:11) (reply to #4)
I agree, the buildservice has a complete different and unique view on
the repository setup. It's better to ignore the information in the .
kiwi file in this case instead of changing the user data behind the
scene
we would also be able to get rid of the obs:// type which confused many
people in the past

#6: Adrian Schröter (adriansuse) (2013-07-01 09:50:50) (reply to #4)
We can do that, but that means we need updated kiwi versions for all
distros. Also, we need somewhere some setting if the old or the new way
should be used. Otherwise we would break all old (and current) builds.
What I fear also is that users don't understand then why their
repositories in the kiwi files are ignored. But maybe this can we
solved by some large and loud print lines

#7: Adrian Schröter (adriansuse) (2013-07-01 09:56:34) (reply to #4)
Another option is that this mode becomes action, when a obsrepositories:
/ directive is added to the .kiwi file:
* OBS replaces it with the project definitions and adds further
optional obs:/ defined repos, but ignores all other definitions.
* Plain kiwi ignores all obsrepositories:/ and obs:/ repositories, but
uses the rest of them.
In that way a user still can build one .kiwi file for OBS and non-OBS
usage, but we do not break former stuff and it is still kind of
transparent (from my POV).

#8: Adrian Schröter (adriansuse) (2013-07-01 10:03:40) (reply to #7)
JFYI, currently OBS refuses to build a .kiwi file with non obs:/
repositories because it looks to be incomplete. That check would be
disabled if an obsrepositories:/ directive is inside.

+ #26: Adrian Schröter (adriansuse) (2014-02-13 10:49:26) (reply to #4)
+ build script is now using the switches you mentioned for appliance
+ builds (not for product, but that is entire different story)

#9: Adrian Schröter (adriansuse) (2013-07-02 13:38:45)
JFYI, this is currently deployed on build.opensuse.org. While we still
can modify or remove this functionality (since it is just in master
branch so far), I like to blog about it to let the users try it.
However, this makes no sense if you disagree on this approach and we
need to remove it again. Any opinion?

#10: Marcus Schaefer (sax2) (2013-07-02 16:05:55) (reply to #9)
The ignore-repos / add-repo options exists for all kiwi versions the
buildservice use. I don't think we have to change/update kiwi here just
the buildservice needs to change the way it calls kiwi. I'm more in
favour to discuss and decide this in a meeting also with Robert. If I
would have to decide this on my own I wouldn't add any new url type
like obsrepositories:// or obs:// In order to explain a change in that
area to the user I think we only can document it properly or print a
warning at build time that repos in the .kiwi file will not have any
effect It's clear to me that such a change would break current image
builds also my own containment builds would break but I would accept
that and move the repo defintition from the .kiwi file into the project
configuration I can't tell how many other users would be affected by
this but somehow we should cleanup the code and I think adding yet
another complexity just to stay compatible will not make the situation
any better. Thus I vote for a clear and maybe hard cut but again it
should be discussed
if you announce this as new feature we might receive more resistance if
it is changed after a short time again. Thus I wouldn't talk about it
until we had the chance to discuss it
just my 2cents

#11: Adrian Schröter (adriansuse) (2013-07-02 16:54:22) (reply to #10)
well, breaking former builds, including the ones in released products
is an absolute no-go. If the kiwi file can not contain the switch
(obsrepositories:/) we need to put it elsewhere. No idea where right
now.

#12: Adrian Schröter (adriansuse) (2013-07-03 14:08:30) (reply to #10)
I have now changed the code to use the kiwi command line parameters
instead of patching the kiwi file. However, the trigger is still the
obsrepositories:/ path.

#13: Robert Schweikert (rjschwei) (2013-07-07 00:49:38) (reply to #12)
I understand that we should not break current builds. This implies that
we need a smooth transtition.
How about this?
If OBS finds any "http://"; or "opensuse://" repositories in the .kiwi
file, than OBS ignores the repos in the .kiwi file, i.e. it uses the --
ignore-repositories kiwi command line argument. If the "obs://" prefix
is found than OBS operates as it does today. Thus, we still do not need
a new switch.
If in addition we can create a version check in OBS such that the use
of "obs://" for Factory, i.e. 13.1, in a .kiwi file triggers an error,
we can slowly transition to the new implementation. Unfortunately we
still have to carry the old stuff until SLE 10 and SLE 11 are dead.

#14: Adrian Schröter (adriansuse) (2013-07-08 08:29:26) (reply to #13)
The handling of the URLs is possible, the only disadvantage would be
that you could not combine a fixed defined repository given via obs://
with the ones defined in the OBS repository. A feature which I
personally think it makes sense, but that would not be a regression on
the other hand.
The second thing regarding the version check is not easy to do (it
would be a recursion, because the kiwi version defines then the
repositories to use, but the used repositories also defining the kiwi
version...)

#15: Robert Schweikert (rjschwei) (2013-07-08 14:54:02) (reply to #14)
Well, IMHO, we should not try to find a "combined" solution, i.e.
consider the repositories defined in OBS and the repositories in the
kiwi config file. This gets confusing very fast, not just to the user
but also to us on the development side and thus leads to brittleness.
It was my understanding that the idea was to use the repositories as
defined in OBS and that those would make it "easy" to move/copy
projects within OBS. Allowing a mix will than break the "easy" feature
again.
From my point of view we should just use the --ignore-repos command
line argument in OBS.
Within the OBS code you are of course free to parse the repos and add
any repo that is prefixed with "obs://" to the "--add-repo" command
line option. Thus, you get the "mixed" implementation you are after.
However, I think this is complicated and confusing. This feels more
like having a feature for the feature's sake.
I would prefer that we have the rule as suggested in my previous
comment:
If OBS finds a "obs://" repo, operate as it does today, if OBS finds
"http://"; or "opensuse://" than use the --ignore-repos command line
switch.
This is easy to explain, easy to understand and will allow us to
eventually remove the "obs://" directive from the KIWI schema, mybe in
version 5.7 or 5.8.
Never mind my comment on the version. I fogot that projects that build
images are version dependent by the repository definition, not by the
settings in OBS.

#16: Adrian Schröter (adriansuse) (2013-07-09 08:17:17) (reply to #15)
While I still think the combined feature is good for some setups to
test updates (like new kiwi versions) based on the same file, can I ask
you first why are you so keen to get rid of the obs:// URL's?
I mean, kiwi files with obs:// URLs would still be usable with plain
kiwi when specifing the download root repository via some command line
switch (and the default could be download.opensuse.org/repositories).
So, one could see it as just yet-another repository format supported by
kiwi.

#17: Robert Schweikert (rjschwei) (2013-07-09 06:30:40) (reply to #16)
Using "obs://" in a kiwi file is confusing because it does not actually
resolve to OBS, rather it resolves to "this://" meaning it is a local
repository.
If "obs://" could be made to resolve to
"http://download.opensuse.org/repositories/"; such that one can just
provide a project name i.e. "obs://Cloud:EC2/SLE_11_SP3" than I'd be
fine with "obs://". However making this change in KIWI would also break
current builds, which is not an option. Therefore, I'd rather get rid
of it in KIWI in the long run.

#18: Adrian Schröter (adriansuse) (2013-07-09 13:57:17) (reply to #17)
kiwi could check if it exists locally and otherwise it would use the
remote URL. That was at least my original idea behind that.
In that way it would be IMHO transparent (even more than just silently
ignoring all repos in file) and not breaking anything

#22: Robert Schweikert (rjschwei) (2013-07-10 09:52:57) (reply to #21)
If OBS does not consider the repositories in the KIWI configuration
file than this is a choice we make, that will be documented and thus
establishes a convention. Therefore, we would not "silently" ignore the
repositories in the configuration file, but rather we would ignore
those by documented choice.
I would like to see OBS getting out of the business of comprehending
KIWI configuration files. This is consistent with the goal of this
request, i.e. use the repositories defined for the project to build the
image. We then have a nice easy to understand and use setup:
Build in OBS uses the repositories defined in the project, the
repositories in the configuration file are ignored.
Build outside of OBS uses the repos to in the configuration file.
The advantage of this is that I can have a configuration file that will
build inside and outside of OBS.

#23: Adrian Schröter (adriansuse) (2013-07-12 14:43:48) (reply to #22)
while working on the kiwi project test setups it showed that dropping
support for obs:/ urls from kiwi would
* require to create an own repository per image build, which would also
break release number syncing and make the handling way way more
complicate and error prone.
* make the test setups to test and compare the different kiwi (and
other tool chain packages) versions basically impossible.
So from my POV dropping the obs:// URL support is absolut no option
without an idea how to solve these critical features.

#24: Robert Schweikert (rjschwei) (2013-07-12 09:31:08) (reply to #23)
OK,
I think we managed to mangle two related but distinct topics.
Lets focus on the original request, "Make it easy..." and forget about
the kiwi handling of "obs://"
For implementation of this request in OBS I still believe the following
to be true:
* OBS should get out of the business of comprehending kiwi
configuration files, the only thing that is needed is a quick grep for
http(s), iso, opensuse in the "source" element
** When one of the entries above is found OBS builds with the --ignore-
repositories command line argument
* * If non of the above is found OBS builds as it does today
While fishing through the kiwi configuration is still more than I would
like OBS to do, it is better than it is today in that OBS does not have
to understand the complete URL of the source element. Additionally this
is needed to avoid breaking existing builds.
In the long run we should be able to phase this out and create warnings
in OBS that complain when a kiwi file does not contain http etc.
repositories.

#25: Adrian Schröter (adriansuse) (2013-07-12 15:43:55) (reply to #24)
Okay, most of that what you suggested is done today already, and I
could also use --ignore-repositories switch when iso/http/.. URLs are
available (instead of just delivering an error as today).
Regarding "comprehending kiwi files": we only look for the type of
images, the package lists and the repository pathes, but everything
else gets ignored.
However, I still need to have trigger when it is okay to re-use the
repositories from the OBS project repository configuration to allow
people to write .kiwi files which use different repositories in their
test and stable projects. This is what I am currently doing via the
obsrepositories:/ trigger.




--
openSUSE Feature:
https://features.opensuse.org/315250

< Previous Next >
This Thread
  • No further messages