[openFATE 315250] Make it easier to move around kiwi files in OBS

Feature changed by: Adrian Schröter (adrianSuSE) Feature #315250, revision 26 Title: Make it easier to move around kiwi files in OBS Buildservice: Implementation Milestone: 2.5 Priority Requester: Mandatory Projectmanager: Mandatory Requested by: Adrian Schröter (adriansuse) Partner organization: openSUSE.org Description: We have often problems when moving around kiwi files in OBS, since they often need a changed repository list then. OBS will support a directive to use the repositories from the project repository definition instead. This can be triggered via defining an "obsrepositories:/" repository in the kiwi file. OBS (and osc) will patch then the kiwi file when a build starts to use the expanded repository list from the project definition. It will work therefore with all kiwi version, but we should add some proper error message to kiwi, if someone tries to run kiwi manually on such a file. Discussion: #1: Adrian Schröter (adriansuse) (2013-06-28 10:18:06) It is half implemented and testable at build-test.opensuse.org. Missing is: * product image support * osc support * correct kiwi error message #2: Robert Schweikert (rjschwei) (2013-06-28 08:48:07) As the feature is partially implemented I guess it is best to move forward with this. However, I think we should re-think this and the existing setup. At present we have: opensuse and obs acronyms comprehended by both KIWI and the build service. With this we get another acronym obsrepositories This acronym however is only comprehended by the build service, if I understand correctly. The build service will change obsrepositories into obs:// with the appropriate path before handing the config file to kiwi. I have the following concerns about this: 1.) We mangle the user data 2.) It is really difficult to explain the difference between obs:/ and obsrepositories:/ 3.) Building images in the build service is getting ever more complicated and drifting father away from a kiwi build on the command line #3: Adrian Schröter (adriansuse) (2013-06-28 14:57:14) (reply to #2) we can still drop the entire feature (it is just in the master branch for now), if there is a better solution. I just do not see one :) Yes, we mangle the user data, but only when the user decided that we should do so by using the obsrepositories:/ directive. Please note that the file in the source repo will be untouched, the mangling does only happen on the build host. The general problem here is that the repository list is in the kiwi file usually while OBS users expect it in the project meta. That is what I want to solve with the directive. #4: Robert Schweikert (rjschwei) (2013-06-28 10:21:31) (reply to #3) If the actual goal is to have the image build fit better into the concepts prevalent in OBS then it would be easier to tell kiwi to ignore the configured repositories and add new repositories that are taken from the project config on the command line. Thus when OBS invokes kiwi the command line switch --ignore-repos would be use as well as --add-repo, --add-repoalias, --add-repopriority, and --add-repotype. This has the following advantages: 1.) It is very easy to explain and document "During an image build in OBS all repositories configured in the kiwi configuration file will be ignored. Repositories for the image build are added on the kiwi command line and are taken from the project configuration." 2.) It gets OBS out of the business of having to comprehend the repository configuration of the kiwi configuration file 3.) A user can check in a configuration file that will build inside and outside of OBS The only thing that's left to solve in this case is how one build images for multiple distributions in the same project. But this can probably be deducted from some attribute settings. #5: Marcus Schaefer (sax2) (2013-07-01 09:42:11) (reply to #4) I agree, the buildservice has a complete different and unique view on the repository setup. It's better to ignore the information in the . kiwi file in this case instead of changing the user data behind the scene we would also be able to get rid of the obs:// type which confused many people in the past #6: Adrian Schröter (adriansuse) (2013-07-01 09:50:50) (reply to #4) We can do that, but that means we need updated kiwi versions for all distros. Also, we need somewhere some setting if the old or the new way should be used. Otherwise we would break all old (and current) builds. What I fear also is that users don't understand then why their repositories in the kiwi files are ignored. But maybe this can we solved by some large and loud print lines #7: Adrian Schröter (adriansuse) (2013-07-01 09:56:34) (reply to #4) Another option is that this mode becomes action, when a obsrepositories: / directive is added to the .kiwi file: * OBS replaces it with the project definitions and adds further optional obs:/ defined repos, but ignores all other definitions. * Plain kiwi ignores all obsrepositories:/ and obs:/ repositories, but uses the rest of them. In that way a user still can build one .kiwi file for OBS and non-OBS usage, but we do not break former stuff and it is still kind of transparent (from my POV). #8: Adrian Schröter (adriansuse) (2013-07-01 10:03:40) (reply to #7) JFYI, currently OBS refuses to build a .kiwi file with non obs:/ repositories because it looks to be incomplete. That check would be disabled if an obsrepositories:/ directive is inside. #9: Adrian Schröter (adriansuse) (2013-07-02 13:38:45) JFYI, this is currently deployed on build.opensuse.org. While we still can modify or remove this functionality (since it is just in master branch so far), I like to blog about it to let the users try it. However, this makes no sense if you disagree on this approach and we need to remove it again. Any opinion? #10: Marcus Schaefer (sax2) (2013-07-02 16:05:55) (reply to #9) The ignore-repos / add-repo options exists for all kiwi versions the buildservice use. I don't think we have to change/update kiwi here just the buildservice needs to change the way it calls kiwi. I'm more in favour to discuss and decide this in a meeting also with Robert. If I would have to decide this on my own I wouldn't add any new url type like obsrepositories:// or obs:// In order to explain a change in that area to the user I think we only can document it properly or print a warning at build time that repos in the .kiwi file will not have any effect It's clear to me that such a change would break current image builds also my own containment builds would break but I would accept that and move the repo defintition from the .kiwi file into the project configuration I can't tell how many other users would be affected by this but somehow we should cleanup the code and I think adding yet another complexity just to stay compatible will not make the situation any better. Thus I vote for a clear and maybe hard cut but again it should be discussed if you announce this as new feature we might receive more resistance if it is changed after a short time again. Thus I wouldn't talk about it until we had the chance to discuss it just my 2cents #11: Adrian Schröter (adriansuse) (2013-07-02 16:54:22) (reply to #10) well, breaking former builds, including the ones in released products is an absolute no-go. If the kiwi file can not contain the switch (obsrepositories:/) we need to put it elsewhere. No idea where right now. #12: Adrian Schröter (adriansuse) (2013-07-03 14:08:30) (reply to #10) I have now changed the code to use the kiwi command line parameters instead of patching the kiwi file. However, the trigger is still the obsrepositories:/ path. #13: Robert Schweikert (rjschwei) (2013-07-07 00:49:38) (reply to #12) I understand that we should not break current builds. This implies that we need a smooth transtition. How about this? If OBS finds any "http://" or "opensuse://" repositories in the .kiwi file, than OBS ignores the repos in the .kiwi file, i.e. it uses the -- ignore-repositories kiwi command line argument. If the "obs://" prefix is found than OBS operates as it does today. Thus, we still do not need a new switch. If in addition we can create a version check in OBS such that the use of "obs://" for Factory, i.e. 13.1, in a .kiwi file triggers an error, we can slowly transition to the new implementation. Unfortunately we still have to carry the old stuff until SLE 10 and SLE 11 are dead. #14: Adrian Schröter (adriansuse) (2013-07-08 08:29:26) (reply to #13) The handling of the URLs is possible, the only disadvantage would be that you could not combine a fixed defined repository given via obs:// with the ones defined in the OBS repository. A feature which I personally think it makes sense, but that would not be a regression on the other hand. The second thing regarding the version check is not easy to do (it would be a recursion, because the kiwi version defines then the repositories to use, but the used repositories also defining the kiwi version...) #15: Robert Schweikert (rjschwei) (2013-07-08 14:54:02) (reply to #14) Well, IMHO, we should not try to find a "combined" solution, i.e. consider the repositories defined in OBS and the repositories in the kiwi config file. This gets confusing very fast, not just to the user but also to us on the development side and thus leads to brittleness. It was my understanding that the idea was to use the repositories as defined in OBS and that those would make it "easy" to move/copy projects within OBS. Allowing a mix will than break the "easy" feature again. From my point of view we should just use the --ignore-repos command line argument in OBS. Within the OBS code you are of course free to parse the repos and add any repo that is prefixed with "obs://" to the "--add-repo" command line option. Thus, you get the "mixed" implementation you are after. However, I think this is complicated and confusing. This feels more like having a feature for the feature's sake. I would prefer that we have the rule as suggested in my previous comment: If OBS finds a "obs://" repo, operate as it does today, if OBS finds "http://" or "opensuse://" than use the --ignore-repos command line switch. This is easy to explain, easy to understand and will allow us to eventually remove the "obs://" directive from the KIWI schema, mybe in version 5.7 or 5.8. Never mind my comment on the version. I fogot that projects that build images are version dependent by the repository definition, not by the settings in OBS. #16: Adrian Schröter (adriansuse) (2013-07-09 08:17:17) (reply to #15) While I still think the combined feature is good for some setups to test updates (like new kiwi versions) based on the same file, can I ask you first why are you so keen to get rid of the obs:// URL's? I mean, kiwi files with obs:// URLs would still be usable with plain kiwi when specifing the download root repository via some command line switch (and the default could be download.opensuse.org/repositories). So, one could see it as just yet-another repository format supported by kiwi. #17: Robert Schweikert (rjschwei) (2013-07-09 06:30:40) (reply to #16) Using "obs://" in a kiwi file is confusing because it does not actually resolve to OBS, rather it resolves to "this://" meaning it is a local repository. If "obs://" could be made to resolve to "http://download.opensuse.org/repositories/" such that one can just provide a project name i.e. "obs://Cloud:EC2/SLE_11_SP3" than I'd be fine with "obs://". However making this change in KIWI would also break current builds, which is not an option. Therefore, I'd rather get rid of it in KIWI in the long run. #18: Adrian Schröter (adriansuse) (2013-07-09 13:57:17) (reply to #17) kiwi could check if it exists locally and otherwise it would use the remote URL. That was at least my original idea behind that. In that way it would be IMHO transparent (even more than just silently ignoring all repos in file) and not breaking anything #22: Robert Schweikert (rjschwei) (2013-07-10 09:52:57) (reply to #21) If OBS does not consider the repositories in the KIWI configuration file than this is a choice we make, that will be documented and thus establishes a convention. Therefore, we would not "silently" ignore the repositories in the configuration file, but rather we would ignore those by documented choice. I would like to see OBS getting out of the business of comprehending KIWI configuration files. This is consistent with the goal of this request, i.e. use the repositories defined for the project to build the image. We then have a nice easy to understand and use setup: Build in OBS uses the repositories defined in the project, the repositories in the configuration file are ignored. Build outside of OBS uses the repos to in the configuration file. The advantage of this is that I can have a configuration file that will build inside and outside of OBS. #23: Adrian Schröter (adriansuse) (2013-07-12 14:43:48) (reply to #22) while working on the kiwi project test setups it showed that dropping support for obs:/ urls from kiwi would * require to create an own repository per image build, which would also break release number syncing and make the handling way way more complicate and error prone. * make the test setups to test and compare the different kiwi (and other tool chain packages) versions basically impossible. So from my POV dropping the obs:// URL support is absolut no option without an idea how to solve these critical features. #24: Robert Schweikert (rjschwei) (2013-07-12 09:31:08) (reply to #23) OK, I think we managed to mangle two related but distinct topics. Lets focus on the original request, "Make it easy..." and forget about the kiwi handling of "obs://" For implementation of this request in OBS I still believe the following to be true: * OBS should get out of the business of comprehending kiwi configuration files, the only thing that is needed is a quick grep for http(s), iso, opensuse in the "source" element ** When one of the entries above is found OBS builds with the --ignore- repositories command line argument * * If non of the above is found OBS builds as it does today While fishing through the kiwi configuration is still more than I would like OBS to do, it is better than it is today in that OBS does not have to understand the complete URL of the source element. Additionally this is needed to avoid breaking existing builds. In the long run we should be able to phase this out and create warnings in OBS that complain when a kiwi file does not contain http etc. repositories. + #25: Adrian Schröter (adriansuse) (2013-07-12 15:43:55) (reply to #24) + Okay, most of that what you suggested is done today already, and I + could also use --ignore-repositories switch when iso/http/.. URLs are + available (instead of just delivering an error as today). + Regarding "comprehending kiwi files": we only look for the type of + images, the package lists and the repository pathes, but everything + else gets ignored. + However, I still need to have trigger when it is okay to re-use the + repositories from the OBS project repository configuration to allow + people to write .kiwi files which use different repositories in their + test and stable projects. This is what I am currently doing via the + obsrepositories:/ trigger. -- openSUSE Feature: https://features.opensuse.org/315250