Hi Adrian,

Yes, really yet another obs git integration, but it might be the last one...

might be ;) Thanks for sharing the information. You asked for feedback and I'd like to take the opportunity to share how I use git+obs so far and also have a few questions with regards to the information you provided. The way I connect git+obs with some of my projects is as follows: 1. The Project in obs is setup in a way that it only contains a _service file. obs should only be used to build. 2. The _service file makes use of one ore more source services mostly obs_scm. These source services connects to the git which contains data. * For some projects I wrote my own source service because it was required to process the data in git prior obs can actually consume it. Of course to make these source service effectice, access to the obs backend is needed to install it there, such that 'osc service remoterun' becomes happy. Obviously this limits the scope to my own private instance of obs and to the places were collaboration between R&D and infrastructure happens to drive projects forward. 3. A change in git must trigger the service. I do this with help from github actions and a TOKEN (osc token) which is stored as a secret variable. For my particular projects this works nicely. Also the improved github workflow run for forked projects improved the security aspects of github secret variables. I'm sure there are voices against this concept but as I said for my particular use cases it did the job well. This is how I combine git with obs today. Regarding your information and especially because you said you plan to make incompatible changes my first and foremost question is: * Is the above workflow in danger regarding your upcoming changes ? Second you asked for feedback and I guss you are aiming for use cases. The biggest pain point in the integration as I use it so far is the fact that an obs service still commits static data at the time of the service invocation into the obs storage backend. This means some sort of update mechanism to update the data stored in the obs backend with the data on the git side is required. I explained my way of doing this in step 3) above. An interesting aspect for me would be if it would be possible to just eliminate step 3) completely. Other applications e.g codacy or gitlab use Webhooks to implement the communication between services. From the information provided here I still have the feeling that this is not supported, but I might be wrong ? Other than that I'm happy with the options obs offers to integrate with git. I would only change my concepts if it simplifies the current procedure. From the given examples I have the feeling it would not simplify it, e.g the magic bits in .spec ... what if there is no .spec ? Hope this makes sense. Merry Christmas to everyone and a good start into 2022. Stay safe Best Regards, Marcus -- Public Key available via: https://keybase.io/marcus_schaefer/key.asc keybase search marcus_schaefer ------------------------------------------------------- Marcus Schäfer Am Unterösch 9 Tel: +49 7562 905437 D-88316 Isny / Rohrdorf Germany -------------------------------------------------------

Adrian Schröter writes:

Yes, really yet another obs git integration, but it might be the last one...

***** This is a disruptive change since it would offload really all source management and workflows into any git, gitlab, pagure or github instances. *****

OBS will just following defined git repositories for all the packageing files (like spec, kiwi or Dockerfiles, but also the upstream sources in form of tar balls or scm repositories). Any native OBS mechanics may *NOT* be usable anymore though.

As this would affect more or less all development workflows and tooling, we like to get feedback from you. Esp. in the regard how you would use any the functionality.

Some stuff to touch ===================

mls and me worked on something the last two weeks what we wanted to have since a long time. We consider this version 0.1, meaning some parts are already working via some workarounds in some cases. We may change it incompatible at any time until the next OBS release happened.

There are example setups in OBS to demonstrate this. And also hopefully enables us to discuss about possible setups and workflows for future projects.

***** Just to be clear, do *NOT* use this atm for any package which needs to be submitted to any of our active distributions! So defintive not in any official devel project. *****

If you want to test the existing examples you need to use a forked osc. You get it the tooling via the following commands, but be aware that I did not test much (even though I use it now permanent on my workstation):

# zypper ar https://download.opensuse.org/repositories/home:/adrianSuSE:/OBSGIT/15.3/ OBSGIT # zypper dup --from OBSGIT

This should update osc and the build package. Also the new obs-scm-bridge package gets installed.

Afterwards at the absolute the minimum the follow commands should work:

# osc co home:adrianSuSE:OBSGIT/git-example-1 # cd home:adrianSuSE:OBSGIT/git-example-1 # osc build

What happens here is that the checkout is detecting that this is a scm bridged package. This is currently implemented as temporary hack via the special string in the package meta title element:

<title>obs-scm:https://github.com/adrianschroeter/git-example-1</title>

This currently defines the git repository which provides the sources. This will go into a sepecial structure later, but since this affects also remote OBS instances we need a longer prepared update for the stable versions before.

osc is therefore not downloading any source files on his own but calls the bridge instead. The bridge is doing two things:

1) clone the git repository into the home:adrianSuSE:OBSGIT/git-example-3 directory. You can use any git command there now. You can *NOT* use osc to submit any change!

2) downloading all the assets. Assets are referenced files in the build descriptions. Esp. tar balls should not become part of these git repositories as this would not scale. But also upstream SCM can be refernced here. Please find out more about assets, how they can setup and how verification can be done in our pbuild documentation:

http://opensuse.github.io/obs-build/pbuild.html#_remote_assets

We have three examples in OBS to demonstrate how packages could get maintained this way:

git-example-1 =============

This is one has our packaging specific files in git and is downloading the upstream files, like the tar ball via the asset mechanism.

git-example-2 ============= This one is building directly from some git repository. Everyhing needed is part of this single git repository.

The spec might got added in a branch controlled by the package maintainer. Updating the source would work via git merge or rebase.

git-example-3 ============= This is basically building from 2 independend git repositories. The packaging specific files are in our own git. The upstream sources come from another git repository, specified inside of the spec file.

Note: when you do a checkout using osc, you get currently the upstreams source inside of an .obscpio archive. I will change this to get the git repository as clone.

Another example? Please try it on your own, all what you need to have is a git repository and set the <title> element in any package meta....

Please note that IBS has not yet deployed this new code, only build.opensuse.org is working atm.

On Project Level ================

You can use the same <scmsync> element to specify a git repository in project meta. Any directory there is considered to be a package.

Any build configuration parts (what you have in "osc meta prjconf" usually) can be stored in a file called '_config'.

We support this in multiple ways:

1) One plain single repository providing everything. You can find a real life example here:

https://build.opensuse.org/project/show/home:adrianSuSE:OBSGIT:Project-examp...

or the git repo here:

https://github.com/geckito/image-RaspBerryPi4-pi-hole

As you notice, it is a repositoy from our Geckito project. This means the same project can get build inside of OBS or it can be build entirely independend using the "pbuild" tool.

2) An example which is more likely to suite larger projects, like our distributions is using git submodules. You can find an example here:

https://build.opensuse.org/project/show/home:adrianSuSE:OBSGIT:Project-examp...

or the git repo here:

https://github.com/adrianschroeter/git-project-example-submodules

This is just referencing two git repositories from our earlier examples.

3) A possible later option: We may support URLs on project level pointing to github/gitlab/pagure projects. And listing all repositories inside of these to build these.

Tell if you think this would be useful.

Disadvantage would be the lack of a signed reference to a git repository. The advantage would be that one would not need to deal with git submodules.

In general some first documentation of this functionality can not be found here:

https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.scm_bridge....

--

Adrian Schroeter Build Infrastructure Project Manager

SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev

-- Dan Čermák Software Engineer Development tools SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany (HRB 36809, AG Nürnberg) Managing Director/Geschäftsführer: Ivo Totev

On Montag, 17. Januar 2022, 09:26:00 CET Dan Čermák wrote:

Hi Adrian,

first, apologies for the previous empty email, I hit the wrong key…

Adrian Schröter writes:

...

If you want to test the existing examples you need to use a forked osc. You get it the tooling via the following commands, but be aware that I did not test much (even though I use it now permanent on my workstation):

# zypper ar https://download.opensuse.org/repositories/home:/adrianSuSE:/OBSGIT/15.3/ OBSGIT # zypper dup --from OBSGIT

This should update osc and the build package. Also the new obs-scm-bridge package gets installed.

How radical are the changes that you had to implement?

Not exactly sure what you mean here, but this support is a parallel code path in most places. So it is new functionality, but not changing any existing. Beside osc everything has been merged meanwhile. The packages in my project just may be newer sometimes.

...

On Project Level ================

You can use the same <scmsync> element to specify a git repository in project meta. Any directory there is considered to be a package.

Any build configuration parts (what you have in "osc meta prjconf" usually) can be stored in a file called '_config'.

We support this in multiple ways:

1) One plain single repository providing everything. You can find a real life example here:

https://build.opensuse.org/project/show/home:adrianSuSE:OBSGIT:Project-examp...

or the git repo here:

https://github.com/geckito/image-RaspBerryPi4-pi-hole

As you notice, it is a repositoy from our Geckito project. This means the same project can get build inside of OBS or it can be build entirely independend using the "pbuild" tool.

2) An example which is more likely to suite larger projects, like our distributions is using git submodules. You can find an example here:

https://build.opensuse.org/project/show/home:adrianSuSE:OBSGIT:Project-examp...

or the git repo here:

https://github.com/adrianschroeter/git-project-example-submodules

This is just referencing two git repositories from our earlier examples.

Both of the projects that you linked here do not have any packages visible via the obs webui. Is that a bug or a current limitation?

well, a current limitation. However, there is already a first step to allow access to build results and avoiding webui errors is here already: https://github.com/openSUSE/open-build-service/pull/12071 -- Adrian Schroeter Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev

On Donnerstag, 20. Januar 2022, 09:51:04 CET Dan Čermák wrote:

Hi Adrian,

Adrian Schröter writes:

...

On Montag, 17. Januar 2022, 09:26:00 CET Dan Čermák wrote:

...

Hi Adrian,

first, apologies for the previous empty email, I hit the wrong key…

Adrian Schröter writes:

...

If you want to test the existing examples you need to use a forked osc. You get it the tooling via the following commands, but be aware that I did not test much (even though I use it now permanent on my workstation):

# zypper ar https://download.opensuse.org/repositories/home:/adrianSuSE:/OBSGIT/15.3/ OBSGIT # zypper dup --from OBSGIT

This should update osc and the build package. Also the new obs-scm-bridge package gets installed.

How radical are the changes that you had to implement?

Not exactly sure what you mean here, but this support is a parallel code path in most places.

So it is new functionality, but not changing any existing.

Exactly this: does it alter existing functionality or just adds new code paths.

it only adds code paths atm. But the new code paths can not be considered stable yet. -- Adrian Schroeter Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev