ALP Git-Packaging-Workflow group created
Git Packaging Workflow Exploration A few colleagues from SUSE, namely the build service team and others like myself are exploring the options on what the best way would be to use Git, rather than the current open Build Service source handling, for storing openSUSE / SLE package sources. This was originally launched under the ALP Workgroup umbrella but for now is timeline wise decoupled from this effort, except that if we would decide to switch, that it would have to happen before the first commercially supported Alp release would be branched off openSUSE Factory. For now, all of this is in the exploration and experimentation phase and no decision in either way has been taken. Meanwhile, we do however feel confident and organized enough to be open to wider feedback. If you're interested in any of the topics please let the work package driver know, or if in any doubt contact me. Here's the current list of work packages and their state: * Exploration of the Git Hosting options (run by me) openSUSE already has some experience with Pagure.io (used by https://code.opensuse.org/ for example) and GitHub (used by various openSUSE projects under https://github.com/openSUSE) as well as some projects also in GitLab (used by a few community efforts as well as SUSE-internally for hosting) so we explored another option named Gitea, which is now available to the community under https://gitea.opensuse.org/. Please note that this is not a production service, and can lose data or disappear at any point in time, it is only being operated in the context of the Git packaging workflow exploration. * Exploration of the history import from the SUSE and openSUSE build service source history (run by Stephan Kulow). Stephan Kulow has managed to import the majority of existing history of packages that landed in released products or in openSUSE Tumbleweed as history into package-git repositories and published under https://gitea.opensuse.org/rpm . As an example the history of zlib is here: https://gitea.opensuse.org/rpm/zlib/commits/branch/factory which is significantly more detailed than a similar effort implemented by Bernhard Wiedemann a couple of years ago on our code.opensuse.org instance: https://code.opensuse.org/package/zlib/commits/master Your feedback on the history import quality and how it could be made better is very much appreciated. The key difference compared to the pagure setup is that we use Git LFS for storing large files (like tarballs for example) and currently these large files are mapped via a small redirector proxy directly to the openSUSE build service source service. With that the git repository history is extremely small and very manageable for local cloning. * Exploration of the git packaging workflow going forward. Alberto Planas, with contributions by others, has written up ideas and options in the following document: https://github.com/aplanas/openSUSE-git/blob/main/notes-about-git.md Please provide feedback either as a pull request or as a discussion entry in the forum here: https://github.com/aplanas/openSUSE-git/discussions * Exploration of build and status integration. This one is less well-defined, but for now it is about automatically syncing new changes in the git repository with the state of the Build Service instance that is used for building as well as exploring the options for reporting back the build status into the Git Hosting infrastructure. This is currently worked on by Adrian Schroeter. We store the meeting time and minutes under https://en.opensuse.org/openSUSE:ALP/Workgroups/Git-Packaging-Workflow In case there are any questions, feel free to reach out to related work package owners or myself. Greetings, Dirk
On Wednesday 2022-07-20 14:13, Dirk Müller wrote:
* Exploration of the Git Hosting options (run by me) openSUSE already has some experience with Pagure.io (used by https://code.opensuse.org/ for example) and GitHub (used by various openSUSE projects under https://github.com/openSUSE) as well as some projects also in GitLab (used by a few community efforts as well as SUSE-internally for hosting) so we explored another option named Gitea, which is now available to the community under https://gitea.opensuse.org/.
gitea with the lfs approach works well. I wonder how the ipfs symlinks stored in pagure would be handled. Fuse mount, or another global hook like git-lfs?
* Exploration of the history import from the SUSE and openSUSE build service source history [...] Your feedback on the history import quality and how it could be made better is very much appreciated.
I would make the case that OBS history is not particularly interesting for its messages, because (a) we have had .changes files, (b) the SVN-ish nature of the OBS SCM meant that you can't go back and edit messages or commits, so there is little incentive to write up much if you missed it in the first commit.
On Wed, Jul 20, 2022 at 8:44 AM Jan Engelhardt <jengelh@inai.de> wrote:
On Wednesday 2022-07-20 14:13, Dirk Müller wrote:
* Exploration of the Git Hosting options (run by me) openSUSE already has some experience with Pagure.io (used by https://code.opensuse.org/ for example) and GitHub (used by various openSUSE projects under https://github.com/openSUSE) as well as some projects also in GitLab (used by a few community efforts as well as SUSE-internally for hosting) so we explored another option named Gitea, which is now available to the community under https://gitea.opensuse.org/.
gitea with the lfs approach works well.
I wonder how the ipfs symlinks stored in pagure would be handled. Fuse mount, or another global hook like git-lfs?
Right now, we don't have any blob storage system attached to Pagure. We can do whatever we want for this. Fedora's Dist-Git setup[0] leverages a wrapper tool called fedpkg[1] built on the rpkg[2] framework. We don't have to do it the same way, but there are a lot of nice things from having a swiss army knife tool for packaging. We have this today with OBS through osc, for example. After working with Git-LFS for years for normal development, I'm not a huge fan of it. A couple of issues I have with LFS: * It's incredibly difficult to mirror LFS, and there's no straightforward way to do it with git repos, given how the integration works * LFS easily breaks when modifications occur across forks, and it's difficult to identify the problems there and fix them There are other issues that happen specific to LFS server implementations. For example, GitLab's implementation imposes ACLs on each blob which makes it difficult for forks made by non-members to work reliably. GitHub's implementation has issues where smudging can be broken when doing complex merges that have LFS objects touched. For some of the awkwardness around Dist-Git, these problems don't happen because it's a transparent blob cache and the only thing written to Git is a plain text file with the filename and checksum, which is used as the blob reference. Authorization to upload and download those blobs is controlled by packager ACLs or whatever you wish to use to control that, making it effectively global, so it works properly across forks, or even cross-server forks. With my downstream hat, that's also useful if I'm building patched derivative packages, because I can use the Dist-Git server references even without my own Git repo on my own Git server. It makes CI and all kinds of other things easier too. [0]: https://github.com/release-engineering/dist-git [1]: https://pagure.io/fedpkg [2]: https://pagure.io/rpkg
* Exploration of the history import from the SUSE and openSUSE build service source history [...] Your feedback on the history import quality and how it could be made better is very much appreciated.
I would make the case that OBS history is not particularly interesting for its messages, because (a) we have had .changes files, (b) the SVN-ish nature of the OBS SCM meant that you can't go back and edit messages or commits, so there is little incentive to write up much if you missed it in the first commit.
The history is better than nothing because things like Git blame work reasonably okay. When Bernhard was working on the original setup, he didn't bother because it was a read-only setup. If we transition to Git, we probably want the attribution and messages to be more accurate because the Git history is useful for working through the package's development. It would also open the door for things like rpmautospec to be used in the future[3]. [3]: https://docs.pagure.org/Fedora-Infra.rpmautospec/ -- 真実はいつも一つ！/ Always, there's only one truth!
On Wednesday, July 20, 2022 4:02:12 PM CEST Neal Gompa wrote:
GitHub's implementation has issues where smudging can be broken when doing complex merges that have LFS objects touched.
I do not think that LFS is set into stone yet, and this kind of interaction should be tested. IMHO one of the advantage of LFS is that is a very simple protocol, that can behave very nice with OBS. For example, there is currently in the PoC a proxy in place that routes back to OBS for old tgz, avoiding duplication. Future commits can delegate into OBS or another LFS server from this point, without much notice from the users PoV. The dist-git / fedpkg idea was commented too. I personally do not like it. The point is a bit irrelevant, but as a developer, when I am using a model that is like the one in OBS (tarball, spec, changes and patches), I do not like the context switch that requires fedpkg to manage the tgz at a different level. In that regard "osc" is better, and with git-LFS there is a chance to replicate this experience using a mostly pure git model. In any case this should be stressed, validated and revisited.
The history is better than nothing because things like Git blame work reasonably okay.
+1 on this. -- SUSE Software Solutions Germany GmbH Frankenstrasse 146 90461 Nuremberg Germany Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman (HRB 36809, AG Nürnberg)
On Mittwoch, 20. Juli 2022, 16:39:55 CEST Alberto Planas wrote:
On Wednesday, July 20, 2022 4:02:12 PM CEST Neal Gompa wrote:
GitHub's implementation has issues where smudging can be broken when doing complex merges that have LFS objects touched.
I do not think that LFS is set into stone yet, and this kind of interaction should be tested.
yes, definitivly. It was mainly used to represent the file blobs from OBS source server commits. We wanted to avoid to duplicate all these files and therefore created a _read-only_ LFS bridge to OBS source server. This means also that * the .lfsconfig will also work when you move a git repository to another git hoster, as the URL to the LFS storage is hardcoded * the .lfsconfig needs to be removed/adapted when replacing the assets. IMHO using the pbuild/OBS remote assets would be a good way to go here, but it does not mean that this is the only allowed way. Using the remote assets we would avoid the mentioned problems when pushing the git repo to another host and we also would keep tracking data from where the resources came (something what is not working with LFS out of the box). It would also help us to create proper provenance files for reproducibility. Something what can only increase the trust in our code base ...
IMHO one of the advantage of LFS is that is a very simple protocol, that can behave very nice with OBS. For example, there is currently in the PoC a proxy in place that routes back to OBS for old tgz, avoiding duplication. Future commits can delegate into OBS or another LFS server from this point, without much notice from the users PoV.
Right, you can find an example package here btw: https://gitea.opensuse.org/adrianSuSE/git-example-lfs The .lfsconfig is pointing to a (non final) OBS LFS provider. But again, this is only intended for past commits, not for future ones when git will be the master source. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
Adrian Schröter wrote:
[...] * the .lfsconfig will also work when you move a git repository to another git hoster, as the URL to the LFS storage is hardcoded
Isn't that already the argument against doing it that way? :-) What you are saying is that one can switch the git hosting location but not the one for the binary blob storage. That sounds like a layering violation that defeats the distributed nature of git. It should be possible to host the sources as well as the binary blob storage somewhere else without having to rewrite all sources, right? So the information about where the blob storage is would have to be somehow carried with the transport mechanism used to clone the sources. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Software Solutions Germany GmbH, GF: Ivo Totev HRB 36809 (AG Nürnberg)
Am 25.07.22 um 10:29 schrieb Ludwig Nussel:
What you are saying is that one can switch the git hosting location but not the one for the binary blob storage. That sounds like a layering violation that defeats the distributed nature of git. It should be Your argument sounds very academic. Note that other distributions contain source URLs in their git repos - it's not very different to what we do here.
Greetings, Stephan
On Mon, Jul 25, 2022 at 4:34 AM Stephan Kulow <coolo@suse.de> wrote:
Am 25.07.22 um 10:29 schrieb Ludwig Nussel:
What you are saying is that one can switch the git hosting location but not the one for the binary blob storage. That sounds like a layering violation that defeats the distributed nature of git. It should be Your argument sounds very academic. Note that other distributions contain source URLs in their git repos - it's not very different to what we do here.
No other distribution relies on the source URLs continuously to actually get the sources. They exist as reference information for packagers to fetch sources and upload them to trusted blob storage (Dist-Git, ABF, binrepo, etc.). -- 真実はいつも一つ！/ Always, there's only one truth!
Am 25.07.22 um 12:01 schrieb Neal Gompa:
No other distribution relies on the source URLs continuously to actually get the sources. They exist as reference information for packagers to fetch sources and upload them to trusted blob storage (Dist-Git, ABF, binrepo, etc.). What makes your "trusted blob storage" different to our OBS? I'm not talking about upstream URLs.
Greetings, Stephan
On Mon, Jul 25, 2022, 9:30 AM Stephan Kulow <coolo@suse.de> wrote:
Am 25.07.22 um 12:01 schrieb Neal Gompa:
No other distribution relies on the source URLs continuously to actually get the sources. They exist as reference information for packagers to fetch sources and upload them to trusted blob storage (Dist-Git, ABF, binrepo, etc.). What makes your "trusted blob storage" different to our OBS? I'm not talking about upstream URLs.
Source services change everything. They fetch and store sources without regard. Nobody else does that.
On Wed, Jul 20, 2022 at 10:40 AM Alberto Planas <aplanas@suse.de> wrote:
On Wednesday, July 20, 2022 4:02:12 PM CEST Neal Gompa wrote:
GitHub's implementation has issues where smudging can be broken when doing complex merges that have LFS objects touched.
I do not think that LFS is set into stone yet, and this kind of interaction should be tested.
IMHO one of the advantage of LFS is that is a very simple protocol, that can behave very nice with OBS. For example, there is currently in the PoC a proxy in place that routes back to OBS for old tgz, avoiding duplication. Future commits can delegate into OBS or another LFS server from this point, without much notice from the users PoV.
The dist-git / fedpkg idea was commented too. I personally do not like it. The point is a bit irrelevant, but as a developer, when I am using a model that is like the one in OBS (tarball, spec, changes and patches), I do not like the context switch that requires fedpkg to manage the tgz at a different level. In that regard "osc" is better, and with git-LFS there is a chance to replicate this experience using a mostly pure git model.
Git LFS is a bolt-on to Git, it just so happens that GitLab and GitHub know how to dereference it when the blobs are stored on *their* LFS servers. It is not hard to bolt on Dist-Git the same way. There's just been no drive to do that in Fedora because the "fedpkg" porcelain is more than sufficient. Git LFS does it with the .gitattributes file and storing extra goop in the git repository metadata, Dist-Git does it with the "sources" file. Six one way, half dozen the other. That said, if OBS has its own LFS server, then it really doesn't matter. Git repos in Pagure can be pointed to it like any other. Right now, Pagure has no specific plugin/integration for *any* blob storage system. The biggest annoyance I have with Git LFS from a general workflow perspective is that "git clone --mirror" produces unusable git repos with it, which makes downstream usage annoying. -- 真実はいつも一つ！/ Always, there's only one truth!
Am 21.07.22 um 12:30 schrieb Neal Gompa:
The biggest annoyance I have with Git LFS from a general workflow perspective is that "git clone --mirror" produces unusable git repos with it, which makes downstream usage annoying.
Unless there is a .lfsconfig in the repo pointing to the upstream LFS server and that LFS server does not require authentification. At that point the LFS server is basically like the lookaside cache of fedora. Greetings, Stephan
Am 21.07.22 um 19:07 schrieb Stephan Kulow:
Am 21.07.22 um 12:30 schrieb Neal Gompa:
The biggest annoyance I have with Git LFS from a general workflow perspective is that "git clone --mirror" produces unusable git repos with it, which makes downstream usage annoying.
Unless there is a .lfsconfig in the repo pointing to the upstream LFS server and that LFS server does not require authentification. At that point the LFS server is basically like the lookaside cache of fedora.
BTW: git clone https://gitea.opensuse.org/rpm/ltrace.git cd ltrace && git lfs fetch --all -> you have the perfect offline mirror Greetings, Stephan
Am 20.07.22 um 16:02 schrieb Neal Gompa:
There are other issues that happen specific to LFS server implementations. For example, GitLab's implementation imposes ACLs on each blob which makes it difficult for forks made by non-members to work reliably. GitHub's implementation has issues where smudging can be broken when doing complex merges that have LFS objects touched. I can confirm that gitlab's restriction there makes LFS usage painful if you're not expecting the problem. Once you realize what's going on, it's easy to work around.
And IMO it outweighs the benefits of having a mechanism built into git itself. Greetings, Stephan
Hi Jan,
I wonder how the ipfs symlinks stored in pagure would be handled. Fuse mount, or another global hook like git-lfs?
For now we're not looking at the pagure setup at all. I personally have no experience with ipfs and it wouldn't work for the certified part of the SUSE product building, so it is a dead end. We're taking Git with LFS as plan A, unless there are compelling reasons against it, we will not look back.
I would make the case that OBS history is not particularly interesting for its messages, because (a) we have had .changes files, (b) the SVN-ish nature of the OBS SCM meant that you can't go back and edit messages or commits, so there is little incentive to write up much if you missed it in the first commit.
Right, the *.changes files are still part of the commit content, so you get these as part of the commits (and follown commits). There are some suggestions around using git notes for the end-user changelog, but that hasn't been evaluated in detail yet. Any suggestions or input welcome. Greetings, Dirk
On Wednesday 2022-07-20 16:37, Dirk Müller wrote:
I would make the case that OBS history is not particularly interesting for its _messages_ [..]
[.. ]There are some suggestions around using git notes for the end-user changelog, but that hasn't been evaluated in detail yet. Any suggestions or input welcome.
Adam Majer held a conversation with a few individuals ahead of the openSUSE conf 2022 for thoughts, I'd like to repaste some of that since it was originally off-list and still hope to see this coming to build.opensuse.org soon. --- Date: Tue, 31 May 2022 15:33:31 +0200 From: Jan Engelhardt To: adamm Cc: stroeder On Tuesday 2022-05-31 12:50, Adam Majer wrote:
https://build.opensuse.org/project/show/devel:ALP https://code.opensuse.org/adrianSuSE/Alp https://code.opensuse.org/adrianSUSE/zypper https://code.opensuse.org/package/zypper
Internally, most of the feedback from maintainers I've talked to is along the lines of "ok, so I'll have to adapt to this I guess", ie. not enthusiastic :-)
I would like to get your thoughts about this and any suggestions of what you would expect this "new" approach to have.
* Because the git checkout does not have the .osc directory, the osc build line is a bit longer than usual. osc build --alternative-project openSUSE:Factory standard x86_64 pdns.spec * Checking out /package/zypper: the tarball is a (dangling) symlink to /ipfs. ALP is "just" a spin/a distribution -- let's deploy this git scheme for an "actual" development prj and see how that works out. I could think of a few to spare for the experiment that have low volume (security:netfilter, maybe network:telephony).
Also, any sticking points when it comes to current (current OBS) workflow that git could improve?
* Some command-line equivalent for `osc sr` will be needed if git is used as a backend. `git push` to /jengelh/zypper is (would be) easy, but making the actual SR/MR/PR.. if I don't know better, would require opening a browser, and one shouldn't have to go to a browser.
On Donnerstag, 21. Juli 2022, 09:34:17 CEST Jan Engelhardt wrote:
On Wednesday 2022-07-20 16:37, Dirk Müller wrote:
I would make the case that OBS history is not particularly interesting for its _messages_ [..]
[.. ]There are some suggestions around using git notes for the end-user changelog, but that hasn't been evaluated in detail yet. Any suggestions or input welcome.
Adam Majer held a conversation with a few individuals ahead of the openSUSE conf 2022 for thoughts, I'd like to repaste some of that since it was originally off-list and still hope to see this coming to build.opensuse.org soon.
---
Date: Tue, 31 May 2022 15:33:31 +0200 From: Jan Engelhardt To: adamm Cc: stroeder
On Tuesday 2022-05-31 12:50, Adam Majer wrote:
https://build.opensuse.org/project/show/devel:ALP https://code.opensuse.org/adrianSuSE/Alp https://code.opensuse.org/adrianSUSE/zypper https://code.opensuse.org/package/zypper
Internally, most of the feedback from maintainers I've talked to is along the lines of "ok, so I'll have to adapt to this I guess", ie. not enthusiastic :-)
I would like to get your thoughts about this and any suggestions of what you would expect this "new" approach to have.
* Because the git checkout does not have the .osc directory, the osc build line is a bit longer than usual.
osc build --alternative-project openSUSE:Factory standard x86_64 pdns.spec
the current osc can checkout a scmsync package. You get a local git clone, but osc remains working with local builds inside. The downside atm is that it still creates a .osc which need to be ignored by git. I hope we will find a better solution for that in future. Try osc co home:adrianSuSE:OBSGIT git-example-1 for an example package which defines a git url. At least osc-0.179 is required for this though.
* Checking out /package/zypper: the tarball is a (dangling) symlink to /ipfs.
ALP is "just" a spin/a distribution -- let's deploy this git scheme for an "actual" development prj and see how that works out. I could think of a few to spare for the experiment that have low volume (security:netfilter, maybe network:telephony).
Also, any sticking points when it comes to current (current OBS) workflow that git could improve?
* Some command-line equivalent for `osc sr` will be needed if git is used as a backend. `git push` to /jengelh/zypper is (would be) easy, but making the actual SR/MR/PR.. if I don't know better, would require opening a browser, and one shouldn't have to go to a browser.
A sr would not transfer the git information atm (and on purpose). That means you can work with such a git setup in your devel or home project and submit request the result to factory for example. However, the long term plan here is that the sr gets actually replaced by a pull request in the git hosting instance. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Frankenstraße 146, 90461 Nürnberg, Germany (HRB 36809, AG Nürnberg) Geschäftsführer: Ivo Totev
On 20/07/2022 14.44, Jan Engelhardt wrote:
I wonder how the ipfs symlinks stored in pagure would be handled. Fuse mount, or another global hook like git-lfs?
The fuse filesystem is part of the go-ipfs package: mkdir /ipfs /ipns ; ipfs daemon --mount However, I never added the files referenced there to a server, so it will timeout on access. However, you can 'ipfs add $tarball' (with the right options[1]) locally and if the SHAsums work out, it will be there. Since that is not very user-friendly, there is this other option I never tested so far: IPFS has a urlstore feature where you can add URLs instead of local files, so we could use the same OBS-redirector that coolo used and make all tarballs available from OBS over IPFS without storing them another time. Pro: * allows for easy mirroring and local caching * allows to keep references independent of git (e.g. in tarballs) * uses cryptographic hashes to ensure the file you get is what was checked in. Con: * Alpha version with occasional annoying bugs [2] * poor performance (IMHO too many threads from goroutines) * Symlinks need root permission once (in a container might be enough) [1] https://github.com/bmwiedemann/openSUSE/blob/master/scripts/mappkgs.pl#L45 [2] https://github.com/ipfs/go-ipfs/issues/9044 https://github.com/ipfs/go-ipfs/issues/2167
participants (8)
-
Adrian Schröter -
Alberto Planas -
Bernhard M. Wiedemann -
Dirk Müller -
Jan Engelhardt -
Ludwig Nussel -
Neal Gompa -
Stephan Kulow